S/PDIF or USB? (3)
USB decoders
So, starting from the bottom, the worst jitter performer I've come across so far was Philips UDA1321. It actually showed the highest jitter I have ever measured, not only in PC audio but in audio generally: in fact, when I saw this result, my first thought was that something was wrong... but it wasn't: its jitter is about 1ms (yes, millisecond). And it is not only relatively high absolute value but it is also mostly not benign 1/f jitter. (I thought this would be a good candidate for the world record, but when I posted this result a few months ago to diyhifi.org I was told that some of the first DVD players actually had even higher jitter.) Yet, this chip is used in some pieces marketed as “audiophile” ones. One will also come across enthusiastic owners reporting that they've sold all the expensive classic equipment they used to have, finding USB PC system based on this device superior, and tweaks (cryogenic treatment and mechanical damping...) “bring it even to the higher level”. It may look like total mess, but one doesn't have to wonder if we find the same enthusiastic owners of UDA1321 explaining how all this PC stuff is good "because it is jitter free".

[click to see a higher res graph]
And taking such events into account, the caution regarding PC audio shown by many serious audio consumers is understandable. Fortunately, PC stuff can indeed do better.
According to the measurements posted by John Westlake to diyhifi.org in 2006, a C-Media CM108 is better but still with relatively high figure of 3ns and with probably more important problem of bimodal recovered clock distribution.
As shown by the same author, the TUSB3200 from Texas Instruments reveals a bit different approach in designing; overall recovered clock jitter is a bit lower @ 2ns, and it has so-called spread spectrum.
Additionally, the measurement of TAS1020 has been submitted by Ergo Esken. This chip apparently performed worse than TUSB3200. The things being said, it is not clear in what extent it was the programming that determined results achieved by these Texas chips.2
PCM2704/5/6/7 units confirm Texas Instruments leadership in this domain. This series perform apparently better than any of the chips above, and it doesn't show the problems of bimodal distribution or spread spectrum or any other problems of apparently poor PLL or small buffer. I ran the frequency analysis of J-signal, using a PC running under Windows XP, and I used the Foobar2000 player with ASIO output. The result shows relatively wideband jitter content (supposedly benign) with total amount I estimated at about 1ns (RMS), and with very low data related artifacts – the point at which a USB link apparently easily overcomes S/PDIF related issues. The shape of the wideband content is not entirely "random" (Gaussian) but includes a certain unevenness, even some parts that can be considered discrete ones, and prominent parts in fact float a bit over time. The two curves shown below were captured within a couple of tens of seconds. (This effect is in my experience associated with certain asynchronous work of the source and conversion clock. More on internal structure of PCM2704/5/6/7, including its "Sampling Period Adaptive Controlled Tracking System" (SpAct), can be found in [2] and [3].) Also, the jitter content still shows a couple of low frequency artifacts as well as certain 1/f skirt, however it is still much lower than that of the previously considered chips. The absolute number of 1ns is still too high to be considered a definitive solution though. Also, the total amount I estimated based on the frequency analysis didn't match completely to the result reported by John Westlake, who used the same printed circuit board of mine to analyze PCM2706 MCK out directly. (It isn't necessarily the full explanation, but the difference may be caused by the use of a different PC; we'll get back to this topic a bit later.)

[click to see a higher res graph]
So, why do these things look like that?
The best feature of USB interface, its "non return to zero inverted", "clockless" protocol, means that the results may be free of the source jitter, but at the same time this is its main problem: unless the host PC is slaved to the peripheral device, it is not clear how to synchronize the peripheral device to the host PC. This is of no concern for the most purposes that USB is normally used for, but here it is, because conversion is performed in the real time. A data buffering comes to mind as a solution, but unless complete files are buffered, a buffer will be either overflowed (if PC clock is faster), or emptied (if PC clock is slower).
One way to solve this is to design the audio unit to use USB packets, which are sent every millisecond, as time basis. This unfortunately brings exactly what was meant to be avoided, results totally dependant on the source jitter, and very high jitter should be expected.
However, the time interval of USB packets can be also used only as a basis for PLL, which locks to it, slightly adjusting own frequency, and thus achieving certain jitter immunity with regard to the actual jitter of the USB stream. Such a use of PLL, which works together with certain data buffering, is often called an "adaptive" USB mode, and is what the majority of today's USB devices are using. Designing such a PLL is however not a trivial task. Further, it normally needs some data buffer i.e. memory, and memory is not cheap. As reported, results achieved by different USB decoder manufacturers differ both in the overall quality and in nature, depending on the approach of the particular manufacturer and their abilities, but apparently also on their understandings of (un)important parameters.
How about the PC used as S/PDIF source?
Checking this may seem redundant at the moment, but it is not. The previous paragraph in fact points out why the final jitter is dependent on the jitter of the source. So I've run the same test again to see if this jitter is associated to the PC itself, rather than to its USB interface as such. So I used again the same DAC (The Model) but this time measuring both via its USB and S/PDIF inputs, later being fed by the soundcard (E-MU) hosted by the same PC. Indeed, a bulk of this jitter figure is actually related to the PC, since these two curves have very much in common.
(Another change in the shape of USB associated jitter displays another shift, as pointed out earlier.)

[click to see a higher res graph]
___________________
[2] Hitoshi Kondoh - "The D/A diaries", Planet Analog, 2002-2005
[3] Hitoshi Kondoh (Texas Instruments Incorporated) - US Patent # 6,724,264:
"Controller for a device with change in characteristics", 2004
2 - According to the available data, some devices use this chip and own proprietary firmware to slave the host PC.