I'm just reporting what I see, and it makes sence that it would take longer because extra requests would have to be made, although, maybe it is faster after the first load. If the css remains in the cache, then it won't have to be loaded a second time, so you save that time, but the first load will be slower.
everything is a trade off. Stop using photos and rotating headers and fancy effects and ad's and google tracking and java scripts and your site will load FAST...might look like h*** but it would load fast. Everything is a trade off.