So, now I have found out why it sounds so much different and "boring" with the O2. It's because it's mono!
I've had my suspicions when soldering the cables between ODAC and O2 but I thought I'd better follow the guide... It turns out that in the guide the author had reversed the white and black cables which means that I have done it totally wrong and will have to open it up and reverse my cables. I have of course told Stefan Walter and Headnhifi about this and hopefully he will correct it.
I will fix this tomorrow and take some pictures of the process. Hopefully I'll end up with a kickass DAC+amplifier when I'm done :)