I'd say it depends in honesty there are some things that are worse like the music player, it's a lot more buggy and crashes now if the album art is to big and sometimes loses your music for no reason. If you want it mainly for music you are probably better were you are. Also depends on the size of your iPod and how full it is mine was 64gb and almost full resulting in the update taking about 5 hours and messing up most of my stuff anyway. As for features Notifications are nice the rest you will probably want to switch off I've had nothing but problems with wifi sync.
I would not, unless there are some apps you want that would require 5.0 to run. It doesn't add much from 4.0, and from what I've heard it makes the iPod run slower.
Not to sound stupid (Too late. Doh!), but is this the same IOS as the iPhone? I'd swear my iPhone has been slower and crabbier since 5... DBC