The problem is any time I try and move around I get attacked by the random groups war raid parties, and I can't ever build up a decent army.
Learn to avoid them when it's just you by yourself! Run around to all the cities, and go in the taverns of every city. Talk to any heroes (NPC's with unique names) that are willing to join your party.
I don't mean the mercenaries (eg "4 mercenary knights are willing to join you"), but the actual unique heroes that you can level up and assign skillpoints to just like your main character. Some will join you for free, some will join you for something like 100-500 gold, and others won't join you unless you build up a big reputation first. But check all the taverns in all the cities and grab perhaps 5 of the "free" (or cheap) heroes, and start levelling them up. Remember that "party skills" (like tracking, spotting, engineering, first aid, surgery) only need to be high on one party member, so you can have 1 hero working on surgery and first aid, 1 hero working on engineering and tracking, 1 hero on spotting and... etc.
Also make sure you max the "training" skill on
every character including yourself. This skill grants "free" exp to every character in your group (including the soldiers) that are below the level of the trainer, for every trainer. So if you have 5 heroes at level 25 with 10 training (10 training gives 80 exp per day) every single troop below level 25 will be getting exp - so if you have 100 peasants, you'll get 5 * 100 * 80 exp on the peasants - which is an instant level up in 1 day without having to do anything.
A group with you and 4 heroes in, especially if you have a "heavy charger" horse, will be able to take down the smaller sea-raider parties (and with some practise, even the huge sea raider parties) you find on the coast, and you get good exp and loot from them. Keep at that and keep levelling up just you and your heroes, until you're around level 30 and your heroes are around level 20+. Once you hit this stage, you can think about starting to hire an army (hiring peasants at villages and training them up). Don't bother with an army before you've got some high level heroes and you are high level yourself though.
Now you have an army built up, and once the army troops are all quite high level (eg 150+ Nordic Berserkers) - you can think about joining another faction (you'll get offered to join/fight for a faction randomly when your reputation is high enough), or just going to war on a faction all by yourself and taking over some castles/cities
