I've had my 5 year old playing with my RC cars for a few years now, and it's surprising how well he's able to control it. With the speed dialled down and the space to get used to it, they soon pick it up.
He started with Lunchbox/Heavy Dump sessions, when really all he could do was throttle or steering. The Heavy Dump was great for those as it didn't get stuck too much, and looked interesting to him as he could interact with it with his other construction toys. I used the EPA settings on the controllers to keep the speed down, allowing me to set it to the appropriateness of the terrain.
As he's got older, he wanted something quicker, so he's tried out Hotshots, TT02Bs, Top Force, Optima etc. He tended to get quite opinionated about what they looked like, so that might be a key to the success, whether your nephew likes the look of it, the colour. I still have the speed wound down to 75%, but he's had a go on full power and actually prefers to be in control when the space is tight. On rugby pitches he was much happier with it at 100% on mild motors.
Reliability wise, the worse option (thankfully something you've not picked) is touring cars. When playing with his cousin, it can often turn into a destruction derby, and the touring cars really don't get on well there. The buggies however seem ok. TT02B vs Hotshot hasn't really resulted in any damage to person or car.
I think the big wheels would be a great success, I don't think the bumpy ground would hinder them. I think the increased ride height would reduce grass pickup on the driveshafts etc, which can bring playtime to a halt. Buggy wheels would be more performant, but more likely to get stuck in the mud you refer to.
I've not had a Madbull, but the simplicity, ruggedness and big wheels (always popular) is great choice. You could stick big tyres on a TT02B if one of those bodies was more appealing, but the price starts to creep up.