This is a really funny issue to me actually, because mostly we are only talking a matter of accent.
My husband and I are both involved in financial markets education & present in the US on a regular basis. Almost unfathomable to me is that they have a really hard time understanding us, even in just regular conversation. All they ever want to talk about in the beginning is our 'funny' accent & you can see by the looks on people's faces that frequently they have NO idea what we have just said.
It seems that once they register that you aren't from the States their brains switch into panic mode & tell them "oooooh I can't understand this person!" & they don't even try. (Luckily I have perfected nice little Cali accent that I can use at will when the going gets really tough!)
Conversely we have absolutely NO problem understanding 99.9% of Americans - including those from the deep South & a number born & bred guys from the Bronx who we have come across.
The only person I ever had a real problem with was an African American from LA when he switched into his 'street talk' to show me that I can't understand everyone who speaks English! Naturally a lively debate arose about whether it was really English or actually pidgin!
The problem doesn't seem so great with those from the UK & Ireland - they seem to have no problem understanding us. Though admittedly a heavy Scotish brogue can be very difficult to follow.
The key in understanding other English speakers with broad accents is context, until I get used to their speech pattern.