If anyone likes being OCD and maximizing all the options, check out http://www.forum-software.org/
It’s not even complete, but it lists a couple dozen PHP-based FLOSS forum options and some other proprietary things.
There’s other stuff out there too, but not a lot that is competing with Discourse as a more modern-style option.
Note that Reddit is FLOSS and you can host your own Reddit site or use any of dozens of Reddit-style clones as another alternative.
I’m sure there’s lots more I don’t even know about. I also have some things to say about the discussion system in-progress at my site Snowdrift.coop, but we haven’t (yet) pulled it out to be a tool others could use outside of our system, and it’s still early days. Of course, I’m totally biased but think that we’re doing some valuable things I haven’t seen elsewhere.
Oh, and the new Mailman 3 UI being worked on by RedHat / Fedora folks is pretty nifty and makes Mailman work like a forum in many ways. I think that is a modern option that looks really promising! Check that out too!