One feature of RL games is the random placement of monsters and
objects.  While this is good in general, it leads to setups that are
highly improbable at best, and ludicrous at worst.  You know what I'm
talking about.  A barrack of soldiers with one exit leading to a room
containing half a dozen giant spiders.  Orcs frolicing in one room while
a lone elf wanders next door.  Wargs who ignore the juicy smell of the
nearby ratling.  And so forth.

A good AI helps, but if the initial placement of the monsters makes it
unlikely the would encounter each other by their default behavior, the
problem isn't fixed.  The community has done a fantastic job developing
dungeons that are physically continuous, but whose content is not.  

First of all, we need some additional data for our generic monsters. 
Something which indicates which monsters tend to be next to which
monsters.  Call it a Theme.  For a Goblin Lair theme it might look like:

1     Goblin King and 3 advisors (1-3 hobgoblins) (unique, mandatory)
2     Goblin guards (1-6)
2     Hobgoblin guards (1-2)
3     Goblin Wolfriders  (1-3)
3     Goblin Shopkeeper (unique)
3     1-3 Goblins and 1-2 Goblin thieves
4     1-2 Goblins and a Crate
4     1-2 Goblin Guards
4     Goblin pig farm - 1 goblin and 3 pigs
4     Empty Room
5     Empty Room
6     Empty Room - random monster possible
7     Empty Room -random monster or stairs allowed

First, as you;re building your level, check if there's a theme.  Not all
levels should have one.  If your level has a theme then start at the top
of the list.  

In this case, the first room that you plunk down should have a Goblin
King (#1) in it.  When you draw your doors/hallways, draw them leaving
from this room, and carry with it the number on the left.    Moving to
the next room, move down the list (75% chance) or back up the list
(25%).  For the example above, there are two possibilities, either some
Goblin or Hobgoblin guards.  Add these to the next room.  Draw your
doors/halls from there and continue.  

You may even choose to add some entries to the hallways themselves, if
they're large enough.  The Goblin King might have a few guards in the
outside hall, but it would be difficult to have a pig farm in a
hallway.  If you wanted to use this option, the simple solution is
staggering your entries so that odd ones were in rooms and even ones
were the hallways between.

The nice thing is that you end up with a small goblin community, with a
few empty rooms between them and any wandering monsters.  The Goblin
King sits at the middle, furthest from the stairs where the hero will
enter, which is nice.

How to implement?  If you're carving your dungeon this is easy. 
Starting with an empty dungeon level, you maintain a separate list of
doorways:
1) Start with a dummy doorway with a content value of 1
2) From the doorway, try and add a room filling with content value from
the doorway; if successful, add both the room and doorway to the map.
3) Add doorways from the room, migrating the content (1->2, etc)
4) Step through your list of doorways and go back to step 2;  You're
done when you;re out of doorways

If you're using the grid to plunk down your rooms you either need to
wait until you draw the hallways (and draw them carefully), or use a
flood fill algorithm.

If you don't always use fixed rooms (like Crawl) I suppose you could
also flood fill with each entry having a certain radius before it
transitions to the next one.  It's trickier.

Other thoughts:  
Not all levels should have a theme.  Some themes should be smaller than
others, maybe only a few connected rooms, with the rest of the level
being more random.  
To set up the themes you could make them more abstract.  Make the sample
above generic to humanoids (xxx lair) and fill in with the details fro
the appropriate level at run time (Orc Lair, Goblin liar, etc).  Or you
could make a large Monster Map connecting the likely monsters/groups. 
One whole edge of the map should be 'random monster'.
I recommend that no more than about 1/3 to 1/2 of your dungeon be filled
with this kind of theme, to keep the replay value.
If you include stairs with doors as transition points, it's possible to
spread the theme in 3d, and no longer required to keep the stairs so far
from the center (just treat it like you would a door).
Along with the monsters, make the objects more continuous too.  The
Goblin Guards might have an extra sword lying around, but the peasants
aren't going to have 50gold for very long.  The empty rooms near the
Lair might thave signs of the local occupants - grafitti in the goblin
tongue, for example.