To put it bluntly, trolls are just a later incarnation of the jotuns. They aren't distinct species. Of course, this hasn't stopped modern writers from trying to force them to coexist in the same stories. I too am guilty of this.
Having jotuns and trolls in the same story sounds redundant, doesn't it? When you get right down to it, the only difference is that jotuns are always gigantic whereas trolls may range in size from giants to dwarves (in the dictionary and folkloric sense, not modern fantasy gaming's definition taken from Tolkien). In fact, trolls seem to be closer to more generic conceptions of fairies and nature spirits as compared to other cultural mythologies of Europe (e.g. the "wood nymph with a hollow back" is considered a type of troll). So you could easily argue that jotuns are a subset of trolls, specifically the biggest.
Of course, since we aren't making any pretensions of being accurate (insofar as that ever made sense for myths that weren't consistent in the first place) then maybe we could devise different criteria for distinction? Trudvang Chronicles, for example, maintains the distinction that jotuns are elemental beings whereas trolls are not. However, trolls are commonly considered giants in fantasy gaming. Could this be reconciled?
I think that, mirroring the evolution of Norse mythology into Scandinavian mythology as a result of Christianity, the trolls could be explained as degenerate jotuns that have lost their elemental power and sometimes shrunk in size. One side effect of this is that trolls are photosensitive, often turning to stone under sunlight.
Although it isn't mentioned nowadays, jotuns were originally thought to turn to stone under sunlight. This explains both why they aren't around anymore and why a lot of rock formations resemble faces (in reality, the human brain is simply wired to see faces where none exist). Since trolls are an evolution of the jotuns in folklore, they inherited this weakness.
However, not all trolls myths describe them as photosensitive. Depending on the country of origin, the trolls in stories may be entirely photosensitive, not photosensitive, or photosensitive only sometimes. In fact, aside from the names being cognates there are seemingly no traits that all trolls in all Scandinavian tales share in common.
Of course that's really annoying to us modern folks with our anal-retentive obsessive-compulsive taxonomies, so lets just pretend that for the purposes of our modern stories and games that all trolls have common properties selected from a single list. That list is:
- Tufted tails
- Horns (and/or tusks and/or other bony growths)
- Huge noses (and/or ears and/or other cartilaginous growths)
- Photosensitive (variable severity ranging from being nocturnal to dying)
- Turn to stone upon death (and/or entering any comatose state)
- Like other giants, may have extra limbs and/or heads
Trivia: The "wood nymph with a hollow back" has a number of different names in the Scandinavian languages.
- elverpige (elver "an elves" + pige "a girl", "a she-elf")
- ellepige (elle "an alders" + pige "a girl", "a dryad")
- huldra ("the fairy")
- skogsrå (skog "a forest's" + rå "a fairy", "a dryad")
