Ideally, they should be outdoors, use naturally hot water directly extracted from a natural volcanic spring, and they are often embellished with (or, in the cheaper varieties, replaced by) a wide variety of extravagant spa baths, artificial waterfalls and saunas. The essential difference between an onsen and a Sento (communal bath house) is that the water in an onsen must be volcanic spring in origin, even if reheated, whereas a sento may use ordinary heated water. Onsen water is often thought to have healing powers according to its mineral properties and onsens often have several different baths, each augmented by the addition of different minerals or the composition of the tub.
Note that at an onsen, as in Sento you wash your body and rinse yourself thoroughly before you enter the hot water. This is vital in a public place as entering the onsen while still dirty or covered in soap will cause uproar. At the very least, if you have not washed (perhaps because you showered half an hour earlier) use the scoop provided to splash water over your genitals and feet, thus symbolically cleansing them.
The most important features of the onsen by far are the water and the bathing facilities, which is why many bathers simply come for an hour or so to soak in the waters even if they do not stay. Probably the next most important issue for Japanese guests is the food; a good onsen inn (or Ryokan) will offer what it claims is something special in the way of the evening meal. Because Ryokan tend to gently pressure people towards eating their evening meal at set times (e.g. 6pm) the baths are often deserted around this hour of the day; this is a good time to hit the tub. While massage and other services are often offered, they are peripheral. Some of the most (deservedly) famous onsen in Japan are the most rustic and spartan establishments, such as Tsuru-no-Yu in Akita prefecture.
If you go with your colleagues to an onsen it is certainly possible that the experience of being buck naked together in a pool of hot water will break down some of the hierarchical stiffness inherent to Japanese work life. However, most visitors to onsen are not work groups but friends, couples and families. It is not unusual to see a father or mother introducing a small child to the onsen for the first time. Very small children of either sex up to about 3-4 years old can often be seen in both male and female baths. Mixed-sex bathing is a tradition that persists at onsen in the more rural areas of Japan. Wearing swimsuits is quite rightly forbidden at most of the better onsen; if nudity offends you then it is better to stay at home.
Onsen are places to relax and although the baths are usually silent it is a silence punctuated by the odd sigh or grunt of satisfaction. There is nothing wrong with talking quietly to friends or partners. People in the same bath or pool may well strike up a conversation with you (at least if you are a foreigner). For Japanese, initiating conversation with a stranger is fairly unusual but for nearly all Japanese people onsen are not a usual part of life but a respite from it. Essentially, the onsen should be the diametric opposite of everything in normal, hectic day-to-day Japanese life.