Many archaeologists believe that a stylistic feature is likely to appear only once in the archaeological record - once it disappears it is unlikely to reappear. The implication is that a seriation with a feature present for a number of consecutive artifacts and then absent for a number of subsequent artifacts and then present again for even later artifacts is less realistic than a seriation where the artifacts having a feature are not interrupted by some that do not.
The transition penalty is a means of enforcing this (unimodality is another). The transition penalty for a feature is incurred each time the feature transitions from absent to present or vice versa in a seriation. The larger the penalty, the less likely OptiPath is to create a seriation with interrupted occurrences of a feature.
The transition penalty for a feature can be set in the Features table. Right clicking on the column brings up a popup menu that allows you to set all features with one click.