Do good summaries exist?
Some comments on defining summary as a textual genre.
Istituto Discipline della Comunicazione
Università di Bologna
One question that frequently recurs in debates about text summarizing is how to evaluate summaries. What should the evaluating criteria be, or, one might even ask, is it possible to set any criteria at all? Given the wide variety of texts that can be summarized and the different purposes which are served, it may seem reasonable to conclude that summaries cannot be evaluated at all.
In this paper, however, I would like to argue that some general principles do exist which can enable us to evaluate summaries. Before we can specify these principles, though, we must examine the way summaries are constructed and the constraints inherent in their textual structure.
In other words, before we can discuss what makes a "good" summary, we must first decide what a summary is [1].
According to one standard dictionary definition, a summary is: "a comprehensive and usually brief abstract, recapitulation, or compendium of previously stated facts or statements". (Random House, The Unabridged Edition)
This is not very satisfactory for our purposes, since we must now look up "abstract", "compendium" and "recapitulation", only to find that "abstract", for example, is defined as "a summary of a statement, document, speech etc." (ibid.). Furthermore, the inclusion of scalar terms in the definition, such as "comprehensive" and "brief", leaves us uncertain as to just how these terms should be applied. "Comprehensive" is a very subjective feature, and "brief" is both vague and relative: many texts are in fact shorter than others without being summmaries of them.
At this point, instead of trying to find out what a summary is, let me ask a different question: What kind of text is a summary? This slightly different formulation implies an important shift in perspective, since it forces us to look at a summary as a particular textual genre. Textual studies on genre have revealed that each genre is endowed with very specific constraints on its structure. A personal letter, for example, does not have the same structure as a document or a scientific report: the genre imposes linguistic, stilistic, semantic, structural choices on the texts that belong to it. [2]
One specific feature of a summary is that it cannot be defined autonomously; that is, a summary always derives its contents from some other text (a statement, document, speech, etc. - see below). I would like to suggest that a summary can be defined as a "parasitic" text that may be evaluated only in relation to the source text.
Thus, "summary" as a genre crosses over other traditional genres, making it a kind of hyper-genre. Both narrative and scientific texts can be summarized; these summaries will exhibit relevant differences and, presumably, will satisfy different purposes.
However despite these differences, I believe that it is possible to identify some constraints on the genre "summary", as defined by its relationship with the source text. These constraints affect the textual structuring of a summary and determine what portions of the source text will be selected, a process which is clearly not arbitrary or unconstrained. This fact of the existence of constraints implies that it is possible to make an evaluation of a summary. Constraints may be weaker than rules as a measure for judgement, but when they are violated, we recognize that the summary is deficient in some way. This "negative" approach may not allow us to define what the good summary of a given text should be, but it gives us some criteria for judging what a summary should not be.
In examining the constraints operating on the structure of a summary, I suggest that they are of two kinds: extratextual and textual.
By extratextual constraints, I mean a hierarchy of saliency that is not specific to any given text but is generally valid [3]. Let us look at how these constraints operate in the following text taken from a newspaper [4]:
"Two bank robbers were shot dead last Friday when a high-speed car chase ended in a gun battle in the city centre. IRA involvement has not been ruled out by detectives. The drama began shortly after 10 am when the men held up a bank 5 miles west of Dublin. They made off with 2,500 pounds but their car was spotted by armed detectives on the edge of the city and a high-speed car chase followed. The Garda car came under repeated fire from the raiders. The chase ended when the robbers tried to smash through a second Garda car that was blocking the road. A prolonged shootout followed as more armed Gardai arrived. The two men inside were reported dead on arrival in hospital. Gardai could not say who fired the first shot."
Common sense advises us that a summary of this text should report the fact that the two men died, and omission of this fact would make the summary deficient. It should be noted that this judgement does not depend on any quantitative statistics; since the Gardai are, in fact, mentioned more often in the article than the death of the two robbers, which appears only twice, at the beginning and at the end of the text. The reason why we feel that the death is essential information lies not in the text itself or its structure, but rather in an extratextual scale of relevance. We feel that the information about death should be kept in a summmary because death is in itself relevant for human beings. Life and death, as anthropology tells us, are basic categories of our experience and as such they are endowed with a text-independent saliency. These are the kind of elements I call extratextual constraints.
Textual constraints, on the other hand, are strictly dependent on the structure of the text itself. Looking back at the above text, we note that the fact that the two men were killed by Garda men is textually more relevant than the fact of the robbers' possible involvement in the IRA. Let me note here that in discussions about the possibility of evaluating summaries, it is frequently objected that summaries tend to be task-oriented. That is, the hierarchy of relevance of a summary's elements depends on the particular task and therefore any "objective" evaluation is impossible. For example, a summary based on a search for possible crimes connected with the IRA would contain very different information from a summary of Garda actions. However, we are no longer really talking about summaries at this point. Obviously, it is always possible to construct texts according to a preselected topic (this is the principle that guides information retrieval) but the result does not necessarily have to be a summary of the source text. A "summary" refers to a set of texts (i.e. a textual genre) that exhibit certain features in relation to their source texts, and in particular a summary reproduces in its internal structure the same hierarchy of relevance among its parts that the source text exhibits.
Let us consider another more complex example. A few years ago, I conducted an experiment with first-year university students, asking them to write a one-page summary of one of the most well-known Italian novels: I promessi sposi by Alessandro Manzoni. All Italian students are acquainted with this novel, since it is part of the mandatory final year high-school program. This 19th century novel narrates the story of a young peasant couple who want to get married and cannot because of the opposition of a local squire who lusts after the joung lady, but who do finally get married in the end. This is a very rough summary of the main plot that could be further reduced and seen as the transition between two states:
initial state: not-married
final state: married.
In the approximately three hundred pages that it takes the couple to get married, many minor characters are introduced and many subplots are developed. Some of these minor plots are actually autonomous and self-defined stories, sometimes more interesting and perhaps artistically superior to the main plot with its rather boring central couple. But the novel is about that couple, and a good summary of the novel should, therefore, give predominance to the main plot over all subplots.
A summary has to respect the same proportional distribution of elements which is present in the source text. This is not always easy to do in writing a summary. Most of the students involved in the experiment had trouble keeping a balance between all the different parallel stories in the novel. They tended to give too much weight to the subplots; but a summary that deals 90% with the subplots and only 10% with the main plot cannot, in my opinion, be considered a reasonable summary of the original text. The same principle holds true in non-narrative texts as well; a good summary, in this case, must reflect the relative importance of the various subjects and parts of the text.
So far I have suggested that there are two kinds of saliency: an external saliency determined by our experience of the world and an internal saliency determined by the text and the relative importance of its different parts. Though operating at different levels, both kinds of saliency can be seen as constraints on the structure of a summary. On the one hand, the extratextual constraint induces us to believe that a summary should include certain high priority elements (such as death). The textual constraint, on the other hand, leads us to judge that a summary should reproduce in its structure a similar hierarchy of relevance among components to that exhibited by the source text.
Let me now discuss one aspect of summary structure, namely the question of length. For the same source text, we can have summaries of different length and therefore also with very different levels of abstraction. A one-paragraph summary of a novel will necessarily be much more abstract and general than a two-page summary of the same text. One may ask, at this point: how concise should a "good" summary be? Or, to put the question another way: is there an ideal or "spontaneous" level of abstraction for summarizing?
In answer to this question I can offer only some suggestions based on my own experiences. In a journalism class where I was teaching writing techniques, I asked students to make five summaries of the same newspaper article, which was about 1000 words long and a report of an event. The summaries were all of different lengths, starting with a two-page summary and ending up with a single sentence. Of these five summaries, the longest version was the most difficult to write, and the results were generally judged as the least satisfactory by the students themselves. At the other extreme, the one-sentence summaries were not even really summaries, but were constructed more as possible titles. ([5]) The "best" summaries were those of medium length, approximately 20-25% of the original.
Though this data is not systematic and needs to be confirmed with further research, it reveals certain interesting tendencies, at least with regard to the kind of texts that we were looking at, namely short reports of events. The students' results seem to indicate that some implicit norm or convention operating for the summary genre may lead us to write summaries of a certain length. Jerry Hobbs (personal communication) suggests, in fact, that the reduction 4 to 1 is not casual. In his opinion, a typical exposition has a summary sentence and then three sentences decomposing the event into finer-grained subevents. Of course, such a schema is not directly applicable to all types of texts, but would seem to apply to the ones we examined.
The summary represents the top node of the tree.
In general it is a well known fact that we prefer to describe events or objects at a certain level of abstraction. Studies in human categorization [6] have shown that for categories such as natural kinds or manufactured objects there is a basic level of categorization. We are more likely to say: "I have a cat" than "I have an animal" or "I have a Siamese". It is not unreasonable to assume that a similar principle operates for actions and events as well. We generally "lump together" certain sequences of actions and refer to them as a single event. We say "This morning I got up early and had breakfast". "To get up" and "to have breakfast" are events that we perceive as unitary, and we do not break them down into smaller units. That this grouping of actions into a larger event is a natural tendency can be perceived more readily if we describe a situation such as the one above at a very unusual level of details. For example instead of saying "I had breakfast", we could list a sequence of actions, such as "I picked up a cup, I picked up the coffee-pot, tipped the coffee-pot, poured the coffee into the cup, took a spoon, put the spoon into the coffee and stirred the coffee" (and we could go on breaking down each of these actions into even smaller subactions). The result is disconcerting because it is so out-of-the-ordinary. This technique was exploited in France in the 1960's by the école du régard and in particular by Robbe-Grillet to create an unusual fine grain of description.
This kind of description produces a distancing effect on the reader - an effect of "estrangement" as was called by the Russian Formalists - the result of the fact that the author has broken the implicit norms that govern the level of abstraction. This leads us to believe that a basic level of abstraction exists for actions and events, as well as for natural categories. Such reasoning could also lead to the conclusion that one level of abstraction is more natural than others in all kinds of abstractive processes.
That some sort of general device does operate in summarizing was indicated by the way students constructed their summaries in the experiment with my journalism students. In fact, they generally followed one of two different strategies. Some of them made summaries out of their previous summaries, taking as the source text the result of their previous reduction, while others always referred back to the original source text. The latter group clearly produced better summaries, and we tried to analyse the reason for this. Setting aside the possibility that the latter group was simply better at making summaries, we arrived at an interesting hypothesis. It appears that summarizing involves more than just leaving out progressively more text; that is, a three-sentence summary cannot be produced by simply deleting two sentences of a five sentence summary. A good summary is, in effect, a condensed form of representation of the original text. For this reason, it is always necessary to go back to the original text and construct a more abstract version. This idea is consistent with the claim of a basic level of abstraction in summaries. It is clearly more productive to access this level directly than through progressive reductions.
Thus, we come to the question of the relationship between summarizing and text interpretation which we may now view from a new perspective. If we take summarizing and interpretation as cognitive processes, it should be clear that the two are strictly connected and cannot be separated, at least as far as human summarizing is concerned. Automatic summarizing may be quite different and may not require a previous understanding of the source text, but humans need to understand a text in order to make a summary, and understanding always implies making some kind of interpretation. However, if we look at summary and interpretation not as cognitive processes, but as different textual genres, they can, and indeed should be kept distinct. A "good" summary not only should respect the original balance among parts of the source text, but also should not introduce judgements, evaluations, or causal explanations that are not explicitly stated in the source text. These additions, however, are precisely what characterizes a textual interpretation. Text interpretation as a specific kind of discourse is in this sense less constrained than summary and more dependent on the reader's point of view - whether that means knowledge base or subjectivity.
To refer once again to Manzoni's novel, a text that focuses only on a minor detail of a minor character of a subplot of the novel and makes that the main reading of the whole novel could be an interesting interpretation, but definitely could not be a summary of the novel. Thus, we can say that summary and interpretation, as textual genres, differ in the amount of inferential content that may be included. "Good" summaries tend to exclude extra-textual information, while "good" interpretations are based on just that kind of inferential information, i.e., on a set of associations that was not in the source text.
Let us now turn to another aspect of the issue of summary evaluation which is the supposedly task-oriented nature of a summary. There are several questions that come up here: Does the textual structure of a summary depend on the purpose for which the summary was written? Do we need summaries for different purposes? What, for that matter, is the purpose of a summary?
Notice that I am talking here of "having" a summary and not "writing" a summary, which is a different question. Indeed we can write summaries for many different reasons, one of the most common is to help our process of understanding and memorizing of a given text. However, what I am considering here is the possible use of a summary as an already produced text, which is the question addressed by automatic summarizing. In other words, what are summaries good for?
It seems to me that there is only one possible answer to the last question: we need summaries in order to have a shorter text; in other words we use summaries to save time and economize on the processing of large amounts of information.
This may seem a trivial answer, but it becomes more interesting if we ask ourselves not why we make summaries, but when we make summaries: when, that is, we want to save time. We want to save time when we are dealing with texts that we have to read (or we have to decide whether to read them or not), but do not particularly enjoy reading, texts, in other words, that we do not read out of the barthesian "pleasure of the text" but out of the need to acquire more information on a given topic. This is the case with scientific papers, technical reports, most newspaper articles, and so on.
When, on the other hand, we read something out of the pure pleasure of reading, we do not need, or indeed want, summaries, because we do not want to shorten the pleasure of reading (if anything we want to draw it out). This is why we find the idea of reading a summary of Giulietta and Romeo to be somewhat absurd. After all a summary would deprive us of the full in-depth experience of the original text, which is the main point of reading it. Indeed, in the case of literature, summaries do not make much sense, unless they are to serve as writing practice for students or as an aid in such tasks as rapid learning of the plot. Our goal, in this case, is no longer enjoyment of the text, but the need to get certain information out of it.
The reluctance to summarize certain texts is thus independent of the quality or the artistic level of the source text and depends only on the attitude of the reader. As long as the reader's goal in reading a particular text is enjoyment, then it will not matter whether the text is a Shakespearian play or a detective story. [7]
To conclude, a summary can be seen as a particular kind of text defined by its relationship with the source text. Given that the source text can belong to any different textual genre, the summary genre will intersect with all other traditional genres, and become a kind of hyper-genre. This implies that its definition cannot be based on content elements, but only on formal elements (i.e. the specific structural features necessary for a text to be a summary of another text). At least in principle, we can have summaries of any kind of text, regardless of its format, structure or quality. In practice, however, we use summaries only when we are especially interested in acquiring information from a given text, for example for the purpose of selecting which texts to read, rather than in enjoying the reading itself. [8] In these cases, the main task in summarizing is always to save time while preserving the most general level of informational structure. Such a general task imposes specific constraints on the textual structure of summaries, as I have tried to show, and characterize the summary genre as a whole.
References
Dijk T. van, 1980, Macrostructures, Hillsdaly, New Jersey, LEA,
Eco U., 1979, The Role of the Reader, Bloomington, Indiana University Press.
Fillmore C., 1976, "Topics in Lexical Semantics", in Cole R. (ed.), Current Issues in Linguistic Theory, Bloomington, Indiana University press.
Fillmore C., 1977, "The case for Case reopened", in Cole P. and Sadock J. (eds), Syntax and Semantics. Grammatical Categories, vol.8, New York, Academic Press.
Fløttum K., 1986, "Methodological problems in the analysis of student summaries", in L.S. Evensen (ed.), Nordic Research in Text Linguistics and Discourse Analysis, Trondheim, Tapir.
Genette G., 1982, Palimpsestes, Paris, Seuil.
Moreau J., 1977, La contraction et la synthèse des textes, Paris, Nathan.
Morfaux L., Prévost R., 1984, Résuméet synthèse de textes, Paris, Armand Colin.
Rosch E. and Lloyd B. (eds.), 1978, Cognition and Categorization, Hillsdale, Lawrence Erlbaum.
Todorov T. 1978, Les genres du discours, Paris, Seuil