Generally Speaking

Think about these two examples:

1. The hummingbird interviewed the penguin.
2. The the interviewed penguin hummingbird.

It's unlikely that you have seen either of these examples before. Nonetheless, you can probably tell that example A is a possible English sentence (it's certainly strange, but it is grammatical—that is, it obeys all the rules of English sentence structure), while example B could not be an English sentence.

How are we able to make judgments about sentences we have never seen before? This question relates to the broader topic of generalization—applying what you have learned to new situations or examples. Generalization is an important concept in computational linguistics because we want to build computers that can effectively generalize to sentences they have never seen before.

At NACLO Labs, to study generalization, we have created four different computers that are able to learn from data. The computers are named C1, C2, C3, and C4, and they have never encountered any English sentences before our experiment. We first show the computers the 17 sentences below, which we will refer to as the training set—the set of examples that the computers learn English from. We tell the computers that all 17 of these sentences are grammatical.

Training set:

1. The linguist met the programmer.
2. The cartoonist saw the spy.
3. The watchmaker is asleep.
4. The happy concierge met the ballerina.
5. The spy saw the woodcarver.
6. The spy is knowledgeable.
7. The main detective visited the ballerina.
8. The main calligrapher saw the astronaut.
9. The programmer visited the blacksmith.
10. The detective met the yodeler.
11. The spy is happy.
12. The ballerina is famous.
13. The linguist is tall.
14. The talented watchmaker visited the astronaut.
15. The famous programmer visited the blacksmith.
16. The cheerful concierge met the ballerina.
17. The haberdasher saw the linguist.

Note that all of the nouns in these sentences are types of occupations (for example, a haberdasher is someone who sells men's clothing), but for this problem you do not need to know the meanings of these words.

We now show the computers 21 more sentences, which are listed in the table below. For each sentence, we ask each computer whether it thinks the sentence is grammatical (G) or ungrammatical (U). Their answers are recorded in the table. We have also included the answers that a human provided when shown the same sentences.

D1. Some of the cells in the table are empty (marked (a)–(n)). Provide the missing entries.

D2. The table below contains some more sentences, along with C2 and C3's judgments for those sentences. In each sentence, one word has been hidden from you by replacing it with the string HIDDEN_WORD_1, HIDDEN_WORD_2, or HIDDEN_WORD_3. Write down what each HIDDEN_WORD stands for, choosing from the following options: asleep, happy, or main. Additional notes:

• You should use each option (asleep, happy, or main) exactly once.
• Each HIDDEN_WORD stands for the same word in both of the sentences where it appears.

D3. For each sentence in the table above, think about what your own labels would be. Out of these 6 sentences, how many are there where C2 produced the same label as you? How many are there where C3 produced the same label as you?

NOTE: Different people may have different judgments about these sentences. Therefore, as long as you write something for D3, you will get credit—it doesn't matter what you write! We just want you to think a bit about which computer matches your own judgments the best.

Ideally, we would want our computers to make exactly the same predictions as a human would. If you look back at sentences 18 to 38, you will see that C1 and C2 often undergeneralize—there are sentences, such as sentence 25, that these computers label as ungrammatical (U) even though the human called them grammatical (G).

In the other direction, C4 sometimes overgeneralizes—it labels some sentences as grammatical (G) even though the human labeled them as ungrammatical (U). Depending on how you answered question D3, there may also be some examples of undergeneralization or overgeneralization in sentences 39 to 44 as well.

The examples of overgeneralization that we've seen so far were about the domain of sentence structure. Overgeneralization can also happen on the level of individual words. For example, based on pairs of words like walk and walked, or laugh and laughed, a computer might overgeneralize and produce the word thinked (based on think) or bringed (based on bring).

How might word-level overgeneralization affect our interactions with a computer? As an illustration, below are some interactions between a human and a hypothetical robot. The robot has been trained on a very large amount of English text, and it often overgeneralizes when forming words. Each word that resulted from overgeneralization has been hidden from you by being encoded in a cipher, where each letter has been replaced with a different letter. The same code was used for all examples—for example, if X stands for w in the first encoded word, then it will also stand for w everywhere else.

D4. Decode each encoded word from the conversations below. We are not providing the robot's training set, so it will not be possible to understand the robot as precisely as C1, C2, C3, and C4 from the previous part of the problem. However, if you carefully analyze what the robot says and also figure out the code, it should be possible to determine what each encoded word stands for.

An important challenge in computational linguistics is figuring out how to get computers to generalize in the right way. Generalizing can also be a challenge for humans acquiring languages. For example, when acquiring English, children often overgeneralize by producing words such as thinked or bringed.