Language is beautiful. The study of language, linguistics, less so.
Consider the following, from the Internet Encyclopedia of Philosophy:
Compositionality of stereotype: the stereotype associated with a complex expression E in a natural language is determined by (and only by) (i) E’s morphosyntactic structure and (ii) the stereotypes associated with E’s morphemes.
Compositionality of semantic features: the semantic features (e.g. [+male] or [+animate], as they attach to ‘he’ and ‘who,’ respectively) of a complex expression E in a natural language is determined by (and only by) (i) E’s morphosyntactic structure and (ii) the semantic features of E’s morphemes.
It goes like this for each possible type or dimension of meaning.
The philosophical question is which, if any, of these theses is true.
I for one don’t have much interest in figuring out which if any of those thangs (technical term) are true. But here’s the thing: it doesn’t matter. All that matters is: what do we need to give a neural network in order for it to recognize the hidden dimensions of meaning? By which I mean, whatever it is that we understand every day, all day.
If I say, “Can you go easy on me?” or “Give me a break” or “Lay off me, would ya?” or riddle you with expletives laced with ‘off’ or ‘out,’ or even just give you a certain look, they all communicate the same thing.
What you’re doing when you and I use different ways of saying the same thing is that you’re recognizing something which is essentially the same. Sometimes it may be exactly the same.
You recognize exactly what I’m saying. Even though it it may be expressed by any number of different words, sentence rearrangements. It is as though we toss clumps of meaning over a wall to one another, like dung. And words are the dung-bag we use. We want not to focus on the bags, but on what really counts: the dung.
Well. There’s a definition of linguistics that’s a little easier to stomach. So to speak.
That is very, very similar to what happens in image recognition. I (or you, or the machine) recognizes peaches even though they’re sometimes on a tree, sometimes on the ground, sometimes in a bowl, sometimes close, sometimes far, sometimes in a pie, sometimes in jam (more difficult) etc.
What is the difference? Well one could say that a peach is a peach, and has certain properties (size range, color range, flavor, texture, etc.) whereas the word groups may share little in common, i.e. none of the words may be the same. However, what we’re trying to do is to get the computer to recognize meaning. Meaning may be operationally defined as that which is similar across multiple word groups: a recognizable pattern across word groups, all of which or some of which may be unique, ie not contained in other word groups.
How would we do that? I think by re-thinking word embeddings: how they’re developed, what they’re based on, and how they might more accurately reflect, in their relationships to one another, the real world.
The one in the clumps that we hurl to one another, over that fence.