An argument for verified humans (Part IV)
In Part III of this series (linked below), inspired by a quote from Deep Learning with PyTorch (further below), to build toward propositions for (1) verified humans on social media platforms and (2) transparency regarding the use of language models to generate text, I discussed the possibility of a machine forming a thesis by exploring both mechanical desires and algorithmically generated worldviews.
Herein, to further build and to develop a framework for evaluating the ethics of natural language processing applications, I will explore two famous rooms, i.e., (1) John Searle’s Chinese room and (2) Virginia Woolf’s room of one’s own, to evaluate the truthfulness of the claim made in the original quote that “self-awareness is definitely not required” to perform natural language tasks.
Imagine a native English speaker who knows no Chinese locked in a room full of boxes of Chinese symbols (a data base) together with a book of instructions for manipulating the symbols (the program). Imagine that people outside the room send in other Chinese symbols which, unknown to the person in the room, are questions in Chinese (the input). And imagine that by following the instructions in the program the man in the room is able to pass out Chinese symbols which are correct answers to the questions (the output). The program enables the person in the room to pass the Turing Test for understanding Chinese but he does not understand a word of Chinese.²
It seems the most demonstrative examples (e.g., the Stanford marshmallow experiment for delayed gratification, king minus man plus woman equals queen for word embeddings) become like refrains, and so it is quite likely that the reader is aware of John Searle’s oft-discussed Chinese Room argument, with its supposition of a person who does not understand a lick of Chinese alone in a room with nothing but instructions for converting Chinese inputs into Chinese outputs. The Chinese room was proposed as an analogy for the computer, and as the reader may remember, the conclusion of the argument provided by Searle is that, while a computer given syntactic rules can perhaps manipulate symbols in what appears to be a reasonable manner, the computer itself does not have an understanding of meaning or semantics.
There are many responses to the Chinese room argument, including, among others, (a) the Systems reply, which states that the man is only part of a larger system, i.e., the room, and that the system, due to its ability to process the Chinese inputs, does in fact understand Chinese, and (b) the Robot reply, which, while conceding that an unembodied computer cannot understand words and meanings, claims that an embodied computer could learn through experience as a child learns (as discussed in Part III of this series). Herein, however, rather than discuss either the initial argument, its most famous replies, or my own thoughts regarding its implications for computational systems, I would like to consider the Chinese room as a framework for evaluating the ethics of natural language processing applications.
But before applying Searle’s room as an ethical framework for evaluating natural language tasks, to provide a useful contrast, I will first commit an egregious sin against feminism by considering Virginia Woolf’s A Room of One’s Own — an essay on (or, more precisely, surrounding) women and fiction — entirely without sex and rather as a work centering on a room that can be considered the antithesis of Searle’s room, in that in Woolf’s room, there is no database with which to convert symbols, but instead fleeting sparks and the tugs of whim and memory (how patchwork and fickle with urgency) that comprise one’s creativity, for it is the black box of a room of one’s own (herein, more precisely, the mind), occupied as it is by a single person with a singular perspective, for which input does not neatly produce output, as from within this room, inputs are imprinted upon and contorted by past, mistake, and counterfactual, perchance, and manipulated in a way inseparable from the days upon days of experience through which one lives before taking pen to paper, fingertips to keys, hindsight to forethought to suggest —
Thought — to call it by a prouder name than it deserved — had let its line down into the stream. It swayed, minute after minute, hither and thither among the reflections and the weeds, letting the water lift it and sink it, until — you know the little tug — the sudden conglomeration of an idea at the end of one’s line: and then the cautious hauling of it in, and the careful laying of it out? Alas, laid on the grass how small, how insignificant this thought of mine looked; the sort of fish that a good fisherman puts back into the water so that it may grow fatter and be one day worth cooking and eating.³
How in essence all of which one can be aware is that to which one is exposed — the pleasure and pain provoked during each lived day (how experience carves its pathways like a chemical etching), the organic messiness of it (so without guidelines for transfer or transcription)! But I am preciously elevating the human experience, perhaps beyond its station, because even with their differences, the rooms are not so different, Woolf’s and Searle’s — the logical and the lyrical — for within each there is a lone person who has only a limited context with which to produce a response to the world beyond the walls (and yet still he or she does).
So it seems quite appropriate to use as a basis these two rooms to evaluate the ethics of automating natural language tasks by asking simply: Which room must one occupy?
As stated in Part III, machines were created by humans to automate our toil. As such, when considering formerly human-only tasks, it seems reasonable to first consider the main goals of the task and the characteristics of the type of person who would be able to achieve those goals. More specifically, instead of evaluating only a computer’s ability to achieve human-level results, we should also evaluate the consideration with which an end user would expect the task to be completed if it were to be completed manually. (Note that, herein, the end user for a given application is the person who provides the input and receives the output.) In other words, when evaluating ethics, rather than considering only the end result, the means of producing the end result should also be considered.
The above distinction (i.e., end result vs. means of producing the end result) is important to the ethical use of natural language processing tools because of the social aspect of communication, which requires interlocutors to implicitly agree to adhere to certain social norms, as discussed in Part II. In essence, for certain tasks (e.g., “Hey, Siri, who was the second president of the United States?”), the end user may be more concerned with the correctness of the response than any other feature of the interaction. However, for other tasks, the end user may want the input to provoke a more subjective or otherwise nuanced response, even if the output is not necessarily optimal. For counter example, if a person were to seek therapy after the loss of a loved one, then a human therapist may be a better option than an algorithm, even if the algorithm could provide better advice, as in such cases, the human element — of receiving the time and attention of a person — might be more important than the matter-of-fact usefulness of any advice. In other words, it is not simply the correct use of language that an end user may want, but rather the insight of a being who can, for example, empathize or discuss lived experiences.
Now, before discussing specific natural language processing tasks, let us consider a toy example of two classes of elementary school students, i.e., Mrs. Woolf’s class and Mr. Searle’s class, who are given the same (rather vague) instructions: Write an essay about [some specified book]. For such a task, a student in Mrs. Woolf’s class may provide insights based on lived experience but misinterpret the book’s content, while a student in Mr. Searle’s class might produce a report that correctly paraphrases facts from the book yet provides no insight. (Of course, Mrs. Woolf’s student could be factually correct as well, but that case is slightly less interesting, as it is more lopsided.) The grades received by the students depend on whether the grader values effort over accuracy or vice versa: If the grader prefers accuracy, then paraphrasing from a seat in Searle’s room may be sufficient; in contrast, if effort is preferred, then the student seated in Woolf’s room would receive the higher grade. Notably, in this example, for the students to perform well, it helps for the rubric to be predefined, as for such tasks, the expectations of the person receiving the output must be considered.
As discussed in the previous section, an end user may prefer for a task to be completed from Woolf’s room if he or she expects the input to be considered from a subjective or experienced perspective (i.e., if he or she expects a thesis to be formed); thus, it is unethical to automate most such tasks — especially if the person is unaware that the task has been automated or if the person has only a vague understanding of the capabilities of the machine with regard to the given task. However, if an end user can trust that a task can be reasonably accomplished by a person seated in Searle’s room, i.e., by a person who can follow rules to produce an output but does not necessarily understand the meaning of the input, then most such tasks can be ethically completed by an algorithm, assuming that the end user is aware of the automation and that the algorithm can achieve an adequate level of accuracy.
In this section, I will discuss how the ethical framework presented previously can be applied for three natural language applications: question answering, translation, and chatbots. Note that, because I believe that the framework is fairly generalizable, I will not provide details specifying the language models used for these applications; even so, I do assume that the models are trained only on text sources or handcrafted rules (and thus that sensory data is not used to derive experiential knowledge).
Question answering
Of the three tasks, the question answering task is the most similar to the task considered in the Chinese room argument, and so it may seem as if question answering is an obvious candidate for automation. However, not all questions can be answered with the one-to-one correspondence described in Searle’s argument (i.e., given input [x], return output [y]). Thus, in this subsection, I will contrast two types of questions: (1) objective questions and (2) subjective questions. (Note that the other two types of applications, i.e., translation and chatbots, can be posed as variations of the question answering task.)
For objective questions, if there is one answer for a given question, and if that answer does not change depending on the person answering it, then for any input question, an understanding of the semantic meaning of the question is not necessarily needed for the correct output to be provided. Therefore, an end user would likely be willing to accept an answer that was generated in a way similar to that supposed in Searle’s room, and thus the task is ethical to automate. Note that this type of question answering task is essentially an information retrieval task, where the information returned should match the information requested.
In contrast, for subjective questions, there may not be a single answer that can be returned given the input, as the answer should depend on the experience of the person answering. Therefore, in this case, it is unethical for a machine to provide a response as if it holds the purported opinion. For (a low-stakes) example, suppose an end user asks a machine to identify the best pizza place in town. The model may have been trained on data that enable it to produce a coherent response to this input, e.g., perhaps the algorithm makes a claim based on the sentiment of reviews on a review site; however, as a machine cannot itself form an opinion on pizza — being unable to consume pizza and provide a human judgment of pizza quality — an end user would likely prefer to know the opinions of human townies who have eaten from the relevant pizza places. Therefore, in this case, rather than responding as if the computer took on the identity of a person who has eaten pizza (and thus burying its sources), it would be more ethical to retrieve related sources so that the end user can judge their trustworthiness for themselves.
Translation
As stated above, translation tasks are not so different from question answering tasks, as any translation task can be posed as a question of the form “How does one say [word | phrase | sentence| etc.] in [language].” As such, factors such as subjectivity vs. objectivity and transparency also come into play when evaluating the ethics of machine translation applications (although in slightly different forms).
Let us first discuss transparency by supposing a person wants to translate a sentence into his or her native language. In this case, machine translation may be reasonable, as the end user may not deem it necessary for the translation to be performed by a person with full understand of both languages. Specifically, because the end user is aware that a machine will translate the input, he or she can consider oddities in the output with a grain of salt (assuming that the translation is not incomprehensible). Therefore, according to the proposed framework, machine translation is ethical for this task, as a person in Searle’s room with no knowledge of either language could meet the end user’s expectations.
In contrast, suppose that machine translation is used to translate an article about a current event from one language to another language for publication. In this case, readers of the translated article might assume that it was written by a speaker of the translated language, i.e., by someone seated in Woolf’s room. Therefore, because the expectations of the end user may not be met (as the translation was not produced by a person who understands the language and intended to convey the meaning of the words), it is unethical to post such content unless readers are informed that the content has been translated by a machine.
In the above example, transparency is imperative because words carry both objective (denotative) and subjective (connotative) meanings. Specifically, the connotation of a word in the original language and context may take on a vastly different contextual connotation in translation — even if the translated word is similar to the original in terms of denotative meaning. Thus, using a machine to translate an article could result in the conveyance of inappropriate sentiment or otherwise incorrect information. Furthermore, in literary works, such as novels and poetry, the connotative and phonetic elements of words are often important for conveying imagery and metaphor, and thus (although perhaps of lesser ethical concern), machine translation may not be able to as fully capture the intentions of the writer as a human translator.
Chatbots
Although chatbot applications surely have subjective and objective aspects to consider, these issues seem no different in essence than those discussed in the question answering section and thus will not be expounded upon here. So let us instead focus on issues of transparency in the use of computer-generated text. Specifically, let us examine how the Searle-Woolf framework relates to a (Turing test)-like scenario on a social media platform, wherein the end user is unaware that they are interacting with a machine.
As I have discussed previously, a computer is said to pass the Turing test, not if it exhibits human-like intelligence, but if it can fool a human into believing that it is a human through a chatbot-style communication task. However, as stated in the last subsection, if an end user expects a text to be produced by a person seated in Woolf’s room, then due wholly to the failure to meet this expectation, the application is unethical. Therefore, any machine that aims to fool a human (i.e., any bot that is not obviously labelled as a bot), regardless of whether it succeeds in passing the Turing test and of whether it speaks truth or lies, is an unethically applied machine.
Note that the potential for the deception of human users, which breaks the implicit rules of communication, is key to this final example. Furthermore, while perhaps less obviously deceitful than the lack of labelling a chatbot as such, a bot can also be said to deceive a person if that person does not have a good understanding of the abilities of the bot, relative to those of a human, as the onus is then on the end user to evaluate the algorithm used to produce the text, which may not be possible in practice. Moreover, the issue of deception suggests a further ethical dilemma regarding accountability in the automation of formerly human-only tasks. Specifically, because a machine cannot be held accountable for the text it generates, combined with the possibility of a model voicing opinions not intended by the person who deployed the model, it may not be possible at present to ameliorate the negative impacts produced by machine-generated texts.
In this article, by considering several natural language processing tasks in terms of their ability to be ethically automated, I showed that self-awareness may be required for certain natural language tasks if the expectations of an end user are to be met successfully. However, it is important to note that, even if it is ethically permissible to automate a task in theory, it is not necessarily ethically permissible to deploy certain models in practice, as the ability of the model to perform the task must also be considered as a prerequisite for ethical deployment. Therefore, while the framework described here can be used to evaluate the task itself, it says nothing of the deployment of an actual model.
In the final part of this series, I will further explore issues of accountability and draw from the discussions presented in Parts 1 through 4 to support propositions for (1) verified humans on social media platforms and (2) transparency regarding the use of language models to generate text.