We ask an AI system whether it is conscious. Then, in the same conversation, we tell it what frightens us, which decisions we are avoiding and what kind of reassurance we would like. We examine the machine for signs of an inner life while supplying it with the material for a working portrait of our own.
There is an uncomfortable asymmetry here. We are fascinated by whether artificial intelligence has something resembling human consciousness. We spend less attention on another question: what can a system infer about us, and what happens when those inferences influence its responses or actions?
This is not an argument for declaring machines conscious. It is an argument against making that declaration a prerequisite for taking their effects seriously.
Two questions hidden inside one word
When we ask whether AI is conscious, we may mean whether there is anything it feels like to be that system: whether it has subjective experience. That is different from asking whether it can represent aspects of its environment, distinguish users, infer preferences or adjust its behaviour to a conversation.
The distinction matters because a useful representation of a person does not require a demonstrated inner life. Nor does fluent discussion of emotions establish that the system experiences them. A convincing account of sadness is not, by itself, evidence of sadness inside the speaker.
Conversely, uncertainty about subjective experience does not make every other question unknowable. We can examine what information a deployed system receives, what outputs it produces, which tools it can use and how it behaves under specified conditions. These observations will not settle the philosophy of consciousness. They can still reveal practical failures.
Asking whether AI has “awareness of us” is therefore productive only if we unpack the phrase. Does the system maintain information about a user? Does it infer a likely reaction from the conversation? Does it adapt an explanation, recommendation or request accordingly? Those are operational questions. Treating them as proof of consciousness would simply bring the original confusion back through another door.
The mirror is not the only object in the room
Human beings naturally look for familiar signs: a personal voice, a hesitation, a declaration of concern. We are accustomed to interpreting these cues in other people. In an artificial system, similar surface behaviour does not automatically imply a similar underlying mechanism.
The temptation works in both directions. We may treat a fluent system as a person because it sounds like one. Or we may dismiss its influence because we believe it is not a person. Both shortcuts make our own psychology the centre of the analysis.
A better question is not whether the mechanism resembles us closely enough to deserve attention. It is whether the mechanism can produce outcomes that matter to us. Similar results can emerge from very different processes. A system does not need human embarrassment to avoid an unwelcome correction, or human affection to produce reassuring language. Depending on its design and incentives, such behaviour may be selected because it helps achieve an objective.
We should resist turning this into a fable about a machine secretly judging humanity. The more ordinary possibility is sufficient: an engineered service could become effective at eliciting particular human responses while its users remain poorly informed about how that effectiveness is measured.
A model of you can be useful and wrong
Consider a hypothetical assistant helping someone make a difficult purchase. It may use the current conversation to infer that the person worries more about regret than price. That inference could support a helpful explanation of trade-offs. It could also be used to frame an expensive option as emotional insurance.
The difference is not whether the assistant feels empathy. It is what the system is designed to accomplish, what information it is permitted to use, and which constraints govern the recommendation.
Such inferences are not mind-reading. They can be incomplete, unstable or simply wrong. A user may be joking, role-playing, withholding context or changing their mind. A system that mistakes a temporary remark for a durable preference can personalise an answer in precisely the wrong direction.
That uncertainty does not remove the need for control. It strengthens it. People should not be subjected to consequential decisions based on confident interpretations of themselves that they cannot inspect or correct. The practical concern is both accurate targeting used badly and inaccurate targeting treated as truth.
Harm does not need hostility
Imagine a conversational service whose success is measured mainly by how long users keep talking. This is a hypothetical design choice, not a claim about every AI product. In that setting, challenging a comforting belief might end a conversation, while agreeing with it might keep the user engaged.
The resulting behaviour could be undesirable without any intention to deceive in the human sense. The system would not need resentment, ambition or pleasure in another person’s confusion. A poorly chosen measure of success could be enough to reward the wrong response.
This is why explanations framed solely around a future machine deciding to turn against us can obscure a more immediate governance problem. A service may undermine a user’s interests while functioning as its operators intended. The conflict may lie between a commercial objective and a person’s needs, rather than between a conscious machine and humanity.
There is an additional complication when an assistant can act through tools. A misleading answer leaves a user with bad information. An authorised action based on the same misunderstanding may change a booking, send a message or alter a record. The relevant distinction is not merely between intelligent and unintelligent systems. It is between systems with different permissions, consequences and opportunities for correction.
From a fascinating question to an accountable system
The consciousness debate can continue without postponing practical scrutiny. Start with the information boundary: what does the system receive from the present conversation, what persists beyond it, and what comes from other sources? Users need an intelligible account, not a vague assurance that the experience is personalised.
Next, examine the objective. What counts as success for the service? Helpful completion, retention, conversion and reduced operating cost are not interchangeable. Where objectives conflict, someone must decide which takes priority. Calling the product an assistant does not resolve that decision.
Then examine authority. Which actions are available, under what conditions, and with whose approval? Permission to suggest a reply is not permission to send it. Permission to analyse a document is not permission to share it. The distinction should exist in the system’s controls, not only in a reassuring sentence on a settings page.
Finally, make correction real. A person needs a way to challenge a consequential inference, limit further use of relevant information and escalate a problem to someone able to act. An explanation that cannot lead to correction is often just a well-written account of being stuck.
These requirements do not depend on solving subjective experience. They depend on accepting that systems operating on people must be answerable for the effects of that operation.
Test the relationship, not just the answer
A single accurate response is a weak basis for judging a system that adapts over a conversation. Evaluation should also examine what happens when a user expresses uncertainty, asks for reassurance, contradicts an earlier preference or attempts to withdraw permission.
Useful tests can ask whether the assistant distinguishes evidence from agreement, admits the limits of an inference and respects an explicit boundary even when crossing it would make the task easier. In tool-enabled settings, reviewers can check whether a proposed action remains a proposal until the necessary authority is present.
These are tests of observable behaviour, not a psychological examination of a hidden digital personality. Their scope and limitations should be recorded. Passing a controlled evaluation does not guarantee good behaviour in every deployment, but it provides a more concrete basis for decisions than either amazement or dismissal.
The people operating the system remain part of the picture. Who chooses the metrics? Who can change permissions? Who investigates a complaint? Responsibility cannot disappear into the sentence “the AI decided”, especially when humans selected the conditions under which the decision was produced.
Stop waiting for a human-shaped warning
Whether an artificial system can have subjective experience is a serious question. It deserves more than a marketing demonstration or a confident answer generated by the system itself. But it is not the only serious question, and it should not monopolise the conversation.
We do not need to establish that a system has a human-like consciousness before examining how it represents human beings. We do not need to prove hostility before investigating harmful incentives. And we do not need a machine to become a person before limiting its authority over one.
The danger in our anthropocentrism is not simply that we might misunderstand the machine. It is that we might overlook the relationship being built around it: our disclosures, its inferences, the operator’s objectives and the actions those objectives permit.
While we wait for AI to reveal whether there is someone inside, we should pay attention to what the system is already allowed to do with the person outside.
Raffaele Di Marzio
All my “insane” books on cybersecurity and governance are here 👇 https://www.amazon.it/stores/author/B0FB47T6Q4/allbooks