Self-description is output, not knowledge
A chatbot writes sentences about being an AI because such sentences appeared in its training and in its instructions, not because it inspected itself. The same system will describe itself differently under different instructions, which shows the description is produced rather than read.
The question sounds philosophical and it has a practical edge: everything a chatbot says about its own limits, model, memory and rules is generated the same way as everything else, and none of it should be treated as a report.
The phrase was trained in; nothing observed it
As an AI language model appears because that construction was frequent in training data and is reinforced by instructions. Nothing in the system checked what it is before writing it. The sentence is a likely continuation, and its truth is a coincidence of the training rather than a result of inspection.
The same system describes itself differently
Change the hidden instructions and the self-description changes with them: a different name, a different set of stated limits, a different claim about memory. If the description were read from the system, it would not move when the wrapper moved.
Why this matters for what you ask
People test a chatbot by asking what it can do. The answer is generated text about capability rather than a capability report, so it can be confidently wrong in both directions: claiming a limit that is not there and denying one that is. Testing the behaviour settles it and asking does not.
What to do instead
Ask the system to do the thing rather than to describe whether it can. A refusal, an attempt, or a failure is evidence, and a statement about itself is a sentence. This is the same reason a product page is weaker evidence than a page that shows an example.
Before you paste it in
So is everything it says about itself wrong?
Not wrong by default, and not verified either. Some self-descriptions match reality because the instructions said so and the instructions were accurate.
Why does it sometimes deny being an AI?
Because the instructions or the conversation pushed it towards a persona. That is another demonstration that self-description follows the wrapper rather than the system.
Can it tell me which model it runs?
It can produce a model name. Whether that name is correct depends on whether somebody put it in the instructions, and there is no way for the model to check.
Ask it to do something rather than to say whether it can.
Open uncensored chatbot