I still have my Nokia 5800 lying around. It runs Symbian. That platform is quite different from Android. For one thing the internal storage size is 81 MB - less than what your single android app takes up, even in RAM, and it works faster than modern smartphones - e.g. the SMS app opens instantaneously. Another peculiarity of the latter is that unlike any modern chat app, including the android SMS app, it doesn't actually divide your messages into chats and presents them in a single "inbox" - like what you see in most e-mail clients.
Aside from stimulating coherent communication (where you'd want to write your thought as a single coherent message, not bombard your interlocutor with two-word brainfarts like zoomers do) it meant that communications probably went at a different pace. It was not the only type of messaging interface - IRC clients certainly didn't look like that - but from the fact that such interface was acceptable we can conclude that messaging culture looked different. That's the era when xmpp was developed. Oh wait, actually xmpp was conceived yet another decade before this phone would be released. Smartphones didn't even exist back then. Texting went at a yet different pace.
That's why it's so "buggy" today: the devs did not have the modern use case in mind. Who could've known twenty years ago we'd all be carrying our electronic dopamine stimulants in our pockets, maybe also have a slightly bigger portable pleasure/espionage rectangle lying around just because, maybe also pay extra to have espionage built in to our watches and so on and so forth, and reasonably expect all those to synchronise between each other?
Everything that we expect today - chat history, reliable message delivery, multi-device synchronization - is an afterthought in XMPP.
If you take any modern chat platform, it will basically be a database. You don't just send messages to your pal, you two share a chat history together which you two edit. There are some rules to this editing of course, like you can't spoof message date and so on, but ideally you can edit any message any time, you can delete any message any time, and that will affect both your profiles. Today you expect a messenger to be a database. Such is very much the case with Telegram, and with Matrix the database is also distributed. People call out a metadata disaster in Matrix, but it functions as it should - it is a database, it stores data.
Xmpp per se, without extensions, does NOT see your chats as a (distributed) database. To the xmpp protocol, in essence, only messages exist that are passed around and forgotten. It may store your message temporarily if the recipient is not currently online - or it may not. This isn't even specified by the RFC.
See excerpt from the Matrix docs:
Matrix defines APIs for synchronising extensible JSON objects known as "events" between compatible clients, servers and services.Ctrl+F "event" on the Matrix page and you find 73 matches, just on that page alone, without following any links. Do the same for the core xmpp rfc - what few matches you find will be irrelevant. Xmpp doesn't think in such terms.
Ctrl+F "synchronis" on that page and you'll get 8 matches. If you do the same for the xmpp rfc you'll get zero matches - xmpp didn't have this in mind.
Similar to Matrix in this regard is Telegram. But xmpp wasn't designed to compete against Telegram. It was designed to compete against IRC if anything.
Whenever I try bringing someone over to XMPP we stumble into messages being lost - again, XMPP didn't account for OSes like android that kill apps at random and do other "optimizations". Only my most patient friends remain, and while the situation has improved over the years, we still occasionally realize that some message was not delivered. And it's good when we realize it at all.
Today there's an ongoing competition between companies at producing as slow and as buggy webpages as possible while employing as many people as possible. But while ReactJS was invented less than a decade ago, bullshit jobs in general have existed much longer. In the late nineties there was a competition at using a certain already bloated markup format as a serialization format, in as silly and as ugly a way as possible. First time reading XML-RPC examples made me smile - for a moment I thought it was a joke like IP over Avian Carriers. XMPP is also a child of that competition, or at least greatly inspired by it. And while XMPP's go at the competition, unlike XML-RPC, doesn't seem as bad at first sight and won't make you think it's a prank, you'll soon learn that in practice it is actually much more wicked. XML-RPC at least works in complete documents transferred over http, as laughable as those documents may be, so you may use one of the many XML libraries already available for virtually every language, (good chunk of them vulnerable and never fixed).
XMPP streams XML over tcp. An unitiated person might think XML streaming refers to length-prefixed or record-separator documents over tcp, but hey, remember about the competition?
Instead the entire XMPP stream is one big XML document that is never closed:
<stream:stream from="gmail.com" id="{ID}" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features>
<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"><required/></starttls>
<mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
<mechanism>X-GOOGLE-TOKEN</mechanism><mechanism>X-OAUTH2</mechanism>
</mechanisms>
</stream:features>
At that level, the document goes on forever. No </stream:stream> is seen ever, except in the unlikely scenario where your client or server disconnects and decides to send this closing tag as well, for no practical reason. An XMPP protocol message is an immediate child element of that document and is referred to as stanza. Doesn't seem that bad? See parsing.
<starttls> alone means, without xmlns="urn:ietf:params:xml:ns:xmpp-tls"? And there would definitely be name collisions without it! Now, unfortunately for all the advantages this feature provides, there's a slight disadvantage: wasting bandwidth. This is especially laughable in XMPP's attempt at reliability, see:
<a xmlns='urn:xmpp:sm:3' h='1'/>
There are 2 meaningful characters here: "a" (it's an ack) and "1" (1 stanza ack'd). That xmlns='urn:xmpp:sm:3' takes 19 characters.
To solve this totally unavoidable problem, XML offers us namespace prefixes. In the previous stanza example, note that <stream:features> doesn't have to specify xmlns="http://etherx.jabber.org/streams" (ah, that http:// is an especially useful inclusion!) - instead the namespace is associated with the prefix stream in the very top tag once and now whatever element that has this namespace can specify the shorter prefix.
Luckily now <a xmlns='urn:xmpp:sm:3' h='1'/> can be shortened to <a:a h='1'/>. Sike! In practice, many xmpp servers and clients will break should you feed them any prefixes other than the strandard xmlns:stream="http://etherx.jabber.org/streams". And if you don't specify this one, they may also break. And if you don't specify namespaces at all, they'll break too. So yes, <a xmlns='urn:xmpp:sm:3' h='$N'/> will be sent all over again all the time. It also means you get lovely stanzas like
<message id='aeb213' to='juliet@capulet.lit/chamber'>
<result xmlns='urn:xmpp:mam:2' queryid='f27' id='28482-98726-73623'>
<forwarded xmlns='urn:xmpp:forward:0'>
<delay xmlns='urn:xmpp:delay' stamp='2010-07-10T23:08:25Z'/>
<message xmlns='jabber:client' from="witch@shakespeare.lit" to="macbeth@shakespeare.lit">
<body>Hail to thee</body>
</message>
</forwarded>
</result>
</message>
Where <forwarded> just isn't enough to convene that the message is forwarded, <delay> just isn't enough to convene that there's a delay and so on.
<b>Hello</b> <i>world</i>!Neat right? Something XML is actually good for! XHTML-IM was however deprecated. What XMPP went for instead is this:
<message>
<body>There is really no reason to worry.</body>
<markup xmlns="urn:xmpp:markup:0">
<span start="9" end="15">
<emphasis/>
</span>
</markup>
</message>
What... what the hell did I just read?
There is a less terrifying alternative that again doesn't use xml to do the actual markup. This one is actually not pointless: after all, if you type it by hand, >quote is much handier than say <q>quote</q>. So maybe XML just has no place in instant messaging to begin with?
It can come in quite handy to be able to uniquely identify a message. Like with a hash, you know. But you can't hash stanzas in XMPP. Well, theoretically you could, but in practice you won't, and if you do, others won't. Thus you'll probably want to assign randomly generated ids to stanzas. Luckily, there's already an id attribute:
<message id="{e.g. a UUID}">...</message> But again, XMPP is wary of collisions - and there really can be UUID collisions. So they came up with an idea of allowing several entities to assign their UUIDs simultaneously:
<message xmlns='jabber:client'
to='room@muc.example.com'
type='groupchat'>
<body>Typical body text</body>
<stanza-id xmlns='urn:xmpp:sid:0'
id='5f3dbc5e-e1d3-4077-a492-693f3769c7ad'
by='room@muc.example.com'/>
<origin-id xmlns='urn:xmpp:sid:0' id='de305d54-75b4-431b-adb2-eb6b9e546013'/>
</message>
Ah sweet, those repeating namespaces again, that verbosity... Now, in this particular example the ids take up 180 characters, while the actual message a human supposedly wrote is 18 characters. And this is not unrealistic, and this is their own example. That's a ten time difference in size between data and its id, and the id is the bigger one. At this point one could as well be just comparing the message itself - it'd actually be more effecient. And what percentage of instant texts exceed 180 characters?
Note that this overhead achieves nothing. Readability and ease of use are actually impeded by XML here no less than bandwidth effeciency is. The problem (inability to hash) is created by overdosing on XML in the first place.
Now the question comes up, how do you parse this? There most probably are ready made solutions for parsing xml one "stanza" at a time, but this (ab)use-case is so rare it's unlikely that you'll find a library that is available for your particular language, has been updated in the last decade, and also isn't tightly-coupled to an existing XMPP client.
Of course they don't reimplement the actual parsing - this isn't bencode where you could write a complete parser in a couple of hours just for fun, and besides there's no need to, since libs like expat are available in virtually every language, complete with a ton of features you won't need, a history of some vulnerabilities or insane defaults to the point where Gajim still tells Expat explicitly not to use DTDs to not leak IP (although this in particular should no longer be necessary).
What they do reimplement is DOM. Problem is, XML by itself doesn't map nicely into native language structures like lists or dicts or even bools. How do you find if the server requires starttls from the mess below?
<stream:features>
<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"><required/></starttls>
<mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl">
<mechanism>X-GOOGLE-TOKEN</mechanism><mechanism>X-OAUTH2</mechanism>
</mechanisms>
</stream:features>
You could use xpath: boolean(/tls:starttls/tls:required). Oh yeah, xpath, basically yet another language. Do you want to spend time implementing it? If you don't... you're still going to come up with and create helper functions to e.g. find nodes by name, etc. Or you could manually loop over the DOM every time. Either way that's still not a 1 hour project.
For this brief note's purposes we'll go with a mixture of 3rd and 5th options: we'll use expat, but we'll tell it to ignore namespaces, because we're lazy. It won't be 100% correct xml and we'll need a dom parser that can ignore namespaces as well - luckily, tdom can do that. And yes, we too will break when encountering non-standard ns prefixes.
The wicked logic will boil down to this:
...
method OnElemStart {tag attrs} {
append Stanza "<$tag [join [lmap {k v} $attrs {set - $k='[xesc $v]'}]]>"
if {$Depth==0} {
lappend Queue $Stanza
set Stanza {}
}
incr Depth
}
method OnElemEnd tag {
append Stanza $tag>
incr Depth -1
if {$Depth==1} {
lappend Queue $Stanza
set Stanza {}
}
}
method OnCdata cdata {
append Stanza [xesc $cdata]
}
...
XMPP uses port 5222 and is by default not encrypted. Just picking a standard port for encrypted XMPP, like it is done with HTTP (80 vs 443) would be too simple for the XML mindset - there are SRV records for specifying a port, but there is no standard port. There's STARTTLS as well and it is described in the RFC itself, so that's what we'll go for, to really get a taste of the XMPP core.
The way it's supposed to go is:
<!--You send this unholy mess to the server over plaintext:-->
<stream:stream to="{host}" xml:lang="en" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:stream from="{host}" id="{ID}" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client">
<stream:features><starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'><required/></starttls></stream:features>
<!--You tell the server that you want to enable tls-->
<starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>
<failure xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>. Because obviously the server can randomly change its mind and the protocol accounts for that -->
<proceed xmlns="urn:ietf:params:xml:ns:xmpp-tls"/>
Now you start TLS. Don't forget to destroy the XML parser you previously set up for that socket. All of this instead of a standartized port.
You may be wondering why it is important for the server to specify that it supports STARTTLS (because if we're serious about secuirty, we'll just ignore it anyway, or refuse to connect), or why it is important for the server to additionally confirm it's ok with tls again straightaway. I'm wondering too.
All in all, so far our code for starting tls can look like this: set chan [socket -async $host $port]
XmlStream create stream $chan
stream write "<stream:stream to='[xesc $host]' xml:lang='en' version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>"
stream read ; # opens stream, don't care what it says
stream read ; # <features/>, don't care what it says
stream write "<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>"
stream read ; # <proceed/>, don't care what it says
stream destroy
set chan [tls::import $chan] ; # start tls
XmlStream create stream $chan ; # create parser anew
stream write "<stream:stream to='[xesc $host]' xml:lang='en' version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>"
stream read ;# again opens stream, nothing of importance for now
puts [stream read] ;#finally stream features
All of this instead of a standard port. And this is a very bare implementation, disregarding half the specification.
Nothing insane here. Quite simple really, just make sure our half-assed sasl implementation is compatible with the server and send it the first and only sasl step, get a <success xmlns='...'/>, realize that tcl's xml library, tdom, is quite awkward at this use case, almost making me want to do something like exec xmllint -xpath $xpath - << $val, but xmllint can't shut up about namespaces.
set features [connect $host $port stream]
# yikes, I understand XML sucks, but like, couldn't this be *slightly* more convenient?
set d [dom parse -ignorexmlns $features]
set mechanisms [lmap n [[$d documentElement] selectNodes {mechanisms/mechanism/text()}] {$n nodeValue}]
$d delete
if {"PLAIN" ni $mechanisms} {
error "We don't support anything but PLAIN, and server offers us: $mechanisms"
}
# authenticate
set saslPlain [base64::encode "\0$login\0$password"]
stream write "[xesc $saslPlain] "
set authResp [stream read]
set d [dom parse -ignorexmlns $authResp]
if {[[$d documentElement] nodeName] ne "success"} {
error "Couldn't authenticate: $authResp"
}
$d delete
# the stream is started over *again*
set chan [stream getChan]
stream destroy
XmlStream create stream $chan
stream write ""
stream read ;# opening tag
stream read ;# features
Now we can finally send a <message to="juliet"></message>. Sike, still more rituals.
Up to this point we could read the stream linearly, but now we need to support XMPP's request-response facility, <iq/>, among other things. I'll leave the implementation details out, let's just say I add a new class to encapsulate more advanced features like said <iq/>, which we don't need until this step.
Assuming our previous steps are incapsulated in a single login function:login $login $password $host $port $client.stream
XmppClient create $client $client.stream
set reply [client iq set "<bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'>\
<resource>poorly-written-client</resource></bind>"]
$client.stream write <presence/>
Note how awkward that xml looks in code thanks to the xmlns. It's like they're trying to make it worse.
It is important that before you start texting you also send that <presence/> tag. At least the server I'm using, draugr.de, outright didn't react to attempts of sending messages before this - not even an error reported.
<presence/> is in fact so important that it takes up 70% of all stanzas sent. In response the server will let us know what other devices we have, but we don't care for now
Anyway, now we can send that <message></message>. Let's send one to ourselves:
$client.stream write "<message to='wusspuss@draugr.de' from='wusspuss@draugr.de' type='chat'><body>Hi!</body></message>"And get it back:
<message xml:lang='en' to='wusspuss@draugr.de' from='wusspuss@draugr.de/half-assed-client' type='chat'><body >Hi!</body></message>Note how XMPP so prudently specifies that the message is in English with that
xml:lang='en'.
We also hear a notification sound if we have another client connected. Or we don't. Conversations shows a notifications, Dino doesn't, and I don't think either is considered a bug. It is an eXtensible Messaging Protocol, not a Well-Defined one or whatnot.
I made the laziest attempt possible at a user interface:
hook bind ::$client <Message> me {apply {stanza {
set d [dom parse -ignorexmlns $stanza]
puts "[$d selectNodes string(/message/@from)] texts: [$d selectNodes string(/message/body)]"
$d free
}}}
while 1 {
set input [coroutine::util gets stdin]
...
$client.stream write "<message to='[xesc $recipient]' from='[xesc $login@$host]' type='chat'>\
<body>[xesc $input]</body></message>"
}
We can try poking around and sending a message from a different client to ourselves and our client will receive it. We will notice that:
<message/> may not be a message at all, and telling what it actually means apparently isn't all straightforward.
c2s-s2s-s2c chain work flawlessly, given that implementations are different.
yay -S tcllib tdom tcltls
apt install tcllib tdom tcl-tls
Note username, host and password need to be specified in the source code, in proc ui
To send a message just type it and press enter; to pick the addressee type /pick N where N is a number from the list displayed upon startup or a jid
foreach m {coroutine hook oo::util tdom tls base64} {package require $m}
# escape xml
proc xesc {content} {
string map {
< <
> >
& &
\" "
' '
} $content
}
# break xml into "stanzas"
oo::class create XmlStream {
variable Expat Stanza Depth Chan Queue
constructor {chan} {
set Expat [expat [self].expat -final no \
-elementstartcommand [mymethod OnElemStart]\
-elementendcommand [mymethod OnElemEnd] \
-characterdatacommand [mymethod OnCdata]]
set Depth 0
set Chan $chan
set Data ""
set Queue ""
fconfigure $Chan -blocking 0
}
method read {} {
while {$Queue eq ""} {
fileevent $Chan readable [info coroutine]
yield
$Expat parse [read $Chan]
fileevent $Chan readable {}
}
set stanza [lindex $Queue end]
set Queue [lreplace $Queue [set Queue end] end]
return $stanza
}
method write {data} {
puts $Chan $data
flush $Chan
}
method OnElemStart {tag attrs} {
append Stanza "<$tag [join [lmap {k v} $attrs {set - $k='[xesc $v]'}]]>"
if {$Depth==0} {
lappend Queue $Stanza
set Stanza {}
}
incr Depth
}
method OnElemEnd tag {
append Stanza </$tag>
incr Depth -1
if {$Depth==1} {
lappend Queue $Stanza
set Stanza {}
}
}
method OnCdata cdata {
append Stanza [xesc $cdata]
}
method getChan {} {set Chan}
destructor {$Expat delete}
}
oo::class create XmppClient {
variable Stream IqHandlers IqId
constructor {stream} {
set Stream $stream
set IqHandlers {}
set IqId 0
coroutine [self].run my run
}
method run {} {
while 1 {
set stanza [$Stream read]
set d [dom parse -ignorexmlns $stanza]
switch [[$d documentElement] nodeName] {
iq {
switch [$d selectNodes string(/iq/@type)] {
result {
set id [$d selectNodes string(/iq/@id)]
{*}[dict get $IqHandlers $id] $stanza
dict unset IqHandlers $id
}
error {"Got unsupported iq: $stanza"}
}
}
message {
hook call [self] <Message> $stanza
}
presence {
hook call [self] <Presence> $stanza
}
default {error "Unknown stanza: $stanza"}
}
$d delete
}
}
method iq {type payload {cb ""}} {
set cb [info coroutine]
incr IqId
dict set IqHandlers $IqId $cb
$Stream write "<iq id='[xesc $IqId]' type='[xesc $type]'>$payload</iq>"
yield
}
method write {$stanza} {
$Stream write $stanza
}
}
# Step one: starttls, restart stream
proc connect {host port stream} {
set chan [socket -async $host $port]
XmlStream create $stream $chan
$stream write "<stream:stream to='[xesc $host]' xml:lang='en' version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>"
$stream read ; # opens stream, don't care what it says
$stream read ; # <features/>, don't care what it says
$stream write "<starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>"
$stream read ; # <proceed/>, don't care what it says
$stream destroy
set chan [tls::import $chan] ; # start tls
XmlStream create $stream $chan ; # create parser anew
$stream write "<stream:stream to='[xesc $host]' xml:lang='en' version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>"
$stream read ;# again opens stream, nothing of importance for now
return [$stream read] ;#finally stream features
}
#Step two: auth, restart stream again
proc login {login password host port stream} {
set features [connect $host $port $stream]
# yikes, I understand XML sucks, but like, couldn't this be *slightly* more convenient?
set d [dom parse -ignorexmlns $features]
set mechanisms [lmap n [[$d documentElement] selectNodes {mechanisms/mechanism/text()}] {$n nodeValue}]
$d delete
if {"PLAIN" ni $mechanisms} {
error "We don't support anything but PLAIN, and server offers us: $mechanisms"
}
# authenticate
set saslPlain [base64::encode "\0$login\0$password"]
$stream write "<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='PLAIN'>[xesc $saslPlain]</auth>"
set authResp [$stream read]
set d [dom parse -ignorexmlns $authResp]
if {[[$d documentElement] nodeName] ne "success"} {
error "Couldn't authenticate: $authResp"
}
$d delete
# the stream is started over *again*
set chan [$stream getChan]
$stream destroy
XmlStream create $stream $chan
$stream write "<stream:stream to='[xesc draugr.de]' xml:lang='en' version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client'>"
$stream read ;# opening tag
$stream read ;# features
}
# Step three: bind (also presence)
proc bind {login password host port client} {
login $login $password $host $port $client.stream
XmppClient create $client $client.stream
set reply [client iq set "<bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'>\
<resource>half-assed-client</resource></bind>"]
$client.stream write <presence/>
# uncomment to see incoming presences:
# hook bind ::$client <Presence> test puts
}
# laziest user interface imaginable, specify your login data in the code
proc ui {client} {
set login YOURLOGIN ;#without host
set password YOURPASSWORD
set host YOURSERVER;# e.g. draugr.de
set port 5222
bind $login $password $host $port client
set d [dom parse -ignorexmlns [$client iq get "<query xmlns='jabber:iq:roster'/>"]]
set roster {}
set recipient $login@$host
puts Roster:
foreach n [$d selectNodes /iq/query/item/@jid] {
puts "[llength $roster]. [lindex $n 1]"
lappend roster [lindex $n 1]
}
hook bind ::$client <Message> me {apply {stanza {
set d [dom parse -ignorexmlns $stanza]
puts "[$d selectNodes string(/message/@from)] texts: [$d selectNodes string(/message/body)]"
$d delete
}}}
puts "Default recipient is yourself, $recipient"
while 1 {
set input [coroutine::util gets stdin]
if {[lindex $input 0] eq "/pick"} {
if {[string is digit [lindex $input 1]]} {
set recipient [lindex $roster [lindex $input 1]]
} else {
set recipient [lindex $input 1]
}
} else {
$client.stream write "<message to='[xesc $recipient]' from='[xesc $login@$host]' type='chat'>\
<body>[xesc $input]</body></message>"
}
}
}
coroutine m ui client
vwait forever