Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| en:iot-open:networking2:applicationnetworkprotocols [2024/05/27 13:56] – pczekalski | en:iot-open:networking2:applicationnetworkprotocols [2024/08/28 18:56] (current) – pczekalski | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| + | ====== Application Protocols ====== | ||
| + | {{: | ||
| + | The host layers protocols include session (SES), presentation (PRES) and application (APP) levels. In particular, the APP (application) layer in regular Internet communication is dominated by the HTTP protocol and XML-related derivatives, | ||
| + | Although advanced and more powerful IoT devices frequently use these protocols, this is problematic to implement in the constrained IoT devices world: even the simplest HTTP header occupies at least 24 + 8 + 8 + 31 bytes without payload! \\ | ||
| + | There is also a problem of crossing firewall boundaries when communication between subnetworks of IoT devices is expected to occur. \\ | ||
| + | \\ | ||
| + | Some IoT-designed protocols are reviewed below. | ||
| + | |||
| + | == MQTT == | ||
| + | MQTT protocol ((https:// | ||
| + | |||
| + | MQTT uses a TCP connection, so it requires an open connection channel (this is opposite to UDP connections, | ||
| + | |||
| + | The standard MQTT Message header is composed of just two bytes only (table {{ref> | ||
| + | |||
| + | <table MQTT_header> | ||
| + | < | ||
| + | ^bit ^ 7 ^ 6 ^ 5 ^ 4 ^ 3 ^ 2 ^ 1 ^ 0 ^ | ||
| + | |byte 1 | Message Type |||| DUP flag | Qos level || RETAIN | ||
| + | | byte 2 | Remaining length | ||
| + | </ | ||
| + | |||
| + | MQTT requires a centralised MQTT Broker located outside firewalls and NATs, where all clients connect, send and receive messages via the **publish/ | ||
| + | <figure mqtt_broker> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | |||
| + | **MQTT Message**\\ | ||
| + | MQTT is a text-based protocol and is data-agnostic. | ||
| + | A message comprises a Topic (text) and a Payload (data). | ||
| + | The Topic is a directory-like string with a slash ("/" | ||
| + | The subscriber can subscribe to a specific, single Topic or a variety of Topics using wildcards, where: | ||
| + | * # stands for the entire branch, | ||
| + | * + stands for the single level. | ||
| + | |||
| + | //Example Scenario// | ||
| + | |||
| + | Publishers deliver some air quality information data in separate MQTT messages and for various rooms at the department Inf of the Universities (SUT, RTU, ITMO) to the Broker: | ||
| + | ^ Topic (publishers): | ||
| + | | SUT/ | ||
| + | | SUT/ | ||
| + | | SUT/ | ||
| + | | RTU/ | ||
| + | | ITMO/ | ||
| + | | RTU/ | ||
| + | | SUT/ | ||
| + | | RTU/ | ||
| + | |||
| + | Subscriber 1 is willing to get all sensor data for SUT University, Inf (informatics) department only, for any space: | ||
| + | ^ Topic (subscription): | ||
| + | | SUT/ | ||
| + | Subscriber 2 is willing to get only Temperature data virtually from any sensor and in any location in TalTech University: | ||
| + | ^ Topic (subscription): | ||
| + | | TalTech/#/ | ||
| + | Subscriber 3 is willing to get any information from the sensors, but only for the RTU | ||
| + | ^ Topic (subscription): | ||
| + | | RTU/# | | ||
| + | |||
| + | The message' | ||
| + | |||
| + | **MQTT Broker**\\ | ||
| + | MQTT Broker is a server for both publishers and subscribers. The connection is initiated from the client to the Broker, so assuming the Broker is located outside a firewall, it breaks its boundaries.\\ | ||
| + | The Broker provides QoS (Quality of Service) and can retain message payload. MQTT Broker QoS (supplied at the message level) has three levels. | ||
| + | * Unacknowledged service: Ensures that MQTT message is delivered at most once to each subscriber. | ||
| + | * Acknowledged service: Ensures message delivery to every subscriber at least once. The Broker expects acknowledgement to be sent from the subscriber. Otherwise, it retransmits data. | ||
| + | * Assured service: This two-step message delivery ensures the transmission is delivered exactly once to every subscriber. | ||
| + | Providing unique packet IDs in the MQTT frame is vital for Acknowledged and Assured services.\\ | ||
| + | |||
| + | The DUP flag (byte 1, bit 3) represents information sent by the publisher, indicating whether the message is a "first try" (0) or a retransmitted one (1). This flag is primarily for internal purposes and is never propagated to the subscribers. | ||
| + | |||
| + | MQTT offers a limited set of features (options): | ||
| + | * clean session flag for durable connections: | ||
| + | * if set //TRUE//, the Broker removes all of the client subscriptions on client disconnect, | ||
| + | * otherwise, the Broker collects messages (QoS depending) and delivers them on client reconnecting; | ||
| + | * MQTT " | ||
| + | * message retaining: it is a feature for regular messages. Any message can be set as retaining; in such case, the Broker will keep the last one. Once a new client subscribes to a topic, they will receive a retained message immediately, | ||
| + | |||
| + | Interestingly, | ||
| + | |||
| + | MQTT security is relatively weak. The MQTT Broker can offer user and password verification sent in plain text. However, all communication between the client and Broker may be encapsulated in an SSL-encrypted stream. | ||
| + | |||
| + | A short comparison of MQTT and HTTP protocols is presented in table {{ref> | ||
| + | <table http_vs_mqtt> | ||
| + | < | ||
| + | | ^ MQTT ^ HTTP ^ | ||
| + | ^ Design | ||
| + | ^ Pattern | ||
| + | ^ Complexity | ||
| + | ^ Message size | Small, with 2 byte binary header | ||
| + | ^ Service levels | ||
| + | ^ Implementation | ||
| + | ^ Data distribution models | ||
| + | </ | ||
| + | |||
| + | == CoAP == | ||
| + | |||
| + | CoAP protocol (RFC7252) originates from the REST (Representational State Transfer). CoAP does not use a centralised server as MQTT does, but every single device " | ||
| + | CoAP uses URIs to address services: | ||
| + | * %%coap://< | ||
| + | * a secure, encrypted version uses " | ||
| + | It supports various content types, can work with proxy and can be cached.\\ | ||
| + | The protocol is designed to be compact and straightforward to implement. The stack implementation takes only about 10 kB of RAM and 100 kB of storage. The header is only 4 bytes. | ||
| + | |||
| + | CoAP protocol has a binary header to decrease overhead, but the payload depends on the content type. The initial, non-exclusive list of the payload types includes: | ||
| + | * text/plain (charset=utf-8) (ID=0, RFC2046, RFC3676, RFC5147), | ||
| + | * application/ | ||
| + | * application/ | ||
| + | * application/ | ||
| + | * application/ | ||
| + | |||
| + | CoAP endpoint services are identified by unique IP and port number. However, they operate on the UDP instead of TCP (like, e.g. HTML does). The transfer in CoAP is made using a non-reliable UDP network, so a message can appear duplicated, disappear, or be delivered in another order than initially sent. Because of the nature of datagram communication, | ||
| + | CoAP messages can be (non-exhaustive list): | ||
| + | * CON - Confirmable, | ||
| + | * NON - Non-Confirmable, | ||
| + | * ACK - an acknowledgement message, | ||
| + | * RESET - sent if CON or NON was received, but the receiver cannot understand the context, e.g. part of the communication is missing because of device restart, messages memory loss, etc. | ||
| + | Empty RESET messages can be used to " | ||
| + | |||
| + | Because of the UDP network characteristics, | ||
| + | * stop and wait for retransmission with exponential back-off for CON messages, | ||
| + | * duplicate message detection for CON and NON-messages. | ||
| + | The request-response pair is identified by a unique " | ||
| + | Sample request-response scenarios are presented in the images below. | ||
| + | Sample CoAP message exchange scenarios between client and server are presented (two per image) in figure {{ref> | ||
| + | <figure CoAP1> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | <figure CoAP2> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | The scenario in figure {{ref> | ||
| + | The scenario in figure {{ref> | ||
| + | The scenario in figure {{ref> | ||
| + | The scenario in figure {{ref> | ||
| + | |||
| + | == AMQP == | ||
| + | In its principles, the AMQP (Advanced Message Queuing Protocol) somehow recalls MQTT: it is message-oriented and uses a central broker. There are data publishers and consumers (that, in the case of the MQTT, are called subscribers). Messages are routed from publishers to the Broker, where they hit so-called exchanges, and then they are copied to the queues (0, 1 or more) from which the consumer can later read. A diagram of the message' | ||
| + | <figure amqp1> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | AMQP uses TCP/IP. AMQP is intended to work in non-reliable networks; thus, the protocol has a message acknowledgement mechanism to ensure delivery. A message is removed from the queue only if it has been acknowledged. Besides acknowledged delivery, it is also possible to use an unacknowledged one that does not involve acknowledgements. If a message cannot be routed (for any reason), it can be returned to the publisher, dropped or placed in the "dead letter queue" | ||
| + | |||
| + | **Queues**\\ | ||
| + | AMQP is a programmable protocol, so bindings are not defined by the Broker but rather by the publisher. Queues are also created on-demand via external actors (mostly consumers). Routing via bindings is provided with a message, and the Broker analyses it to provide correct message handling and delivery.\\ | ||
| + | Consumers can subscribe to the exchange and define a queue. Bindings then act as filters so they receive only selected messages. A single queue is intended to handle one consumer, but there is a possibility of letting many consumers use a single queue in the round-robin model. | ||
| + | As in the protocol version 0.9, queues have the following properties: | ||
| + | * Name, | ||
| + | * Durable flag - the queue and its contents are persistent across broker restarts, | ||
| + | * Exclusive flag - used exclusively by one consumer only, | ||
| + | * Auto-Delete flag - the queue is removed if the last consumer unsubscribed, | ||
| + | * Arguments - optional. | ||
| + | A queue name can be selected explicitly, or a broker may deliver one on demand. A queue has to be expressly defined. An existing queue can be silently redeclared with an exact attributes set. A declaration of the existing queue with different attributes set throws an exception code '' | ||
| + | |||
| + | ** Exchanges**\\ | ||
| + | Specification 0.9 of the AMQP protocol creates 4 exchanges (exchange types): | ||
| + | * Direct Exchange (its name is empty string or '' | ||
| + | <figure amqp2> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | * Fanout Exchange ('' | ||
| + | <figure amqp3> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | * Topic Exchange ('' | ||
| + | <figure amqp4> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | * Headers Exchange ('' | ||
| + | <figure amqp5> | ||
| + | {{ : | ||
| + | < | ||
| + | </ | ||
| + | |||
| + | **Bindings**\\ | ||
| + | Bindings are rules defining how to route from exchange to queues. | ||
| + | Depending on the exchange type the publisher interacts with, the messages are routed using algorithms that consider both exchange type, arguments and bindings. The publisher is responsible for providing information on which queue will receive the messages from the exchange. | ||
| + | |||
| + | **Consumers**\\ | ||
| + | Consumers subscribe to the queue to " | ||
| + | |||
| + | **Messages**\\ | ||
| + | The essential, immutable AMQP frame size is 8 bytes, and the payload is up to 2GB. Besides the header frame, messages have several virtually freely definable attributes, but the Broker uses some predefined ones. Common attributes cover: | ||
| + | * content type, | ||
| + | * content encoding, | ||
| + | * routing key, | ||
| + | * persistence flag, | ||
| + | * message priority, | ||
| + | * message publishing timestamp, | ||
| + | * content expiration period, | ||
| + | * publisher app id. | ||
| + | Header attributes (used, e.g. by the Headers Exchanges) are optional and similar to X-Headers in HTTP ((https:// | ||
| + | A payload in the AMQP message is a byte array. Broker does not process or review the content, and it can be even zero length. | ||
| + | |||
| + | <note tip> | ||
| + | * a queue is like your home, | ||
| + | * an exchange is a nearby train station, | ||
| + | * bindings are routes from the train station to your home, and usually there are one, many, or eventually there are none. | ||
| + | </ | ||
| + | |||
| + | The address of the Broker is referenced with URI, similar to the CoAP (e.g.): | ||
| + | *%%amqp://< | ||
| + | *" | ||
| + | |||
| + | < | ||
