As you saw in my previous post I was able to get the same mqttc example from NuttX running on Linux.
Unfortunately the NuttX application is not working, then I decided to run the WireShark to compare what the Linux application was doing different from the NuttX application.
It should be easier to sniffer the network if it was Ethernet instead WiFi, so I used the ESP32-Ethernet-Kit board to test with NuttX and a RaspberryPI 3 to test with Linux (I tried to use other computer with Linux, but there are so many packages from Linux services that it was also impossible to compare).
Initially I connected both boards to my WiFi router RJ45 ports, but the router seems to isolate each port, so it is working like an Ethernet Switch instead of working like an Ethernet HUB.
Then I decided to use my laptop to share Internet with those board (one by time). I remember that I did it almost 15 years ago and documented it here (sorry in Portuguese).
That idea worked fine, that is a good solution for future test!
Initially I noticed some strange things: first the NuttX was with “MSS=536” (the minimum MSS size supported by IPv4) where the Linux was using “MSS=1460”
NuttX:
3 0.050060828 192.168.1.100 75.2.83.130 TCP 60 4097 → 1883 [SYN] Seq=0 Win=5488 Len=0 MSS=536
Linux:
7 1.359660410 192.168.1.100 99.83.172.119 TCP 74 40264 → 1883 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=2409185516 TSecr=0 WS=128Maybe these differences could explain why NuttX was not working, but unfortunately increasing the MSS to 1460 didn’t help, I still receiving a Ack=0 from the server:
6 34.353497310 192.168.1.100 75.2.83.130 TCP 60 4097 → 1883 [SYN] Seq=0 Win=5488 Len=0 MSS=1446
7 34.496251127 75.2.83.130 192.168.1.100 TCP 58 1883 → 4097 [SYN, ACK] Seq=0 Ack=0 Win=65535 Len=0 MSS=0So, let to test other ideas, I also noticed that the Linux application was using a big Window (Win=64240), the “Selective Acknowledgements” (SACK_PERM=1), time stamp (TSval=2409185516 TSecr=0) and Window Size Scaling (WS=128)
$ sudo sysctl -w net.ipv4.tcp_sack=0
$ sudo sysctl -w net.ipv4.tcp_timestamps=0
$ sudo sysctl -w net.ipv4.tcp_window_scaling=0These modifications also didn’t solve the issue.
Increasing the Window to 65KB on NuttX doesn’t sound like a good idea, let’s to modify this other Linux TCP stack parameters to become similar to NuttX, we can reduce the Win to 4KB:
# echo "4096 4096 4096" > /proc/sys/net/ipv4/tcp_rmem
# echo "4096 4096 4096" > /proc/sys/net/ipv4/tcp_wmemAnother nice idea is to reduce the MTU to 576 and consequently the MSS becomes 536:
$ sudo ifconfig eth0 mtu 576 upIt also didn’t make the issue show up on Linux, so there is some other detail, see complete Linux log:
1 0.000000000 192.168.1.100 8.8.8.8 DNS 72 Standard query 0x6055 A mqtt.tago.io
2 0.041538839 8.8.8.8 192.168.1.100 DNS 104 Standard query response 0x6055 A mqtt.tago.io A 99.83.172.119 A 75.2.83.130
3 0.043238286 192.168.1.100 99.83.172.119 TCP 60 59880 → 1883 [SYN] Seq=0 Win=1460 Len=0 MSS=1460
4 0.190940161 99.83.172.119 192.168.1.100 TCP 58 1883 → 59880 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1416
5 0.191451199 192.168.1.100 99.83.172.119 TCP 60 59880 → 1883 [ACK] Seq=1 Ack=1 Win=1460 Len=0
6 0.192109946 192.168.1.100 99.83.172.119 MQTT 117 Connect Command
7 0.371321172 99.83.172.119 192.168.1.100 TCP 54 1883 → 59880 [ACK] Seq=1 Ack=64 Win=65535 Len=0
8 0.371805531 192.168.1.100 99.83.172.119 MQTT 111 Publish Message [tago/data/post]
9 0.467783276 99.83.172.119 192.168.1.100 MQTT 58 Connect Ack
10 0.468174062 192.168.1.100 99.83.172.119 TCP 60 59880 → 1883 [ACK] Seq=121 Ack=5 Win=1456 Len=0
11 0.515294572 99.83.172.119 192.168.1.100 TCP 54 1883 → 59880 [ACK] Seq=5 Ack=121 Win=65535 Len=0
12 5.098640506 20:7b:d2:0d:a8:d0 Raspberr_65:2a:1a ARP 42 Who has 192.168.1.100? Tell 192.168.1.1
13 5.099071927 Raspberr_65:2a:1a 20:7b:d2:0d:a8:d0 ARP 60 192.168.1.100 is at b8:27:eb:65:2a:1a
14 5.193192947 192.168.1.100 99.83.172.119 MQTT 60 Disconnect Req
15 5.193699389 192.168.1.100 99.83.172.119 TCP 60 59880 → 1883 [FIN, ACK] Seq=123 Ack=5 Win=1456 Len=0
16 5.338709513 99.83.172.119 192.168.1.100 TCP 54 1883 → 59880 [ACK] Seq=5 Ack=123 Win=65535 Len=0
17 5.338714704 99.83.172.119 192.168.1.100 TCP 54 1883 → 59880 [FIN, ACK] Seq=5 Ack=124 Win=65535 Len=0
18 5.339267333 192.168.1.100 99.83.172.119 TCP 60 59880 → 1883 [ACK] Seq=124 Ack=6 Win=1455 Len=0After many tests I realized with help from my friend Marcelo Barros that the issue was because the mqttc didn’t wait the MQTT ACK before sending data and it confuse the server that refuses to accept further data.
I submitted a fix to let mqttc on NuttX to wait for the MQTT ACK: https://github.com/apache/nuttx-apps/pull/1748
Alan C. Assis 