Mongoose with Mbedtls gives an error when receiving packets larger than a certain size.

[miguelibarra-hub]

Hello, I'm having the following problem while testing Mongoose with Mbedtls. It seems to only work when the server response is less than 536 bytes. Otherwise, if it's greater than that value, it only processes those 536 bytes and then throws an error.
Environment:

• Mongoose version: 7.17
• Compiler/IDE and SDK: GCC 13.2 (arm-none-eabi-gcc), Raspberry Pi Pico SDK 2.1.1, Mbed TLS 2.28.8 (this is the version bundled with the Pi Pico SDK)
• Target board: RP2040
• Connectivity chip/module: Wiznet w5100s, Mongoose built-in TCP
• Target RTOS : Freertos SMP .

The application initially performs a POST to obtain a token, this is the part that works correctly, then that token is used to access a resource by performing a GET, this is where it fails because the server response is greater than 536 bytes.

Enabling debugging in Mongoose and mbedtls, when receiving the TLS OK frame I get the following:

5762 2 httpClientInterface.c:131:Even --- TLS handshake done! ---
5768 2 mongoose.c:12843:debug_cb 1 2 => write
576c 2 mongoose.c:12843:debug_cb 1 2 => write record
5772 2 mongoose.c:12843:debug_cb 1 2 => encrypt buf
5777 2 mongoose.c:12843:debug_cb 1 2 <= encrypt buf
577c 2 mongoose.c:12843:debug_cb 1 2 => flush output
5781 2 mongoose.c:12843:debug_cb 1 2 message length: 343, out_left: 343
5788 4 mongoose.c:4630:tx_tcp TCP 192.168.100.13:40607 -> 133.146.104.56:8443 fl 18 len 343
579a 4 mongoose.c:4258:settmout 1 0 -> 10646
579b 4 mongoose.c:12809:mg_net_send 1 n=343 e=0
57a0 2 mongoose.c:12843:debug_cb 1 2 ssl->f_send() returned 343 (-0xfffffea9)
57a7 2 mongoose.c:12843:debug_cb 1 2 <= flush output
57ad 2 mongoose.c:12843:debug_cb 1 2 <= write record
57b2 2 mongoose.c:12843:debug_cb 1 2 <= write
5860 4 mongoose.c:4917:rx_ip TCP 133.146.104.56:8443 -> 192.168.100.13:40607 len 244
5865 4 mongoose.c:4258:settmout 1 0 -> 10821
586a 4 mongoose.c:4798:read_conn 1 SEQ 6dd109e9 -> 6dd109e9
5870 4 mongoose.c:4800:read_conn --- 244 Bytes -> rtls ---
5876 4 mongoose.c:4258:settmout 1 1 -> 58ef
587a 2 mongoose.c:4727:handle_tls_rec TLS pending 0
587f 2 mongoose.c:4733:handle_tls_rec TLS recv len 0 size 0
5885 2 mongoose.c:12843:debug_cb 1 2 => read
5889 2 mongoose.c:12843:debug_cb 1 2 => read record
588e 2 mongoose.c:12843:debug_cb 1 2 => fetch input
5894 2 mongoose.c:12843:debug_cb 1 2 in_left: 0, nb_want: 5
5899 2 mongoose.c:4024:mg_io_recv !!!!--- c->rtls.len = 244 len = 5 c->rtls.size = 8192 ---!!!!
58a2 4 mongoose.c:12818:mg_net_recv 1 n=5
58a7 2 mongoose.c:12843:debug_cb 1 2 in_left: 0, nb_want: 5
58ac 2 mongoose.c:12843:debug_cb 1 2 ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
58b5 2 mongoose.c:12843:debug_cb 1 2 <= fetch input
58ba 2 mongoose.c:12843:debug_cb 1 2 => fetch input
58bf 2 mongoose.c:12843:debug_cb 1 2 in_left: 5, nb_want: 244
58c5 2 mongoose.c:4024:mg_io_recv !!!!--- c->rtls.len = 239 len = 239 c->rtls.size = 8192 ---!!!!
58ce 4 mongoose.c:12818:mg_net_recv 1 n=239
58d2 2 mongoose.c:12843:debug_cb 1 2 in_left: 5, nb_want: 244
58d9 2 mongoose.c:12843:debug_cb 1 2 ssl->f_recv(_timeout)() returned 239 (-0xffffff11)
58e1 2 mongoose.c:12843:debug_cb 1 2 <= fetch input
58e6 2 mongoose.c:12843:debug_cb 1 2 => decrypt buf
58ec 2 mongoose.c:12843:debug_cb 1 2 <= decrypt buf
58f0 2 mongoose.c:12843:debug_cb 1 2 <= read record
58f6 2 mongoose.c:12843:debug_cb 1 2 <= read
58fa 2 mongoose.c:12933:mg_tls_recv -- 1 n=0 mbedtls_ssl_read max read = 0 --
5901 1 mongoose.c:422:mg_error 1 -1 TLS recv error MG_IO_ERR
5908 2 httpClientInterface.c:73:Event --- HttpClient MG_EV_ERROR ---
590e 1 httpClientInterface.c:75:Event 0xffffffff TLS recv error MG_IO_ERR
5915 2 mongoose.c:4727:handle_tls_rec TLS pending 215
591a 2 mongoose.c:4733:handle_tls_rec TLS recv len 0 size 8192
591f 2 mongoose.c:12843:debug_cb 1 2 => read
5924 2 mongoose.c:12843:debug_cb 1 2 <= read
5928 2 mongoose.c:12933:mg_tls_recv -- 1 n=215 mbedtls_ssl_read max read = 8192 --
5930 2 httpClientInterface.c:139:Even --- HttpClient MG_EV_HTTP_HDRS ---
5937 1 mongoose.c:2434:http_cb Content length missing from response
593d 2 httpClientInterface.c:134:Even --- HttpClient MG_EV_READ 215 ---
5944 2 httpClientInterface.c:135:Even --- HttpClient MG_EV_READ HTTP/1.1 200
X-Frame-Options: SAMEORIGIN
Content-Type: application/json
Date: Mon, 07 Apr 2025 11:31:28 GMT
Connection: close

{"access_token":"bK-rx76hUsc5vClHHRLz9w","token_type":"bearer","expires_in":3600}

For the case where the TLS frame is about 1K, I get the following:

aa1f 2 httpClientInterface.c:131:Even --- TLS handshake done! ---
aa25 2 mongoose.c:12843:debug_cb 3 2 => write
aa2a 2 mongoose.c:12843:debug_cb 3 2 => write record
aa2f 2 mongoose.c:12843:debug_cb 3 2 => encrypt buf
aa35 2 mongoose.c:12843:debug_cb 3 2 <= encrypt buf
aa3a 2 mongoose.c:12843:debug_cb 3 2 => flush output
aa3f 2 mongoose.c:12843:debug_cb 3 2 message length: 258, out_left: 258
aa46 4 mongoose.c:4630:tx_tcp TCP 192.168.100.13:40608 -> 133.146.104.56:8443 fl 18 len 258
aa55 4 mongoose.c:4258:settmout 3 0 -> 15904
aa57 4 mongoose.c:12809:mg_net_send 3 n=258 e=0
aa5c 2 mongoose.c:12843:debug_cb 3 2 ssl->f_send() returned 258 (-0xfffffefe)
aa63 2 mongoose.c:12843:debug_cb 3 2 <= flush output
aa69 2 mongoose.c:12843:debug_cb 3 2 <= write record
aa6e 2 mongoose.c:12843:debug_cb 3 2 <= write
aa75 4 mongoose.c:4896:rx_ip UDP 192.168.100.11:5353 -> 224.0.0.251:5353 len 40
ab23 4 mongoose.c:4917:rx_ip TCP 133.146.104.56:8443 -> 192.168.100.13:40608 len 536
ab29 4 mongoose.c:4258:settmout 3 0 -> 15adf
ab2d 4 mongoose.c:4798:read_conn 3 SEQ 5b3693c -> 5b3693c
ab33 4 mongoose.c:4800:read_conn --- 536 Bytes -> rtls ---
ab39 4 mongoose.c:4258:settmout 3 1 -> abad
ab3e 2 mongoose.c:4727:handle_tls_rec TLS pending 0
ab42 2 mongoose.c:4733:handle_tls_rec TLS recv len 0 size 0
ab48 2 mongoose.c:12843:debug_cb 3 2 => read
ab4c 2 mongoose.c:12843:debug_cb 3 2 => read record
ab52 2 mongoose.c:12843:debug_cb 3 2 => fetch input
ab57 2 mongoose.c:12843:debug_cb 3 2 in_left: 0, nb_want: 5
ab5d 2 mongoose.c:4024:mg_io_recv !!!!--- c->rtls.len = 536 len = 5 c->rtls.size = 8192 ---!!!!
ab66 4 mongoose.c:12818:mg_net_recv 3 n=5
ab6a 2 mongoose.c:12843:debug_cb 3 2 in_left: 0, nb_want: 5
ab70 2 mongoose.c:12843:debug_cb 3 2 ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
ab78 2 mongoose.c:12843:debug_cb 3 2 <= fetch input
ab7d 2 mongoose.c:12843:debug_cb 3 2 => fetch input
ab82 2 mongoose.c:12843:debug_cb 3 2 in_left: 5, nb_want: 990
ab88 2 mongoose.c:4024:mg_io_recv !!!!--- c->rtls.len = 531 len = 985 c->rtls.size = 8192 ---!!!!
ab92 4 mongoose.c:12818:mg_net_recv 3 n=531
ab96 2 mongoose.c:12843:debug_cb 3 2 in_left: 5, nb_want: 990
ab9c 2 mongoose.c:12843:debug_cb 3 2 ssl->f_recv(_timeout)() returned 531 (-0xfffffded)
aba4 2 mongoose.c:4024:mg_io_recv !!!!--- c->rtls.len = 0 len = 454 c->rtls.size = 8192 ---!!!!
abad 4 mongoose.c:12818:mg_net_recv 3 n=-2
abb1 2 mongoose.c:12843:debug_cb 3 2 in_left: 536, nb_want: 990
abb8 2 mongoose.c:12843:debug_cb 3 2 ssl->f_recv(_timeout)() returned -26752 (-0x6880)
abc0 2 mongoose.c:12843:debug_cb 3 1 mbedtls_ssl_fetch_input() returned -26752 (-0x6880)
abc8 2 mongoose.c:12843:debug_cb 3 1 ssl_get_next_record() returned -26752 (-0x6880)
abd0 2 mongoose.c:12843:debug_cb 3 1 mbedtls_ssl_read_record() returned -26752 (-0x6880)
abd9 2 mongoose.c:12933:mg_tls_recv -- 3 n=-26752 mbedtls_ssl_read max read = 0 --
abe0 2 mongoose.c:4739:handle_tls_rec TLS recv error MG_IO_WAIT
abe6 4 mongoose.c:5073:mg_tcpip_poll 3 ack a0ef 5b36b54
abeb 4 mongoose.c:4630:tx_tcp TCP 192.168.100.13:40608 -> 133.146.104.56:8443 fl 10 len 0
abf6 4 mongoose.c:4258:settmout 3 0 -> 15bae
acac 4 mongoose.c:4917:rx_ip TCP 133.146.104.56:8443 -> 192.168.100.13:40608 len 485
acb2 4 mongoose.c:4258:settmout 3 0 -> 15c6a
acb7 4 mongoose.c:4258:settmout 3 4 -> e73a
acbb 4 mongoose.c:4630:tx_tcp TCP 192.168.100.13:40608 -> 133.146.104.56:8443 fl 11 len 0
ad73 4 mongoose.c:4917:rx_ip TCP 133.146.104.56:8443 -> 192.168.100.13:40608 len 0
-- LED TASK --b8b5 2 network_manager.c:106:timer_fn Ethernet: ready, IP: 192.168.100.13, rx:68, tx:47, dr:0, er:0, ip = 224700608
c313 4 mongoose.c:4896:rx_ip UDP 192.168.100.11:5353 -> 224.0.0.251:5353 len 40

Any idea what might be going on?
Best regards.

[scaprile]

It is impossible to see what you are doing wrong without seeing what you are doing.
Our applications and examples do work, and are described in our tutorials.
Please see our documentation, and follow the guidelines in our tutorials.
Please sniff your network (using Wireshark, for example) and observe the traffic.
If, after your network inspection, you find an actual issue, feel free to open an issue, fill the issue template, and provide us with pertinent information (including useful network capture files; no snapshots, please attach capture files) in order to reproduce and fix the issue. We don't inspect your code for errors, please send the minimum piece of code that can reproduce the issue

[miguelibarra-hub]

Hi, thanks for your response. I was setting up a basic example that shows the problem. When I was capturing it with Wireshark, I think I found the problem.
The server sends two Application Record frames, almost at the same time in succession. Mongoose always decodes the first frame and returns an MG_IO_ERR error. It then closes the connection due to this error, and the second frame isn't processed. Sometimes the second frame isn't even detected.
The MG_IO_ERR error is due to the mbedtls_ssl_read() function returning 0. By making a small change to the mg_tls_recv() function so that it doesn't return the error when mbedtls_ssl_read() returns 0, the second frame could be processed when detected.

long mg_tls_recv(struct mg_connection *c, void *buf, size_t len) {
struct mg_tls *tls = (struct mg_tls *) c->tls;
long n = mbedtls_ssl_read(&tls->ssl, (unsigned char *) buf, len);
MG_INFO(("-- %lu n=%ld mbedtls_ssl_read max read = %d --", c->id, n,len));
if (n == MBEDTLS_ERR_SSL_WANT_READ || n == MBEDTLS_ERR_SSL_WANT_WRITE)
return MG_IO_WAIT;
#if defined(MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET)
if (n == MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET) {
return MG_IO_WAIT;
}
#endif
if (n < 0) return MG_IO_ERR; // if (n <= 0) return MG_IO_ERR;
if(n > 0) {
MG_INFO(("!--- %lu %.*s ---!", c->id, (int) n, (char *) buf));
}
return n;
}
This change only works when both frames are received. Sometimes, even though the frames arrive, they are visible in Wireshark and are not detected in Mongoose.
My test hardware is a W5100S-EVB-Pico.
I've attached the test program, Ethernet captures, and logs, as requested for analysis.
The test program performs a POST to an IoT test server.
mongoose-tls-pico-w5100s.zip
Logs-captures.zip
Any help is appreciated, regards.

[scaprile]

I think I have an idea of what is going on, thing that rarely (if ever) happens except in a high-delay low-throughput like yours. I'd like to see what are those two frames and why is your server sending two TLS records, whatever they are. I can guess, but I'd rather see what it is. Mongoose supports the SSLKEYLOGFILE strategy by defining MG_TLS_SSLKEYLOGFILE, but as you don't have a filesystem and don't control the other end, I'd have to simulate that and pray it happens, or devise another strategy to get the keys out.

I'm converting this to an issue for further investigation, I'll see what I can do. We'll definitely address this, but it will take some time.

[miguelibarra-hub]

Yes, the server thing is strange, I don't know why it sends two Application record packets. The content of the first package is as follows:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
Server:

and the content of the second package is the continuation of that answer:

**Thinger.io
Connection: Close
Content-Length: 53
Content-Type: application/json

{"OutA":false,"OutB":false,"OutC":false,"OutD":false}**

The response size is 302 bytes, according to Postman. I don't understand why the server splits it in two.
This response is always the same unless I change something in the dashboard, but I'm not going to do that to keep the response the same.
Any other information you may need, please ask me.

[scaprile]

Yes, given that this is TLS is weird, as records are usually sized by the application, so TCP doesn't seem to be a player here.
BTW, can you connect with no TLS ?
And, if this site supports TLS1.3, try with MG_TLS=MG_TLS_BUILTIN

[miguelibarra-hub]

Yes, I can connect without TLS, and I initially tried using TLS 1.3 with the MG_TLS=MG_TLS_BUILTIN option. But it doesn't let me connect without using certificates. So far, I haven't been able to find any certificates to use, but it's something I wanted to avoid anyway, so I wouldn't run the risk of having to update them in the future.

[scaprile]

        case MG_EV_READ:
            MG_INFO(("CLIENT got data: %.*s", r->len, r->buf));
            r->len = 0;  // Tell Mongoose we've consumed data
            break;  

Please see our documentation, and follow the guidelines in our tutorials.

We do NOT do this, it interferes with the normal HTTP handling. Don't do this.

[miguelibarra-hub]

Thanks for your clarification, I'll correct it now.