Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open'

[XCiteForever]

Describe the bug

Endless sending of messages to a non-existent exchange when PublisherConfirmationsEnabled = false or PublisherConfirmationsEnabled = true and PublisherConfirmationTrackingEnabled = false.
Attached is a sample code that causes the error. Is this normal behavior or not?
And why?

At
PublisherConfirmationsEnabled = true
PublisherConfirmationTrackingEnabled = true
It works correctly

RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open'', classId=60, methodId=40

Reproduction steps

1. Run Code
2. See log

Expected behavior

Works stably

Additional context

    static async Task Main(String[] args)
    {
        var factory = new ConnectionFactory
        {
            HostName = "localhost",
            Port = 5672,
            UserName = "guest",
            Password = "guest"
        };

        var connection = await factory.CreateConnectionAsync();
        connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;

        var options = new CreateChannelOptions(false, false);
        var json = "{ \"A\" : \"X\"}";
        var body = Encoding.UTF8.GetBytes(json);
        var bp = new BasicProperties
        {
            ContentType = MediaTypeNames.Application.Json,
            ContentEncoding = "UTF-8",
        };

        IChannel channel = await connection.CreateChannelAsync(options);
        channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;

        while (true)
        {
            try
            {
                await channel.BasicPublishAsync("NotExistsExchange", String.Empty, true, bp, body);
                Console.WriteLine("send");
            }
            catch (AlreadyClosedException)
            {
                Console.WriteLine("not send");
            }
            
            if (channel.IsClosed)
            {
                channel = await connection.CreateChannelAsync(options);  // The error occurs here
                Console.WriteLine("Recreate");
                channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;
            }
        }
    }

    private static async Task Connection_ConnectionShutdownAsync(object sender, RabbitMQ.Client.Events.ShutdownEventArgs @event)
    {
        Console.WriteLine("Connection: " + @event.ReplyCode);
    }

    private static async Task Channel_ChannelShutdownAsync(object sender, RabbitMQ.Client.Events.ShutdownEventArgs @event)
    {
        Console.WriteLine("Channel: " + @event.ReplyCode);
    }
}

[XCiteForever]

Some log RabbitMQ

2025-05-16 15:50:38 2025-05-16 12:50:38.294140+00:00 [error] <0.102182.0> Channel error on connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest'), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.294140+00:00 [error] <0.102182.0> operation basic.publish caused a channel exception not_found: no exchange 'NotExistsExchange' in vhost '/'
2025-05-16 15:50:38 2025-05-16 12:50:38.343681+00:00 [error] <0.102189.0> Channel error on connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest'), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.343681+00:00 [error] <0.102189.0> operation basic.publish caused a channel exception not_found: no exchange 'NotExistsExchange' in vhost '/'
2025-05-16 15:50:38 2025-05-16 12:50:38.391125+00:00 [error] <0.101966.0> Error on AMQP connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest', state: running), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.391125+00:00 [error] <0.101966.0> operation basic.publish caused a connection exception channel_error: "expected 'channel.open'"
2025-05-16 15:50:38 2025-05-16 12:50:38.393535+00:00 [info] <0.101966.0> closing AMQP connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest')

[lukebakken]

Hello, thanks for this report and for code to reproduce the issue.

I created a complete project here: https://github.com/lukebakken/rabbitmq-dotnet-client-1835

Here is the problem:

• The code publishes as fast as possible, and does not wait for confirmations from the server.
• Since the code publishes to a non-existent exchange, with mandatory: true, every publish results in a channel exception.
• When RabbitMQ closes the channel due to the exception, it must send channel.close back to the client application, which, remember, is publishing as fast as possible.
• Sometimes, this channel.close message makes it back at just the right time to initiate channel closure in this library BEFORE the next publish.
• However, if a basic.publish is sent after RabbitMQ closes the channel, but before the channel.close message makes it back to this client, you will see the exception reported.

I'm not sure if it's possible to "do better" in this scenario. It is always possible that, without using publisher confirmations, a client application could send basic.publish to a closed channel.

I'm going to remove the "bug" label for now while I investigate.

[Gsantomaggio]

Ok, I can reproduce the problem too. Will start looking.

[Gsantomaggio]

@lukebakken I don't think this is a client issue.
The client's asynchronous nature requires some external synchronisation in this case, which can be achieved through a publish confirm or a wait, as shown in the code below.

My opinion:
Attempting to reconnect a channel too quickly results in an application error. All our reconnection implementation provides a delay during the reconnection.

I would not change the client to deal with a non-real use case like this.

The code like this is enough to solve the problem.

using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using RabbitMQ.Client.Exceptions;
using System.Net.Mime;
using System.Text;

var factory = new ConnectionFactory
{
    HostName = "localhost",
    Port = 5672,
    UserName = "guest",
    Password = "guest"
};

var connection = await factory.CreateConnectionAsync();
connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;

var options = new CreateChannelOptions(false, false);
var json = "{ \"A\" : \"X\"}";
var body = Encoding.UTF8.GetBytes(json);
var bp = new BasicProperties
{
    ContentType = MediaTypeNames.Application.Json,
    ContentEncoding = "UTF-8",
};

IChannel channel = await connection.CreateChannelAsync(options);
channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;

while (true)
{
    try
    {
        await channel.BasicPublishAsync("NotExistsExchange", String.Empty, true, bp, body);
        Console.WriteLine("after BasicPublishAsync");
    }
    catch (AlreadyClosedException)
    {
        Console.WriteLine("caught AlreadyClosedException");
    }

    if (channel.IsClosed)
    {
        int retry = 0;
        while (retry < 3)
        {
            Console.WriteLine("channel.IsClosed is true, retrying...");
            await Task.Delay(1000);
            Console.WriteLine("channel.IsClosed is true");
            try
            {
                if (!connection.IsOpen)
                {
                    Console.WriteLine("connection.IsOpen is false, retrying...");
                    connection.ConnectionShutdownAsync -= Connection_ConnectionShutdownAsync;
                    connection = await factory.CreateConnectionAsync();
                    connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;
                }

                channel.ChannelShutdownAsync -= Channel_ChannelShutdownAsync;
                channel = await connection.CreateChannelAsync(options); // The error occurs here
                channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;
                Console.WriteLine("channel recreated successfully");
                break;
            }
            catch (Exception e)
            {
                Console.WriteLine("Exception during channel recreation: " + e.Message);
            }
            
            retry++;
        }
    }
}

static Task Connection_ConnectionShutdownAsync(object sender, ShutdownEventArgs @event)
{
    Console.WriteLine("Connection shutdown: " + @event.ReplyCode);
    return Task.CompletedTask;
}

static Task Channel_ChannelShutdownAsync(object sender, ShutdownEventArgs @event)
{
    Console.WriteLine("Channel shutdown: " + @event.ReplyCode);
    return Task.CompletedTask;
}

[michaelklishin]

Attempting to reconnect a channel too quickly results in an application error. All our
reconnection implementation provides a delay during the reconnection.

(I'm mentioning this for the readers who are not core team members, not insulting Gabriele's intelligence ;)) All RabbitMQ clients maintained by our team perform connection recovery after a delay.

Are you suggesting a small delay between channel recovery? That's a reasonable idea but with a large number of channels — whether we recommend it or not, there is no shortage of such apps — can result in significant recovery delays.

Or is your recommendation against such a delay?

[Gsantomaggio]

Are you suggesting a small delay between channel recovery?

Yes. I suggest adding a delay (a back-off would be better), as we do for other clients. I am thinking of the stream clients experience.
We have a back-off pattern, ex:
Stream .NET Client
stream Golang client
Java stream client

That's my opinion, based on the experiences of other clients, but I am open to discussing

[Gsantomaggio]

I will close the issue. Feel free to reopen in case needed.