Scaling Phoenix Instances with Docker and Libcluster

Initial Requirements

Make sure you have Erlang and Elixir installed on your machine. For this tutorial, I’m using the following versions:

Erlang/OTP 26 [erts-14.2.5.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]
Elixir 1.17.3 (compiled with Erlang/OTP 26)

You’ll also need Docker up and running. I’m using Docker Compose as well, so make sure that’s installed too.

Setting Up the Environment

To get started, we’ll create two separate Phoenix instances:

mix phx.new first_instance --no-ecto
mix phx.new second_instance --no-ecto

Before running the instances, there’s still something to do. Navigate into both directories and let’s set up some important files:

Create docker-compose.yml:

in root directory, create docker-compose.yml:

version: "3.8"

services:
  app:
    container_name: ${APP_NAME}
    build:
      context: .
    volumes:
      - .:/app
    environment:
      - APP_NAME=${APP_NAME}
      - APP_PORT=${APP_PORT}
      - APP_PUBSUB=${APP_PUBSUB}
      - APP_CLUSTER_COOKIE=${APP_CLUSTER_COOKIE}
    networks:
      - example_network
    restart: "on-failure"
    ports:
      - "${APP_PORT}:${APP_PORT}"

networks:
  example_network:
    external: true

Make sure the network name is the same across all instances.

Create Dockerfile:

in root directory, create Dockerfile:

FROM elixir:1.17.0-otp-27-alpine AS build

RUN apk add --no-cache build-base git
RUN apk add inotify-tools

WORKDIR /app

RUN mix local.hex --force && \
    mix local.rebar --force

COPY . .

COPY entrypoint-dev.sh /usr/local/bin/entrypoint-dev.sh

RUN chmod +x /usr/local/bin/entrypoint-dev.sh

CMD ["/usr/local/bin/entrypoint-dev.sh"]

Create entrypoint-dev.sh:

in root directory, create entrypoint-dev.sh:

#!/bin/sh

cd /app

mix deps.get
mix deps.compile

PORT=$APP_PORT elixir --sname app@$APP_NAME --cookie $APP_CLUSTER_COOKIE -S mix phx.server

Create .env:

in root directory, create .env:

APP_NAME=example_one
APP_PORT=4000

# Make sure these two are the same between all instances
APP_PUBSUB=ExampleApp.PubSub
APP_CLUSTER_COOKIE=example_cookie

Edit dev.exs:

To make both instances accessible via localhost, navigate to config/dev.exs and edit line 12.

http: [ip: {127, 0, 0, 1}, port: 4000],
http: [ip: {0, 0, 0, 0}, port: System.get_env("APP_PORT")],

Running the Instances

Open your terminal in the root folder of the project and create the network that will connect the containers:

docker network create example_network

Finally, start the containers with the following command:

docker compose up -d --build

Now, the applications should be running on the ports specified in your .env file. In my case, they’re accessible at:

• localhost:4000
• localhost:4001

Configuring Libcluster

Now, let’s set up the cluster for the instances.

Install libcluster:

To start, we need to install libcluster as a dependency. To do this, add it to your mix.exs file:

defp deps do
  [
    ...
    {:libcluster, "~> 3.3"}
  ]
end

Setup Topology

Next, let’s configure the topology. In this example, we’ll use the Gossip strategy, which automatically discovers nodes within a network.

In the config/config.exs file, add the following at the end, right before import_config:

config :libcluster,
  topologies: [
    example_gossip: [
      strategy: Elixir.Cluster.Strategy.Gossip,
      config: [
        port: 45892,
        if_addr: "0.0.0.0",
        multicast_addr: "255.255.255.255",
        broadcast_only: true
      ]
    ]
  ]

If you want more control over the nodes, you can also use EPMD Strategy. Be sure to check the libcluster documentation for other available options.

Initializing Libcluster

Now we need to activate this during the application initialization.

Add the following lines to lib/*_instance/application.ex:

def start(_type, _args) do
  topologies = Application.get_env(:libcluster, :topologies) || []

  children = [
    FirstInstanceWeb.Telemetry,
    {DNSCluster, query: Application.get_env(:first_instance, :dns_cluster_query) || :ignore},
    {Phoenix.PubSub, name: FirstInstance.PubSub},
    {Cluster.Supervisor, [topologies, [name: FirstInstance.ClusterSupervisor]]},
    ...

Checking connection

Now, let’s rebuild the instances and watch the logs. Run the following commands in both directories:

docker compose up -d --build
docker logs --follow instance_name

When both are running, you should see something like this in one of them (usually the one that started first):

[info] [libcluster:example_gossip] connected to :app@example_two

This means our instances have successfully connected and are ready to interact with each other via PubSub!

Configuring PubSub

Now it’s time to set up PubSub to allow interaction between the instances.

In this tutorial, I’ll separate the tasks into two instances:

• Instance 1 will send messages.
• Instance 2 will receive them.

But first, we need to ensure that both instances are using the same PubSub.

To do this, open config/config.exs in both instances and edit line 22 (or wherever the endpoint configuration is):

config :first_instance, FirstInstanceWeb.Endpoint,
  ...
 pubsub_server: FirstInstance.PubSub,
 pubsub_server: System.get_env("APP_PUBSUB") |> String.to_atom()
  ...

Next, in lib/*_instance/application.ex, update the start function:

def start(_type, _args) do
  topologies = Application.get_env(:libcluster, :topologies) || []

  children = [
    ...
    {DNSCluster, query: Application.get_env(:first_instance, :dns_cluster_query) || :ignore},
   {Phoenix.PubSub, name: FirstInstance.PubSub},
   {Phoenix.PubSub, name: System.get_env("APP_PUBSUB") |> String.to_atom()},
    ...

Setting Up the Sender

This step is pretty straightforward; all we need to do is call the PubSub.broadcast_from/3 function from somewhere. We’ll use Phoenix LiveView for this.

Create the following files:

lib/first_instance_web/live/sender_live/index.ex:

defmodule FirstInstanceWeb.SenderLive.Index do
  use FirstInstanceWeb, :live_view

  alias Phoenix.PubSub

  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_event("send_signal", _params, socket) do
    pubsub_identifier = System.get_env("APP_PUBSUB") |> String.to_atom()
    pid = self()

    PubSub.broadcast_from(pubsub_identifier, pid, "topic", %{message: "Hello from Sender"})

    {:noreply, put_flash(socket, :info, "Signal sent")}
  end
end

lib/first_instance_web/live/sender_live/index.html.heex:

<button phx-click="send_signal" class="bg-gray-200 px-4 py-2">Send signal</button>

Now, add the new live route in the browser scope in lib/first_instance_web/router.ex:

scope "/", FirstInstanceWeb do
  pipe_through :browser

  get "/", PageController, :home

  live "/sender", SenderLive.Index, :index
end

Visit localhost:4000/sender and test clicking the button. If everything is set up correctly, you should see a success message.

Configuring the Receiver

For this tutorial, I’ll initialize the PubSub subscription along with the application to avoid creating another page. With PubSub, you can send messages to any page, but only users connected to that page will receive the messages. By initializing PubSub with the application, it allows me to use it like a messaging app (e.g., RabbitMQ), enabling interactions with events on the receiver regardless of whether there are connected users.

Create the file lib/second_instance/listener.ex with the following content:

defmodule SecondInstance.PubSubListener do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
  end

  @impl true
  def init(state) do
    pubsub_identifier = System.get_env("APP_PUBSUB") |> String.to_atom()

    Phoenix.PubSub.subscribe(pubsub_identifier, "topic")

    {:ok, state}
  end

  @impl true
  def handle_info(%{message: message}, state) do
    IO.puts("Message received: #{message}")
    {:noreply, state}
  end

  @impl true
  def handle_info(_, state) do
    {:noreply, state}
  end
end

Finally, in application.ex, add the listener to the list of children:

def start(_type, _args) do
  ...
  children = [
    ...
    {Cluster.Supervisor, [topologies, [name: SecondInstance.ClusterSupervisor]]},
    SecondInstance.PubSubListener,
    ...

Final test

Great! Now that you’ve completed the setup, the system should work as expected:

• Open the Docker logs for the receiver (like example_two in my case) using the following command:

docker logs --follow example_two

• Go to the web page of the first instance (e.g., example_one) in your browser: localhost:4000/sender

• Click the button to send the signal. Every time you click the button, you should see the following message appear in the receiver’s logs:

Message received: Hello from Sender

This means the PubSub is working correctly, and the instances are communicating via LibCluster and Phoenix PubSub. Now, your two instances are ready to interact with each other, sending and receiving messages across the cluster.

You can use this method for a variety of use cases, such as:

• Real-time notifications: Send notifications between different nodes in your cluster, like alerting users across instances about new messages, system events, or live updates.

• Distributed job processing: Divide tasks across multiple nodes for parallel processing. For example, each instance can handle different parts of a large dataset or workload, and the nodes can coordinate via messages.

• Multi-instance chat applications: Build a chat system where users connected to different nodes can still exchange messages in real-time using PubSub for communication between instances.

• Live updates in dashboards: If you have multiple servers handling different data streams, you can send updates to user dashboards in real-time, even if the user is connected to a different node from the one where the data originated.

I’m using this setup with 30+ instances, none of which are aware of each other’s existence, but all receive events from a “central” instance.

Feel free to explore your own ideas and adapt this setup to fit your project’s needs!