How to create your own asynchronous runtime in C++

Last time I wrote about how to implement coroutines in C++ (https://rhidian-server.com/implementing-c-coroutines/).

The next logical step would be how to implement your own asynchronous runtime. After all, something needs to trigger your coroutines.

Choosing your runtime

When implementing your runtime for your coroutines, you'll have to consider a couple of points:

• Should my coroutines run single- or multi-threaded?
• How will my coroutines interact with my runtime (Is the runtime some globally accessible singleton?)
• ...

All of these are good questions and very much depend on your scenario, but I'll give out some considerations to help you along:

Should my coroutines run single- or multi-threaded?

If you have coroutines in a hot-loop or need to handle many asynchronous events (for example, a webserver handling hundreds of incoming requests per second) then you'll want to gravitate towards a multi-threaded runtime as you'll be able to squeeze out more performance (at the cost of added complexity of course).

If you don't need that much performance, I would highly recommend keeping your runtime single-threaded to save yourself from deadlocks, race conditions and all those multi-threaded goodies.

How will my coroutines interact with my runtime (Is the runtime some globally accessible singleton?)

There's a couple of ways to design your runtime. One way is to declare some Eventloop variable early on in your code and pass it around everywhere. This makes it clear to which runtime your coroutines belong (and allow you to start new threads with their own runtime). The obvious downside is that you'll be passing your Eventloop around everywhere.

A globally accessible singleton prevents you from passing your Eventloop around everywhere at the cost of sacrificing having multiple runtimes (or adding a lot of complexity in your runtime to prevent race conditions, at which point, you might want to just switch over to a thread-pool based runtime).

Implementing the runtime

For this blog, I'll be making a single-threaded runtime that is passed around. This is because this is a simple example that can be easily extended, and gives you the required insight to proceed with your own custom needs.

For my runtime, I'd like to make my Promise object copyable so that the lifetime becomes easier to manage. I'll do this by grabbing the Promise we made in the previous post and slightly changing it to internally keep a std::shared_ptr to its state.

class Promise
{
 private:
  struct SharedState
  {
    bool m_isReady{false};
    std::vector<std::function<void()>> m_callbacks;
  };

 private:
  std::shared_ptr<SharedState> state;

 public:
  struct Awaiter
  {
   private:
    std::shared_ptr<SharedState> state;

   public:
    explicit Awaiter(std::shared_ptr<SharedState> state)
      : state(state)
    {
    }

    bool await_ready()
    {
      return state->m_isReady;
    }

    void await_suspend(std::coroutine_handle<> handle)
    {
      state->m_callbacks.push_back([handle]() { handle.resume(); });
    }

    void await_resume() {}
  };

 public:
  Promise()
    : state(std::make_shared<SharedState>())
  {
  }
  Promise(const Promise& other) = default;
  Promise& operator=(const Promise& other) = default;
  Promise(Promise&& other) = default;
  Promise& operator=(Promise&& other) = default;

  bool IsReady() const
  {
    return state->m_isReady;
  }

  void Set()
  {
    // we can only execute a Promise once
    if (state->m_isReady) return;

    state->m_isReady = true;
    for (auto const & cb : state->m_callbacks)
    {
      cb();
    };
  }

  Awaiter operator co_await()
  {
    return Awaiter{state};
  }
};

Eventloop

Our coroutines will run on a single-threaded Eventloop based on epoll_wait. epoll_wait takes in a list of file descriptors to monitor for changes. The function blocks until a given file descriptor becomes readable / writable (depending on the flags you give it) at which point the function returns.

Our Eventloop needs to be capable of adding events to it and then running all of those events. This leads us with a very simple class declaration:

class Eventloop
{
 private:
  // How much oustanding work do we have?
  int m_workCount{0};

  // The file descriptor for our epoll instance.
  int m_epollFd{-1};

 public:
  Eventloop();
  ~Eventloop();

  // Our eventloop is not copyable or moveable. Our epoll instance is unique to this eventloop.
  Eventloop(Eventloop const&) = delete;
  Eventloop& operator=(Eventloop const&) = delete;
  Eventloop(Eventloop&&) = delete;
  Eventloop& operator=(Eventloop&&) = delete;

  // Add an event to the eventloop. The promise will be set once the event completes.
  void AddFD(int fd, Promise promiseToSet);

  // Run the event loop until no work remains on it
  void Run();
};

Creating the Eventloop

We start by creating our epoll instance by calling epoll_create1.

Eventloop::Eventloop()
{
  m_epollFd = epoll_create1(0);
  if (m_epollFd == -1)
  {
    // Log an error if we fail to create our epoll instance.
    LERROR(LOGGER, std::format("Failed to create epoll instance: {}", strerror(errno)));
    throw std::runtime_error("Failed to create epoll instance");
  }
}

epoll_create1() returns a file descriptor pointing to our epoll instance. We'll be using this file descriptor to add, remove or modify file descriptors of events that we want to watch.

Adding events to the Eventloop

To add events to our epoll instance, we need to call epoll_ctl. epoll_ctl allows us to add, modify or remove a file descriptor to/from our epoll instance and setting an event to monitor.

The most basic case would be:

struct EventData
{
  Promise promise;
  int fd{-1};
};

void Eventloop::AddFD(int fd, Promise promiseToSet)
{
  LDEBUG(LOGGER, std::format("Adding fd {} to eventloop", fd));
  epoll_event ev{};

  // Check for read and write events.
  ev.events = EPOLLIN | EPOLLOUT;

  // Add our data to the event so we can trigger our promise once it completes.
  ev.data.ptr = new EventData{.promise = promiseToSet, .fd = fd};

  if (epoll_ctl(m_epollFd, EPOLL_CTL_ADD, fd, &ev) == -1)
  {
    LERROR(LOGGER, std::format("Failed to add fd {} to epoll instance: {}", fd, strerror(errno)));
    throw std::runtime_error("Failed to add fd to epoll instance");
  }

  // Ensure that our eventloop is aware of outstanding work.
  m_workCount++;
}

epoll_ctl expects an event of type epoll_event which looks like:

typedef union epoll_data
{
  void *ptr;
  int fd;
  uint32_t u32;
  uint64_t u64;
} epoll_data_t;

struct epoll_event
{
  uint32_t events;	/* Epoll events */
  epoll_data_t data;	/* User data variable */
};

A basic use case is to check for read/write events on our file descriptor, which maps to EPOLLIN | EPOLLOUT. All flags can be found on the epoll_ctl documentation and should be OR'ed together.

We can also pass data to the epoll_event by filling in the data.ptr field, which expects a pointer to some heap-memory that we provided. In our case, we want to be able to trigger our Promise object after it is completed, so we create an EventData object to keep track of our Promise and the file descriptor we're working on, as we'll need it later to remove it from the epoll instance.

We then add our epoll_event and file descriptor to our epoll instance using epoll_ctl with the operation EPOLL_CTL_ADD. The provided epoll_event instance is copied by the kernel, so no need to declare it on the heap and keep track of it manually.

Finally, we tell our Eventloop that there is some outstanding work to be handled by incrementing the work counter after we've successfully added more work to our epoll instance.

Running the Eventloop

Now that we have a way to add events to our Eventloop we need to be able to actually wait for them and trigger our promises.

We do this by calling epoll_wait and passing it a list of empty epoll_event. epoll_wait will block the current thread until one of the previously added file descriptors has one of its events triggered, at which point the epoll_wait call will return the number of triggered events and populate the list of passed epoll_event.

void Eventloop::Run()
{
  // Keep waiting on events as long as we have outstanding work to do.
  while (m_workCount > 0)
  {
    // 10 is an arbitrary number of events to handle. This depends on your needs and can be tuned.
    epoll_event events[10]{};
    LTRACE(LOGGER, std::format("Work count is {}. Running epoll_wait", m_workCount));

    // Wait for events to occur. This blocks until an event has occurred.
    int ret = epoll_wait(m_epollFd, /* events = */ events, /* maxevents = */ 10, /* timeout = */ -1);
    if (ret == -1)
    {
      throw std::runtime_error("epoll_wait failed");
    }

    LDEBUG(LOGGER, std::format("Eventloop: got {} events", ret));

    for (int i = 0; i < ret; ++i)
    {
      // Get our EventData.
      auto* event = static_cast<EventData*>(events[i].data.ptr);
      LTRACE(LOGGER, std::format("Eventloop: Handling event {}", event->fd));

      // Before we trigger our promise make we sure we remove the fd from the epoll instance.
      // This is important as we'll otherwise keep getting this exact event over and over again.
      if (epoll_ctl(m_epollFd, /* op = */ EPOLL_CTL_DEL, /* fd = */ event->fd, /* event = */ nullptr) == -1)
      {
        throw std::runtime_error("Failed to remove fd from epoll instance: " + std::string(strerror(errno)));
      }

      // Trigger our promise.
      event->promise.Set();

      // Delete the event-data we heap-allocated earlier in 'AddFD()'.
      delete event;

      // We've handled an event, so we can decrease our work count.
      --m_workCount;
    }
  }
}

Asynchronous timer

Now that we have a functioning Eventloop we actually need to provide some asynchronous work to test our Eventloop.

I'll create a simple asynchronous timer as it easy to test and implement.

class Timer
{
 private:
  int m_timerFd{-1};

 public:
  Timer() = default;
  ~Timer();

  // Promises should not be discarded.
  [[nodiscard]] Promise Wait(Eventloop& eventloop, int timeoutSeconds);
};

All we need is to keep track of the file descriptor we'll get and provide a Wait() function that takes in our Eventloop and a time to sleep.

Our timer is based on timerfd_create (similar to timer_create, but gives us a file descriptor to work with).

Promise Timer::Wait(Eventloop& eventloop, int timeoutSeconds)
{
  Promise promise;

  itimerspec timerData{};

  // Set the timer to expire after 'timeoutSeconds' seconds.
  timerData.it_value.tv_sec = timeoutSeconds;
  // Our timer should not repeat, so we set the interval to 0.
  timerData.it_value.tv_nsec = 0;

  // Our timer should not repeat, so we set the interval to 0.
  timerData.it_interval.tv_sec = 0;
  timerData.it_interval.tv_nsec = 0;

  // Create our timer file descriptor. We use CLOCK_MONOTONIC to avoid issues with system clock changes and TFD_NONBLOCK to make our timer non-blocking.
  m_timerFd = timerfd_create(CLOCK_MONOTONIC, TFD_NONBLOCK);
  if (m_timerFd == -1)
  {
    LERROR(LOGGER, std::format("Failed to create timerfd: {}", strerror(errno)));
    throw std::runtime_error("Failed to create timerfd");
  }

  // Set our timer with the earlier created 'timerData'.
  if (timerfd_settime(m_timerFd, 0, &timerData, nullptr) == -1)
  {
    LERROR(LOGGER, std::format("Failed to set timerfd: {}", strerror(errno)));
    throw std::runtime_error("Failed to set timerfd");
  }

  // Add our timer fd and promise to the eventloop.
  eventloop.AddFD(m_timerFd, promise);

  return promise;
}

We simply create a timer, set it to expire after timeoutSeconds and pass it to the Eventloop where it will be triggered with a EPOLLIN event after at least timeoutSeconds.

Testing our Eventloop

We can easily test our asynchronous timers and Eventloop by creating 2 detached coroutines, creating some timers in them, awaiting them and checking our output.

Promise Work(Eventloop& eventloop)
{
  Timer timer;

  LINFO(LOGGER, "Work: First timer");
  co_await timer.Wait(eventloop, 1);

  LINFO(LOGGER, "Work: Second timer");
  co_await timer.Wait(eventloop, 3);
  LINFO(LOGGER, "Work: coroutine is finishing");
  co_return;
}

Promise Work2(Eventloop& eventloop)
{
  Timer timer;

  LINFO(LOGGER, "Work2: First timer");
  co_await timer.Wait(eventloop, 1);

  LINFO(LOGGER, "Work2: Second timer");
  co_await timer.Wait(eventloop, 2);
  LINFO(LOGGER, "Work2: coroutine is finishing");
  co_return;
}

Promise AsyncMain(Eventloop& eventloop)
{
  // Start some detached coroutines
  std::ignore = Work(eventloop);
  std::ignore = Work2(eventloop);

  co_return;
}

int main()
{
  LDEBUG(LOGGER, "Creating eventloop");
  Eventloop eventloop;

  LINFO(LOGGER, "Starting AsyncMain");
  auto promise = AsyncMain(eventloop);

  LINFO(LOGGER, "Running eventloop");
  eventloop.Run();

  return 0;
}

Running code gives us the following output:

[2026-02-21 15:14:05.791438907] - [INFO]: Starting AsyncMain
[2026-02-21 15:14:05.791483637] - [INFO]: Work: First timer
[2026-02-21 15:14:05.791497898] - [INFO]: Work2: First timer
[2026-02-21 15:14:05.791506698] - [INFO]: Running eventloop
[2026-02-21 15:14:06.791516841] - [INFO]: Work: Second timer
[2026-02-21 15:14:06.791553451] - [INFO]: Work2: Second timer
[2026-02-21 15:14:08.791587399] - [INFO]: Work2: coroutine is finishing
[2026-02-21 15:14:09.791576815] - [INFO]: Work: coroutine is finishing

As you can see, we first start our first coroutine (Work) followed by the second coroutine (Work2).

We then sleep twice in each coroutine, with Work sleeping longer than Work2. As we would expect, Work does not block Work2 from executing and Work2 finishes before Work.

Github Repository

Thanks for reading! If you have any comments and/or want to talk about coroutines, feel free to reach out to me at rhidiandewit@gmail.com

The full code example for this blog post can be found here: https://github.com/Rhidian12/AsynchronousRuntime

The next blog post will most likely be about memory safety in coroutines and how to implement the core guidelines as best as possible.