Game with ESP32 and a LED matrix connected to Twitch

Today we are going to see a fun project to do as a Twitch community. It is a collaborative game in which users can write colored pixels on a LED matrix through commands written in the chat.

If you remember a few months ago, there was a game posted on Reddit that became very popular, which consisted of users having to write a colored pixel.

In this way, collaborative drawings and images were formed among all the users in the world. It was an event that lasted a couple of days and was very entertaining.

The idea is to do something similar, but controlled through Twitch chat commands, and put it as an overlay in OBS… Nothing especially complicated. In fact, several streamers have made similar developments.

But obviously, since our blog is a site dedicated to electronics and “tinkering”, we are going to take a turn to make the project more interesting.

So we are going to connect a WS2812B LED matrix of 16 x 16 pixels to an ESP32. This will be controlled via WiFi, through a NET6 application connected to the Twitch chat, and the communication will be done through MQTT.

For this, in previous posts we have already seen how to make a Twitch bot in NET6, how to use MQTT from a NET6 application, and how to control a WS2812B LED matrix very easily from an ESP32.

With that, we have all the necessary components to carry out the project. I am going to use an M5StickC Plus, because you know I like this device. But any ESP32, Arduino with WiFi, or similar, will serve you equally well.

NET6 Code

We start by looking at the code with the backend part. In our case it is done with a NET6 application. Remember that NET6 is compatible with multiple systems, including Windows and Linux. So it would be possible to host the backend application on a Raspberry Pi, for example.

We already saw the basics of how to create a bot in NET6 that connects to Twitch in this post. Therefore, we are only going to put the part related to managing the Twitch command, and its sending by MQTT. However, I leave all the code in a repository at the end of the post.

The part that interests us in the Twitch bot is the handler for the received command event, which has the following form.

if (e.Command.CommandText == "pixel" && e.Command.ArgumentsAsList.Count == 3)
{
    string x = e.Command.ArgumentsAsList[0];
    string y = e.Command.ArgumentsAsList[1];
    string color = e.Command.ArgumentsAsList[2];

    bool valid_x = int.TryParse(x, out int x_position);
    bool valid_y = int.TryParse(y, out int y_position);
    bool valid_color = ColorMapper.TryParse(color, out string color_hex);

    if (valid_x && valid_y && valid_color)
    {
        if (x_position >= 0 &amp;&amp; x_position < GRID_SIZE_X &amp;&amp;
            y_position >= 0 &amp;&amp; y_position < GRID_SIZE_Y)
        {
            SendByMqtt_AddPixel(x_position, y_position, color_hex);
        }
    }                
}
else if (e.Command.CommandText == "clear")
{
    SendByMqtt_Clear();
}
else if (e.Command.CommandText == "help")
{
    client.SendMessage(e.Command.ChatMessage.Channel, "DarkestRed - Red - LightRed - Orange - Yellow - Paleyellow - DarkGreen - Green - LightGreen - DarkTeal - Teal - LightTeal - DarkBlue - Blue - LightBlue - Indigo - Periwinkle - Lavender - DarkPurple - Purple - Palepurple - Magenta - Pink - LightPink - DarkBrown - Brown - Beige - Black - Darkgray - Gray - LightGray - White");
}

Basically, we are checking that the data received by the command is correct, and if so, we execute the clear command or the set command, with the appropriate parameters.

Depending on the command, we send it by MQTT. For this, we use the functions related to sending via MQTT, which are as follows.

private void SendByMqtt_AddPixel(int x_position, int y_position, string color)
{
    dynamic product = new ExpandoObject();
    product.x = x_position;
    product.y = y_position;
    product.c = Utils.ColorUtils.Hex_to_RGB565(color);

    string json = JsonConvert.SerializeObject(product);
    clientMqtt.Publish("twitch/pixel/set", Encoding.UTF8.GetBytes(json), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
}

private void SendByMqtt_Clear()
{
    clientMqtt.Publish("twitch/pixel/clear", Encoding.UTF8.GetBytes("clear"), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
}

JS Code

On the other hand, we have the frontend application in HTML, which we are going to use as an overlay in OBS. It is a very simple website, consisting of a canvas and a script with the necessary functions to make the drawing. You have the code in the repository, here again we will only take care of the part related to the communication in JavaScript.

function onMessageArrived(message) {   
    let json = JSON.parse(message.payloadString);
    let command = json.command;

    if(command === "clear")
    {
        clearCanvas();
    }
    else if(command === "set")
    {
        let x = json.x;
        let y = json.y;
        let color = json.color;
        drawPixel(x, y, color);
    }
}

In the part related to communication, we simply receive the command through MQTT, parse the received JSON. With this, we execute the appropriate drawing functions on the canvas, defined in its own independent JavaScript file.

ESP32 Code

Finally, we have the part that interests us the most for this post, which is the part related to the ESP32 and the WS2812B LED Matrix. Again, you have all the code in the repository, here we are going to focus on the part of managing the message received by MQTT.

void OnMqttReceived(char* topic, byte* payload, unsigned int length)
{
  String topicStr = topic;
  content = GetPayloadContent((char*)payload, length);

  if (topicStr == "twitch/pixel/clear")
  {
    ClearScreen();
  }
  if (topicStr == "twitch/pixel/set")
  {
    StaticJsonDocument<200> doc;
    DeserializationError error = deserializeJson(doc, content);
    if (error) return;

    uint8_t x = doc["x"];
    uint8_t y = doc["y"];
    uint32_t color = doc["rgb565"];
    DrawPixel(x, y, color);
  }
}

As we can see, fundamentally it is very similar to the case of JavaScript. In summary, we receive the JSON through the appropriate topic, we obtain the parameters from the JSON, and we perform the necessary actions to draw on the TFT screen and send to the LED matrix.

Conclusion

That’s how simple we have seen how to make a collaborative game that connects to the Twitch chat to make a board similar to the one on Pixel Reddit. For this, we have made a NET6 application, an overlay in HTML and connected an ESP32 with a TFT and an RGB LED matrix of 16 x 16 pixels.

The complicated part of the communication we have done through MQTT, which has made it a very simple part. However, it is a representative project of what we could find in a real IoT application, although in this case the notification service would be hosted in the cloud (on a VPS, on AWS, Azure… etc). But the architecture is similar.

An interesting and fun project to do both as an electronics project and as an exercise in integration with a third-party application (in this case, as a Twitch bot).

Download the code

All the code in this post is available for download on Github.