Skip to main content

XiaoZhi on Kamoji: voice assistant firmware, end to end

How to build xiaozhi-esp32 for Kamoji: add a new board to the upstream source, patch the mic path, set a wake word of your own, build and flash.

The result is Kamoji as a voice assistant. Say the wake word, talk, and the server answers in Vietnamese through the speaker. Wi-Fi is set up over a hotspot; devices are managed at xiaozhi.me.

Demo​

Signing in to the server, answering in Vietnamese, then the right touch pad taking volume to 100.

The log excerpts further down tell you the firmware reached each state. The clip tells you something they cannot: how long each step takes on real hardware, and how the panel behaves while it happens. Wi-Fi scan to a spoken reply is about ten seconds, the volume overlay draws over a reply that is still being spoken, and Vietnamese diacritics render correctly at 240×240 in the bundled font. That last one was the open question before the clip existed.

VietBot is the other guide in this section. It speaks the same protocol against a Vietnamese-hosted server, so the board definition and the microphone patch below carry over; the pairing flow and the wake word options are what differ.

1. Hardware​

PartKamoji
MCUESP32-S3
PSRAM8 MB, octal mode
Flash16 MB
DisplayST7789 240×240, SPI, write-only
MicI2S MEMS (INMP441 / SPH0645)
Speakerclass-D amp over I2S
Buttons1 BOOT button + 3 touch pads
LEDWS2812

Wake word needs the esp-sr AFE, and the AFE requires an ESP32-S3 with PSRAM. Kamoji qualifies.

2. Toolchain​

ESP-IDF ≥ 5.5.2. Take the version from xiaozhi's main/idf_component.yml, not from the README, which lists a different one. Verified build: v5.5.4.

git clone -b v5.5.4 --recursive https://github.com/espressif/esp-idf.git ~/esp/esp-idf-v5.5.4
cd ~/esp/esp-idf-v5.5.4 && ./install.sh esp32s3

You also need python3 and pyserial on the host to read the serial log.

Kamoji uses USB-Serial-JTAG, so there is no separate USB-UART chip. Plugging in gives you /dev/cu.usbmodem* (macOS) or /dev/ttyACM* (Linux). If no port shows up:

ioreg -p IOUSB -l -w 0 | grep -i "USB JTAG"     # macOS
lsusb | grep -i espressif                        # Linux

3. Get the source​

mkdir -p ~/kamoji-xiaozhi && cd ~/kamoji-xiaozhi
git clone https://github.com/78/xiaozhi-esp32.git src
cd src

No --recursive needed: xiaozhi uses the IDF Component Manager, and dependencies are fetched on the first set-target.

4. Create the kamoji board​

Three files in main/boards/kamoji/.

4.1 config.h, the pin map​

#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_

#include <driver/gpio.h>

// ---------------------------------------------------------------
//  Audio: I2S simplex, no I2C codec
// ---------------------------------------------------------------
#define AUDIO_INPUT_SAMPLE_RATE  16000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000

#define AUDIO_I2S_METHOD_SIMPLEX

#define AUDIO_I2S_MIC_GPIO_WS   GPIO_NUM_15   // LRCLK
#define AUDIO_I2S_MIC_GPIO_SCK  GPIO_NUM_17   // BCLK
#define AUDIO_I2S_MIC_GPIO_DIN  GPIO_NUM_16   // mic DOUT

#define AUDIO_I2S_SPK_GPIO_LRCK GPIO_NUM_21
#define AUDIO_I2S_SPK_GPIO_BCLK GPIO_NUM_48
#define AUDIO_I2S_SPK_GPIO_DOUT GPIO_NUM_47   // amp DIN

// The Kamoji mic sits in the LEFT slot. Reading the RIGHT slot returns all zeros.
#define AUDIO_I2S_SPK_SLOT_MASK I2S_STD_SLOT_LEFT
#define AUDIO_I2S_MIC_SLOT_MASK I2S_STD_SLOT_LEFT

// ---------------------------------------------------------------
//  Buttons: touch pads idle low, go high on touch
// ---------------------------------------------------------------
#define BOOT_BUTTON_GPIO        GPIO_NUM_0
#define TOUCH_BUTTON_GPIO       GPIO_NUM_5    // top pad
#define VOLUME_UP_BUTTON_GPIO   GPIO_NUM_6    // right pad
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_2    // left pad

// ---------------------------------------------------------------
//  WS2812 LED
// ---------------------------------------------------------------
#define BUILTIN_LED_GPIO        GPIO_NUM_4

// ---------------------------------------------------------------
//  Display: ST7789 240x240, CS tied to ground so it is write-only
// ---------------------------------------------------------------
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_13
#define DISPLAY_MOSI_PIN      GPIO_NUM_11
#define DISPLAY_CLK_PIN       GPIO_NUM_12
#define DISPLAY_DC_PIN        GPIO_NUM_14
#define DISPLAY_RST_PIN       GPIO_NUM_10
#define DISPLAY_CS_PIN        GPIO_NUM_NC

#define LCD_TYPE_ST7789_SERIAL
#define DISPLAY_WIDTH   240
#define DISPLAY_HEIGHT  240
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y true
#define DISPLAY_SWAP_XY  false
#define DISPLAY_INVERT_COLOR true
#define DISPLAY_RGB_ORDER    LCD_RGB_ELEMENT_ORDER_BGR
#define DISPLAY_OFFSET_X 0
#define DISPLAY_OFFSET_Y 80
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
#define DISPLAY_SPI_MODE 3

#endif // _BOARD_CONFIG_H_

Four display values are the easiest to get wrong, and here is why they have to be what they are:

ValueReason
MIRROR_X / MIRROR_Y = true, BGRThe Kamoji panel runs MADCTL 0xC8 = MY | MX | BGR
OFFSET_Y = 80The controller is 240×320; the visible 240×240 window sits 80 px down
SPI_MODE = 3The panel latches data on this clock edge
CS_PIN = NCCS is tied to ground on the board, so there is no chip-select signal
Don't use the stock 240×240 preset

XiaoZhi ships CONFIG_LCD_ST7789_240X240_7PIN. It sets OFFSET_Y = 0 and RGB colour order, so the image lands 80 px off with the colours wrong.

4.2 kamoji_board.cc, the board implementation​

#include "wifi_board.h"
#include "codecs/no_audio_codec.h"
#include "display/lcd_display.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "led/single_led.h"
#include "assets/lang_config.h"

#include <esp_log.h>
#include <esp_lcd_panel_vendor.h>
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <driver/spi_common.h>

#define TAG "KamojiBoard"

class KamojiBoard : public WifiBoard {
private:
    Button boot_button_;
    Button touch_top_button_;
    Button touch_right_button_;
    Button touch_left_button_;
    LcdDisplay* display_;

    void InitializeSpi() {
        spi_bus_config_t buscfg = {};
        buscfg.mosi_io_num = DISPLAY_MOSI_PIN;
        buscfg.miso_io_num = GPIO_NUM_NC;      // write-only panel
        buscfg.sclk_io_num = DISPLAY_CLK_PIN;
        buscfg.quadwp_io_num = GPIO_NUM_NC;
        buscfg.quadhd_io_num = GPIO_NUM_NC;
        buscfg.max_transfer_sz = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t);
        ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
    }

    void InitializeLcdDisplay() {
        esp_lcd_panel_io_handle_t panel_io = nullptr;
        esp_lcd_panel_handle_t panel = nullptr;

        esp_lcd_panel_io_spi_config_t io_config = {};
        io_config.cs_gpio_num = DISPLAY_CS_PIN;
        io_config.dc_gpio_num = DISPLAY_DC_PIN;
        io_config.spi_mode = DISPLAY_SPI_MODE;
        io_config.pclk_hz = 80 * 1000 * 1000;   // the Kamoji panel handles 80 MHz
        io_config.trans_queue_depth = 10;
        io_config.lcd_cmd_bits = 8;
        io_config.lcd_param_bits = 8;
        ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI3_HOST, &io_config, &panel_io));

        esp_lcd_panel_dev_config_t panel_config = {};
        panel_config.reset_gpio_num = DISPLAY_RST_PIN;
        panel_config.rgb_ele_order = DISPLAY_RGB_ORDER;
        panel_config.bits_per_pixel = 16;
        ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io, &panel_config, &panel));

        esp_lcd_panel_reset(panel);
        esp_lcd_panel_init(panel);
        esp_lcd_panel_invert_color(panel, DISPLAY_INVERT_COLOR);
        esp_lcd_panel_swap_xy(panel, DISPLAY_SWAP_XY);
        esp_lcd_panel_mirror(panel, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y);

        display_ = new SpiLcdDisplay(panel_io, panel,
                                     DISPLAY_WIDTH, DISPLAY_HEIGHT,
                                     DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
                                     DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY);
    }

    void InitializeButtons() {
        auto toggle_chat = [this]() {
            auto& app = Application::GetInstance();
            if (app.GetDeviceState() == kDeviceStateStarting) {
                EnterWifiConfigMode();
                return;
            }
            app.ToggleChatState();
        };

        boot_button_.OnClick(toggle_chat);
        touch_top_button_.OnClick(toggle_chat);

        touch_right_button_.OnClick([this]() {
            auto codec = GetAudioCodec();
            auto volume = codec->output_volume() + 10;
            if (volume > 100) volume = 100;
            codec->SetOutputVolume(volume);
            GetDisplay()->ShowNotification(Lang::Strings::VOLUME + std::to_string(volume));
        });
        touch_right_button_.OnLongPress([this]() {
            GetAudioCodec()->SetOutputVolume(100);
            GetDisplay()->ShowNotification(Lang::Strings::MAX_VOLUME);
        });

        touch_left_button_.OnClick([this]() {
            auto codec = GetAudioCodec();
            auto volume = codec->output_volume() - 10;
            if (volume < 0) volume = 0;
            codec->SetOutputVolume(volume);
            GetDisplay()->ShowNotification(Lang::Strings::VOLUME + std::to_string(volume));
        });
        touch_left_button_.OnLongPress([this]() {
            GetAudioCodec()->SetOutputVolume(0);
            GetDisplay()->ShowNotification(Lang::Strings::MUTED);
        });
    }

public:
    KamojiBoard() :
        boot_button_(BOOT_BUTTON_GPIO),
        // second argument = active_high: Kamoji pads go high on touch
        touch_top_button_(TOUCH_BUTTON_GPIO, true),
        touch_right_button_(VOLUME_UP_BUTTON_GPIO, true),
        touch_left_button_(VOLUME_DOWN_BUTTON_GPIO, true) {
        InitializeSpi();
        InitializeLcdDisplay();
        InitializeButtons();
        if (DISPLAY_BACKLIGHT_PIN != GPIO_NUM_NC) {
            GetBacklight()->RestoreBrightness();
        }
    }

    virtual Led* GetLed() override {
        static SingleLed led(BUILTIN_LED_GPIO);
        return &led;
    }

    virtual AudioCodec* GetAudioCodec() override {
        static NoAudioCodecSimplex audio_codec(
            AUDIO_INPUT_SAMPLE_RATE, AUDIO_OUTPUT_SAMPLE_RATE,
            AUDIO_I2S_SPK_GPIO_BCLK, AUDIO_I2S_SPK_GPIO_LRCK, AUDIO_I2S_SPK_GPIO_DOUT,
            AUDIO_I2S_SPK_SLOT_MASK,
            AUDIO_I2S_MIC_GPIO_SCK, AUDIO_I2S_MIC_GPIO_WS, AUDIO_I2S_MIC_GPIO_DIN,
            AUDIO_I2S_MIC_SLOT_MASK);
        return &audio_codec;
    }

    virtual Display* GetDisplay() override { return display_; }

    virtual Backlight* GetBacklight() override {
        if (DISPLAY_BACKLIGHT_PIN != GPIO_NUM_NC) {
            static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
            return &backlight;
        }
        return nullptr;
    }
};

DECLARE_BOARD(KamojiBoard);

Use the 10-argument NoAudioCodecSimplex overload. The 8-argument one takes no slot mask and will read the wrong mic channel.

4.3 config.json, the board identity​

{
    "type": "kamoji",
    "target": "esp32s3",
    "builds": [
        {
            "name": "kamoji",
            "sdkconfig_append": [
                "CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y",
                "CONFIG_PARTITION_TABLE_CUSTOM=y",
                "CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/16m.csv\""
            ]
        }
    ]
}

type is the device name reported to the server. CMake rejects anything outside ^[a-z0-9.-]+$, so lowercase letters, digits, . and - only.

5. Register the board with the build system​

XiaoZhi does not auto-discover board directories. Two places, and missing either one breaks the build.

main/Kconfig.projbuild, inside choice BOARD_TYPE:

    config BOARD_TYPE_KAMOJI
        bool "Kamoji (ESP32-S3, ST7789 240x240, I2S MEMS mic + class-D amp)"
        depends on IDF_TARGET_ESP32S3

main/CMakeLists.txt, as the first branch of the if/elseif chain that picks BOARD_DIR:

if(CONFIG_BOARD_TYPE_KAMOJI)
    set(BOARD_DIR "kamoji")
    set(BUILTIN_TEXT_FONT font_noto_sans_basic_20_4)
    set(BUILTIN_ICON_FONT font_material_symbols_20_4)
    set(DEFAULT_EMOJI_COLLECTION noto-color-emoji_64)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI)
    ...

The 20_4 fonts with 64 px emoji are what the other 240×240 boards use. Font names are checked against ^font_noto_sans_basic_([0-9]+)_([0-9]+)$ by regex. Keep elseif spelled correctly: elsif makes CMake report a syntax error on a line far away from the mistake.

Miss the CMake branch and the build stops immediately with:

FATAL_ERROR: The selected board does not define BOARD_DIR

6. Patch the mic path, and this one is mandatory​

Edit NoAudioCodec::Read() in main/audio/codecs/no_audio_codec.cc.

An I2S MEMS mic carries a very large DC bias. On Kamoji it measures around 2.5·10⁸ on the 32-bit scale. The stock code only shifts:

int32_t value = bit32_buffer[i] >> 12;

2.5·10⁸ >> 12 = 61035, past INT16_MAX of 32767, so every sample clips. The wake-word detector sees a flat constant: the screen still says it is listening, but it never recognises speech.

Replace it with a one-pole high-pass, time constant ≈ 16 ms, and shift by 11, not 12:

    samples = bytes_read / sizeof(int32_t);
    static int32_t dc_estimate = 0;
    for (int i = 0; i < samples; i++) {
        int32_t raw = bit32_buffer[i];
        dc_estimate += (raw - dc_estimate) >> 8;
        int32_t value = (raw - dc_estimate) >> 11;   // 11, not 12
        dest[i] = (value > INT16_MAX) ? INT16_MAX
                : (value < -INT16_MAX) ? -INT16_MAX : (int16_t)value;
    }
    return samples;

Two changes from upstream, both needed. Removing the DC bias stops the clipping. The shift then sets the level, and 12 is too quiet to recognise. Same speaker, same distance:

ShiftSpeech peakServer returns text
>> 121520 / 32767 (−26 dBFS)never
>> 118000–26000yes

At >> 12 every log line still looks healthy. The mic reads its full 100 frames/second, the AFE passes that level through, and VAD even flags speech. It is simply too quiet for recognition. Don't go to >> 10 either: the loudest sample measured here is 26039 at >> 11, still under the ceiling, but 52078 at >> 10, which is heavy clipping.

This is a property of the microphone, not a board fault: Kamoji's stock firmware has an equivalent filter in its own mic driver.

7. Build configuration​

Create sdkconfig.kamoji at the source root:

CONFIG_BOARD_TYPE_KAMOJI=y

# Vietnamese UI and voice prompts
CONFIG_LANGUAGE_ZH_CN=n
CONFIG_LANGUAGE_VI_VN=y

# 16MB flash, v2 partition table (has an assets partition)
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions/v2/16m.csv"

# Wake word, see section 8
CONFIG_USE_AFE_WAKE_WORD=y
CONFIG_SR_WN_WN9_ALEXA=y
CONFIG_SR_WN_WN9_NIHAOXIAOZHI_TTS=n

# AFE audio processing (AEC + VAD)
CONFIG_USE_AUDIO_PROCESSOR=y

# Package and flash the assets image (fonts, emoji, wake-word model)
CONFIG_FLASH_DEFAULT_ASSETS=y

# Wi-Fi provisioning over hotspot
CONFIG_USE_HOTSPOT_WIFI_PROVISIONING=y

Then an env.sh next to the source so you don't have to remember it:

source ~/esp/esp-idf-v5.5.4/export.sh
export SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.defaults.esp32s3;sdkconfig.kamoji"

sdkconfig.defaults.esp32s3 has to be listed explicitly: once you set SDKCONFIG_DEFAULTS yourself, idf.py stops appending the per-target file, and that file holds the octal 80 MHz PSRAM settings.

Change config in this file, not in menuconfig

Once sdkconfig exists it overrides SDKCONFIG_DEFAULTS; the overlay files only supply defaults for keys sdkconfig doesn't already have. So after editing sdkconfig.kamoji you must rm sdkconfig and rebuild, otherwise you get the old firmware with no error at all. Editing via menuconfig leaves sdkconfig.kamoji no longer describing the real firmware, and anyone rebuilding the project on another machine loses those changes.

8. Wake word​

Two decisions here: which detector, and whether the phrase is one of esp-sr's or one of yours. The stock xiaozhi phrase is Chinese; you do not have to live with that.

§6 is a hard prerequisite for everything below: with the mic still at >> 12, no detector recognises anything, and the log gives you no hint that this is the problem.

8.1 Which engine​

esp-sr has two detectors. They share no code, and they take completely different config.

WakeNetMultiNet
What it isa network trained for exactly one phrasea general command recogniser
The phrase lives inthe model weightsa string you type in config
You typenothing, just pick a modelletters or phonemes
Conversion stepnoneyes, and this is where it breaks
Sensitivityhighnoticeably lower
Your own phrasenoyes

One question decides it: does the wake phrase have to be your own words?

  • No → WakeNet. Far more reliable, one config line, no code changes.
  • Yes → MultiNet. You get any phrase you like, at the cost of exact formatting and lower sensitivity.

WakeNet is more sensitive because it only has to tell one phrase from everything else, so it recognises the overall shape. MultiNet has to decode each unit and match the whole string, so one wrong sound misses.

8.2 A built-in English phrase​

Two lines, nothing else:

CONFIG_USE_AFE_WAKE_WORD=y
CONFIG_SR_WN_WN9_ALEXA=y
Config linePhraseTrained on
CONFIG_SR_WN_WN9_ALEXA=y"Alexa"real voices
CONFIG_SR_WN_WN9_HIESP=y"Hi ESP"real voices
CONFIG_SR_WN_WN9_HIMFIVE=y"Hi M Five"real voices
CONFIG_SR_WN_WN9_JARVIS_TTS=y"Jarvis"synthesised
CONFIG_SR_WN_WN9_SOPHIA_TTS=y"Sophia"synthesised
CONFIG_SR_WN_WN9_HEYKIRA_TTS3=y"Hey Kira"synthesised
CONFIG_SR_WN_WN9_MYCROFT_TTS=y"Mycroft"synthesised
CONFIG_SR_WN_WN9_HEYWILLOW_TTS=y"Hey Willow"synthesised

The _tts suffix means the model was trained on synthesised speech and is less reliable than one trained on real voices. Of esp-sr's 64 models, only three English ones lack that suffix: wn9_alexa, wn9_hiesp, wn9_himfive. Start there.

Enable exactly one. The model-loading loop in afe_wake_word.cc has no break, so enabling several only bloats the assets image, and the extra phrases don't work.

Correct log:

AfeAudioEngine: Model 0: wn9_alexa
AFE: AFE Pipeline: [input] -> |VAD(WebRTC)| -> |WakeNet(wn9_alexa,)| -> [output]
AfeAudioEngine: Initialized FD AFE, detector: WakeNet

8.3 Your own phrase, no code changes​

This is the one to use if you want HEY KAMOJI. MultiNet6 English takes plain letters, so there is no conversion step to get wrong.

CONFIG_USE_AFE_WAKE_WORD=n
CONFIG_USE_CUSTOM_WAKE_WORD=y
CONFIG_CUSTOM_WAKE_WORD="HEY KAMOJI"
CONFIG_CUSTOM_WAKE_WORD_DISPLAY="Kamoji"
CONFIG_CUSTOM_WAKE_WORD_THRESHOLD=10

CONFIG_SR_MN_CN_NONE=y
CONFIG_SR_MN_EN_NONE=n
CONFIG_SR_MN_EN_MULTINET6_QUANT=y

Letters are MultiNet6's native unit. esp-sr's docs recommend all caps for MultiNet6. The phrase is read with English spelling rules, so XIN CHAO comes out sounding English, not Vietnamese. To get a Vietnamese-ish sound you have to misspell it deliberately, e.g. SIN CHOW.

CONFIG_CUSTOM_WAKE_WORD_DISPLAY is only the label shown on screen and in the log.

Enable one language only

If both a Chinese and an English MultiNet model are on, get_language_from_multinet_models() in scripts/build_default_assets.py returns cn, and your English phrase never matches.

Assets image sizes: mn6_en 3.6 MB, mn7_en 2.6 MB.

There is also a Chinese path (mn7_cn, pinyin syllables like ga mo ji). English speakers don't need it, so skip it.

8.4 mn7_en, your own phrase via phonemes​

MultiNet7 English works on phonemes, not letters. Its config:

CONFIG_CUSTOM_WAKE_WORD="hey kamoji"

CONFIG_SR_MN_CN_NONE=y
CONFIG_SR_MN_EN_NONE=n
CONFIG_SR_MN_EN_MULTINET7_QUANT=y

esp-sr's docs say an internal grapheme-to-phoneme tool is called at runtime when you supply no phonemes, and esp_mn_commands_add() does contain that branch, but on this board it rejects the phrase. Feeding the phonemes in directly works. In main/audio/wake_words/custom_wake_word.cc:

#ifdef CONFIG_SR_MN_EN_MULTINET7_QUANT
// flite_g2p.h ships without an extern "C" guard, so including it straight
// from C++ mangles the symbol and the prebuilt C library never matches.
extern "C" {
#include <flite_g2p.h>
}
#endif

and in the command-loading loop of CustomWakeWord::Initialize:

     for (int i = 0; i < commands_.size(); i++) {
+#ifdef CONFIG_SR_MN_EN_MULTINET7_QUANT
+        char* ph = flite_g2p(commands_[i].command.c_str(), 1);
+        ESP_LOGW(TAG, "G2PDIAG '%s' -> '%s'", commands_[i].command.c_str(),
+                 ph != nullptr ? ph : "(null)");
+        esp_err_t err = esp_mn_commands_phoneme_add(i + 1, commands_[i].command.c_str(), ph);
+        ESP_LOGW(TAG, "G2PDIAG phoneme_add ret=%d (0=OK)", (int)err);
+        // phoneme_add copies the string into the phrase, unlike
+        // esp_mn_commands_add() which frees it for us, so release it here.
+        if (ph != nullptr) { free(ph); }
+#else
         esp_mn_commands_add(i + 1, commands_[i].command.c_str());
+#endif
     }

Three details that each cost real time to find:

  • flite_g2p.h has no extern "C". Include it from C++ and the link dies with undefined reference to '_Z9flite_g2pPKci', because the C++-mangled name never matches the prebuilt C library. The guard is mandatory.
  • esp_mn_commands_phoneme_add() does not free the phoneme string, unlike esp_mn_commands_add() which frees it internally. Call free(ph) yourself or leak every boot.
  • On-device G2P differs from the offline tool. esp-sr/tool/multinet_g2p.py uses g2p_en (CMUdict); the runtime uses Flite. Same phrase, two answers: hd KcMoqm ("hay ker-MOO-ji") offline versus hd KaMbqm ("hay ka-MOH-ji") on the board. Don't predict pronunciation from the offline tool. Read the G2PDIAG line in the log, that's the truth.

Correct log:

AfeAudioEngine: Model 0: mn7_en
CustomWakeWord: G2PDIAG 'hey kamoji' -> 'hd KaMbqm'
CustomWakeWord: G2PDIAG phoneme_add ret=0 (0=OK)
                1 active speech commands:
                Command 1: hd KaMbqm

8.5 Phrase rules​

Syllablesthree or more
Charactersno digits, no punctuation, no accented letters
Languagenever mix Chinese and English in one phrase
Casemn6_en ALL CAPS; mn7_en lowercase

The middle two are esp-sr rules: mixed Chinese and English is unsupported, and command words cannot contain Arabic numerals or special characters.

One- and two-syllable phrases false-trigger constantly, which is why HEY KAMOJI beats plain KAMOJI. A longer phrase puts more constraints in the FST, so there is less to match by accident.

8.6 Where the phrase actually lives​

CONFIG_CUSTOM_WAKE_WORD does not go straight into the C++.

sdkconfig.kamoji     CONFIG_CUSTOM_WAKE_WORD="HEY KAMOJI"
   │  rm sdkconfig + idf.py build
   ▼
sdkconfig
   │  scripts/build_default_assets.py reads sdkconfig
   ▼
index.json           { "language":"en", "threshold":0.1,
                       "commands":[{"command":"HEY KAMOJI","action":"wake"}] }
   │  packed with the model, fonts and emoji
   ▼
generated_assets.bin ──esptool──▶ assets partition at 0x800000

At runtime it reads back the other way: ParseWakenetModelConfig() loads index.json from assets, calls esp_mn_commands_add(), and MultiNet builds the FST from that list.

The direct consequence: the phrase is in the assets partition, not in the app. Flash only 0x20000 and the old phrase stays live. idf.py flash writes everything, so this only bites when flashing regions by hand.

Check the phrase made it into the image before flashing:

python3 -c "
d = open('build/generated_assets.bin','rb').read()
i = d.find(b'\"multinet_model\"')
print(d[i:i+240].decode('utf-8','replace') if i>=0 else 'NOT FOUND')"

8.7 Sensitivity​

CONFIG_CUSTOM_WAKE_WORD_THRESHOLD is a percentage floor on the beam-search score.

SymptomDo this
Doesn't respond when calledlower it to 10, then 5
Wakes up on its ownraise it to 20, 30, 40

The xiaozhi default is 20; this board uses 10. There is no universally correct value. It depends on the room, the background noise and how far away you usually stand. Measure it: call ten times and count the hits, then sit silent for five minutes and count the false wakes.

MultiNet is fast enough that performance is never the culprit. Inference on a 512-sample (32 ms) chunk takes 17–25 ms and the backlog stays flat, so if the wake word doesn't fire, look at the phrase format or the threshold, not at CPU.

9. Build​

cd ~/kamoji-xiaozhi/src
source ../env.sh
idf.py set-target esp32s3     # first time only, to fetch dependencies
idf.py build

The first set-target downloads a lot of components and needs network. A full build takes about 5–10 minutes.

What a correct build looks like, near the top:

wakenet models: wn9_alexa (will be packaged)

and at the end:

xiaozhi.bin binary size 0x2bb150 bytes. Smallest app partition is 0x3f0000 bytes. 31% free.

10. Flash​

idf.py -p /dev/cu.usbmodem11301 flash

That writes five regions: bootloader, partition table, ota_data, app, and the assets image. NVS is not erased, so saved Wi-Fi and the device binding survive a reflash.

To start over from scratch:

esptool.py --chip esp32s3 -p /dev/cu.usbmodem11301 erase_flash

This wipes NVS too, meaning both the saved Wi-Fi and the device identity: the board generates a new UUID and has to be re-added at xiaozhi.me. To clear only Wi-Fi:

esptool.py -p /dev/cu.usbmodem11301 erase_region 0x9000 0x4000

11. First-run setup​

On first boot there is no Wi-Fi yet, so the board opens its own hotspot:

StateMachine: State: starting -> wifi_configuring
WifiConfigurationAp: Access Point started with SSID Xiaozhi-XXXX
esp_netif_lwip: DHCP server started ... IP: 192.168.4.1
  1. Join the Wi-Fi network Xiaozhi-XXXX (last four characters come from the MAC).
  2. Open http://192.168.4.1.
  3. Enter your home Wi-Fi. Leave the Advance tab empty. Filling it writes a custom server address into NVS that then overrides the default server.
  4. The board reboots, connects, and shows a 6-digit code.
  5. Add the device with that code at xiaozhi.me.

The assistant's name and personality live at xiaozhi.me, not in the firmware, so changes take effect immediately with no reflash.

In the wifi_configuring state the wake word does not work yet: the firmware only starts the detector after it is on a network. No response before Wi-Fi is configured is expected.

12. Verify from the log​

idf.py -p /dev/cu.usbmodem11301 monitor      # Ctrl+] to exit

Lines worth watching at boot:

Log lineMeaning
Board: UUID=… SKU=kamojiboard detected correctly
AfeAudioEngine: Model 0: wn9_alexawake-word model loaded from assets
AFE Pipeline: [input] -> |VAD(WebRTC)| -> |WakeNet(wn9_alexa,)| -> [output]wake word is in the pipeline
Initialized FD AFE, detector: WakeNetrunning in WakeNet mode
AudioCodec: Set input enable to truemic is on
StateMachine: State: activating -> idleserver connected, ready

Saying the wake word produces the full sequence:

Application: Wake word detected: Alexa
StateMachine: State: idle -> connecting
StateMachine: State: connecting -> listening
Application: >> Alexa                      <- what you said, as the server heard it
StateMachine: State: listening -> speaking
Application: << Xin chào, mình nghe đây.   <- the reply
StateMachine: State: speaking -> listening

A >> line means mic, WakeNet, network and speech recognition are all working. A << line means synthesis and the speaker work too. That is the exchange in the demo above.

12.1 Extra lines when you use a custom phrase​

MultiNet reports whether your command list loaded. Working:

1 active speech commands:
Command 1: HEY KAMOJI

Rejected:

Command HEY KAMOJI is invalid, please check format.
No commands available, not updating.
49 active speech commands:
Command 1: TfL Mm c qbK

49 is the tell. Your command list came out empty and the model fell back to esp-sr's built-in set (TELL ME A JOKE, SING A SONG, …). See 49, and you know your phrase never landed.

A successful wake also prints a score line: Custom wake word detected: command_id=1, string=..., prob=....

13. Troubleshooting​

SymptomCause
FATAL_ERROR: The selected board does not define BOARD_DIRmissing branch in main/CMakeLists.txt (§5)
Board not listed in menuconfigmissing declaration in Kconfig.projbuild (§5)
Image offset by 80 pxDISPLAY_OFFSET_Y is not 80
Wrong colours, red shows as blueLCD_RGB_ELEMENT_ORDER_BGR missing
White screen or garbageDISPLAY_SPI_MODE is not 3, or CS_PIN is not NC
Says it's listening but never recognises speechDC filter not patched (§6)
Wake word fires but the server never returns textmic shift left at >> 12 instead of >> 11 (§6)
Mic completely silentwrong slot mask, must be I2S_STD_SLOT_LEFT
Config changed but firmware identicaldidn't rm sdkconfig (§7)
Custom phrase rejected, log shows 49 commandswrong unit type for the model, letters vs phonemes (§8.3, §8.4)
Custom English phrase never matchesa Chinese MultiNet model is also enabled (§8.3)
Link error undefined reference to '_Z9flite_g2p...'missing extern "C" guard (§8.4)
Changed wake word, old phrase still worksassets region not reflashed (§8.6)
Wake word ignored before joining a networkby design (§11)
Touch pads fire random eventsactive_high = true missing in the Button constructor

14. Partition table​

partitions/v2/16m.csv:

NameOffsetSizeUsed for
nvs0x90000x4000Wi-Fi, device UUID, settings
otadata0xd0000x2000boot partition selection
phy_init0xf0000x1000RF calibration
ota_00x200000x3f0000app
ota_10x4100000x3f0000spare app for OTA updates
assets0x8000008Mfonts, emoji, wake-word model

The app uses about 2.9 MB of 4.1 MB, roughly 31% free, enough room for features added later.

15. Updating the source later​

The whole delta against upstream is six files:

PathChange
main/boards/kamoji/config.hnew
main/boards/kamoji/kamoji_board.ccnew
main/boards/kamoji/config.jsonnew
main/Kconfig.projbuildedit, one block added
main/CMakeLists.txtedit, one branch added
main/audio/codecs/no_audio_codec.ccedit, DC filter + gain patch
main/audio/wake_words/custom_wake_word.ccedit, only for mn7_en (§8.4)
cd ~/kamoji-xiaozhi/src
git stash
git pull
git stash pop

Two spots to watch when there are conflicts:

  • main/CMakeLists.txt, where the first if() branch conflicts easily. If you lose it the build fails loudly (FATAL_ERROR ... BOARD_DIR), so it can't slip through.
  • no_audio_codec.cc, where resolving in favour of upstream drops the DC filter and the mic goes silent with no error. Check with:
grep -c dc_estimate main/audio/codecs/no_audio_codec.cc     # must print 3

16. Limits​

No Vietnamese model. esp-sr has neither a WakeNet nor a MultiNet model for Vietnamese, so the wake phrase has to borrow English or Chinese sounds. This applies to the wake phrase only. The conversation afterwards is normal Vietnamese, because recognition and synthesis both run on the server.

Your own words with WakeNet reliability means having Espressif train a model. The model directory has a wn9_customword slot for exactly that; today it holds a copy of a Chinese model and has no Kconfig entry.

Still unexplained: why esp_mn_commands_add() rejects the phrase while esp_mn_commands_phoneme_add() accepts the same phoneme string. Ruled out: the Kconfig macro is present in sdkconfig.h and esp_mn_speech_commands.c does get recompiled; only the G2P branch's format string exists in the ELF; flite_g2p returns a valid string, not NULL or garbage; and both functions store identically via esp_mn_phrase_alloc(). The patch in §8.4 routes around it rather than fixing the cause.