Edit: Aug 22. Necessary changes

Edit: Neovim/LSP keybindings added

I’ve come to accept that I’m an old-school vi user. With about forty years of muscle memory behind that sentence, I’ll say that I’m an effective vi user, but in today’s world of C++ overrides, sentence-long symbols, and huge lists of available methods, I _have_ felt the occasional pang of jealosy and tried to move beyond bare bone vi. I’ll then sit back down at the terminal two days later, completely forget about that snazzy new editor that I told myself I’d learn, and immediately be back to a couple of iTerms with ack-grep just from habit. Making matters worse for myself, most of the time these days I work on ESP32 projects, so I’m dealing with cross compilers. Well, I’m not unhappy about that – having basically the same compiler on my embedded device as I do on my desktop is actually pretty sweet, but they’re always just different enough to be annoying. One difference that surfaces is that clangd-like services for Language Server Protocol (LSP) are usually just out of reach. One recent day when alternating between fighting CLion and vi (OK, it was probably actually vim; though I have flirtations with nvim.) I set out to win this battle.

It turns out there’s a slew of annoying behaviour you must overcome to get niceties to work. As is my style, I don’t declare this is the One True Way or even the Only Way. It’s just the journey that I took and I’m sharing it in the hopes that it saves you a few hours in your own journey. I’ll focus first on CLion and ESP32 because from there, “winning” with nvim is just an exercise in persistence. If it’s your goal to ignore CLion and focus on nvim or other clangd consumers, the steps to skip should be self-evident.

CLion claims to be able to use a platformio.ini. This is at best partially true. It can read the list of [env:] projects and give you a list of -e options that it will pass when shelling out to the (already slow and often broken) Platformio ‘pio’ command. By default, when given my (admittedly non-trivial, but hardly back-breaking) platformio.ini of 53 environments, we’re met with a red “Index 0 out of bounds for length 0”. No hint is given on what array was being indexed, what source file might be responsible, or even what language might have issued this error. All #included files are red squigglies because it can’t find them. No symbols have hover text unless they’re declared in the file you’re looking at, and syntax highlighting is about as effective as regex highlighting ever is on C++. Good job, Jetbrains.

Populate CLion’s shadow build system:

$ pio project init --ide clion

Resolving demo dependencies... (demo is the name of one environment in my platformio.ini that I'll use throughout)
Already up-to-date.
Updating metadata for the clion IDE...
Project has been successfully updated!

Restart CLion to see the changes.

$ open -a clion . (or whatever pointy clicky equivalent gives you joy)

It’s my experience, after probably a hundred restarts while developing this recipe, that this will fail on odd numbered invocations and pass on even numbered invocations… (Did I yet bitterly say “Good job, Jetbrains?”)

Now the bottom will show a blue progress bar during an ‘importing’ step that for me, takes about four minutes (!!) to complete. I have no idea why it takes long to analyze than it takes to completely build one or two of our targets, but I’m just a lowly bitslinger.

Restart CLion (twice) to see the changes.

Now it’ll still give the error and it still won’t convince you that it’s read platformio.ini, but it’s apparently necessary ritual sacrifice.

Build the Compilation “database”.

$ pio run -t compiledb -e demo

[ ... ]

Building in release mode
Building compilation database compile_commands.json

========================= [SUCCESS] Took 34.44 seconds =========================
Environment    Status    Duration
-------------  --------  ------------
demo           SUCCESS   00:00:34.439

========================= 1 succeeded in 00:00:34.439 =========================

Note that it will download and install all the packages required for all the builds (which is why I’m restricting it to ‘-e demo’ here or I’d be typing this until the cows came home. (Where do the cows GO anyway?) On my project, that’s about 20 libraries times 53 -e environments, so even though it’s not actually BUILDING much of anything, it’s still an exercise in persistence.

You now have a shiny new `compile_commands.json` file that describes what it would have invoked to build the software. Well, it doesn’t include the critical linking step, so it’s still not REALLY helpful if you’re trying to do what a ‘pio run’ would do, but it would tell you what the first 99% of a build will do. In my case, it’s 830 lines of blindingly dumb JSON that makes no attempt to factor out redundancy. These files ARE handy if you want to see what flags really really get applied to your build without looking at `-v`.

Purge the old CLion config

rm -fr .idea 

Now tell CLion to open that directory. It will recognize `compile_commands.json` and open it. The bottom of your tools menu will now have an option for Compilation Database. Your source tree will now show your source AND the zillion files in `~/.platformio/packages/framework-arduinoespressif32/libraries/*/src` that are ACTUALLY built. It also now knows where all your includes are as well as the system includes.

Notice that you can do OK living a bit of a lie here. The include paths for a lot of the system stuff actually change slightly depending if you’re targeting LX6 (ESP32-Classic), LX7 (ESP32-S2 or ESP32-S3) or RISC-V (everything else), but unless you’re working at the very bottom of the stack and NEED to know which source is used for the system stuff, it’s probably OK.  If it matters, you can repeat the pio run -t compiledb with a different -e and then reload the compilation database.

So this is great, right?

No. This is OK for editing in CLion, but you can’t actually BUILD in CLion this way. Despite giving it agonizing details of every invocation of g++, every one of the hundreds of `-I` flags and all that other stuff (look in `compile_commands.json`, seriously!) it won’t actually build code. You still have to open a window and `pio run -t upload -e foo` for that. There may be workarounds for that, but we’ll come back because vim is where work gets done while CLion is nice for fix-ups and clang warnings. Let’s forge on.

We have a great compile_commands.json, we can surely now rely on LSP support in nvim, right?

No.

We’re thwarted by many things. It seems that, despite having these agonizingly detailed roadmaps describing how to compile (almost) everything, both `clangd` and CLion have their own internal whitelists of toolchains. If your g++ binary isn’t whitelisted, it will silently fail and then try to invoke the native compiler. So the first time you hit, say `<vector>` or `<cstdio>` everything hits the fan because your MacOS or native Linux `/bin/g++`’s headers are in the “wrong” place and they call things that aren’t in the `-I` paths that are given. For MacOS, `_ansi.h` seems to be the poison pill, but if it weren’t that file, it would be something else.

I’ll skip ahead about two hours of debugging, but the solution lies in creating a .clangd in the root of your project’s directory to add the flags that you need and to strip the flags that nvim tries to add to the build that would choke your cross compiler, like anything starting with -m because you’re presumably compiling for a different machine architecture.

CompileFlags:
  Remove:
    - -m*
    - -fno-tree-switch-conversion
    - -fstrict-volatile-bitfields
    - -Wno-frame-address

  Add:
    - -nostdlibinc
    - -nostdinc++

I’ll fast-forward you through another two hours, though, and tell you that the `clangd` on Mac is so hell-bent on developing for Mac that it will ignore the above and STILL add things to its own include path that are toxic to the cross compiler. This is because Apple’s `/usr/bin/clangd` is a stripped-down, lobotomized version tied to Xcode that lacks standard features and stubbornly injects Apple-specific flags. The solution to this is to install `clangd` from Homebrew (`brew install llvm`) and then add a wrapper file (remember to mark it executable) in a directory in your `$PATH` that’s before the system binary. I already had `~/.local/bin/` in my PATH, so I’ll just add that.

My new ~/.local/bin/clangd:

#!/bin/sh

# Find the real clangd by listing ALL instances and skipping this wrapper
#REAL_CLANGD=$(which -a clangd | grep -v "\.local/bin/clangd" | head -n 1)

REAL_CLANGD=/opt/homebrew/opt/llvm/bin/clangd

# Execute it with the query-driver argument pointing to the global PlatformIO directory
exec "$REAL_CLANGD" --query-driver="$HOME/.platformio/packages/**/bin/*g++*,$HOME/.platformio/packages/**/bin/*gcc*" "$@"

Testing this mess without the editor

You’ll know this is in the game when you can do something like:

$ clangd --background-index --check=src/main.cpp 2>&1 | less

(Lipe-pro-tip: `–background-index` is generally a good flag to add in your editor setups so `clangd` can eagerly cache your project and make cross-file navigation instant.)

and see the  presence of:

I[01:22:30.648] Loading compilation database...
I[01:22:30.667] Loaded compilation database from ...
I[01:22:30.668] Compile command from CDB is:...

…and output that ends like:

I[01:23:24.137] All checks completed, 6 errors

It’s not important that it be zero. It’s important that it not be so many that clangd aborts. In fact, it’s even normal to have hundreds of lines like

E[01:23:24.130] IncludeCleaner: Failed to get an entry for resolved path : No such file or directory

or

E[01:25:02.807]     tweak: ExpandDeducedType ==> FAIL: Could not deduce type for 'auto' type

Despite these starting with “E:” and LOOKING like errors, this is expected.

** Note: if you tinker much with .clangd, it seems that a rm -fr .cache/clangd is sometimes needed. Since it’s not clear WHEN it’s needed and it tends to be pretty fast to just rebuild, if you’re working on this, just keep a rm -fr .clangd/cache ; nvim .clangd in your recall buffer.

NOW, you’ve slain the final boss between you and glorious nvim assisted by clangd via LSP.

<screenshot goes here>

Now, tab completion, hover, and auto clang-tidy all work sensibly. Victory!

But nvim screws up copy-paste?!?! Yeah, well…future battles to fight. *(Though if you want to skip the wrapper script entirely in Neovim, you can actually pass `–query-driver` directly in your `nvim-lspconfig` setup for `clangd` using the `cmd` table!)*

But I want to BUILD in CLion!

Since we bypassed CLion’s native PlatformIO integration (because it was choking on our 53 environments) and loaded the project as a bare Compilation Database, CLion doesn’t actually know *how* to compile your code. It just knows how to parse it. If you hit the hammer icon, nothing good happens.

To fix this, we need to teach CLion how to shell out to `pio`. We do this using Custom Build Targets.

1. Go to **Preferences / Settings -> Build, Execution, Deployment -> Custom Build Targets**.
2. Click the `+` to add a new target and call it something creative like “PIO Build”.
3. Next to the “Build” field, click the `…` to add an External Tool. 
4. Click `+` to add a new External Tool.

   – **Name**: `pio run demo`
   – **Program**: `pio` (or the full path like `~/.platformio/penv/bin/pio` if CLion’s `$PATH` is lacking)
   – **Arguments**: `run -e demo` (or whatever environment you want to actually build)
   – **Working directory**: `$ProjectFileDir$`

5. (Optional but helpful) Do the exact same thing for the “Clean” tool, passing `run -t clean -e demo` as the arguments
6. Save your new target.

Now, go to the Run/Debug Configurations dropdown in the toolbar (or **Run -> Edit Configurations**).

1. Click `+` and select **Custom Build Application**.
2. Set the **Target** to the “PIO Build” target you just created.
3. You can leave the Executable blank if you just want to build, or point it to your compiled `.elf` file if you have ambitions of setting up an Embedded GDB Server later.

Now you can smash that hammer icon or hit `Cmd+F9` (or your OS equivalent), and CLion will obediently shell out to PlatformIO, build your project, and dump the output in the console window.  If the native Platformio support worked, that’s all it would do for you anyway.

You’ve finally got the best of both worlds: CLion’s shiny graphical UI, proper code insight and navigation powered by your compilation database, and the actual build process correctly handled by PlatformIO. And when you’re ready to get real work done, your `.clangd` and wrapper script mean Neovim is sitting right there in your terminal, ready to go with full LSP support.

Take a victory lap. (And/or a beverage of your choosing.) You’ve earned it.

Shifting sands. Article changes.

I admittedly don’t use this configuration every day. I’d noticed that it had quit working, but I also wasn’t certain that I’d tested it on this exact configuration so I let it coast until that little rock in my shoe annoyed me enough to tackle it. Still, I knew my approach was sound but I couldn’t suss it out. Eventually, I asked my friend Gemini (Antigravity) to help. I’m leaving the above unedited because that probably works for some configurations. Here’s what we discovered.

• On this system, my $HOME is a symlink to an external SSD. (Thank you, Apple for your extortionistic storage prices.) My ~/.local/bin/clangd suffered from a one-two punch of overly strict matching and an update to PlatformIO moving the internal platform file.• What changed: We pointed REAL_CLANGD away from Homebrew’s LLVM and directly to the esp-clangd binary installed by EIM (~/.espressif/tools/esp-clangd/…).
  • The sneaky fix: EIM actually splits esp-clangd and esp-clang into two completely separate directories. This means the clangd binary doesn’t know where its own built-in headers are! We had to append –resource-dir=”~/.espressif/tools/esp-clang/…/lib/clang/21″ so it
  could find basic things like stddef.h. While we are here, we tweaked things so we only apply all this cleverness if we have a .platformio.ini containing the word “espressif” (so nvim on native projects still work) AND we dig around in the espressif tree to find the most recent esp-clangd.

  Updated file:

  $ cat ~/.local/bin/clangd
  
  
  #!/bin/sh
  
  # 1. Detect if we are inside an ESP32 PlatformIO project
  
  if [ -f "platformio.ini" ] && grep -qi "espressif" "platformio.ini"; then
  
      # 2. Dynamically find the newest esp-clangd and resource dirs
      ESP_CLANGD_DIR=$(ls -rd "$HOME"/.espressif/tools/esp-clangd/* 2>/dev/null | head -n 1)
  
      REAL_CLANGD="$ESP_CLANGD_DIR/esp-clangd/bin/clangd"
      ESP_CLANG_DIR=$(ls -rd "$HOME"/.espressif/tools/esp-clang/* 2>/dev/null | head -n 1)
      RESOURCE_DIR=$(ls -rd "$ESP_CLANG_DIR"/esp-clang/lib/clang/* 2>/dev/null | head -n 1)
  
      # 3. If the ESP tools exist, use them
      if [ -x "$REAL_CLANGD" ] && [ -d "$RESOURCE_DIR" ]; then
  
  exec "$REAL_CLANGD" \
  
       --query-driver="$HOME/.platformio/packages/**/bin/*g++*" \
       --resource-dir="$RESOURCE_DIR" \
       "$@"
      fi
      # If the ESP tools were missing, it safely falls through to the system default below
  fi
  
  # 4. We are NOT in an ESP project (or ESP tools are missing).
  # Fall back to standard desktop C++ clangd.
  # Find the first clangd that IS NOT this wrapper script itself.
  SYSTEM_CLANGD=$(which -a clangd | grep -v "\.local/bin/clangd" | head -n 1)
  
  if [ -n "$SYSTEM_CLANGD" ]; then
      exec "$SYSTEM_CLANGD" "$@"
  else
      echo "No system clangd found." >&2
      exit 1
  fi
  
  if [ -n "$SYSTEM_CLANGD" ]; then
      exec "$SYSTEM_CLANGD" "$@"
  else
      echo "No system clangd found." >&2
      exit 1
  fi

• NeoVIM (AstroNVIM?) changed direction, requiring tweaks to the Neovim Config (~/.config/nvim/lua/plugins/astrolsp.lua)
  • What changed: We explicitly set cmd = { “/Users/robertlipe/.local/bin/clangd”, … } instead of just “clangd”.
  • Why it broke your previous setup: AstroNvim recently integrated Mason very tightly. If you just ask for “clangd”, Mason intercepts your $PATH and silently launches its own vanilla LLVM clangd. This completely bypassed your wrapper script, dropping your –query-driver argument, and putting you back to square one. Hardcoding the absolute path forces Neovim to respect your wrapper. Solution:Add a section under “servers…pyright” stanza

      servers = {
        -- "pyright"
      },
  

      -- customize language server configuration options passed to `lspconfig`
      ---@diagnostic disable: missing-fields
      config = {
        clangd = {
          cmd = {
            "/Users/robertlipe/.local/bin/clangd",
            "--background-index",
          },
          capabilities = { offsetEncoding = "utf-8" },
        },
      },
  

      -- customize how language servers are attached
      handlers = {
        -- a function without a key is simply the default handler, functions take two parameters, the server name and the configured options table for that server
  

• I moved to C++26 and this made PlatformIO grumpy – Changes to .clangd required
  A few months ago, I bumped my minimum platform requirements to C++26 (because I can). PlatformIO adds its own –std= lines into the build and depending on the time of day, that can come before or after my own entry. GCC parses arguments left to right, so the last one standing ‘wins’. When calling clang-format, it was getting reset, so my
  static_assert(__cplusplus > 202002L, "This project requires C++26");
  was sending the parse down in flames.
  • What changed: We keep the Remove: block to strip out the GCC-specific optimization flags that cause warnings. However, we were able to delete -nostdlibinc and -nostdinc++. We have to add back the -std=gnu++2b flag.
  • Why it’s better: Because you’re now using Espressif’s custom clangd, it natively understands the Xtensa target. You no longer have to manually nuke the Apple standard headers because Espressif’s parser doesn’t confusingly fall back to macOS Mach-O headers anymore!You officially have the ultimate setup. No cosmetic #error in limits.h, native understanding of .iram1 ELF sections, and perfect C++ standard library resolution!

Appendix: Neovim/LSP Keybindings (the reason we do this…) 

Since this is largely documentation for myself, here, at no extra cost to you, is a subset of the keybindings that are most useful.