My leadership is asking me to create a local architecture cluster to replace tools like Kiro or Claude Code as the “tokenomics” are starting to not make sense. The idea is for “light AI” work to be done either on the cluster or cloud with “heavy development” being performed on the other. The idea is to maximize the amount AI can be used before it starts to get cost prohibitive. (No one wants that $50 million bill).
As someone who is not exactly the technologies biggest fan, I have not done a lot of research in the current landscape. Has anyone done this successfully?
In my humble opinion your best bet is getting some NVIDIA/AMD GPUs (ROCm versions required for AMD) with 24Gb VRAM or whatever open models you want to run locally. ollama.com+ollama are good for exploring the different models. Maybe running gemma4/qwen3.5 with hermes-agent or using AI routers to forward to best models for the task but tbh even with cloud models like kimi-k2.6:cloud and gemma4 i’m pretty disappointed. In my opinion it is like “Oh, neat” but not production ready especially when compared with large parameter, token heavy frontier models so I think everyone building their own Claude Code/Cursor may not be an entirely feasible solution. Not that i’ve built a cluster but that is what i’d check out.