run AI models locally

Hi, this topic is about running AI models locally on your own hardware. Whether it is a text-generation model or a music generator, everything is welcome. If any local AI enthiusiasts are in this group, they can talk about it here. I for myself have some text models like Gemma3-27b and dolphin. For music generation I have Ace-Step.

I should've gone with the RTX5060TI 16gb instead of the 12gb 4070. I could have run larger models like GPT OSS 20b, but I can only run up to Gemma 3 12b. I came from a 3060 with 12gb vram so not much of an upgrade. Oh well. I bought it before the GPU shortage so I guess I could sell for some profit and buy the 5060.

I have a 30 60 with 8GB Vram, aka nothing ueful. Considering to get a Mac Mini or something to at least run some lightwaight stuff, I think getting a good GPU would be a bigger deal at the moment. It's on my list though should the prices ever go down again, which, if ever, will take it's while for sure.

I also only have the 4060 with 8GB of VRAM, but I can run Gemma327b and GPT-oss20b easily. I just allocated some of my RAM to the GPU to get shared VRAM.

so glad I got a 50 70before all these prices went haywire

LOL Didn't you get yur pc just a month ago? The prices already were quite high then as far as I know.
Anyway, I have a 4070 super with 12 gb VRAM. Back then I used it primarly for RVC, a bit more recently for Ollama models such as GPT-OSS 20b and Quen 2.5-coder. However, my current take is that it's, while funny to play around with, not something you can get advantage of. The models are way to small to actually show they can be relied on and promt injection is real in case you decide to use them in combination with something like Openclaw. Hence I nowadays just would use the card for video games.

yes, but I did not build it myself, and the company still had stock from when prices were cheaper

Fair, makes sense.

Does anyone have a good guide on getting RVC working? I tried and just couldn't.

A guide not, but if you want you can DM me and I can explain to you all the things you don't understand. However, if you have no Nvidia GPU it will not work.

I would be interested in this as well, I would also like to get it working

This sort of thing should be able to work with whatever GPU model, whether it be Nvidia or AMD, as long as it is powerful enough.

I'm not sure if that is a clarification or a wish, but in case this is the former this is not true.

^Yeah, you can run RVC with AMD, but you can only train models with an NVIDIA GPU according to their documentation. Never got it working on my non-nvidia laptop.

ROCM is getting there, but sadly it's only for Linux on Pytorch at least of February 2026

It was kind of an assumption as things shouldn't depend on a certain brand of GPU, as long as its powerful enough.
-- (rudolf):
I'm not sure if that is a clarification or a wish, but in case this is the former this is not true.

--

I see. I don't know the exact reason either, but their structure still is different from each other so there might be a reason why.

Sadly, it's not about power, but instead it's about driver support. Back in th day, certain CPUs were better because they had certain instruction sets or better floating point units. This was like back in the 90s and 00s though. GPUs are somewhat the same for graphics processing, but for AI NVIDIA is much better because of Cuda. If it weren't for that, I would be running GPT OSS 20B on a 9060XT 16gb right now, but my lazy butt doesn't want to learn to take up a Linux distro as my daily driver.

Update on RVC: I was having so many issues because I was running from source when there was a compiled version right in front of me. Jokyboy129's comment about "a guide not" makes much more sense because I assumed everyone had to run from source lol!
I would like to get realtime voice cloning working, but the GUI isn't accessible. Has anyone figured any workarounds?

I would honestly really like to learn how to use it. Is there any guide somewhere? Because honestly I don't even know where to start