Why is a post about a CAD program being downvoted so much?

Flying Snake

TIL.

These are gorgeous photos, by the way. I wonder what camera/lens they were shot with?

Partially configured some parts via LLM but please don’t crucify me for that.

Slap in a spare GPU, and self-host one!

The 30B-class models are unbelievably good now, for being so small. They’re kinda where Claude was like a year ago, if not less. And (with the right backend) they aren’t expensive to host.

It’s not! Use SonoBus; it’s dead simple, and superior to Discord. It’s far lower latency, with customizable filters, peer-to-peer; and totally free.

Now if you want emojis and video and rambling channels and stuff, you will have to go elsewhere.

• For LLM hosting, ik_llama.cpp. You can really gigantic models at acceptable speeds with its hybrid CPU/GPU focus, at higher quality/speed than mainline llama.cpp, and it has several built in UIs.

• LanguageTool, for self run grammar/spelling/style checking.

Keep an eye on Open Web UI. I’ve heard rumblings that it’s starting to enshittify.

To be clear, VMs absolutely have overhead but Docker/Podman is the question. It might be negligible.

And this is a particularly weird scenario (since prompt processing literally has to shuffle ~112GB over the PCIe bus for each batch). Most GPGPU apps aren’t so sensitive to transfer speed/latency.

In my case it’s performance and sheer RAM need.

GLM 4.5 needs like 112GB RAM and absolutely every megabyte of VRAM from the GPU, at least without the quantization getting too compressed to use. I’m already swapping a tiny bit and simply cannot afford the overhead.

I think containers may slow down CPU<->GPU transfers slightly, but don’t quote me on that.

I dunno about Linux, but on Windows I used to use something called K10stat to manually undervolt cores with no access to such via the BIOS. The difference was night and day dramatic, as they idled ridiculously fast and AMD left a ton of voltage headroom back then.

I bet there’s some Linux software to do it. Look up if anyone used voltage control software for desktop Phenom IIs and such.

5557U

Ah, no eDRAM on that one I think, though I’m not 100% sure.

Still, with 16GB its plenty for a server as others said.

So some 2015 MacBooks came with Broadwell-eDRAM chips, great CPUs with (IIRC) a 64MB cache chip the CPU or GPU could use.

https://chipsandcheese.com/p/broadwells-edram-vcache-before-vcache

If that’s what yours has, it’s a gem, and punches way above its age/clockspeed for the power draw. Keep it!

Jellyfin might be able to use its iGPU (and not just the hardware blocks) for some offloading as well, though I’m not sure of Broadwells Vulkan support or what exactly Jellyfin uses it for.

What @mierdabird@lemmy.dbzer0.com said, but the adapters arent cheap. You’re going to end up spending more than the 1060 is worth.

A used desktop to slap it in, that you turn on as needed, might make sense? Doubly so if you can find one with an RTX 3060, which would open up 32B models with TabbyAPI instead of ollama. Some configure them to wake on LAN and boot an LLM server.

You can still use the IGP, which might be faster in some cases.

Oh actually that’s a great card for LLM serving!

Use the llama.cpp server from source, it has better support for Pascal cards than anything else:

https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md

Gemma 3 is a hair too big (like 17-18GB), so I’d start with InternVL 14B Q5K XL: https://huggingface.co/unsloth/InternVL3-14B-Instruct-GGUF

Or Mixtral 24B IQ4_XS for more ‘text’ intelligence than vision: https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF

I’m a bit ‘behind’ on the vision model scene, so I can look around more if they don’t feel sufficient, or walk you through setting up the llama.cpp server, but basically it provides an endpoint which you can hit with the same API as ChatGPT.

OK.

Then LM Studio. With Qwen3 30B IQ4_XS, low temperature MinP sampling.

That’s what I’m trying to say though, there is no one click solution, that’s kind of a lie. LLMs work a bajillion times better with just a little personal configuration. They are not magic boxes, they are specialized tools.

Random example: on a Mac? Grab an MLX distillation, it’ll be way faster and better.

Nvidia gaming PC? TabbyAPI with an exl3. Small GPU laptop? ik_llama.cpp APU? Lemonade. Raspberry Pi? That’s important to know!

What do you ask it to do? Set timers? Look at pictures? Cooking recipes? Search the web? Look at documents? Do you need stuff faster or accurate?

This is one reason why ollama is so suboptimal, with the other being just bad defaults (Q4_0 quants, 2048 context, no imatrix or anything outside GGUF, bad sampling last I checked, chat template errors, bugs with certain models, I can go on). A lot of people just try “ollama run” I guess, then assume local LLMs are bad when it doesn’t work right.

Totally depends on your hardware, and what you tend to ask it. What are you running? What do you use it for? Do you prefer speed over accuracy?

TBH you should fold this into localllama? Or open source AI?

I have very mixed (mostly bad) feelings on ollama. In a nutshell, they’re kinda Twitter attention grabbers that give zero credit/contribution to the underlying framework (llama.cpp). And that’s just the tip of the iceberg, they’ve made lots of controversial moves, and it seems like they’re headed for commercial enshittification.

They’re… slimy.

They like to pretend they’re the only way to run local LLMs and blot out any other discussion, which is why I feel kinda bad about a dedicated ollama community.

It’s also a highly suboptimal way for most people to run LLMs, especially if you’re willing to tweak.

I would always recommend Kobold.cpp, tabbyAPI, ik_llama.cpp, Aphrodite, LM Studio, the llama.cpp server, sglang, the AMD lemonade server, any number of backends over them. Literally anything but ollama.

…TL;DR I don’t the the idea of focusing on ollama at the expense of other backends. Running LLMs locally should be the community, not ollama specifically.

Yeah I mean it really depends on the app.

If it just occasionally peaks, maybe modest zram and low swappiness is all you need.

But sometimes i crack it up (and basically disable zram) because I’m paging like 4x my RAM pool size onto ssds with big files.

Honestly, most LLMs suck at the full 128K. Look up benchmarks like RULER.

In my personal tests over API, LLama 70B is bad out there. Qwen (and any fine tune based on Qwen Instruct, with maybe an exception or two) not only sucks, but is impractical past 32K once its internal rope scaling kicks in. Even GPT-4 is bad out there, with Gemini and some other very large models being the only usable ones I found.

So, ask yourself… Do you really need 128K? Because 32K-64K is a boatload of code with modern tokenizers, and that is perfectly doable on a single 24G GPU like a 3090 or 7900 XTX, and that’s where models actually perform well.

Late to this post, but shoot for and AMD Strix Halo or Nvidia Digits mini PC.

Prompt processing is just too slow on Apple, and the Nvidia/AMD backends are so much faster with long context.

Otherwise, your only sane option for 128K context in a server with a bunch of big GPUs.

Also… what model are you trying to use? You can fit Qwen coder 32B with like 70K context on a single 3090, but honestly its not good above 32K tokens anyway.