I usually don’t even understand the lingo they use. “Open-weighted” is the most recent one, then it usually goes down to specific “models” that everybody is supposed to know about.

These are my thoughts (I will stick to the vague “it” for now, but of course therein lies another question: “and how does all this apply to various specialised AIs”):

  • Is it really feasible to run it 100% locally? I know there’s plenty of people with very powerful rigs indeed, but still. Or are 99% of these people really saying “it would, in theory, be possible to run that locally, therefore your concerns are invalid”?
  • If yes to the previous: the software doesn’t come from nowhere and ultimately still relies on gas-turbine-powered datacenters and stolen IP and stolen personal data, no?

If what I wrote above is true, what exactly are people arguing when they say it’s still possible to use LLMs ethically or true to FOSS philosophy, because … ???

edit

Thanks to all who answered.

I guess it’s my fault for asking several questions in one, but this thread has attracted exactly the type of people I’m writing about; several even used the term “open-weighted models” without explaining it.

Asking to get arguments explained, I got more arguments instead.

  • zlatiah@lemmy.world
    6·
    6 hours ago

    I’ll go one by one… although I do have to say, I can’t answer the main question of whether it can be 100% ethical

    Open-weight model

    It’s actually quite literal. LLMs may look like magic, but under the hood are basically very, very big mathematical/statistical functions that transform one type of input (for example, “User: hey ChatGPT what is an open-weight model?”) to an output (example, “ChatGPT: To understand what an open model is…”)… To simplify, these models function by piecing together a ridiculous amount of tiny, nonlinear mathematical functions, apply a weight to each of these tiny functions, and somehow get a coherent result out of it. The theory behind this has been known in math research for decades before OpenAI.

    Because LLMs are mathematical models, if you know the weights of those tiny functions and how you chain theme together, you can run an exact replica of it. Open-weight models are just that, models where the entire weight and model architecture are released with open, permissive licenses. Many of these end up being modified by other users/hobbyists, and I do believe this sharing/modifying part of AI/LLM is quite true to FOSS philosophy.

    As others have mentioned though, “open-weight” has no bearing on how the model is trained. It just means that the final product is free and open

    Is it really feasible to run it 100% locally?

    Theoretically yes, practically yes but depends on the model.

    • For the really small models designed for mobile phones, absolutely. For example, LFM-2.5 and Ling-3.0-tiny run on just about any potato. Granted these models are… not great. They are probably functional as “search engine wrappers” though, methinks that’s what they were built for.
    • For the intermediate models, yes but with an asterisk. They require at least a decent GPU, so either you have a gaming rig, happened to have snagged one of the AI computers, or try to kneecap the model in some way (limit the amount it generates, make the mathematical operations less precise a.k.a. quantization, be okay with it running for a long time, …). I have a local setup for Qwen3.8-27B with 4-bit quantization, but it runs at 2 tokens/second on my laptop (which has a ton of RAM but not a very good GPU)…
    • For really large models, you can technically run them, but they are realistically designed for organizations with privacy in mind: hospitals, private companies that handle sensitive data, etc…

    The software doesn’t come from nowhere

    I mean IMO that’s the heart of the whole debate. Making a modern LLM requires massive amounts of data and computation… both of these do have to come from somewhere. And my understanding is that, current AI/LLM research didn’t create that much technological breakthrough, so building a better model often does indeed rely on either using more data or bigger architecture (which takes longer to compute)… which then leads to a truckload of ethics issues here and there

    For private companies, we know that OpenAI is blatantly hoarding data from just about everywhere, see the recent Mathematics Millennium Prize scandal. A lot of the open-weight models are trained by Chinese training houses, and Chinese tech companies are not known for respecting intellectual property… and China uses lots of coal energy still.

    Maybe Mistral is more ethical, given that 1) they are based in France and care about GDPR and data regulations a lot more, and 2) France uses lots of nuclear energy, but I can’t say for sure. An European company operating an AI/LLM training house from Iceland using their excess geothermal energy is probably the most ethical we can get on this

    In theory (and take this with a whole jug of salt) someone can run an off-grid farm and train a small LLM all on their own, which would be very ethical… but I doubt any serious organizations are doing it this way. Most ppl just want better performance and/or more optimized models for now

    For a complete FOSS purist with an environmentally conscious angle, I don’t know if there is an AI/LLM that can be considered 100% ethical… but then again Gentoo is a thing. So yeah, I don’t know