I’m looking for experiences and opinions on kubernetes storage.

I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.

I’ve looked at linstore or seaweedfs/garage with juicefs but I’m not sure how well the performance of those options is across the internet and how well they last in long term operation. Is anyone else hosting k3s across the internet in their homelab?

Edit: fixed wording

  • notfromhere@lemmy.mlEnglish
    3·
    6 months ago

    I know Ceph would work for this use case, but it’s not a lighthearted choice, kind of an investment and a steep learning curve (at least it was, and still is, for me).

    • InnerScientist@lemmy.worldOPEnglish
      1·
      6 months ago

      I heard that ceph lives and dies with the network hardware. Is a slow internet connection even usable when the docs want 10 gbit/s networking between nodes?

      • notfromhere@lemmy.mlEnglish
        2·
        6 months ago

        I’m really not sure. I’ve heard of people using Ceph across datacenters. Presumably that’s with a fast-ish connection, and it’s like joining separate clusters, so you’d likely need local ceph cluster at each site then replicate between datacenters. Probably not what you’re looking for.

        I’ve heard good things about Garbage S3 and that it’s usable across the internet on slow-ish connections. Combined with JuiceFS is what I was looking at using before I landed on Ceph.

    • InnerScientist@lemmy.worldOPEnglish
      1·
      6 months ago

      I mean storage backends as in the provisioner, I will use local storage on the nodes with either lvm or just storage on a filesystem.

      I already set up a cluster and tried linstore, I’m searching for experiences with the options because I don’t want to test them all.

      I currently manage all the servers with a NixOS repository but am looking for better failover.