• 1 Post
  • 33 Comments
Joined 3 years ago
cake
Cake day: June 30th, 2023

help-circle

  • all my back ups are what they should be

    Are you sure? While the cloud backups may not affect you the exclusions might, afaict no one even knows what exactly is excluded.

    From the link:

    This annoyed me. Firstly I needed that folder and Backblaze had let me down. Secondly within the Backblaze preferences I could find no way to re-enable this. In fact looking at the list of exclusions I could find no mention of .git whatsoever.

    Which strongly implies that there might be other important folders that aren’t backed up. (Without .git inside a git folder it is no longer a git repository)

    I don’t use backblaze but from the outside it looks like they’re cutting costs by worsening the backups to reduce storage usage.





  • Here:

    server {
        listen 443 quic;
        listen [::]:443 quic;
        listen 443 ssl;
        listen [::]:443 ssl;
        server_name jellyfin.kitsuna.net;
        http2      on;
        http3      on;
        quic_gso   on;
        tcp_nodelay on;
        # You can increase the limit if your need to.
        error_log /var/log/nginx/jellyfin.access.log;
    #    ssl on;
    #    ssl_certificate /etc/nginx/certificate.crt;
    #    ssl_certificate_key /etc/nginx/certificate.key;
    #    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # don’t use SSLv3 ref: POODLE
            ssl_certificate /etc/letsencrypt/live/kitsuna.net/fullchain.pem;
            ssl_certificate_key /etc/letsencrypt/live/kitsuna.net/privkey.pem;
    #        ssl_certificate_key /etc/letsencrypt/live/kitsuna.net/privkey.pem;
            ssl_protocols TLSv1.2 TLSv1.3;
            add_header Alt-Svc 'h3=":$server_port"; ma=86400';
            add_header x-quic 'h3';
            add_header Alt-Svc 'h3-29=":$server_port"';
        location / {
            proxy_pass http://10.159.4.12:8096/;
    #       proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forward-Proto http;
            proxy_set_header X-Nginx-Proxy true;
        }
    }
    


  • It is expected, the users inside the container are “real” users. They just get offset inside the container and some mapping is applied:

    Root inside the container is mapped outside to the user running the container, everything that has the owner “root” inside the container can be read from outside the container as your user.

    Everything that is saved as non-root inside the container gets mapped to the first subuid in /etc/subuids for your user + the uid inside the container.

    You can change this mapping such that, for example, user 1000 inside the container gets mapped to your user outside the container.

    An example:

    You have a postgres database inside a container with a volume for the database files. The postgres process inside the container doesn’t run as root but instead runs as uid 100 as such it also saves its files with that user.
    If you look at the volume outside the container you will get a permission denied error because it is owned by user 100100 (subuids starts at 100000 and usid inside container is 100).

    To fix: Either run your inner processes as root, this can often be done using environment variables and has almost no security impact or add --userns keep-id:uid=100,gid=100 to the cmdline to make uid 100 inside the container map to your user instead of root (this creates a new image automatically and takes a while on the first run)


  • The problem seems to be that when you run restic with systemctl start, it passen on your user environment including the information of where your rclone.conf lies. When the systemd service runs on its own it doesn’t have this. You need to either tell restic the path to your rclone config or set the home environment such that the systemd service checks the right location.




  • I mean storage backends as in the provisioner, I will use local storage on the nodes with either lvm or just storage on a filesystem.

    I already set up a cluster and tried linstore, I’m searching for experiences with the options because I don’t want to test them all.

    I currently manage all the servers with a NixOS repository but am looking for better failover.





  • I have rss feeds for my main service updates so I know what new features I have, the services mostly run in podman containers and update automatically each Monday. I also have daily backups (timed to run just before the update on monday) in case anything does break.

    If it breaks I fix it depending on how much I want/need it, mostly it’s a matter of half an hour to fix it and with my current NixOS/Podman system I haven’t yet needed to fix anything this year so it breaks infrequently.

    Also why are you using Kubernetes on a single host if you want minimal maintenance? XD

    My recommendation is to switch to just managing containers, you should just be able to export the volumes out of kubernetes and import them as normal volumes, as long as they’re mounted in the right place you keep your data and if it doesn’t work just try again. Not like you need to destroy the current system to slowly replace it.

    Edit: I also recommend to update and reboot frequently, this stops updates and unstable configurations from piling up.




  • Yeah it works great and is very secure but every time I create a new service it’s a lot of copy paste boilerplate, maybe I’ll put most of that into a nix function at some point but until then here’s an example n8n config, as loaded from the main nixos file.

    I wrote this last night for testing purposes and just added comments, the config works but n8n uses sqlite and probably needs some other stuff that I hadn’t had a chance to use yet so keep that in mind.
    Podman support in home-manager is also really new and doesn’t support pods (multiple containers, one loopback) and some other stuff yet, most of it can be compensated with the extraarguments but before this existed I used pure file definitions to write quadlet/systemd configs which was even more boilerplate but also mostly copypasta.

    Gaze into the boilerplate
    { config, pkgs, lib, ... }:
    
    {
        users.users.n8n = {
            # calculate sub{u,g}id using uid
            subUidRanges = [{
                startUid = 100000+65536*( config.users.users.n8n.uid - 999);
                count = 65536;
            }];
            subGidRanges = [{
                startGid = 100000+65536*( config.users.users.n8n.uid - 999);
                count = 65536;
            }];
            isNormalUser = true;
            linger = true; # start user services on system start, fist time start after `nixos-switch` still has to be done manually for some reason though
            openssh.authorizedKeys.keys = config.users.users.root.openssh.authorizedKeys.keys; # allows the ssh keys that can login as root to login as this user too
        };
        home-manager.users.n8n = { pkgs, ... }:
        let
            dir = config.users.users.n8n.home;
            data-dir = "${dir}/${config.users.users.n8n.name}-data"; # defines the path "/home/n8n/n8n-data" using evaluated home paths, could probably remove a lot of redundant n8n definitions....
        in
        {
            home.stateVersion = "24.11";
            systemd.user.tmpfiles.rules =
            let
                folders = [
                    "${data-dir}"
                    #"${data-dir}/data-volume-name-one" 
                ];
                formated_folders = map (folder: "d ${folder} - - - -") folders; # a function that takes a path string and formats it for systemd tmpfiles such that they get created as folders
            in formated_folders;
    
            services.podman = {
                enable = true;
                containers = {
                    n8n-app = { # define a container, service name is "podman-n8n-app.service" in case you need to make multiple containers depend and run after each other
                        image = "docker.n8n.io/n8nio/n8n";
                        ports = [
                            "${config.local.users.users.n8n.listenIp}:${toString config.local.users.users.n8n.listenPort}:5678" # I'm using a self defined option to keep track of all ports and uids in a seperate file, these values just map to "127.0.0.1:30023:5678", a caddy does a reverse proxy there with the same option as the port.
                        ];
                        volumes = [
                            "${data-dir}:/home/node/.n8n" # the folder we created above
                        ];
                        userNS = "keep-id:uid=1000,gid=1000"; # n8n stores files as non-root inside the container so they end up as some high uid outside and the user which runs these containers can't read it because of that. This maps the user 1000 inside the container to the uid of the user that's running podman. Takes a lot of time to generate the podman image for a first run though so make sure systemd doesn't time out
                        environment = {
                            # MYHORSE = "amazing";
                        };
                        # there's also an environmentfile option for secret management, which works with sops if you set the owner of the secret/secret template
                        extraPodmanArgs = [
                            "--pull=newer" # always pull newer images when starting, I could make this declaritive but I haven't found a good way to automagically update the container hashes in my nix config at the push of a button.
                        ];
                     # few more options exist that I didn't need here
                    };
                };
            };
        };
    }
    
    

  • I use podman using home-manager configs, I could run the services natively but currently I have a user for each service that runs the podman containers. This way each service is securely isolated from each other and the rest of the system. Maybe if/when NixOS supports good selinux rules I’ll switch back to running it native.