• 2 Posts
  • 15 Comments
Joined 3 years ago
cake
Cake day: July 30th, 2023

help-circle

  • Gotta preach for the cult of ZFS. It’s check summing, copy on write, and zraid features are all exactly what you want for data resilience. Plus you get transparent compression, and snapshots that can provide a bit of a stop-gap for your lack of backups.

    It will normally soak up any and all memory for buffers and caches, but is meant to quickly free up when it’s needed by an app. Linux already does this on any filesystem with its page cache.

    Oh and mounting a ZFS dataset on a new machine is super quick and easy, it stores it’s config on the drives themselves, so you can plug them into a new box and zpool import -af and boom it’s mounted and ready to go.



  • I used a very similar method in a similar situation to albb0920. They describe it as vibe coding too.

    The exact chip that handles everything is undocumented, but similar ones in the same series have datasheets. A maintained version of the linux driver handily collated all of the available datasheets and configurations used by different motherboards. Between that and my microcontroller/hardware experience, that side of things wasn’t too bad.

    What I didn’t know anything about was writing an Illumos driver. I used the chatbot with a free claude account, compiling and running the code manually myself. I was impressed that it was able to build out the boilerplate and get something going at all. Course it took a few tried to get something that compiled and worked somewhat correctly. At some points I needed to look through the generated code and point out exactly what what wrong, but at least it would address it.

    Code running in the context of the kernel is definitely not something I would have autonomously executing by a LLM. The end result is absolutely not something I would want put into the official Illumos source.




  • Ah kay, definitely not a RAM size problem then.

    iostat -x 5 Will print out per drive stats every 5 seconds. The first output is an average since boot. Check all of the drives have similar values while performing a write. Might be one drive is having problems and slows everything down, hopefully unlikely if they are brand new drives.

    zpool iostat -w Will print out a latency histogram. Check if any have a lot above 1s and if it’s in the disk or sync queues. Here’s mine with 4 HDDs in z1 working fairly happily for comparison:

    Here's mine with 4 HDDs  in z1 working fairly happily for comparison

    The init_on_alloc=0 kernel flag I mentioned below might still be worth trying.



  • After some googling:

    Some Linux distributions (at least Debian, Ubuntu) enable init_on_alloc option as security precaution by default. This option can help to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

    Unfortunately, it can lower ARC throughput considerably (see bug).

    If you’re ready to cope with these security risks 6, you may disable it by setting init_on_alloc=0 in the GRUB kernel boot parameters.

    I think it’s set to 1 on Raspberry Pi OS, you set it in /boot/cmdline.txtI think.

    Exhaustive ZFS performance tuning guide


  • sync=disabled will make ZFS write to disk every 5 seconds instead of when software demands it, which maybe explains your LED behavior.

    Jeff Geerling found that writes with Z1 was 74 MB/sec using the Radxa Penta SATA HAT with SSDs. Any HDD should be that fast, the SATA hat is likely the bottleneck.

    Are you performing writes locally, or over smb?

    Can try iostat or zpool iostat to monitor drive writes and latencies, might give a clue.

    How much RAM does the Pi 5 have?


  • I’ve been using pcloud. They do one time upfront payments for ‘lifetime’ cloud storage. Catch a sale and it’s ~$160/TB. For something long term like backups it seems unbeatable. To the point I sort of don’t expect them to actually last forever, but if they last 2-3 years it’s a decent deal still.

    Use rclone to upload my files, honestly not ideal though since it’s meant for file synchronisation not backups. Also they are dog slow. Downloading my 4TBs takes ~10 days.





  • I don’t get the term ‘technical debt’. Most people seem to use it to say “We took shortcuts previously, so now we need to go back and do things properly”.

    FIrst, it’s a bad metaphor. You take on debt to invest in long term things that will provide future benefits. Telling the bean counters that you need to stop working on useful features to ‘pay back technical debt’ is not making things clearer to them.

    Second, you write software, what the heck are you talking about? Compare to civil engineering. If an area gets busier and the existing narrow wood bridge is no longer suitible, engineers don’t say “Wow what idiots built this road with no eye to future growth?” It was built with the needs and resources of the time. To improve it, the bridge needs to be closed, demolished, and rebuilt with planning, labour and materials.

    Instead software is empherial. You don’t need to demolish what’s there. No need to build temporary alternative infrastructure. No need for new materials and disposal of the old. It’s just planning and labour to redo a piece of software. It always seems so whiny when people complain about technical debt, as if switching to a different build system is anywhere close to the difficulty of fixing real life; replacing lead pipes with copper for an entire city, or removing asbestos from buildings.



  • I once had a coworker that used a mini PC instead of a laptop for work. Being lighter, and more powerful worked well for him.

    My understanding is that each brand will contract out the design and manufacturing, with potentially totally different factories and people involved. So the same brand will have models that are built totally differently, in terms of quality and ethics. This happens from the cheapest no name products to big western brands. Make sure to check reviews for the specific model, and not blanket trust one brand to be ‘good’. Notebook check and robtech on youtube do mini PC reviews.

    I feel you on the ethics side, and unfortunately it’s pretty difficult to ever avoid. You can try buying second hand, at least save something from landfill and get a bargain.


  • IPv6 isn’t just larger addresses, it was meant to totally remove the need for layer 2 / MAC addresses, bus networks, DHCP, and broadcasts. Since the plan was to get rid of the 12 byte ethernet header, the 24 byte increase in IP addresses would only be a 12 byte increase in header at the end of the day. WiFi wouldn’t need three MAC addresses in every packet. IPv6 only achieves it’s true potential with a complete switch over.

    I personally don’t think that can ever happen. The opportunity to switch everyone over is absolutely long gone. IPv6 isn’t an extension of v4 or a compatible replacement, like ASCII to UTF-8. It’s more like X to Wayland. The protocol authors went “This is a mess we gotta rethink this from scratch”. But there’s so much already relying on the old protocol, and replacing it with something that doesn’t perfectly match features is difficult for little reward for users.

    The increase in IPv6 nodes has mostly been due to mobile networks. The tragedy is they actually still mostly use layer 2 and bridge networking. IPv4 nor v6 can handle maintaining connections while addresses change. So they set it up so that you keep the same IP address as you travel and move between different towers. This is done with massive virtual layer 2 LANs across towers, with the IP routing happening at a central datacentre. IPv6 is simply used for the larger addresses, and none of the network/protocol simplifications it promised can be used.