From c69bfa64e80658303d1fc6d6e28c563749182f77 Mon Sep 17 00:00:00 2001 From: Nick Dumas Date: Thu, 2 Feb 2023 13:43:27 -0500 Subject: [PATCH] New post --- .../gitea-lfs-and-syncing-obsidian-vaults.md | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 content/gitea-lfs-and-syncing-obsidian-vaults.md diff --git a/content/gitea-lfs-and-syncing-obsidian-vaults.md b/content/gitea-lfs-and-syncing-obsidian-vaults.md new file mode 100644 index 0000000..72a3115 --- /dev/null +++ b/content/gitea-lfs-and-syncing-obsidian-vaults.md @@ -0,0 +1,130 @@ ++++ +draft = false +title = "Gitea, git-lfs, and syncing Obsidian Vaults" +date = "2023-01-05" +author = "Nick Dumas" +authorTwitter = "" +cover = "" +tags = ["obsidian", "git", "gitea"] +keywords = ["obsidian", "git", "gitea"] +description = "A brief overview of how I stood up a gitea instance for the purpose of backing up and syncing my Obsidian vault." +showFullContent = false ++++ +# Outline +## What am I Doing? +I take notes on a broad spectrum of topics ranging from tabletop roleplaying games to recipes to the last wishes of my loved ones. Because of how valuable these notes are, I need to accomplish two things: +1) Back up my notes so that no single catastrophe can wipe them out +2) Make my notes accessible on multiple devices like my phone and various work laptops + +For writing and organizing my notes, I use an application called [Obsidian](https://obsidian.md), an Electron Markdown reader and editor with an emphasis on plaintext, local-only files to represent your notes. This has a lot of interesting implications which are well beyond the scope of this post, but this is the one that's germane: your notes are a textbook use-case for version control. + +Markdown files are plain-text, human-readable content that every modern Version Control System is supremely optimized for handling. In this arena, there's a lot of options ( mercurial, bzr, git, svn, fossil, and more ) but I'm partial to git. + +## Life with git +```bash +nick@DESKTOP-D6H8V4O MINGW64 ~/Desktop/general-notes (main) +$ git log $(!!) +git log $(git rev-list --max-parents=0 HEAD) +commit 18de1f967d7d9c667ec42f0cb41ede868d6bdd31 +Author: unknown <> +Date: Tue May 31 09:44:49 2022 -0400 + + adding gitignore +``` +I've kept my vault under git for all but the first 2 months of my vault's lifetime and I cannot count the number of times it's saved me from a mistake or a bug. + +A few times a day, I'll commit changes to my notes, plugins, or snippets and push them up. This is a manual process, but by reviewing all my changes as they're committed I kill a few birds with one stone: +1) I get a crude form of spaced repetition by forcing myself to review notes as they change +2) I verify that templates and other code/plugins are working correctly and if they aren't, I can revert to a known-good copy trivially +3) reorganizations become much easier ( see point 2, reverting to known-good copies ) + +For convenience, I chose to start off with Github as my provider. I set up a private repository because my notes contain sensitive information of various flavors and had no problems with it, except for attachments. This works great, Github is a fast reliable provider and meets all the requirements I laid out above. +## The catch +There is no free lunch. On Github, free repositories have restrictions: +1) github will warn you if you commit files larger than 50mb and ask you to consider removing them or using git-lfs +2) github will not permit any files larger than 100mb to be committed +3) You're allowed a limited number of private repositories, depending on the type and tier of your account. + +My vault does not exclusively consist of plaintext files, though; there's PDFs, PNGs, PSDs, and more hanging out, taking up space and refusing to diff efficiently. I've got a lot of PDFs of TTRPG content, screenshots of important parts of software I care about for work or my personal life, and a lot of backup copies of configuration files. + +In theory, this is sustainable. None of my attachments currently exceed 100mb, the median size is well under 1mb. +```bash +$ pwd +~/evac/obsidian-vaults/bonk/Resources/attachments +$ ls -lah|awk '{print $5}'|sort -hr|head -n5 +62M +36M +8.4M +3.1M +2.9M +``` + +I'm not satisfied with theoretical sustainability, though. For something this important and sensitive, I'd like to have total confidence that my system will work as expected for the foreseeable future. + +## What are the options? +1) Github has its own [lfs service](https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-git-large-file-storage) with the free tier capped at 2gb of storage. +2) Pay for a higher tier of Github's LFS +3) Managed Gitlab (or similar) instance +4) Host my own + +Options 1 and 2 are the lowest effort solution and rely the most on third parties. I've opted not to go with this because Github may change its private repository or git-lfs policies at any time. + +Option 3 is better; a managed git hosting service splits the difference nicely. Using Gitlab would give me built-in CI/CD. + +I've opted out of this mostly for price and partly because I know for a fact that I can implement option 4. + +## Option 4 +I chose to use what I'm already familiar with: [Gitea](https://gitea.io/en-us/). Gitea is a fork of Gogs, a hosted git service written in Go. It's lightweight and its simplest implementation runs off an sqlite database so I don't even need a PostgreSQL service running. + +I've been using gogs and gitea for years and they've been extremely reliable and performant. It also integrates tightly with [Drone](https://www.drone.io/), a CI/CD system which will help me automate my blog, publish my notes, and more I haven't had the energy to plan. + +## docker-compose and gitea +For my first implementation, I'm going to host gitea using docker-compose. This will give me a simple, reproducible setup that I can move between providers if necessary. + +Hosting will be done on my DigitalOcean droplet running a comically old version of Fedora for now. This droplet is really old and up until now I've had very poor reproducibility on my setups. I'm working on fixing that with [caddy](/posts/automating-caddy-on-my-droplet), and using gitea for code management is next. + +Below you'll see the `docker-compose.yaml` for my gitea instance. This is ripped directly from the gitea documentation so there's very little to comment on. The `ports` field is arbitrary and needs to be adjusted based on your hosting situation. +```yaml +version: "3" + +networks: + gitea: + external: false + +services: + server: + image: gitea/gitea:1.18.0 + container_name: gitea + environment: + - USER_UID=1000 + - USER_GID=1000 + restart: always + networks: + - gitea + volumes: + - ./gitea:/data + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + ports: + - "3069:3000" + - "222:22" +``` + +Starting it up is similarly uninteresting; using detached mode for "production" work because I'm not super interested in watching all the logs. If something breaks, I can start it back up again without detaching and see whatever error output is getting kicked up. +```bash +$ docker-compose up -d +Starting gitea ... done +$ +``` + +Once this is done, you've got a gitea instance waiting to be configured with an admin user and a few other bootstrap settings. Navigate to the URL you chose for your gitea instance while following the docs and you're ready to create a repository for your vault. + +The web UI will guide you from there. + +## Success Story??? + +This solution is only a week or two old so it has not be put under a lot of load yet, but gitea has a good reputation and supports a lot of very high profile projects, and DigitalOcean has been an extremely reliable provider for years. + +Migrating my attachments into git-lfs was trivial, but it did rewrite every commit which is something to be mindful of if you're collaborating between people or devices. + +I don't intend to get more aggressive with adding large media attachments to my vault, I *prefer* plaintext when it's an option. Backing up my notes was only one item on a list of reasons I stood gitea up, in the coming weeks I'm going to work on using Drone to automate blog posts and use that as a springboard into more automation.