Watched this documentary about Ulillillia, the internet figure. It follows him and the small town of Minot, ND in the aftermath of the 2011 Souris River Flood. Despite it being from 2012, it was released at the end of 2019.

While the film moved at a slower pace, I found some of Ulillillia’s quotes from it interesting. In the documentary, Ulillillia lives with his parents:

There’s very little I can do, so I don’t get much enjoyment … I definitely wish I could travel more, or see the world … it’s like I can’t get out of the cage

According to this wiki page, he has since gotten a driver’s license and moved out of his parent’s home to Florida.

I own a Nintendo Switch console. Because I don’t want physical games cluttering my home, I get all my games digitally. I recently started running out of room and needed a new SD card to hold my games. This path ended up being circuitous, taking me through fake flash storage, the Switch’s file structure, and its operating system’s treatment of Unix file attributes.

A new card

I purchased a Samsung 512GB SD card from Amazon for $65. I’m shocked at how cheap SD cards are now. This replaced my old 128GB card.

Here’s the old card:

And here’s the new card:

Flash memory has never been more popular. Unfortunately, there has been a recent rise of fake flash memory. These are USB keys, SD cards, and solid state disks that masquerade as a larger capacity, but will lose your data or fail to read / write properly.

After my new card arrived, I wanted to check its authenticity. Is this a real 512GB SD card? Or a fake?

The f3 tool

f3 is a tool to verify the authenticity of flash memory. It stands for “Fight Flash Fraud” or “Fight Fake Flash”. From the site:

I started this project when I bought a 32GB microSDHC card for my Android phone back in 2010, and found out that this card always fails when one fills it up.

I installed it on my Mac with brew install f3. This installed two binaries: f3write and f3read.

f3write writes contiguous 1GB files to your card until it has exhausted its space. It will also report on the write speed for this operation. You use it by running it on the path you want to write to. It will report progress when running:

$ f3write /Volumes/test
F3 write 7.2
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

Free space: 476.69 GB
Creating file 1.h2w ... OK!                           
Creating file 2.h2w ... OK!                           
...
Creating file 476.h2w ... OK!                        
Creating file 477.h2w ... OK!                       
Free space: 1.38 MB
Average writing speed: 77.17 MB/s

My card got 77MB/s, which comes out to about 2 hours of write time (this is about how long it felt). After doing the write step, you can run f3read to verify that the files made it there successfully:

$ f3read /Volumes/test
F3 read 7.2
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
...
Validating file 476.h2w ... 2097152/        0/      0/      0
Validating file 477.h2w ... 1432983/        0/      0/      0

  Data OK: 476.68 GB (999677335 sectors)
Data LOST: 0.00 Byte (0 sectors)
           Corrupted: 0.00 Byte (0 sectors)
    Slightly changed: 0.00 Byte (0 sectors)
         Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 71.12 MB/s

This will read each of the 1GB files created by f3write, verifying they did not experience any data loss. This will also report on your card’s read speed. Mine was 71MB/s, which also felt like around 2 hours.

Cool, a real SD card that works! Next step: move my games over.

Switch SD file structure

Plugging my old SD card into my machine, diskutil reports an NTFS filesystem:

$ diskutil list
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *127.9 GB   disk3
   1:               Windows_NTFS                         127.8 GB   disk3s1

The hierarchy on the card (named “Untitled”) has a single “Nintendo” folder on it. What’s in there?

$ tree Nintendo
Nintendo/
├── Album
│   ├── 2017
│   ├── 2018
│   └── 2019
├── Contents
│   ├── placehld
│   ├── private
│   ├── private1
│   └── registered
├── save
└── saveMeta

829 directories, 688 files

(heavily truncated)

So, it looks like: - Album, which includes all my screenshots and video recordings arranged by year - Content, which has some subdirectories: - placehld, which has some empty directories - likely placeholders given the filename - private, a 16B file which looks like it contains a key - private1, a 32B file which looks like it contains a different key - registered, which appears to contain all the games

I’ve used the majority of the card’s 128GB space:

$ du -skh /Volumes/Untitled
111G    /Volumes/Untitled

Formatting the card

Next step was to insert the 512GB card into the Switch and ask it to format. This process took about 3 seconds, then the console rebooted itself. I verified that the card was now NTFS:

$ diskutil list
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *512.1 GB   disk3
   1:               Windows_NTFS                         512.0 GB   disk3s1

Backing up the card

I don’t have two SD card slots on my machine, so I’ll need to copy the data first to my computer, then onto the card. We can use rsync to do this:

$ mkdir nintendo_backup
$ rsync -avz /Volumes/Untitled nintendo_backup/

I used rsync in archive mode by passing the -a flag, which according to man rsync will:

…ensure that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved…

Perfect! I also passed -v for verbose, and -z to use compression as the files are moved.

Restoring to the new card

Now it’s time to restore to the new card. After unmounting the smaller card and inserting the new card, it’s a simple matter of rsyncing the files back:

rsync -avz nintendo_backup/Untitled/ /Volumes/Untitled

Trying it out

With the rsync step done, we can test out the card:

It doesn’t work! The Switch responds with error 2016-0247 and is unable to access the card

I then tried copying the contents of nintendo_backup over the rsync‘d backup files with Finder. Perhaps the rsync step missed something? Or my machine fell asleep?

Nope, same error. I tried the original card to verify my Switch was still functional. The original card works fine.

Debugging the issue

So, we’ve got a new SD card that the Switch won’t see:

• it was formatted by the Switch’s software, so any secret partitions or files should be there
• it contains an rsync archive copy of the games and media from the original card
• the Switch read the card successfully when formatting it, but hasn’t since the restore succeeded

The Nintendo support site seems to suggest similar steps to those I followed:

• Open Windows Explorer (for PC) and access the microSD card.
• Highlight the data and drag it to the desktop.
• If you are moving your content to a new microSD card, be sure to first format the new card per our recommendations.
• Once the new card has been formatted, continue with the instructions below.
• Eject the first microSD card, then insert the second microSD card into the slot or reader/writer.
• Using Windows Explorer access the microSD card again.
• Drag the data from the desktop to the new microSD card, then insert the new microSD card into the Nintendo Switch console.

Although it does contain this strange warning:

Important: This process may not be able to copy the microSD card contents correctly in environments other than Windows (such as Mac).

A fix

Turns out I was missing the archive attribute on the card. Simply running sudo chflags -R arch /Volumes/Untitled to fix up the Archive attribute fixed the issue (I also did a sudo dot_clean -mn on the directory). Phew! Look at all that space:

Why did this break?

I’m not totally sure. My hunch is that an archive attribute on the files was confusing the Switch’s SD card parsing. It looks like the Nintendo Switch’s OS is not Unix-derived, but rather an evolution of the 3DS OS. It’s possible that some file attributes from Unix may have confused the Switch.

This is the first in a series of posts I hope to write about building tools for “Peterometer”, a way to visualize stats I’ve collected about myself.

Inspiration for Peterometer

I’ve long been inspired by people who build beautiful visualizations of their gathered metrics. For example, Nicholas Felton’s annual reports. Here is an example from his 2007 report:

I think this format of visualization is really cool. It’s easy to glance at while being information-dense.

Another cool example is Anand Sharma’s April Zero, which is now an app called Gyroscope:

He has some great posts about his creative process here and here. The Iron Man-style HUD influence shines through in the finished product.

These visualizations are really cool ways to show stats gathered about your life. I’ve recently been getting more into tracking my day-to-day life, and experimenting with visualization techniques to showcase this data.

Hydration Tracking

Since December 2018, I’ve been tracking my hydration every day. Every time I finish drinking something, I log the type of drink it was, and how much of it I drank. I log the data using the WaterMinder app on my watch and phone. WaterMinder lets you have saved drinks, which is really helpful if you drink the same thing often (my Nalgene, a cup of coffee from the machine at work, etc.)

Some Visualizations

WaterMinder lets you export your data in CSV format. With a little bit of Python, we can parse this into a “stream” stacked area chart to show how I hydrate myself:

This chart is a bit tough to read due to data density. The legend is sorted in descending order of consumption. By the numbers, this is:

Using the chart and table there are some interesting takeaways: - I drink a lot of coffee. Almost 44 liters during this period! - There’s coconut water consumption in late December - I had this while skiing - In early February, there are a few days with only water, smoothies, and some soda. This corresponds to when I had my wisdom teeth removed - I have energy drinks on the same day as protein shakes. This matches reality - I have a pre-workout energy drink and protein shake on days that I lift - I seldom have beer, but I have more beer than I have protein shakes - I infrequently have wine and liquor. This is good! - You can see wine consumption during the New Years period - Most liquor consumption was during vacations

Dogs are awesome. Since March of 2014, I’ve owned a little dog named Riker:

I adopted her from the Humane Society in Milpitas. She was 8 weeks old. She recently had her fifth birthday. Happy birthday Riker! I love her very much.

Anyways, when I got her, I was curious about what type of dog she was. She looks like a chihuahua, but has some features that don’t match the traditional chihuahua:

• Curled, fluffy tail
• Gradient-banded coat near her front legs
• Sheds mountains of fur
• Generally pleasant demeanor

I thought she might have been part Shiba Inu. These dogs are adorable and have the curled tail and gradient-banded coat. Here’s one from Wikipedia:

Doesn’t this dog look like Riker? With the tail and banded coat? I thought so too.

Whenever other people asked me about Riker, I would tell them she’s “part chihuahua, part shiba inu”. They believed me, but I didn’t have any proof.

Enter Embark

A good friend of mine told me about a service where you can get your dog’s DNA analyzed. It’s called EmbarkVet (disclaimer: this is a referral link). Embark gives your dog a breed report, health analysis, and some cool genetic information. It’s like 23andMe for your dog.

I signed up for the service. A few days later they sent me a DNA swab kit. It’s like an oversized q-tip. After sticking this in Riker’s cheeks for 30 seconds or so (she wasn’t happy about this, but I told her it was for science), I packed the kit in the return shipping and sent it off.

Over the next few weeks, Embark notified me as the analysis progressed. After 3 weeks or so, my pressing question was answered: they told me what type of dog she was!

Less than 50% chihuahua

To my absolute amazement, Riker is not even mostly chihuahua! The results confirm that Riker is a pomeranian! The full breakdown is:

• 50% Pomeranian (this accounts for the curved tail, banded coat, shedding, and how she’s nice to be around)
• 32.5% Chihuahua
• 6.9% Miniature Pinscher (no relation to the larger dog)
• 10.6% “Supermutt” (small amounts of DNA from other dogs, including Cocker Spaniel)

I also learned that she has 1.5% “Wolfiness”, which indicates ancient wolf genes that have survived through to modern domesticated dogs. Cool!

Note: I am not a trained biologist. Regrettably, I spent much of my college biology course playing Pokémon Crystal, although I do remember some parts of the genetics sections.

A family tree

Embark also determines haplotypes, which means they can tell what genes of Riker’s came from which parents. This lets them generate a family tree:

This means Riker’s parents were a chihuahua mix and a purebred pomeranian.

Analyzing my dog’s DNA

Besides finally being able to see my dog’s genetic history, I was super happy that I could download the raw genetic data. When I unzipped the file, I was left with two text files:

riker.tfam // 75B
riker.tped // 6.7MB

(I renamed them for brevity)

I searched for information on these file types. They both seem to be related to the free PLINK genetics software package.

If we download PLINK and run it on these files, we get:

$ plink --tfile riker
Processing .tped file... 77%
Error: Invalid chromosome code '27' on line 166046 of .tped file.
(This is disallowed for humans.  Check if the problem is with your data, or if
you forgot to define a different chromosome set with e.g. --chr-set.)

This is pretty cool! PLINK defaults to human DNA. This file is from a dog, not a person. Looking through the plink --help file, we can see that they have support for lots of species:

--cow/--dog/--horse/--mouse/--rice/--sheep : Shortcuts for those species.

(I wonder why rice is so interesting to the software…)

Anyways, let’s run it with the --dog flag:

$ plink --dog --tfile riker
Processing .tped file... done.
plink.bed + plink.bim + plink.fam written.

PLINK wrote these files to the same directory:

$ ls
plink.bed
plink.bim
plink.fam
plink.log
riker.tfam
riker.tped

These look to be PLINK metadata files that the software uses to do its processing.

We can use PLINK to do some interesting sounding genetic computations. For example, if we run it with the --homozyg flag, we can see homozygosity reports. According to Wikipedia (I must have been in Johto during this part of biology), zygosity is “the degree of similarity of the alleles for a trait in an organism”. Running it produces:

$ plink --dog --tfile riker --homozyg
1 dog (0 males, 1 female) loaded from .fam.
--homozyg: Scan complete, found 27 ROH.
Results saved to plink.hom + plink.hom.indiv + plink.hom.summary .

The software knows that the dog is female, which is pretty cool. The files it generated seem to indicate the degree of homozygosity for her individual genes. Neat!

If we run with the --het flag, we can see inbreeding coefficients. The file it produces show this:

O(HOM)    E(HOM) N(NM)  F
     0 3.051e+04 61022 -1

From this helpful documentation I found online:

O(HOM)    Observed number of homozygotes
E(HOM)    Expected number of homozygotes
N(NM)     Number of non-missing genotypes
F         F inbreeding coefficient estimate

-1 looks like it indicates a sampling error or contamination according to the docs:

The estimate of F can sometimes be negative. Often this will just reflect random sampling error, but a result that is strongly negative (i.e. an individual has fewer homozygotes than one would expect by chance at the genome-wide level) can reflect other factors, e.g. sample contamination events perhaps.

We can also use the software to find what parts of the genotyping are missing with the --missing flag. From this, I was able to gather that Riker only has a missing SNP rate of 0.0009034 (less than 1%). I think this means that Riker’s DNA in this sample is over 99% complete. Cool!

I may make Riker’s DNA available online some day for others to do genetic analysis.

Anyways, I thought this was a fun exercise into how genetic data is stored and processed. It’s really cool that there is open source software to analyze this data. Thanks for reading!

This is more of a technical post

This site is written using Markdown, which:

allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML)

It’s a common technique for writing blogs, as it lets the author write text in the text editor of their choice.

I write this page in vim, which is a command line text editor. I love vim for many reasons that are outside the scope of this post. I’ve been using it as my primary editor for almost 5 years. I wanted a good solution for writing Markdown in vim.

Because vim is a command line editor, it makes it tougher to use it to write rich text. In a GUI program, rich text can be rendered inline and as-you-edit. A command line tool can show you the formatting you have applied to text, but it does not give you an accurate rendering of how your content will look as you edit it.

I wanted to continue to use vim to edit my blog posts (I find vim plugins for other editors to be a worse approximation than “I can’t believe it’s not butter”). This post is about how I solved this problem.

A Markdown preview app

On macOS, you can get apps to render Markdown in a window. One that I like a lot is Marked, which automatically refreshes its preview when the Markdown file changes. This means you can open a .md file in Marked and get live updates as you change it. This can be used with any editor, including vim. The flow is something like:

• touch or :e the file you want to write
• Open Marked
• Open the file in Marked

As you save to it with :w, the preview will be auto-refreshed. I like this, but wanted to open it with a single key combination.

Adding Marked to vim

We can write a vim function to help us with this. Something like this:

function OpenInMarked2()
    " Open the file in Marked here
endfunction

Running a shell command from vim

We can use ! to run a command from our function. Technically, this is the filter command. From :help !:

!{motion}{filter} Filter {motion} text lines through the external program {filter}.

You can try this in your vim. Just run :!pwd, and you should be presented with your shell and your current working directory. Below it, vim helpfully tells us “Press ENTER or type command to continue”.

This is great! Using this and the macOS open command, we can at least launch Marked 2 with something like this:

!open -a Marked\ 2.app

But what about opening the current file? vim lets us use % from Ex commands as a stand-in for the path to the current file. So we can do something like:

!echo %

to see the current file. We can use this with open to open the current file in vim:

!open -a Marked\ 2.app %

This is the body of our OpenInMarked2 function:

function OpenInMarked2()
    !open -a Marked\ 2.app %
endfunction

Once we have this function, we can call it with the Ex call command, like this:

:call OpenInMarked2()

This will open the current file in Marked, but will leave us at the shell with “Press ENTER or type command to continue” there. We still have to press enter before we can edit the file live.

Defining a mapping

We can use a vim mapping to call the function for us. A mapping lets us take a key combination and map it to a command. For the key combination, I like to use <leader>, which is a user-controlled modifier in vim. This means that <leader> _ is wide open for use in your .vimrc. For my leader key, I use , (comma), as it’s near the other modifiers on a US keyboard. For leader mappings, I like a mnemonic key combination, as it makes it easier to remember.

A good choice for this mapping, which is applicable for Markdown and for opening Marked, is <leader>m. Our mapping to call the function might look like this:

nmap <silent> <leader>m :call OpenInMarked2()

Here’s a decoder ring for what these commands mean:

• nmap means a mapping that exists in vim‘s normal mode
• <silent> means the mapping will not be echoed on the command line
• <leader>m is the mapping we’ve picked for this particular command
• :call will call a particular function
• OpenInMarked2() is the name of the function we’d like to call

If you add this mapping and function to your .vimrc, you’ll see that it works kind of. You have to hit enter to get the command to actually run, and then after doing so you’re left at the shell, so you need to hit enter again before returning to the editor.

Hitting enter twice

We can add a press of the enter key to our mapping using <CR>. This acts as though the user has pressed the carriage return. We’ll add two of them, as we need one to run the function and another to return to the editor. Our mapping now looks like this:

nmap <silent> <leader>m :call OpenInMarked2() <CR> <CR>

which is the mapping you’ll find in my .vimrc.

I hope you found this post helpful. I have been using OpenInMarked2 to help me write Markdown for 2 years now, and I think it’s a cool example of how to extend vim to fit your needs.

This is the first post on my new website! You’re reading this on the brand new peterhajas.com. Let’s break it down.

A (not so new) domain

It’s worth noting that this page is also hosted on peter.haj.as. I have had this domain from American Samoa for some time. I love the idea of domains that are people’s names with no added alphanumeric characters, and this one fits the bill for me.

My own static site generator - what could possibly go wrong?

My old page was powered by Hyde. While this project provides a lot of flexibility, I felt that I was not using all of its features. I wanted something dead-simple that could enable me to write more. As a result, I wrote my own.

This site is still statically-generated from Markdown, but it’s now done with a Bash script with only one dependency (a markdown binary). This makes it easier to set up on a new machine. It also lets me really understand how the site works.

The site generates quickly, and it’s been a great learning opportunity! For example, I learned how to format dates using date (see here). I’m still learning (I don’t know web layout very well, for example) but I think it’s better to jump in with both feet when learning something new.

The site is missing some stuff (like RSS support), but I plan to iterate on it over time.

Hopefully I’ll write more

All of this is for naught if I don’t actually use the damn thing to write and think out loud.

I hadn’t updated my old webpage in over 6 years. There’s so much that has happened in that time that I want to write about, like:

• I have a dog!
• Living in the Bay Area
• My obsession with dotfiles
• Music that I’m listening to
• Trips I’ve gone on
• Pictures I’ve taken

Fingers crossed I’ll write about these things in the (near) future on this page.