Arachnoid Web Spinal Surgery

Welp, on January 9th I had a neurosurgeon from the Mayo Clinic crack off a few pieces of my vertebrae (the lamina portion of the vertebrae) from T5-T8, cut through the dura covering my spinal cord, and then remove a 3 cm long arachnoid web that was compressing my spinal cord. They are pretty certain that the arteriovenous fistula that I had previously been diagnosed with and treated for was completely unrelated to the spinal cord issues I’ve been having. Similar to the fistula though, arachnoid webs are also very rare, about 1 in 500k or so. This paper has some good summaries if you really want to read a lot.

As three (and three quarters) year old Morgan has excitedly whispered a few times:

They figured it out!

I’m trying to not get overly optimistic, but it is quite possible that now that my spinal cord is no longer compressed it may just float back to where it is supposed to be and will conduct signals much better. Really only 1-2 years and a lot of rehab will tell though. I’m pretty excited about the idea of rehab where I can actually make progress though.

So now I have about 5 more weeks of recovery where I’m not supposed lift more than 10 pounds. Unfortunately at almost 4 months now, Connor is already bigger than that. I can however do as much walking as I can manage. Right now I can safely do about 300 feet as long as I have hiking poles to help my balance. Still a lot of pain from the surgery, but it is a lot better than it was last week, so seeing some improvements.

Thanks everyone for all the thoughts and notes. The past month or so has been a hectic whirlwind trying to figure out how to get to the bottom of why things were getting worse. Turns out we hadn’t really figured out the right cause in the first place.

Below is a video of a similar surgery, and a few images of my spine.

This is the removal of an arachnoid cyst which is pretty much the same procedure just a slightly different malformation.

Ugh… Ongoing Spinal Cord Damage

Welp, as of last Thursday I feel pretty certain that my spinal cord is still getting damaged. The spinal arteriovenous fistula that presumably caused the spinal cord damage was embolized (glued closed) in September 2017. From what I understand though, embolization is only about 75% successful, so I guess I am in the remaining 25%.

Unfortunately, the only real way to know whether it has been successful is to monitor your symptoms and look for additional problems. Especially new problems or more than one problem changing. This has been pretty difficult given that I have been trying to do spinal cord rehab and constantly trying new medications and treating other side effects from the spinal cord damage (like sleep apnea).

Over the past few weeks though I’ve been slowly coming to the realization that I have more bladder urgency than I used to have (and one time where I didn’t get where I needed to in time which is ummm… “fun”). It would probably be far worse if I didn’t work from home. Then on Thursday my PT redid my muscle strength testing that was done in January at Craig Hospital and it was fairly clear that some of my muscles, which were rated as 5 out of 5 then, are no longer 5s. Given that I’ve been regularly doing PT exercises all year long I definitely shouldn’t be getting weaker.

I don’t really know whether the damage has been occurring slowly all year long and only recently did the damage become enough that it started making the symptoms worse, or maybe something recently changed and I just started getting more damage. I am pretty certain that the embolization did reduce the rate at which things had been getting worse. Definitely the rate at which I was getting worse in 2017 was much greater than 2018 has been.

So I’m off on the hunt through medical systems to find the right doctors and surgeons to diagnose and hopefully actually fix the problem. First step was two hours of MRIs this past Monday which I don’t expect to find anything new. I imagine I won’t really get anything diagnosed until I can get some neurosurgeon cutting into me to take another look.

Forward.

Learning is Painful

This is a post about my damaged spinal cord, but I’ve been working at Automattic for seven years now, so I really can’t help but think about the first line of the Creed:

I will never stop learning.

I strongly identify with this line, but damn it is hard sometimes. The past two to three months I have been in physically terrible shape. Every 3-4 weeks I’ve been learning something new about how my body now works, and this learning is both metaphorically and physically painful.

Aristotle – “Learning is not child’s play; we cannot learn without pain.”

Damn Greek philosophers!

Really I think I’ve been learning something new about my body every few weeks for the past 10 months. Currently feeling hopeful that I can put some of the physical discomfort behind me, but a part of me knows that is probably overly optimistic.

Abdominal Muscles

In early June I went to a work meetup in Denver which contributed to me pausing my PT exercises for about two weeks. Then I tried to pick up where I had left off. It turns out though that muscles atrophy faster when you have a damaged spinal cord, so two weeks off is more like 4-6 weeks off for me. On top of that, it turns out I don’t really have much sensation in my abs. So when trying to do my PT, I didn’t realize that I had badly pulled my abs, so I kept going and pushing through as my exercises got harder. And then for the next week I really couldn’t stand up. About a month later, as I was finally getting better I pulled them again, probably due to dehydration.

My lack of sensation in my abs is so bad that my PT can push on them to get the muscles to release and I am perfectly comfortable where he says anyone else would be in agony. Turns out pain is a really useful feedback mechanism.

I now finally understand why every few years since about 2006 I have badly screwed up my abs to the point of having trouble standing. I just didn’t have enough sensation to tell myself when I needed to back off from whatever I was doing. This problem has been there for a long time. At least I understand it now.

Sleep Apnea

I started on an APAP machine in mid-June. It wasn’t until about two weeks ago that I found a mask configuration where I am pretty comfortable and feel like I can sleep well. There was a lot of bad sleep along the way to figuring that out. But I think I can now see some improvements to two of the fatigue metrics that I have been tracking: grip strength and heart rate variability. This is pretty exciting. My grip strength – yes grip strength! – has increased something like 20% on average. Similarly my heart rate variability has increased by 10-20% which is an indication of less fatigue.

Pretty exciting that I am seeing an improvement in the metrics that I had identified as mattering. For some reason my watch has not been showing an increase in deep sleep, but overall I have also been feeling somewhat less fatigued so I’ll take it.

Unfortunately I have actually been feeling more tired which is probably related to how much sleep debt I have. Supposedly that can take months to catch up on. Just in time for kiddo number two to show up! 😉

Anti-Depressants

The other recommendation I got to deal with fatigue was to start taking Cymbalta which is both an anti-depressant and can reduce nerve pain (maybe allowing me to reduce another medication which can also add to fatigue). I started in early July, and the side effects have been pretty terrible: insomnia, nausea, dizziness (especially looking at a screen), and light headedness due to my blood pressure dropping.

The first week was so bad that after days of nausea and insomnia I repulled my abs because I got very dehydrated. Then ended up calling an ambulance at 3AM when I couldn’t stand up without getting dizzy. Every time I stood up my blood pressure would drop about 40 pts. After they put two liters of fluid in me I was a lot better, but ya it was a bit exciting.

Sidebar: when you are laying on the floor unable to stand, alone in the house at 3AM (Mekayla and Morgan had gone camping), but otherwise ok, it is really hard to know when to call an ambulance. Also, it is really nice when you have a watch on your wrist from which you can make a phone call rather than having to go find your phone.

After about five weeks on Cymbalta, I think I am mostly adjusted to the medication. I am not sure it has helped with fatigue, but I am in an amazing mood. Given how rough the past month has been, I have actually been feeling pretty good about everything.

So… um… yay anti-depressants. I don’t actually think my depression was all that bad. It happened occassionally. What I thought of as maybe a few times a year. The reality is that it was always probably there more than I even now want to admit. Wil Wheaton probably explained this better than I can. But I would say that even though I am unsure just how bad it was, that I wish I had tried this years ago. If you are reading this and thinking about it, you should know that there are other meds besides Cymbalta where the side effects are typically not as bad.

Increased Sensation

This one is really just funny.

So my spine has become a bit crooked (not yet the funny part). It turns out that this was because my shoulder and back muscles had become chronically tight to compensate for my bad balance. My PT has been working on this for many months, and it is improving, but I decided to also add in getting a massage from someone who has experience with spinal cord injury clients. In lots of places my body has reduced sensation, but the sides of my torso are extra sensitive. A little bit of pain is just unbearable even when I know the massage is not that strong.

So, the massage reminded me that my sides are extra sensitive, and then solved a long running bout of marital discord. For years, Mekayla would lay on my shoulder, and then I would yelp and ask why she was digging her fist into my side and she we would plead complete innocence. Ignorance even. She would complain that I was being overly sensitive and I would say something about why she has to twist her knuckles into my ribs. This has been going on for years.

Turns out I was being overly sensitive.

Ooops.

Gnōthi Seauton

 

Relearning to Walk at Forty: Fatigue, Physical Therapy, and Data

I spent most of 2017 trying to diagnose the slow degeneration of my ability to run and then walk. Then the past nine months have been slowly trying to discover how much I can recover from my spinal cord injury. My spinal cord damage is non-traumatic which means that it is a bit different than the typical spinal cord injury – though I’ve rapidly learned that all spinal cord injuries are really pretty unique experiences. In general, for someone who is missing 30% of his spinal cord for a couple of vertebras I am doing really well, but that really doesn’t make it feel any easier most of the time.

My rehab is still a work in progress, but I’ve been wanting to write about the data I’m collecting for a number of reasons:

  1. I learn a lot by writing, and it helps get the details clearer in my head when I write them down.
  2. Hopefully sharing this can give ideas to other people struggling with similar problems. I’ve taken a lot of ideas from folks with multiple sclerosis, chronic fatigue syndrome, and of course spinal cord injury.
  3. Comments on blog posts can often provide a lot of great feedback and other ideas of things to pursue, so hopefully I’ll learn something from you.

So I’m going to cover a good chunk of the data that I am tracking about my day to day symptoms. If you don’t want to know a lot of details about what is going on, you should stop reading now. Also, I should probably warn you that I like data a lot, track a lot of data, and this is going to be a 3,000 word post. 😀

Background on my Symptoms

Probably helps to outline my symptoms around walking/moving:

  • Injury diagnosis: T1 ASIA D – Sensation changes start at Thoracic 1 though most of the visible damage on the MRI is from T5-T7. “D” because I can walk, and don’t need a cane or anything. For comparison “B” would be in a wheelchair but with some sensation/control, and “E” is normal.
  • Walking ability: I was close to falling down a lot (and did once), but the Tizanidine I’m taking has gotten the spasticity under control so that I almost always feel pretty stable on even surfaces. Changing speed is challenging though as is looking around while walking, but both are improving. Otherwise my walking is mostly limited by endurance and how much pain I want to be in. I’m averaging about 4.5k steps a day this month. Typically, less than 5k steps I’ll be fine, and more than that I’ll be in pain later in the day or tomorrow.
  • Muscle control and gait: I have poor control over lots of muscles from my chest down, and it is not symmetric. Certain sides and certain muscles are very hard to activate. For example, my left hip is much harder to control than my right. This resulted in some odd walking problems where I’ve learned (over more than a decade) to balance using my left shoulder which is always really tight, and my right hip which is always sore. Was resulting in a slight limp. Also my ankles have very poor control. Manual manipulation by my PT and lots of focus on being able to activate particular muscles is slowly helping. I have a lot of habits to retrain.
  • Pain: My feet are always numb to some extent. As I get more tired it becomes a burning pain and eventually my legs start twitching which also hurts. Overall pain usually varies from 2 to 6. I haven’t not been in pain for over a year. I’m taking Gabapentin to reduce the nerve pain. Helps a lot, especially with falling asleep. Only take it at night because it was increasing my fatigue during the day though so often in more pain as the day goes on and the drugs wear off. Most of my pain doesn’t happen immediately, it kicks in a few hours after the activity, so it is very hard to have good feedback.
  • Fatigue: My fatigue is delayed by many hours and often builds up over the course of multiple days. This is not muscle fatigue, but more full body fatigue that I now think of as “central nervous system fatigue”. Coping with this is what led me to trying to track my fatigue and various symptoms daily so that I could know when I needed to back off before I ended up just laying on the couch unable to move for hours at a time. This used to happen a lot and at those times, even reading to my three year old was too much.
  • Balance: My balance is pretty bad. I can maybe stand on one foot for 10 seconds. When I have my knee braces on it is a bit better, but still pretty bad.
  • PT recovery: I have trouble recovering from any amount of physical therapy. Especially I think the balance exercises really increases my fatigue. I can do a day or two in a row of my 1.5 hrs of PT, but then I’m very exhausted and in pain for another day or two. Currently I’m getting about two PT sessions a week in, and I’d rather be doing four.
  • Standing: Took me a while to realize that just standing around talking with friends often ends up with a lot of ankle pain later on. I didn’t really understand this until just recently. I think it is a combination of my ankle muscles being weak and my balance being poor which makes me work my ankles more.
  • Running: I can run. Yay! Not far, but mostly because I am out of shape. It hurts at the time though, and it puts me in pain for a few days. The big problem with running is that it increases fatigue and then makes it harder to do my PT exercises. I really want to be running, but it takes so much out of me.
  • Sleep apnea: I was just diagnosed and haven’t really started my machine yet (tonight!), but this is the most exciting diagnosis I’ve ever had because it seems so likely to be solvable. It may also be at the root of a lot of fatigue and recovery problems. 20% of the population has sleep apnea, and a significantly higher percentage of folks with spinal cord injury do (which makes sense given that we have less control of our muscles). Given how easy the test was (basically just take home a device to wear at night) I feel like everyone should get tested.
  • Depression: Ya depression happens a lot. Most of the time it hasn’t been too bad, but sometimes it is completely debilitating. I haven’t started any anti-depressants yet. I’m hoping a lot of this is due to the sleep apnea and lack of sleep (and of course trying to adjust to having paraplegia).

I feel like I keep learning about my symptoms and my body, and I seem to learn something new every few weeks.

Docs, PTs, and Specialization

I am fortunate enough to live near Craig Hospital which is one of the top spinal cord rehab centers in the world. The difference between my docs/PTs at Kaiser (even the ones who know a lot about spinal cord injuries) and Craig has been night and day. I ended up also finding a PT near me who used to work at Craig to go to weekly as well.

Tools and Devices

I’ve been using a bunch of different technologies for tracking and walking.

  • DeRoyal Knee Braces: I’m using these all day long to prevent my knees from hyperextending backwards. Starting with these immediately doubled how far I could walk without ending up in pain. There is a pretty common refrain from folks using wheelchairs that their wheelchair gives them freedom and though my own paraplegia is nowhere near the same, putting on these knee braces gives me an incredible feeling of freedom. I’ve also been doing some of my own modifications to my braces to make them easier and work better for me.
    • Knee sleeves: I found that wearing the braces was causing me a bit of discomfort by the end of the day. Putting on some knee compression bands under them gets rid of that problem almost completely. At first I also tried using Rolyan Cotton Stockinette, but it starts falling apart after about a week so I would constantly need to be making new ones.
    • Marking with silver sharpie where on the straps I am velcroing the cross over straps so I don’t have to spend time trying to figure out how to adjust them each time. I started marking with just some tape and moved to sharpie once I was sure of the location. Simply writing a big L and R on the inside makes getting them on so much faster also.
    • I’m tying a small shoestring from the top flap to the plastic buckle loop to reduce how much the top strap slides down throughout the day. You can see this at the top of my right brace in the picture below.
    • I found that the straps that came with them were not quite the right length so I bought some shorter replacement straps (and then put tape over the logo).

2018-06-15 11.53.31.jpg

  • New Balance 990v4 Running Shoes: I ended up with New Balance running shoes in order to get better stability when walking. They also were wider than the shoes I had been wearing that makes them work better with my orthotics.
  • Orthotics: custom made by my PT at Craig. It was kinda like LEGOs for orthotics, pretty cool.
  • Polar H10 chest strap heart rate monitor: My watch also has a heart rate monitor, but to get good monitoring of heart rate variability (see below) you need a chest strap.
  • Apple Watch: I’m using a bunch of different apps for tracking and building habits. There are a lot of different watches out there, but the breadth of Apple Watch apps is what really helps me:
    • AutoSleep: My only complaint is that I feel like it could have told me six months ago that I should get a sleep apnea test. 🙂
    • Streaks: I’m big into habit building, and this is mostly what I use right now. It is always right in front of my face (on my watch), so lots of reminders throughout the day about what I am trying to work on.
    • Meditate: really nice meditation app that I can start from my watch and is not overly distracting.
    • Apple health for step and distance tracking (also swimming when I can). The HRV data from this is not accurate though.
  • Grip strength tester: I use this as a proxy for how tired my nervous system is (see below).
  • Ankle Stabilizer Brace: Just recently got these. I’m not wearing them all of the time, but if I know I may be standing around a lot or my ankles are in a lot of pain then I’m wearing these. I’m also thinking to try running, hiking, and biking in them to see how much that helps reduce pain. So many experiments…
  • Airex Balance Pad: I use this for a lot of my balance exercises. Makes balancing just a bit harder.
  • BOSU: I haven’t been using this for balance exercises as much recently because it is mostly too hard for me. Need to master things on the Airex first, but expect I will get back to using this too.
  • Other iPhone Apps:
    • Symple: I use this to track symptoms daily and throughout the day
    • EliteHRV: I use this with the chest strap above to measure my heart rate variability
    • YouAte: taking pictures of food
    • MetroTimer: a metronome to help retrain myself to take even steps.

PT Exercises

I’ve gone through a lot of different PT exercises and as I said I’ve worked with a number of different PTs. My current set is focused on getting the exercises really good and solid before moving on to the next set. That was probably one of the biggest mistakes early on with things like balance exercises, I didn’t really have them completely solid before I worked on making them harder.

I’m not going to describe all my exercises, but I will describe how I am rating my ability to do them:

  • Level – I abbreviate this to be something like L1, L2, L3. The levels are arbitrary and a bit dependent on the exercise. For balancing on one foot for 30 seconds for instance the levels are:
    • L1 – eyes open, looking at one point
    • L2 – looking around constantly
    • L3 – eyes closed
    • L4 – eyes on one point, but standing on my Airex
    • L5 – looking around constantly, but standing on my Airex
    • L6 – eyes closed, but standing on my Airex
  • Repetitions – for exercises where I am doing repetitions, this is how many I am doing. Squats for instance.
  • Ability (1, 2, or 3):
    • 1: means that I can’t do something non-stop. For 20 repetitions, I need to stop. For a balance exercise, I fall over.
    • 2: I can do it, but not easily. For balance this means my arms are waving around or that I feel I need to use my shoulders or other muscles I shouldn’t be using.
    • 3: It feels pretty good. Maybe not perfect enough to move on to the next level, but probably will be able to move on in a week or two.
  • Sometimes color coding also – yellow, none, green. Currently I am only using this for balance exercises to indicate whether I am wearing my knee braces and shoes or not.

Ability to do PT exercises doesn’t change very often, so I only record these once a week, and I’m really only looking for improvements over the course of 3-4 weeks at a time. Having some metrics to use and slowly seeing improvement helps with a feeling of having some accomplishment. It will probably also tell me when I have maxed out what I will be able to improve.

Here are 6 weeks of recent tracking.

PT-Sample.png

Daily Fatigue Tracking

I’ve experimented with a number of methods for daily tracking of my fatigue so I would know when to take it easy. I print out spreadsheets for this and tape them to the wall so I can fill them out during the day, and then every week or two transfer it to a spreadsheet on my computer. Here is what I am currently tracking and why:

  • Resting heart rate. Usually goes up above 57 when I am more tired or sick.
  • Hours of sleep (from AutoSleep): varies a lot. Below 4 is bad.
  • Hours of deep sleep (from AutoSleep): if I am not getting deep sleep then my body is not entering the sleep stage where my muscles actually recover
  • Grip Strength: This is used by some elite athletes to track “Central Nervous System Fatigue”. Not great scientific evidence, but for me I find that when my score is low (below 47) then I need to back way off.
  • Heel Raise. 5 point scale: 1 – can’t do it, 2 – very hard, 3 – doable, 4 – have a little bounce, 5 – ready and raring to go
  • Pain on the usual 1-10 scale. At 5 the pain is pretty hard to ignore. If I wake up already at 4, then it is going to be a rough day because my pain almost always increases. This is tricky though because sometimes the pain is due to things unrelated to walking, mostly bowels (that’s a different post).
  • Heart Rate Variability: from the EliteHRV app. Below 47 is usually not very good.
  • Steps (at the end of the day): In general more steps puts me in more pain.
  • Some things I just track in notes, and then summarize later:
    • Fatigue (Y/N)
    • Bad (Y/N)
    • Autonomic Dysreflexia (Y/N) – for me this is triggered by bowel problems

Most of these I record in the morning, except grip strength, pain, and hell raise I do both morning and night. No single metric is enough to say that I am fatigued, but they do give me a sense of whether I need to back off or not. I also track whether I did any PT, walking, running, and notes about my day.

Some recent data:

Fatigue-Sample.png

I’m a pretty big fan of conditional formatting in a spreadsheet to pull out patterns.

Monthly Fatigue/Sensation Tracking

Because my spinal damage is non-traumatic it is possible that it hasn’t fully been fixed and unfortunately the only way to know is if my symptoms start getting worse (at which point they probably won’t get better). So I am doing a bunch of monthly tracking to see if I can find any signals that would indicate that things are getting worse. Most of these are based off of various standard medical questionnaires.

  • Fatigue: I am using the Modified Fatigue Impact Scale (MFIS).
  • Depression: I am also using the PHQ-9 for tracking depression.
  • Sensation changes: I have print outs of the ASIA scale that I am trying to self rate myself with. I basically use a pen cap (perfectly calibrated of course) to poke and rub parts of my body and recored where I am extra sensitive (X), triggers twitching (T), no sensation (O), and numbness (N). Especially looking for differences between each side and changes

https://pt.slideshare.net/dennis43/standard-neurological-classification-of-spinal-cord-injury?nomobile=true

I’m also tracking a bunch of bowel and bladder data from month to month, but I’ll leave that for a separate post since I could probably write 3,000 words on that topic too.

Your Turn

OK, so there is everything I am tracking around sensation and movement so far. I’d be really interested in anything others have found useful. This is already a lot, but since I keep learning new things, I assume there is a lot more to think about that I am missing.

Huh, Seven Years. Look at that…

It turns out that seven years ago today I started my trial project with Automattic. I know this because that is when I signed up for user id 23314024 on WordPress.com

Screen Shot 2018-05-11 at 6.25.55 PM.png

My official Automatticversary – as we call the day you became a full time employee – is not until mid-July, but this notification inspired me to be a bit nostalgic. It also happens to fall on the week of my 40th birthday.

My trial project was building a prototype Elasticsearch system (version 0.16 IIRC) so we could search across our internal p2s (disclaimer: no longer what p2 looks like). At the time there were 291k posts and comments across our 89 p2s and MySQL search was no longer working well. There are now almost 2.8m posts and comments across 639 p2s and Elasticsearch is continuing to do pretty well.

We now have six Elasticsearch clusters the largest of which had 6,782,578,164 posts in 2625 shards a few seconds ago. Those clusters have about 850 separate indices that are powering over 66 different use cases. Pretty exciting and humbling how much the company’s usage has grown.

If you are looking to get in on the Search Wrangling fun, then we are hiring. We have a ton of challenging search and relevancy problems to tackle.

 

WordPress and Democratizing Algorithms

In discussing how the newly released Jetpack Search fits with Core WordPress search I veered off into discussing algorithms in general. That generated some healthy discussion on the WPTavern Podcast. I wanted to expand on what I was saying a bit about democratizing algorithms. This is very much related to what JJJ called “race car technology makes its way down to everybody” on the podcast.

I think some of the negative reaction is a personal preference where many readers loudly prefer reverse chronological because it is easy to understand and they feel in control. I think that is a very common opinion among WordPress developers. I’ve had that discussion many, many times. I also think that user control is important.

However, it is very clear that ON AVERAGE having some algorithm filter or reorder the content is much more engaging. It boosts visitor engagement.

Let’s look at some top examples: Twitter, The New York Times, The Washington Post, and Amazon (30% of humans use Facebook, so let’s stipulate that algorithmic is working pretty well for them).

Twitter

Many people are loudly complaining about Twitter not being completely reverse chronological. However, as a product and service, they just had their first profitable quarter ever. And it wasn’t because they have more users. Monthly active users is flat. But daily active users increased 12% year-over-year. They have steadily made their product more engaging to the average user. So while yes many people dislike the algorithmic changes they are making, they also seem to be working.

New York Times and Washington Post

nytimes-top.pngwashpost.pngBeyond even human “algorithms” that order which articles are on the front page, it is very common to have algorithmic sections of the front page of major news sites. Sometimes it is “most emailed stories”, or sometimes it is the most viewed. The current NYT’s home page has a section which looks like the top posts from across the past few days in a nice scrollable display. Maybe they are hand picked, but I bet there is data influencing them. The Washington Post has “Most Read” and “Live Discussions” sections.

Even on this blog I’m using Jetpack’s top viewed posts widget and Jetpack related posts. Both are things that WordPress Core can’t do.

Amazon

Algorithms are everywhere on Amazon. Let’s look at the dog food I buy. The reviews are clearly sorted by what will get me to buy it.

dog-food.pngHow could I not buy it!

It is about Engagement!

None of these examples are easy (or cheap) to deploy right now. It is still pretty hard to build an engaging website. Maybe building an engaging website is plugin territory. That sounds great for Jetpack which has more and more of these features. I’m not sure it is great for the Web though.

This even comes up with simple websites. Gutenberg opens up very interesting questions about how blocks should be organized on a web page. Chris Lema mentioned this when discussing the future of web publishing:

The future can’t continue to be a unidirectional dynamic where someone in marketing determines the best articulation of their message in a single-focused and static design.

The future of publishing is that different people can get different content depending on their behavior, demographics, interest and more.

I think framing it as personally preferring algorithmic vs chronological is really the wrong way to think about it. The question is, if you could flip a switch and get your visitors to spend 20% more time on your website, then would you?

Right now, only big expensive web sites can toggle that switch. Many of them are effectively monopolies. Let’s democratize algorithms so any website can choose to build a highly engaging user experience.

An Aside

While we can and should talk about filter bubbles and the impact that these algorithms have on the world, a world where only monopolistic tech giants can deploy these algorithms is not one where publishing is democratic.

 

Combining Search Scores: Winning and Failing

Trey Jones at Wikimedia Foundation published some very interesting notes up about how to think about combining scores for search ranking (particularly Elasticsearch). I like this insight a lot:

addition is looking for ways to win, multiplication is looking for ways to fail

This is pretty interesting to me when thinking about how I chose to implement the ranking for the WordPress.org plugin search. Applying this insight to the way I combined signals in that ranking function comes up with a couple of interesting observations:

  • The text matching features (phrases, title matches, etc) are looking for ways to win and boost the score. This was a pretty explicit goal of mine, but also partly driven by decoupling the matching of text from boosting on text.
  • All of the other signals are looking for reasons to fail. Not updating the plugin, not testing it on latest WordPress, not resolving support threads, etc. There is some boosting also, but we do a lot to lower scores which is maybe related to some of the exact matching problems I am still looking at (especially after result number 10).

I’m not sure this is either good or bad, just an interesting model for thinking about it and something I need to think about some more. This somewhat matches the intuition that led me to separate out matching text from boosting text with individual features.

I also need to think more about whether I am using the right operations for weighting different scores. There’s a lot of great thoughts in these notes and Trey has a bunch of other notes that look interesting also.

Also it reminds me how great it is to have notes published for others to look at.

 

Senator Bennet, your email address doesn’t work.

I emailed my Senator a few weeks ago to oppose Gorsuch being appointed to the Supreme Court. His response seems to indicate he thinks Trump nominees are still worthy of consideration. That’s an absurd stance since they have been found to be lying under oath during the confirmation process.

Here is Senator Bennet’s response:

Dear Gregory:

Thank you for contacting me regarding the current vacancy on the Supreme Court.

On January 31, 2017, President Trump nominated Judge Neil Gorsuch to serve on the Supreme Court. He has served on the 10th Circuit Court of Appeals since 2006.  Gorsuch clerked for Judge David Sentelle on the U.S. Court of Appeals for the D.C. Circuit and for Supreme Court Justices Byron White and Anthony Kennedy. He also served as the Principal Deputy to the Associate Attorney General at the U.S. Department of Justice.

I take seriously the Senate’s constitutional responsibility to thoroughly vet Judge Gorsuch. I intend to review his record carefully in the coming weeks. Rest assured, I will keep your thoughts and concerns in mind throughout the confirmation process.

I value the input of fellow Coloradans in considering the wide variety of important issues and legislative initiatives that come before the Senate. I hope you will continue to inform me of your thoughts and concerns.

For more information about my priorities as a U.S. Senator, I invite you to visit my website at http://bennet.senate.gov/. Again, thank you for contacting me.

Below is my response


Senator, thanks for your response.

He has served on the 10th Circuit Court of Appeals since 2006.  Gorsuch clerked for Judge David Sentelle on the U.S. Court of Appeals for the D.C. Circuit and for Supreme Court Justices Byron White and Anthony Kennedy. He also served as the Principal Deputy to the Associate Attorney General at the U.S. Department of Justice.
 
None of this matters. You should be working to delay each and every nomination by Trump because the longer we delay them the less damage he can cause.
 
There should be no more confirmations until Sessions has resigned for lying under oath during his confirmation process. How can you trust anything that any Trump nominee has said in any hearing until it is clear that they take not lying to Congress seriously?

Alas… it seems he doesn’t want to accept email responses…
Your message was sent to a non-monitored mailbox and has not been reviewed. If you would like to contact Senator Michael Bennet please visit his website at http://bennet.senate.gov/contact and fill out the webform for a prompt response. Thank you.
So much for dialog… Hey look I have a blog…

Top Five Posts from 2016

Most of this blog’s 40k visitors a year are looking at the epic Elasticsearch posts that I wrote years ago. For the most part they seem to still be relevant to people even if they are somewhat outdated. Here are my top posts with some commentary about each of them.

2014-emailteaser

1: Elasticsearch: Five Things I was Doing Wrong

79% of my traffic comes from search engines, and almost 50% of all traffic goes to this one post. It’s actually kinda crazy that such a simple post gets so much of my traffic. I blame the clickbait headline. I have a bunch of long winded epic posts and what I should probably be writing is these small tidbits as they come up.

2: Three Principles for Multilingal Indexing in Elasticsearch

This is my all time favorite post. After 2.x and the removal of being able to specify an analyzer in a query it has become a bit outdated, but the overall concepts are still good. I love all the comments this post has generated. I’ve learned so much from this post and the discussions that it generated. We’ve accomplished a lot the past year to adjust our multi-lingual indexing (deployed edgengrams into an A/B test yesterday) and I’m hoping to write up what my latest thinking is soon.

3 and 4: Scaling Elasticsearch Series

The first two parts of this three part series are my third and fourth most popular posts. The indexing post is almost twice as popular as the intro and querying posts. Although these posts are almost three years old now they still describe pretty well how we scale most of our queries. Most of the reason why these posts haven’t been updated is because the methods they describe have worked really well for us.

The original post talks about having 600 million posts in the index and 23m queries a day. We now have 4.3 billion posts and do about 45m queries a day. That’s some good scaling.

Only over the past year have we started to see some problems slowly develop with our global cluster scaling. Currently the cluster runs fine for about a month or so and then heap usage creeps upwards until it starts to cause problems. The solution is just to do rolling restart of the cluster. Not pretty, but it works. Here’s what our average heap usage looks like broken down by data center for the past 30 days.

Screen Shot 2017-01-06 at 9.28.48 AM.png

We think a lot of these are just memory management bugs in the old Elasticsearch version we have been running for years and are hopeful that as we transition to 2.x many of them will be resolved. The other option is just to add more servers which we haven’t done in a few years. Our typical load is not very high though until we reach the point of running out of heap so I haven’t felt very justified in ordering more servers for this cluster yet.

One high point of this cluster is it taught us how to run a multi data center cluster. Every cluster we deploy now is multi-data center and we have successfully survived cases where an entire data center goes down. Currently we are in three data centers spread across the US. It’s likely that in 2017 we will start trying to run intercontinental Elasticsearch clusters (Europe and the US). Should be exciting.

5: Managing Elasticsearch Cluster Restart Time

This post describes how we manage long restart times. 2.x is a bit faster in this regard, but still takes a while to synchronize, so this is still relevant to managing a production ES cluster.

 

 

Notes from The Printing Revolution

I’ve been reading The Printing Revolution in Early Modern Europe by Elizabeth Eisenstein on the transition of Europe from a scribal culture to a printing culture. In referencing Michael Clapham:

A man born in 1453, the year of the fall of Constantinople, could look back from his fiftieth year on a lifetime in which about eight million books had been printed, more perhaps than all the scribes of Europe had produced since Constantine founded his city in A.D. 330.

The printing press had an immense and hard to correlate impact on the past 600 years. How will we build the tools and discover the norms that will shape the next 600 years? Here are some quotes that jumped out at me from the third chapter on the “features of print culture”.

“Increased output and altered intake”

To consult different books it was no longer so essential to be a wandering scholar. Successive generations of sedentary scholars were less apt to be engrossed by a single text and expend their energies in elaborating on it. The era of the glossator and commentator came to an end, and a new “era of intense cross referencing between one book and another” began.

Merely by making more scrambled data available, by increasing the output of Aristotelian, Alexandrian, and Arabic texts, printers encouraged efforts to unscramble these data. Some medieval coastal maps had long been more accurate than many ancient ones, but few eyes had seen either.

Contradictions became more visible, divergent traditions more difficult to reconcile.

Printing encouraged forms of combinatory activity which were social as well as intellectual. It changed relationships between men of learning as well as between systems of ideas.

The new wide-angled, unfocused scholarship went together with a new single-minded, narrowly focused piety. At the same time, practical guidebooks and manuals also became more abundant, making it easier to lay plans for getting ahead in this world – possibly diverting attention from uncertain futures in the next one.

“Considering some effects produced by standardization”

The very act of publishing errata demonstrated a new capacity to locate textual errors with precision and to transmit this information simultaneously to scattered readers.

Sixteenth-century publications not only spread identical fashions but also encouraged the collection of diverse ones.

Concepts pertaining to uniformity and to diversity – to the typical and to the unique – are interdependent. They represent two sides of the same coin. In this regard one might consider the emergence of a new sense of individualism as a by-product of the new forms of standardization.

It’s interesting to think of printing and the uniformity it spread as the birth of individualism. Does the internet, by connecting highly dispersed but like minded people into tight niches bring about a reduction of individualism? Bubbles of conformity within your tribe?

no precedent existed for addressing a large crowd of people who were not gathered together in one place but were scattered in separate dwellings and who, as solitary individuals with divergent interests, were more receptive to intimate interchanges than to broad-gauged rhetorical effects.

There is simply no equivalent in scribal culture for the “avalanche” of “how-to” books which poured off the new presses

“Reorganizing texts and reference guides: rationalizing, codifying, and cataloguing data”

printers with regard to layout and presentation probably helped to reorganize the thinking of readers.

Basic changes in book format might well lead to changes in thought patterns… For example, printed reference works encouraged a repeated recourse to alphabetical order.

Alphabetical ordering. The simplest of sorting algorithms. Today it seems that has been taken by reverse chronological sorting. Only the most recent thing is important.

“From the corrupted copy to the improved edition”

A sequence of printed herbals beginning in the 1480s and going to 1526 reveals a “steady increase in the amount of distortion,” with the final product – an English herbal of 1526 – providing a “remarkably sad example of what happens to visual information as it passed from copyist to copyist.” … data tended to get garbled at an ever more rapid pace. But under the guidance of technically proficient masters, the new technology also provided a way of transcending the limits which scribal procedures had imposed upon technically proficient masters in the past.

fresh observations could at long last be duplicated without being blurred or blotted out over the course of time.

“Considering the preservative powers of print: fixity and cumulative change”

Of all the new features introduced by the duplicative powers of print, preservation is possibly the most important.

as edicts became more visible, they also became more irrevocable. Magna Carta, for example, was ostensibly “published”

Copying, memorizing, and transmitting absorbed fewer energies.

“Amplification and reinforcement: the persistence of stereotypes and of sociolinguistic divisions”

Both “stereotype” and “cliché” are terms deriving from typographical processes developed three and a half centuries after Gutenberg.

an unwitting collaboration between countless authors of new books and articles. For five hundred years, authors have jointly transmitted certain old messages with augmented frequency even while separately reporting on new events or spinning out new ideas.

 

Eisenstein, Elizabeth L. (2012-03-29). The Printing Revolution in Early Modern Europe (Canto Classics). Cambridge University Press. Kindle Edition.