Thinking About Hardware Reviews (Premium)

I’ve been evaluating laptops and other electronics for over 20 years, and it shouldn’t be surprising that I’ve continually updated my approach to writing reviews. I want them to be informative, of course, but also useful to potential buyers, and I take the responsibility of recommending—or not recommending—an expensive product like this very seriously.

But there are limitations. I’m just one person, and my reviews will be colored by my specific experiences and, in part, by my own needs. That is, though I try to expand the discussion beyond my personal preferences, all reviews are, by nature, opinions and are thus subjective. There are things about some laptops or other devices that I don’t like that others will not mind or might even prefer. And vice versa.

With that in mind, I’ve been thinking about these reviews a lot lately for a variety of reasons.

For example, I tend to test PCs and other devices a lot longer than other reviewers, and that has its pros and cons. What I will say is that I don’t believe that my 20+ years of experience reviewing laptops entitles me to make quick judgments and pass them along to others. Using a device like this for 3 days, 5 days, a week, or whatever isn’t enough. And not to criticize my reviewer colleagues from other publications, but that’s what most of them do.

The problem is that reviews need to be timely, as well, to be truly useful. And in many cases, 2, 3, or more months will pass between the time I receive a PC or other device for review (and typically write a first impressions article) and the time that I actually publish the formal review. That’s too long. So I’ve been working this summer to shorten the delta between those two dates. I’m not sure what the “right” time frame is, and the two-week Mexico trip I’m currently on will temporarily push those times out again. But I hope to get it to within a month if possible.

Tied to that previous point, laptops, in particular, are a logistical nightmare for people like me, especially if several of them show up at my house every month, which is often the case. So this year, for the first time, I’ve started saying no to some laptop review requests from PC makers because I’m just too busy to get to them all in a timely manner and I’m now actively trying to catch up and get ahead of it.

Also, we’re planning to move in early 2022, and as part of our Great Decluttering Push of 2021, we’re getting as much crap out of our current house as quickly as possible. It’s a monumental undertaking, and the process of sorting through all the laptop boxes in the cellar, reacquainting them with the right laptops, power cords, and other peripherals and whatnot that go with them is a big part of it. As is resetting the PCs securely, packing them up, and returning them to their owners. I’ve sent back nearly 20 laptops already this summer. Yes, really. I’m a master procrastinator.

Well, I was. Now, I have a new system for dealing with laptop reviews, in particular, and with the packing materials that they come in and will be returned in. Rather than just throwing the boxes in the cellar, where they used to commingle with my own property, and were out of sight and out of mind, I keep them stacked in the corner of my office, in a literal FIFO (first in, first out) stack with the oldest laptop box on the top. (OK, that’s technically a queue, but the boxes are literally stacked. Let’s move on.) This is something I can see every day from where I sit and work, and I don’t want the stack of boxes to get too tall. So there’s some incentive for me to not procrastinate and not take months to finish a review. And when that review is done, I now reset the PC immediately and ship it back that day or the next.

As for the actual reviews that I write, I have templates for my laptop and smartphone reviews that include the major sections you see in each review—Design, Display, Internal components, and so on—and that’s another area that is overdue for improvement. This is something I’ve not yet completed, but I’ve had some recent experiences where I or someone else have noticed that I left out a key component of a particular review. The best and most recent example is my HP Elite Dragonfly Max review, in which I originally forgot to write up its excellent 5 MP webcam, which is a real differentiator in this remote work era. I added that bit to the review a few days later, but many of those who read the review right away won’t know that. Long story short, I’m working to improve my templates with reminders in each section about all the key components to cover.

My reviews tend to be pretty long—2000 to 3000 words on average, I bet—and that’s all real, written content. But some (not all) of the lengthier reviews you’ll find on other sites are only lengthy because of their egregious reliance on benchmark tests and the space-hogging graphical charts they can use. I don’t use or rely on benchmark tests, almost ever, and I don’t think that pointing out that “PC A,” with virtually identical components to “PC B,” is 1.7 percent faster at some task is in any way useful to someone looking to spend $700 to $2500 on a new computer.

Instead, my reviews are purely usage- and experience-based. I care about things like heat and noise and keyboard quality and real-world battery life, not the battery life measurement you get from repeating the same movie over and over until the screen shuts off. I think that’s a superior approach because, let’s face it, the reviewers who barely use a computer over that 3, 5, or 7 day period I mentioned above are almost always the same reviewers who pad their reviews with benchmark charts. (Again, not always.)

But reconciling my need to churn out these reviews more quickly with my need to have a meaningful experience with any particular device is tough. By definition, real-world experience doesn’t just mean using a device, it means really using a device over time. I struggle with this problem, but it’s worth pointing out that no review can provide what is arguably one of the most important criteria for a potential buyer: How well will it work over time? New PCs and other devices typically work great, they’re new. But what’s the experience going to be in a year or two, after you’ve installed multiple apps and system updates and cluttered up the storage?

The best example of how this problem bites reviewers and the people who read their missives is the Samsung Note 7, which was universally hailed as one of the best smartphones ever made by an army of reviewers who spent less than a week with the device. Soon thereafter, these handsets started catching fire and exploding, airlines banned them, and Samsung had to issue a recall. Would reviewers spending even one more week have helped prevent disasters? Yeah, it would have. But those reviewers had already moved on to the next shiny bauble. That’s the business: If you’re not timely, everyone else will be.

Another problem is that PC and smartphone makers tend to seed reviewers with their most expensive and premium products. This makes sense: Hardware makers want positive reviews, and their most expensive products will, of course, provide the best experiences. But whatever you think of my lifestyle, I’m not rich, and I can’t afford most of the devices I review. And like many of you, I’m very much interested in learning about lower- and middle-end products in addition to the occasional premium model.

To be fair, some of the hardware companies I’ve worked with do offer more affordable devices for review, and not to throw others under the bus, but Lenovo and HP have been particularly good about this lately (while still providing plenty of opportunities for higher-end product reviews too). I’ll see what I can do here: Most of the reviews I write start with a company asking me, and not the reverse. But I can still ask, and I will.

There are other more specific things we could discuss with regards to reviews, but our needs and expectations shift over time, so anything I mention here could, and probably will, sound passe a year or more now. For example, I used to refer to a so-called “sweet spot” for smartphone displays, as if this were the one thing that most (or maybe all) of us could agree on. But display sizes kept increasing and increasing, and now the only thing that’s clear is that there is no one display size—or aspect ratio, or resolution, or whatever—that’s right for almost everyone. Oops.

Likewise, I have certain fixations—my hatred of 16:9 displays and numeric keypads, for example, or the bizarro “no Thunderbolt” world in which only Microsoft Surface PCs exist—but here, again, it’s all a matter of subjective taste. It pains me to know that there are people who don’t just like 16:9 displays but prefer them and seek them out. But so be it.

Ultimately, all I can try to do is be honest, which comes easily enough with the reviews I write, and adapt over time as things change. That last bit is trickier and is arguably the high-level impetus for thinking about, and now writing about, this topic.

And while we’re being honest with each other, I should end on this note: I don’t know what the future holds. If you read the Premium newsletters that go out via email each Monday, you know about our plans to sell our home early next year, and that we’re not sure what we’ll do after that. But one big possibility—fueled in part by the uncertain terribleness of the pandemic and our love of travel—is spending some amount of time moving around, perhaps internationally, and that kind of a change would obviously impact my ability to review hardware, especially in a timely fashion.

We’ll cross that bridge when we come to it. But that’s another thing I’ve been thinking about a lot lately. What does this future, assuming it happens, mean to me even being able to write reviews? I don’t know for sure, but it probably means doing so a lot less often and maybe not at all. Or perhaps I’ll be in a position where I have a permanent base of sorts for at least part of the year and that will allow me to continue writing reviews at least some of the time.

Yes, that’s a problem for Future Paul. But it’s one I hope to solve one way or the other, and the sooner the better.