Kelly Sutton 2 days ago

AI Retrospective, Predictions

We’ve entered the 4th year of the Slop Wars. We have colorful short-hand like clanker , vibe-code , one-shot , and you’re absolutely right! . These phrases capture the zeitgeist. Emphasis on geist. It’s been more than 3 years since the release of OpenAI’s ChatGPT, which was the inciting incident that’s upended the world economy and changed how we work. This blog post provides some miscellaneous observations on AI, how it’s being used, and how it might be used going forward. I’m writing this mostly for myself to organize my thoughts, but it might be useful to others. These ideas are mostly drawn from what I’ve seen at my company, Scholarly . Between 2023 - 2025, every interface with AI was a chat interface. LLMs are next-token predictors, and the hello world of a next-token predictor is a chat interface. Words go in, words come out. We flew past the Turing Test with hardly a wave. But a chat interface is only one way of interacting with a next-token predictor. As context windows have grown and model quality increased, we can trust the model with more than the call-and-response staccato. We can ask it to go do things that take longer. The chat surface might not withstand the test of time if it’s not the most appropriate tool for the task. Chat right now is similar to 3D websites, Flash, or an under construction GIF. Novel, but potentially pointless. It ultimately comes down to this: Users don’t care if you use chat or the latest models. They care if you solve their problem. I predict the AI-in-your-face becomes muted. Certain parts of applications might feel more probabilistic because they are driven by LLMs, but still belong to the same application. You won’t be able to tell where discrete ends and probability begins. White collar jobs in general have come to the realization that software engineers have known for a few years: This is going to change how we work dramatically. At the time of writing, the Claude Code/Codex interface seems like the sweet spot of software engineering. This is higher level than where we might have collectively been last year, where it was focused on tab completion. As the models have gotten more capable, we’ve started to trust them with more. What used to be a novel time saver (tab completion) is now unnecessarily slow. The entire nature of software engineering has changed, with far less time with hands on the keyboard entering syntax. We’ve gained a very capable author, so much of our time is now spent reviewing, tweaking, asking it to come back with changes. It’s no surprise that the landscape is incredibly messy. There are incomplete integrations (Why can’t I tag Claude in my Linear ticket?) and half-baked products abound. There are companies that build something compelling, only to be obviated by an Anthropic blog post. Observing this, there will be a field of dead companies in the middle with few survivors at the edges. The model makers (OpenAI, Anthropic) will stick around, with third place subjected to the power law dropoff of consumer choice (not good!). The systems of record that participate in the AI ecosystem will be rewarded handsomely, as OpenAI or Anthropic is the conduit for white collar work. Chat has limited us to asking for tasks to be completed inline. If something was too long for the early meager context windows, it just couldn’t be done. At the time of writing, we rarely think of context windows any more. Claude Code and things like chain-of-thought blew the doors off of the context window. By keeping small artifacts, summarizing as we go (compacting), these models are able to stay on a task for much longer through self-regulation. A strange loop? There’s lots of alpha in just making webhooks work (OpenAI) or providing them at all (Anthropic). As models have gotten better and we can trust them more, we need better ways of sending them off and having them come back with a work product for inspection sooner. I predict we’ll see more ways of asynchronously interacting with models. This is also what makes it feel like a coworker replacement: we’re not chit-chatting, they are going off and accomplishing a task. Agents, skills, etc. feel like fertile ground. Agents communicating with agents, working together to accomplish a task? I’m not sold yet, but that could be where we see the next model improvements take us. Right now when I ask Claude to do something, talented engineers still stand a chance to complete the task before it does. These are engineers that know the codebase, have a high WPM, and know exactly what they are going to do. If we play the tape, I predict the models will eventually start competing on latency. Indeed, there are some things that we use AI for in our application where something taking 30 minutes is not particularly impressive. That same task taking 180ms, now that’s impressive. Between Cerebras and Taalas , there are some promising options out there. As model latency decreases, this will put strange and foreign pressure on conventional hardware. Is there a future where Claude Code is operating on my machine (or a cloud VM) and idling waiting for disk I/O? What if it’s already thought through its next 10 steps, and it’s just waiting for the host to catch up? You should know that I’m financially incentivized to believe that SaaS is not dead as we know it. Scholarly is a SaaS company for colleges and universities. If I believed that LLMs posed an existential threat to SaaS, I should get out of this business. There was a market swing this year with the ethos being: “AI makes it easy to create your own applications, and we believe companies will do that instead of going with a vendor. Therefore existing SaaS has become less valuable.” I think that’s dumb. Mostly because the running code that you buy is just a small piece of software. Are you telling me that an enterprise is going to vibe-upgrade their bespoke application to the latest operating system/library/underlying dependency? I don’t think so. If anything, there are some really exciting properties about LLMs that make being a new entrant into a space great. We are unencumbered by the past with powerful tools for helping us plug in to existing systems. So it’s not that SaaS is dead, it’s SaaS that doesn’t adapt is dead. But that’s always been the case, maybe just more urgent now. AI is having profound impacts on white collar work. Many of the layoffs we are seeing in 2026 may be intertwined with the hiring glut of ~2020. Extrapolating my own behavior, I suspect many service aspects of white collar industries are seeing a softening in demand. Lawyers, for example, I bet aren’t hearing from their clients as much. They are still fielding the important interactions where expertise is required, but a quick turn on a simple contract or proof-read of an NDA might just go to Opus 4.6 or GPT 5.2 for many people. Suddenly or slowly, the billable hours slip. Nothing concrete here, just a hunch. Not sure how it plays out. I think MCP is dumb, but it’s what we’re using. I expect it to stick around for a few years, and then we’ll fall back to more traditional REST APIs. Kind of similar to what happened with GraphQL. One of my favorite podcast episodes ever is the episode on spreadsheets from Planet Money . In it, they discuss how the invention of the digital spreadsheet put bookkeepers (the keeper of paper spreadsheets) out of work. But out of that pain new work was born around financial modeling. Rather than taking a day, it took seconds to answer the question, “What if we decreased our costs by 5%?” Work in predictive financial modeling was born. An entire new job came to be, and with it many positions. Even more than bookkeepers. We’re experiencing a similar creative destruction currently. Lots of what we knew is being destroyed before our eyes and new opportunities are being born. Special thanks to Claude Code for suggesting a few edits to this post.

AI Programming

▲ 0

0 views

Kelly Sutton 7 months ago

How We’re Working, 2025

This blog post goes into detail on how our company, Scholarly , works. This post should serve as a bit of a memory capsule and a good thing to send potential candidates when they are evaluating us. I hope this blog post is an interesting view into what building B2B SaaS can look like in a way that is traditionally agile: personalized, great software built in cooperation with our customers. No focus on frameworks, tools, or processes. This blog post is a follow-up from the post last year, “How We’re Working, 2024” . We are hiring at the moment, which is part of what motivated me to update this blog post. You can see our open roles on our About page. I’ve left parts of the original blog post inline, but block-quoted them. Anything unquoted is new. Hopefully this gives a nice pseudo-diff view of what’s changed over time. For example: Here’s what we did last year, and still do this year. Here’s something new this year. We’re a young company of 12 at the moment. 4 engineers, 3 folks on ops, our sales team, myself (CTO), and Rusty, our CEO. Our day-to-day can look different every day, it really comes down to what the biggest priority is for the business at the moment. We are located in Denver and Seattle, with our sales folks spread throughout North America. We’re continuing to grow the company in Denver and Seattle. We have been in business since June, 2023. We sell to universities within the US. There’s a very specific set of individuals that we sell to within a university. No HRIS or competitor serves them well at the moment. They are really underserved but serve one of the most crucial functions of a modern university. We continue to collect customers, and we’re starting to see customers trust us with larger parts of the faculty lifecycle. In the last 12 months, we saw our first major public institution and our first law school sign. Universities put a premium on what we do, so we are definitely into the “enterprise-level” ticket prices of B2B SaaS products even with our nascent feature set. Much of what our customers buy is the relationship with us, putting their trust in our ability to continue to develop and deliver a valuable product for them. We’re starting to see the tailwinds of this trust that we’ve built up, with all of our customers going to bat as references! It’s hard to overstate how detrimental the newest political administration has been for our customers. The freezing of NIH and NSF funds caused us to wonder if we’d even survive. Scholarly resonates with research-heavy, data-driven organizations. The NIH/NSF funding freezes and cuts affected those institutions. Our app provides a source of truth for departments, colleges, and universities that makes it easier to manage faculty. Management of faculty centers around specialized workflows. These workflows might be annual performance reviews, leave or sabbatical processes, or hiring processes. In the last year, we’ve poured a lot of investment into our workflow engine. It’s now self-service and more capable. What used to require engineers writing code for each workflow can now be configured through our UI or, better yet, our AI. This AI-powered workflow setup saves administrators weeks of time. Our product is a SaaS application written in Ruby on Rails and hosted on Heroku. We use PlanetScale for our database and still use as little JavaScript as we can . Due to customer volume, we recently had to add not just a web dyno but a Sidekiq dyno too! We invested in the last year in building our own design system and component library, but still make use of Tailwind and TailwindUI where necessary. We deploy every time the build goes green, automatically. Every university is different, so the technical depth of the application comes from the sophistication and flexibility of our data model. We adapt to every customer’s data without compromising the integrity required to build sophisticated reports and consistent workflows. We’ve also invested heavily in our reporting capabilities, AI Assistant, and other parts of the product in the last year. Our newly-minted blog recaps product improvements we ship every month. Every piece of work we do begins with the customer. We source problems, come up with solutions, show them a prototype or a riff, put some polish on it, make sure it meets their needs, and then check in a few weeks later to make sure it’s still meeting their needs. We spend a lot of time talking to customers. This was how we worked last year, and it’s still largely how we work this year. However, this year we’re putting more effort into systems to scale our efforts internally. We’ve switched to Linear for project management and had a brief pit stop using Basecamp. Linear’s a great application and is serving us well. We use the “Cycles” feature extensively, planning out what work we plan to do that week. For customer-facing project management and communication, we’ve transitioned our customers to Slack. Many institutions use Slack and we’ve found the quick messaging invaluable for keeping project velocity high, especially during implementation. Since last year, our customer management has gotten more hands-on as our team has grown. Our operations team manages the relationship with customers, and is keeping a much closer eye on implementations than Rusty and I can. Project management still follows a simple kanban approach, but the tickets have accumulated a lot more tags these days. We work off of the topmost tickets, deciding the rough ordering at the beginning of the week. All members of the team are welcome to add things to the top of the board as they find bugs, potential improvements, etc. We try to represent all work as tickets, but don’t always succeed. We’re leaning more into feature flags and some other tools for making it easy to test in production without harming the customer experience or corrupting customer data. Where we had originally started integrating every data source “by hand,” we’ve started to build systems around common integrations that we’ve seen. We did things that didn’t scale and now are reaping the rewards. We’ve got a growing API that more customers are tapping into. We have some solid rails for working with common systems found in higher ed. We’ve put together APIs that are compatible with competitors’ systems to make switching easier on university IT teams. We’re finding the balance of how involved engineering should be in implementation. We’re building more and more tools that our operations team will crank on and inevitably break. Our goal is still to put many of the powerful tools in front of customers so that they can self-serve the configuration and operation of the platform. For many of the most powerful pieces of the application, these are still ops-only tools. It’s still important for everyone in the company to deeply understand the customer pains. We also ask everyone to understand the business bottlenecks, since the best ideas can come from anywhere. We ask everyone to stay plugged in because it helps us learn, and then bake those learnings into the product. We’re having a lot of fun, and so are our customers. We get to work together to build software that solves very real problems of theirs. They like paying us. They are willing to recommend us to other institutions and peers out there, which really greases the wheels of the next sales conversation. Our ability to ship something minutes after a customer requests it gives us a competitive edge. At the competition, customers are encouraged to pay an additional fee on their contract to hire an “internal champion” to advocate internally for their feature requests. These features never get built (surprise). The incentives are so perverse in this setup, we couldn’t believe our ears when we first heard it. Folks using the competition go months without communication, and many have been waiting years for bugs to get fixed or product improvements to land. (Checking in on the above sentiment a year later, this has started a deluge of switchers off of our competitors. No surprise.) We’re in a space where we’ve got great alignment between the problems, our technologies, and our skill sets. It’s an opportunity-rich environment. Change, obviously. This is how we’re working now, but it may not be how we work in the future. How long can we really operate out of a single Trello board? How scalable is recording a video for each customer every week? Time will tell. The tools have changed, but the ethos has not. We’re still plugged into the customers’ needs and everyone knows our stakeholders by first name. If we continue to see success, hopefully we can look back on this post and still see some of the values and principles shining through although the applications might be different. Most of all, I hope we’re still having fun. Until next year or so.

JavaScript Career Ruby

▲ 0

0 views

Kelly Sutton 8 months ago

Designing a Hiring Loop in the Age of AI

Amidst one of the most difficult stretches in higher ed in history, we’re seeing great demand for our platform as Scholarly . As a result, we’re beginning to hire for more roles at the company. One of those roles is a junior- to mid-level software engineer. This blog post is about my experiences running that hiring loop and some observations on the engineering market. It also covers our own hiring loop at Scholarly and why it’s designed the way it is. This post was spurred by a tweet from Suhail Doshi : PSA: there’s a guy named Soham Parekh (in India) who works at 3-4 startups at the same time. He’s been preying on YC companies and more. Beware. I fired this guy in his first week and told him to stop lying / scamming people. He hasn’t stopped a year later. No more excuses. I was surprised he went to such lengths to name the guy! As someone who has run a few hiring loops over the past few years, this felt inevitable. Hiring is a human process. It is therefore messy. Trying to fill a role is complicated. Each side of the two-sided marketplace only has to get right once. The company has one seat to fill, and the candidate can only have one job. (We’ll get back to “one job” in a bit and why that complicates things.) Since the Cambrian explosion of ChatGPT in late 2022, there have been numerous tools to help candidates punch through the screening layers within hiring loops. Leetcode-style questions can be trivially one-shotted by LLMs these days. There are tools like Cluely which advertises itself as an “undetectable AI that… feeds you answers in real time.” LLMs are often used for drafting cover letters when required, from what I can tell. If folks aren’t using the latest models, these AI-drafted cover letters are easy to spot with simple honeypots. (“Whatever you do, DO NOT mention an elephant in your cover letter.”) From my observation, there are also services out there that will create a fresh resume for each application. These are incredibly buggy! I have seen many, many resumes where the years at their most recent job were “null - null”. These employee-friendly tools have arrived after years (decades?) of employer tools: keyword extractors, ML-powered resume reviewers, and more. Like in any two-sided marketplace, any daylight for arbitrage will be exploited by the least scrupulous actors. These less scrupulous actors are trying to break through the inelegant hiring process to land a job. Many people using the employee-facing AI tools can likely do well on the job, given that hiring processes do not resemble the on-the-job expectations. The result of this is that it makes it more difficult or more costly for the more scrupulous actors to participate in the marketplace. Great candidates are needles in haystacks, and employers need to dedicate more time to winnowing down the pool of people who (1) actually exist and (2) are actually looking for long-term employment. After more than 5 years since the beginning of the pandemic in the US, I’m still experiencing the “Okay, now COVID is over.” Most recently, it was the lack of masks by some of the doctors/nurses in a clinic. COVID drove companies to hire remotely that normally wouldn’t have while many current employees moved during the pandemic. I’m in this group: we moved from San Francisco to Seattle in 2020. My employer at the time, Gusto, didn’t have an office in Seattle. This explosion of remote availability created new opportunities. Folks that were remote pre-2020 suddenly had a much larger pool of employers to choose from. That’s great! Employers finally woke up to the fact that not all great software engineers live in 3 or 4 specific cities within the country and can indeed be found just about anywhere. This new opportunity also brought new methods of arbitrage. The Overemployed (OE) subreddit was created in May 2021. Being overemployed is the practice of working for more than one job at once without the employers knowing. There are different versions of being OE. Sometimes employees are simply cycling through jobs, existing just long enough to get PIP-ed and fired while collecting a paycheck for 3-6 months. Other OE folks work those jobs honestly, meet the expectations, and their employers cannot tell. Being OE can have enormous financial impacts for the individual. Collecting concurrent paychecks without adjusting one’s lifestyle can lead to some real hyper-accumulation of wealth in a short amount of time. If the employers can’t tell that their employees are OE, why should it matter? OE exposes inefficiencies in an employer’s operations, where they are paying for an honest 40 hours but not receiving it. Perhaps they would be better off with a 10-hour contract per week at a higher, contractor rate for that individual since that’s what they are receiving. If you spend some time on the subreddit, you’ll realize how deep the OE culture goes. Mouse jigglers. Strategies for moving meetings. What types of roles to apply for. Things to tell your manager when confronted. And honestly? If an employer can’t tell, more power to them (the employees). Overemployment can’t be the only reason for the layoffs in tech we’re seeing, but it may be a contributor. Layoffs and overemployment might be sibling symptoms of a deeper fact: there’s lots of slack in tech companies. The result of all of this is that companies hire fewer people and force return-to-office (RTO) mandates. If you’re a remote employee working for one employer, you’re bearing the brunt of employers trying to close these arbitrage gaps. Given the above AI tooling, OE, and more, how would one design a hiring loop for a software engineer in 2025? We don’t have all of the answers here at Scholarly, but here’s what we’re working with for now. First, we’re only hiring for in-office roles. This limits our pool of potential employees. However, we think this trade-off is worth it. Our existing remote software engineers are grandfathered in and will remain remote. Our in-office requirement isn’t solely driven by the above factors but they do contribute to our decision. Our goal is to create a hiring loop that is human. It’s respectful of everyone’s time while giving us a good signal on who we think will be the best candidate for the role. We’ve run a version of this loop a few times, and the feedback has been mostly positive from candidates. Here’s our hiring loop: All portions of the interview except the lunch can be done remotely. We’ll either do them the same day or spread them out over a few depending on the candidate’s preference and our own availability. Each step is gating for the next, meaning if a candidate doesn’t pass one they do not advance. This loop is partially designed to protect the time of our own team. We ask candidates if they are currently located in Seattle and if what they like about the company. About 50% of applications do not answer these questions, so they are immediately eliminated. Another 10%-20% lie (claiming they are located in Seattle but their resume or LinkedIn disagrees) and are also removed from the pool. This mostly covers the logistics of the interview and what to expect next. This is also a good chance to confirm that the candidate is either located in Seattle or willing to move. We also ask what projects at their previous employers they are most proud of. I haven’t fed this question through Cluely or similar tools, but I feel like this gives a pretty human answer and is harder to fake. This section is designed after how one of my college professors, Dr. Toal , ran his final exams. You could use any tool you wanted (books, Stack Overflow, notes) but they were the hardest goddamn exams in the curriculum. Still have nightmares about those. As a cofounder of Scholarly, I want you to be the most effective you can be without artificial constraints. I want this because this is what the job is going to be like: making the best possible product for our customers with the tools at our disposal. In the interview loop, I want to see what you can do. This session starts with a quick intro and a Gist of a problem statement. This problem is sourced from something we built at Scholarly for our customers, so it’s relevant to the day-to-day work. It’s a big problem but I’m always surprised to see how much progress great candidates make. After receiving the prompt and answering any of the candidate’s questions, I drop from the call and the candidate cooks for 90 minutes. At the end of 90 minutes, I hop back on the call. The candidate walks me through their progress and we have a discussion. Can you tell me about [design decision X]? If you have 10 more hours to work on this, where would you invest and why? I love these conversations, because it really creates the space for the candidate to shine. It can provide signal on everything from code hygiene to architecture to creative problem solving. Failing answers here include: This is the only hands-on-keyboard technical aspect of the interview. This interview section dives into other technical details and design decisions of the candidate’s career. It’s also a great chance for a candidate to get an unfiltered view on our engineering culture and what life is like as an engineer at Scholarly. We’re still tweaking this part of the interview loop for every role we make. Software engineers at Scholarly are full-stack and very customer-focused. All engineers are expected to interact with customers regularly and ensure that what we build meets their needs. Engineers are expected to seek out feedback proactively and find creative solutions to customer requests. To get signal on this, we have candidates chat with our CEO ( Rusty ) and our Head of Operations ( Kari ). They spend the bulk of their time these days interfacing with customers so speaking with our engineering candidates is an important part of the process. For candidates, who doesn’t love a free lunch? We’ve got some great options in Pioneer Square. For Scholarly, this is a good chance to make sure you’re actually human and live in the area. It’s also a good “vibe check” for lack of a better phrase. Great signs here are when the conversation flows smoothly and we all learn about each other. This is also a good time for the candidate to ask, “Can I see myself working with these people?” For folks that make it through the whole loop and pass, we move to the offer stage. We evaluate the candidates that have made it to this stage and extend an offer to our favorite. That’s our hiring process and some of the reflections on the market today. Our process is designed against the backdrop of being more employer-friendly than employee-friendly. Pre-2022, this process likely wouldn’t have worked since there were so few candidates for open roles. Our hiring loop hopes to embrace the chaos of the current AI-assisted marketplace while trying to find great candidates through a realistic and human process. The rise of LLMs has shattered many norms, hiring processes being one of them. As with any new technology, things must change. What never changes though is the need to dive into the nuance and find a process that works. Resume submission/application A 30-minute Zoom call with the hiring manager A 2-hour “take home” technical interview A 45-minute conversation with another engineer A 30-minute conversation with our CEO and VP of Operations Lunch with our engineering team in Seattle Not making sufficient amount of progress, although “sufficient” is calibrated against other candidates. Some version of: “I can’t explain this, because I don’t understand the code the LLM created.”

AI Career

▲ 0

0 views

Kelly Sutton 1 years ago

Letting Others Get Rich

This is a blog post about housing in Seattle, but I hope it leaves the reader with new perspectives in their own cities when it comes to addressing affordable housing and homelessness. We should remove policies that try to decide how and when housing developers profit, because that thinking does more harm than good. I had the fortunate opportunity to spend nearly 7 years of my career at a technology company called Gusto in San Francisco. Gusto provides payroll, benefits, and HR tools to small businesses in the US. During my tenure, Gusto grew from about 20 software engineers to several hundred. It was beautiful chaos. We were constantly trying to invent the future while reinventing ourselves. The organization eventually grew to the point where it could stomach funding multiple, competing priorities. Two bets that were being made at the company were essentially “Will we continue to build our code in a monolith?” and “Will we write code in microservices?” I was Team Monolith and spent more emotional energy than I would like trying to discredit Team Microservices. 1 It didn’t matter and it wasn’t my decision to make. Antagonizing my colleagues on Team Microservices harmed my own reputation and some of my career prospects at the company. Given a few years of reflection, I realize my error was in trying to police others’ behaviors instead of giving them the space to fail or succeed on their own performance. Spending emotional energy, political capital, or whatever you want to call it finding the right way instead of betting on the emergent capability of successful innovation is a failure. In a sense, I was preventing others from getting rich rather than practicing a “same team” mindset. Regardless of who was successful in these competing bets, the company (and my stock) would be better off for it. I should have kept my mouth shut and focused on my own efforts. This led me to an important realization: in a growing and successful startup, you will watch people you like and people you dislike getting rich alongside yourself. 2 Policing who gets rich and on what terms would consume a young startup. You gotta live with it. In South Park, which has had [more permissive] zoning since 2019, developers are building up to six separate, small houses on a single lot, with no room for shade trees to grow. The effect is a colorless, charmless, heat island that makes money for developers at the expense of everyone else. You wouldn’t want to live next to it, and chances are, in the next heat wave, you wouldn’t want to live in it, either. Seattle Times Editorial Board, Dec 5, 2024 3 I recognized a familiar mindset while drinking my coffee and reading the Sunday paper last month. This sentiment is identical to my prior feelings of Team Microservices at my former employer. It says, “You can’t get rich unless I say so.” This belief is incredibly destructive to the life of a city or a company. Even permitting this we-should-decide-who-gets-rich argument is unscalable; given the choice we’d probably get N different opinions for N people in the city. Gridlock. While some might bristle at the idea of housing developers becoming obscenely wealthy, value is accrued by everyone in the city as more housing is built. This also betrays some of the truly great cities out there. I don’t think of New York, Amsterdam, London, Tokyo, Berlin, et al. as being especially green. Their greenery tends to be concentrated into great parks located near population centers. Uniform greenery is a unique attribute of Seattle, but I believe we can maintain that character while permitting more housing. We still have the Arboretum , Discovery , Seward , and more. All of Seattle will not be Manhattanized overnight, but I believe the city would benefit to lean into being Brooklynized. Seattle is one of the more difficult places to construct new housing in the country. According to a 2006 study by Wharton, it’s slightly behind San Francisco in terms of regulatory complexity at the sixth position nationally. 4 Without even looking at the data, one can reason that a longer process for getting housing approved increases the cost of housing. Assuming that there are people with salaries behind each project, longer timelines mean more salaries to be paid. For housing to get built at all, the costs (i.e. those salaries) need to be less than the revenue generated by the new building. Efforts to police what type of housing can be built and how it can be built might even make developers richer! In Seattle, there are 8 design review boards that are part of the process of permitting new housing. 5 These design review boards have authority over a building’s “form, massing, and architectural design”. They do not have anything to do with the safety of the building, a responsibility handled by the Department of Construction and Inspections (SDCI). The mere existence of these design reviews adds to the pre-construction timeline. This logically increases the cost of new construction. Assuming no changes are requested to the design, this adds at least 4-6 months to the process in the simplest case. 6 Why does a design review board get to dictate the look of new housing? Few developers likely want to build an ugly, inaccessible building. They would have trouble selling or renting units. That would be a bad use of funds! Entrusting the look of a building to a static group of individuals narrows the possible results of what housing could look like in the Seattle. This tastemaking function in the permitting process counterintuitively creates the “charmless” city that the Seattle Times editorial board bemoaned. Because the design review has nothing to do with the safety or logistics of new construction, it should be eliminated entirely. We should expect to see shorter housing construction timelines, which will result in more affordable homes for all. What we need in west coast housing policy is comfort with higher variance while not sacrificing safety. Rather than trying to find a “better” version of the design review or policies like it, get rid of them. Let builders go wild and experiment. This should result in a higher variance of housing and increased competition. Rather than be constrained to the 5-over-1 template that the current process produces, we should expect to see more varied construction throughout the city. With these varied options, you might see something that offends your sensibilities. That’s fine! You are not compelled to live there! With minimal constraints, we should see a broader range of architectural and financial models competing to attract residents. The best rise to the top, eventually setting new standards. By enabling higher variance and competition among developers, we would also see some developers getting obscenely rich. They might find a format that everyone wants to live in and call home. We should also see just as many developers striking out and producing mediocre buildings. Over time, the mediocrity will get replaced with better buildings. We have to be comfortable with letting those developers getting rich, and not try to police who gets the spoils of taking a risk. I’m convinced that this high-variance, anything-goes housing policy is part of what makes Tokyo the object of millennial obsession. Tokyo has big apartments wedged next to tiny homes, in a patchwork that feels organic and serves its citizens. It’s beautiful chaos. We should look to cultivate our own version of this beautiful chaos for the betterment of the city. Seattle has also voted for social housing initiatives, such as I-135 in a 2023 special election . This creates a social housing developer authority to create housing for folks earning 120% or less than the median income of the area. Let’s do this, as long as it doesn’t come at the expense for our ability to build housing privately. Let’s assess its success at providing housing alongside other alternatives. Let’s support these things in a portfolio of other experiments and investments. Beautiful chaos has space for a public option. We should not view this as a replacement for privately-funded, speculative housing endeavors. Much like the United States Postal Service (public) competes alongside private alternatives (FedEx, UPS), we should create the space for public alternatives. Yes to social housing experiments, yes to private market experiments, and let’s measure results side-by-side. Seattle is in the process of collectively admitting defeat in its approach to housing and homelessness. I was surprised to see The Stranger publish “Where the Left Went Wrong on Homelessness” this month. The Stranger, for those without context, describes itself as a paper that “defines, defends, and sometimes defies the culture and politics of Seattle.” Hopefully I would not be out-of-line as characterizing it as a more left-leaning publication. This article being published in this venue signifies a willingness to admit defeat in the current policies on housing and homelessness. What we’ve tried isn’t working. It’s time to call this a failed experiment and move on. It’s good to see that saying this is politically viable in the only state that became more blue in the 2024 election. 7 Contrary to what the current administration might message, living in a blue state does not mean we lack brain cells. Now is a great time to assess past sentiments and practices, and move towards one we feel will serve citizens better. It’s time to get rid of the developers-are-evil meme. This meme obstructs our ability to build affordable housing. We may also choose to pay more attention to the impact of policy, not its intent. Design review comes from a good place but achieves the opposite of its goal. A city that is less fixated on choosing which developers get to earn how much is a better, more vibrant city. Seattle and the surrounding area are unique economically. The city has more varied industries than it did in the late 1960’s, when Boeing’s downsizing resulted in a 25% unemployment rate. We should embrace this attention and opportunity to absorb folks wanting to move here. The city will continue to change, and we should welcome that. This will result in those that build housing making money hand-over-fist. That’s fine. If that bothers you, think of it as the price to pay for having an ever-more vibrant city. We should remove the policing of who gets to get rich as a valid opinion in discussions about housing. Seattle is no New York, Berlin, or London. But it might grow into one. Let’s not get in our own way of making that happen. Special thanks to Richard D. Worth for catching a typo. I had erroneously dated the editorial board’s article as 2025 instead of 2024. It has now been corrected. What a monolith and a microservice is isn’t important to this post. You can replace them with “Apple” and “Banana”.  ↩ Compensation in a startup is centralized in its equity. You get more equity the earlier you join, since you have a bigger impact on the future trajectory of the company. Many people on Team Microservices had been at the company longer than me. Every time my stock’s value grew $1, theirs probably grew $10.  ↩ “More paving, fewer trees. So much for a green ‘One Seattle’” , Seattle Times, Dec 5, 2024  ↩ “A New Measure of the Local Regulatory Environment for Housing Markets: The Wharton Residential Land Use Regulatory Index” , 2006  ↩ Design Review - Boards & Staff , Seattle.gov  ↩ “Is It Time for Seattle to Do Away with Design Review?” , Publicola, May 3, 2022.  ↩ “Why Washington appears to be the only state to shift blue in 2024” , Axios  ↩ What a monolith and a microservice is isn’t important to this post. You can replace them with “Apple” and “Banana”.  ↩ Compensation in a startup is centralized in its equity. You get more equity the earlier you join, since you have a bigger impact on the future trajectory of the company. Many people on Team Microservices had been at the company longer than me. Every time my stock’s value grew $1, theirs probably grew $10.  ↩ “More paving, fewer trees. So much for a green ‘One Seattle’” , Seattle Times, Dec 5, 2024  ↩ “A New Measure of the Local Regulatory Environment for Housing Markets: The Wharton Residential Land Use Regulatory Index” , 2006  ↩ Design Review - Boards & Staff , Seattle.gov  ↩ “Is It Time for Seattle to Do Away with Design Review?” , Publicola, May 3, 2022.  ↩ “Why Washington appears to be the only state to shift blue in 2024” , Axios  ↩

Career Business

▲ 0

0 views

Kelly Sutton 1 years ago

Moving on from React, a Year Later

It’s been a busy year for Scholarly. At this time last year, we were just closing our first contracts after having been in business for 6 months. Since then, we’ve raised a seed round , achieved SOC 2 Type II compliance , grown our customer base substantially, and expanded our team. On the technical side, we continue to cut out vectors of accidental complexity by keeping our technology choices simple. One of the blog posts I wrote last year, Moving on from React , resurfaced in conversations on Twitter this week. This post is a retrospective on what’s changed since we wrote that post and reflects on that decision. Since writing the original post, our technology stack has remained largely consistent: Rails, Stimulus, MySQL all in a server-rendered context. We use Turbo and ActionCable at times to add an extra layer of interactivity and responsiveness when required. With Turbo, we utilize prefetching and page caching. This stack feels perfectly suited to our business. You can best think of Scholarly as an HR platform for universities. Here’s how our LOC has changed over time: A few reflections on the numbers above: Maybe it’s the changing interest rates or political winds, but I think the “fat client” era JS-heavy frontends is on its way out. The hype around edge applications is misplaced and unnecessary for building many different flavors of successful businesses. Many interactions are not possible without JavaScript, but that doesn’t mean we should look to write more than we have to. The server doing something useful is a requirement for building an interesting business. The client doing something is often a nice-to-have. One of the many ways this matters is through testing. Since switching away from React, I’ve noticed that much more of our application becomes reliably-testable. Our Capybara-powered system specs provide excellent integration coverage. Because our app is primarily server-rendered and JS takes a backseat role, our system tests flake anecdotally at a much lower rate than other single-page applications I’ve worked on in my career. I’m a little out of the loop, but unit-level testing in JS has always been a bit of a lie: you’re not working with the DOM. If you are, you aren’t seeing the whole picture. Pushing most of the application logic to the server has made our application more testable at a higher level, without cumbersome client-server orchestration to get the full experience represented in test cases. One of the arguments for a SPA is that it provides a more reactive customer experience. I think that’s mostly debunked at this point, due to the performance creep and complexity that comes in with a more complicated client-server relationship. We work with Nate Berkopec from time to time to help us keep an eye on performance. During one of our sessions with him, we were browsing some of our performance metrics. When looking at our route change metrics, he didn’t believe his eyes. Our p50 route change time is 86ms over the last month with a p75 at 350ms. This is due to Turbo prefetch in action. Our server response times are reasonably snappy, so by the time a user clicks we’ve already got most of the page. And best of all, we don’t need to carry the mental overhead of state management on the frontend to enable this experience. Everything is just a page of HTML with some JS sprinkles, so there’s no state to maintain between route changes. There’s no complicated state management on the client. Code is not an asset, it’s a liability. – The wisest programmer probably ever When we view the lines of code as a liability, we arrive at the following operating model: What is the least amount of code we can write and maintain to deliver value to customers? When thinking about the carrying cost of different lines of code, maintaining different levels of robust tests reduces the maintenance fees I must pay. So, increasing my more-difficult-to-test lines of code is more expensive than increasing my easier-to-test lines of code. These liabilities become realized costs when we need to change the code. Change comes with risk, and changing untested code has a higher regression risk. We can compensate for riskier changes with moving more slowly or methodically. Moving more slowly is fine in many cases. But when compared to a world where that change isn’t risky (server-rendered ERB), we are suddenly paying a very pricey tax for making changes in JavaScript. For a young company it’s doubly expensive: There’s the lost time on the change itself plus the opportunity cost of something else we could have worked on. Let’s assume that making a JS change costs twice as much time a Ruby change, which I think is being generous to JS. So we can either… The time investment required to safely make changes in a JavaScript application eclipses that of a server-rendered Rails application. This results in a less-competitive business, unhappier customers, and unhappier developers. Well, we’re sticking with Stimulus and Turbo, that much is for certain. So far, we haven’t come across a scenario where we’ve had to author a worse experience because of these technologies. Our server-rendered, JavaScript-light approach has delivered a faster and more reliable experience. Our engineers love working in this stack as well, often remarking how effective their efforts are. We’re still a young company with plenty of other existential risks. If we fail, it will likely not be due to our choice of technologies. We can, however, do our best to remove that possibility. Thanks for reading. Until next time. We have 61k LOC of Ruby code and 4.3k LOC of JS I’ve added a linear trendline to simulate where we might have been with React. Given my experience with React, this is probably Honest™ for the sake of the argument. You can clearly see where we made the cutover from React to Stimulus in August 2023. Our Ruby code continues to grow superlinearly, which is expected given the addition of engineers, customers, and features. We had 3,690 lines of React code on August 9th, 2023. We didn’t re-cross that threshold until January 3 of this year. Make a single JavaScript change might cost 2 units of time. In that same time, we could make two Ruby changes (1 unit each), or one Ruby change followed by immediate customer feedback (1 + 1). Alternatively, we could make a Ruby change and still have time to sip a mai tai (1 + 1).

Business HTML Ruby Web Development JavaScript

▲ 0

0 views

Kelly Sutton 1 years ago

How We’re Working, 2024

This blog post goes into detail on how our company, Scholarly , works. This post should serve as a bit of a memory capsule and a good thing to send potential candidates when they are evaluating us. I hope this blog post is an interesting view into what building B2B SaaS can look like in a way that is traditionally agile: personalized, great software built in cooperation with our customers. No focus on frameworks, tools, or processes. We’re not hiring at the moment but hope to be soon. We’re a young company of 4 at the moment. 2 engineers, myself (CTO), and Rusty our CEO. Rusty handles the sales, the rest of us handle building stuff. We are located in Denver and the Pacific Northwest, with a small office space in Seattle. We hope to grow the company in Denver and Seattle, but are open and familiar with remote. We have been in business since June, 2023. We sell to universities within the US. There’s a very specific set of individuals that we sell to within a university. No HRIS or competitor serves them well at the moment. They are really underserved but serve one of the most crucial functions of a modern university. We are lucky enough to have collected quite a few customers in our first year, many of them name brands that we’re not quite ready to talk about yet. Universities put a premium on what we do, so we are definitely into the “enterprise-level” ticket prices of B2B SaaS products even with our nascent feature set. Much of what our customers buy is the relationship with us, putting their trust in our ability to continue to develop and deliver a valuable product for them. Our app provides a source of truth for departments, colleges, and universities that makes it easier to manage faculty. Management of faculty centers around specializes workflows. These workflows might be annual performance reviews, leave or sabbatical processes, or hiring processes. Our product is a SaaS application written in Ruby on Rails and hosted on Heroku. We use PlanetScale for our database and use as little JavaScript as we can . We use Tailwind and TailwindUI. We deploy every time the build goes green, automatically. Every university is different, so the technical depth of the application comes from the sophistication and flexibility of our data model. We adapt to every customer’s data without compromising the integrity required to build sophisticated reports and consistent workflows. Every piece of work we do begins with the customer. We source problems, come up with solutions, show them a prototype or a riff, put some polish on it, make sure it meets their needs, and then check in a few weeks later to make sure it’s still meeting their needs. We spend a lot of time talking to customers. Our entire product and engineering process mostly operates out of a single Trello board. Tickets range in size from copy fixes to meaty features. For customers that use Slack, we set up shared channels between our organizations. At the end of each week, we try to record videos for our customers of what’s improved in their account that week. We also use those videos to higlight idiosyncrasies or things we’ll want to discuss the next time we get together. Our app has an ever-present “Submit Feedback” button which customers can use to send us notes about the app. Those notes get dropped in our main Slack channel for all to see. When meeting with a customer, we pop open our Trello board. Each card has a tag for the customer. We filter down to just their cards and go through the list. We make sure the list of work to do is comprehensive. From there, we ensure that the list is ordered in a way that is most valuable to them. The most important stuff goes at the top. When picking up new work, we pull from the top of the list. This simple kan ban approach has been a remarkable learning so far. We try to interleave different customers’ tickets so that everyone gets a little bit of progress. We pull up the ones that will hit multiple customers or are required for a customer-defined deadline (e.g. “Our performance review process kicks off March 1, so we really need this feature by then!”) Rather than promising big bang feature sets, we’ve created a relationship where the throughput of value remains constant. They can and have changed prioritization week-to-week, and we’re right there with them to roll with the punches. Since we are young, we are consciously doing everything “by hand.” We integrate with a lot of different systems like Workday and other HRIS’s. We will pull data from Excel, CSV, or scanned-in documents dating back to before anyone in the company was born. We always work with this data by hand to start, pulling it into our application. This gives us a better sense of the shape of the data as well as exercise our own application. Once we feel we’ve learned enough, we’ll start to build internal tools to solve for the bottlenecks. This usually has to do with implementing a new customer, where we might be pulling in tens of thousands of records spanning decades. Eventually, we put these tools in front of customers so that they may self-serve and drive their own accounts. It’s amazing how much of the app just has the “R” in a CRUD interface and that’s okay; some things only change once during onboarding. Over time, we fill out the C, U, and D as necessary. We do things in this order because it helps us learn, and then bake those learnings into the product. None of us come from a background in higher ed (other than attending), so learning is an incredibly important part of our product development process. We’re having a lot of fun, and so are our customers. We get to work together to build software that solves very real problems of theirs. They like paying us. They are willing to recommend us to other institutions and peers out there, which really greases the wheels of the next sales conversation. Our ability to ship something minutes after a customer requests it gives us a competitive edge. At the competition, customers are encouraged to pay an additional fee on their contract to hire an “internal champion” to advocate internally for their feature requests. These features never get built (surprise). The incentives are so perverse in this setup, we couldn’t believe our ears when we first heard it. Folks using the competition go months without communication, and many have been waiting years for bugs to get fixed or product improvements to land. We’re in a space where we’ve got great alignment between the problems, our technologies, and our skill sets. It’s an opportunity-rich environment. Change, obviously. This is how we’re working now, but it may not be how we work in the future. How long can we really operate out of a single Trello board? How scalable is recording a video for each customer every week? Time will tell. If we continue to see success, hopefully we can look back on this post and still see some of the values and principles shining through although the applications might be different. Most of all, I hope we’re still having fun. Until next year or so.

JavaScript Ruby Career Business

▲ 0

0 views

Kelly Sutton 2 years ago

Moving on from React

Early in my career, I fancied myself a frontend engineer. Frontend engineering felt like it had more going on that backend or application engineering. The relatively young language of JavaScript was seeing more uptake after the fall of Flash. Seeing the fruits of your labor in a web browser rather than a commandline or GUI was more rewarding to me. So, we were all writing JavaScript. I remember when the industry had its first consolidation around jQuery when it came to writing cross-browser capable code. MooTools , Prototype.js , et. al faded slowly into obscurity. Shortly after, a surge in new frameworks came about designed more toward writing full applications (single-page applications) in the browser. These were Angular , Backbone.js , Ember , and React (although it wouldn’t be open-sourced for a little while longer). I hitched my wagon at the time to Ember, since I viewed it as the Rails of the frontend: fully-featured, not controlled by a large company, opinionated, and modern. Participating in the Ember community made me several friends and colleagues who I still keep in touch with. Ember saw some popularity for a few years, but was entirely eclipsed by React. React’s success—in my mind—came from its ability to integrate with existing applications in a more sensible way. It played well with the “sprinkling of JavaScript” approach. You could turn just parts of the page interactive by mounting React components to the right element. It took me some time to realize that how minimal React was made it easier to adopt as a large organization. The onramp to it was smooth and linear. Compare that with frameworks like Angular and Ember, which had a more all-or-nothing approach. The adoption path looked a lot more like a step change than slow integration. This all-or-nothing design meant that the Strangler Fig pattern was much more difficult to apply. Being adaptable to this pattern seems to be an immutable requirement at any large organization. So, the industry consolidated around React. Parallel to this, I had joined Gusto in 2016. Gusto had started out on Backbone.js in ~2011 and had made the decision to move to React in 2015. Progress was slow-going, as we didn’t have a team dedicated to the effort. Our efforts to get us off of Backbone.js and onto React were ad-hoc and best effort. We liked the React mental model a bit better, as it prevents a certain class of error that might be easier to make in Backbone.js. React is at its best when it’s forcing the programmer to think about how data flows through the page. Gusto runs payroll, benefits, and HR for small businesses in the US, so the application can mostly be summarized as some of the most complicated forms you might see in a web application. The incidental complexity of payroll and filing taxes can only be reduced so far. React served us well, but things that were once easy became harder. As we bought more into the ecosystem, simple things began feeling harder. It hit us when adding a single input to a form resulted in a 700-line pull request. Coming from a Rails background, this felt like 690 more lines than we should have needed to write. But, my duties pulled me elsewhere in the company and I began to focus and care less about the frontend of the application. My familiarity with React and its ecosystem peaked in 2020. I hadn’t kept up with its development beyond reading Twitter. I thought the push into server-side components was interesting but duplicative for most non-new companies. Nonetheless, React remained what I knew best. So when I left Gusto mid-2023, we went with React as the means of adding interactivity to our application. 1 After a few weeks of building an initial prototype, it was clear that React of 2023 was very different from the React of 2014. The small, discretely mountable components had been replaced by an all-or-nothing approach. It took me some time to figure out how to hack a sprinkling-based approach, since the library seems to really want to drive the entire page these days. Embracing React fully was a non-starter, since the backend was in Rails, the technology that I’ve spent most of my career with. It was a near-daily occurrence to be fighting React because it was not the one generating the server-side HTML. I only wanted half of what the library had become. It was no longer a gradually-adoptable library, but an all-or-nothing framework. The seed was planted to make a change. When we hired our first engineer, he brought new expertise to the company. When giving him the tour of our nascent application, I highlighted my struggles with React. He suggested we use StimulusJS, since it fits well to the business we’re building and gels well with Rails. Normally, a framework change for a young company is a terrible distraction, but we were in a position where there would never be a cheaper time to make the change as today. Without making the change, we’d put ourselves in a progressively more painful situation as we grew. So we made the decision to cut over and were done within a week. As a result of switching from React to StimulusJS, we deleted about 60% of our JavaScript or about 2k lines at the time. Much of that was due to needing to do the same thing client-side that we were already doing server-side or just the state juggling that comes with using a client-side library that wants to put the HTML on the page. Interestingly, the amount of JavaScript in our application has remained relatively flat since then. StimulusJS lets you consolidate more application logic and state to the backend. Sure, you might not get the reactivity of client-side state but client-side state is a lie. Today, the amount of JavaScript in our application has remained incredibly small. We have 993 lines of JS and 12 npm packages. You can see where we migrated off of React in August 2023. We go weeks without authoring any new JavaScript while still delivering great experiences to our customers. Our application never has to pay the penalty of two pageloads: one for the server-rendered HTML and one for the client-rendered HTML. Although we’ve made little to no optimizations, our app has mostly stayed in a “default fast” state. We’ve discovered some of the sharp edges of StimulusJS, mostly around typos in action names. This first felt like a huge regression when coming from a TypeScript world. StimulusJS takes a contrarian approach when it comes to JavaScript: it wants you to write as little of it as possible. If we take to heart the messages of The Goal and The Phoenix Project , it’s that lines of code (inventory) are a liability not an asset. We have no JavaScript test frameworks, since all interactivity is exercised through Rails system specs. After a false start with React in 2023, we’re now on a tech stack that we’re not fighting against and that maps better to our customers’ domain. The result is that we’re able to be a lot more nimble than our competitors, which we think will result in better products for our customers and better business outcomes for ourselves. Special thanks to Ngan Pham for chastising me years ago to give StimulusJS a try. I blew him off at the time, but he came with receipts when I started writing this post. Lesson: always listen to Ngan. If you’re interested in receiving blog posts like this regularly, join hundreds of developers and subscribe to my newsletter . Our company, Scholarly , is building a Faculty Information System (FIS) for higher ed institutions within the US. Our first product is most similar to performance review software. So, lots of forms and reporting while being light on the interactivity.  ↩ Our company, Scholarly , is building a Faculty Information System (FIS) for higher ed institutions within the US. Our first product is most similar to performance review software. So, lots of forms and reporting while being light on the interactivity.  ↩

Frontend Ruby JavaScript HTML

▲ 0

0 views

Kelly Sutton 2 years ago

What’s Next

After more than 6.5 years at Gusto , I’ve moved on to start a new company. I’m incredibly grateful for the opportunity to work at Gusto and proud of the work we did helping small businesses in the US. The company is better than ever and has an incredibly bright future. We had plenty of fun along the way, breaking apart monoliths or building briefcases that deploy code . What’s next? I’ve teamed up with Rusty Cowher to start building Scholarly . We are building a next generation faculty information system (FIS) for higher ed. We think there’s a substantial opportunity and many customers to make happy. We’ve closed a $1M pre-seed round and began operations this week. But it won’t just be the 2 of us for long! We are looking to hire a founding engineer. If that sounds interesting, you can see more about the role and how to apply on our site .

Career Business Web Development

▲ 0

0 views

Kelly Sutton 2 years ago

Divining Dreams and Exploring LLMs

To better understand large language models (LLMs) , I put together a free dream interpreter called Dream Diviner. This post will talk about building the app, LLMs, and reflect on this iteration of AI. The past 6 months is not the first time computer scientists and the public have been excited about artificial intelligence. Apple’s Siri received much fanfare when it was included with the iPhone 4S in 2011. Our new future of smart assistants was here, voice was the new interface, and the mouse+keyboard combination was headed for the junkpile. Fast-forward 12 years to today and human-computer interaction doesn’t look that much different. Siri is good for setting timers and reminders, but that’s about it. Amazon’s Alexa is internally considered a “colossal failure” that burns $10 billion per year. It appears we got out over our skis, once again. That being said, the applications of artificial intelligence and machine learning have seen some interesting uses for humans-in-the-loop system. For example, one might assume that providing an AI “barrier” for customer support allows humans to get involved in support cases where humans provide differentiation. If a customer just needs a link to help doc, a chatbot is perfectly capable of forwarding such a link. With the release of ChatGPT in November, 2022, OpenAI gifted the world a conversational interface for its GPT-3 language model. Its initial release was in June, 2020, and it wasn’t until it got a chat interface did the world care. Having a great interface instead of a general-purpose tool is a point I’ll come back to in this post. The OpenAI + Microsoft alliance seems to have a headstart, which has Facebook and Google playing catchup. Stock prices move in response to AI demos of each company. Others have pointed out that Web3 grifters are now all-in on AI, the next Big Thing™. The hype cycle is in hyperdrive. And the current technology is definitely impressive. I’ve been personally surprised with its ability to generate boilerplate for things like Rails applications, Terraform resources, or answer esoteric API questions. It does an admirable job of combinatorial programming problems, like figuring out how to have multiple Ruby gems play nicely together. (“Please write a Sidekiq job that iterates over the children of an ActiveRecord object and updates each child’s timestamp to now.”) It also handles the combinatorial editorialized prompts well too, which is part of what it makes it so fun and surprising. (“Re-write this email in the style of Shakespeare.”) But beyond the whimsy of a smart chatbot, I haven’t yet seen a truly novel experience where using ChatGPT, GPT-3, or something like it is demonstrably better than what currently exists. Benedict Evans maybe says it best: Microsoft & Google adding generative AI into office apps is a classic pattern of incumbents making the new thing a feature. But the new thing generally also enables completely new ways to solve the problem. ‘Easier spreadsheets’ is less important than ‘why is that a spreadsheet?’ It will take time to understand who will be the bookkeepers to this technology’s spreadsheet , and if any truly new experiences are possible and sustainable with this technology. Based on my understanding of the underlying technologies, the dangers ( hallucinations and Wailuigi effect , among others), ChatGPT and tools like it excel as tools where humans remain very much in-the-loop. And that’s just fine. Airplanes still have humans in the loop (2-ish), which is far fewer than the 100’s needed to sail a ship in the 17th century. That’s where and how we’ll see AI-driven improvements: by augmenting existing processes with humans in the loop. When a 10x or 100x productivity gain can be realized, this might start to reasonably replace jobs (or just make jobs so much more efficient that one doesn’t need to hire as many software engineers, for example). Because of these models need to be supervised by humans, most applications of this generation of AI will be working in the realm of increasing efficiency rather than something net-new. That may not see as exciting as a robot automating your life, but it is still progress nonetheless. So this round of AI improvements and hype may be overblown for the time being. But with any system, it’s always fun (and sometimes profitable) to exploit its weaknesses. These LLMs have been known to develop overconfidence, where they confidently do something incorrectly. This has been called “hallucination” . This is where your dear author decided to lean in and use this as an excuse to improve my own understanding of what’s out there today. Dream interpretations were raised as something that ChatGPT was particularly good at. My thinking was as follows: So, enter Dream Diviner . It’s a small site that is meant to interpret dreams. I’ve left the door open for charging for this, just in case it starts to become popular. This concept leans into the hallucination flaw and has fun with it. The interpretations that result are pretty reasonable. From a technical level, this is little more than an API call to OpenAI. It’s built on Ruby on Rails (of course) and is all of one controller. The most interesting technical challenge here was likely creating the prompt to send to OpenAI. The responses that one gets from these LLMs are interesting in how unstructured they are. I wanted the site to offer a short and a long interpretation of each dream, so I had to tell it about the structure ahead of time. At the time of writing, here is the prompt that I’m using: That AI dutifully separates the response into and , which is enough structure to string-split the response and save to the database. There are other concepts and knobs that can be tweaked through the API that I do not yet understand and that’s okay. For how powerful this is, I’ve been surprised at how approachable this API is. This year’s evolution of AI is being heralded as a step-change. The FAANG companies are tripping over themselves to upend roadmaps and ship products at a breakneck speed. The amount of competition in this realm has been remarkable. Competition is messy, however, and often doesn’t benefit those second or third place. It does benefit the consumer of these APIs though. The underlying technology itself is indeed remarkable, but we will likely look back on this time as a period driven by hype. In the hype, though, a few sharp folks will likely find some killer applications for this generation of AI tooling. My bet is that it will come in the form of drastically increased efficiencies in places opaque to the outside world: some expensive internal processes of companies will become 10x or 100x more efficient. These savings may either be passed onto shareholders or re-invested into more R&D. Until a version of GPT- N can replace a human, that is. Special thanks to ChatGPT for reading early versions of this blog post and providing feedback. ChatGPT is a general-purpose tool, which make it difficult to approach. Purpose-specific interfaces that map directly to a person’s intent will likely make for much better applications of AI than a master-of-all-trades interface, even if they are powered by the same technology underneath. This is the Mario-Fire Flower problem . It helps when the stakes are low and something can be built in a weekend.

Ruby AI

▲ 0

0 views