Ankur Sethi 2 weeks ago

Your analytics are lying to you

Alistair Davidson writes about migrating a form-heavy web application from a React SPA to a traditional server-rendered HTML-first website . The entire article is worth reading, but I want to draw attention to this bit about analytics (emphasis mine): The results? When we launched, the number of people completing the form doubled. The analytics people didn’t even know where these users were coming from. Of course, your javascript-based analytics package doesn’t see the users you are bouncing because of javascript failures. It was a flood! We also saw my “keep a backend session, never lose user data” approach pay off. In one case, someone completed a form a month after starting it. Web analytics are fragile. They fail in so many ways that making product decisions based wholly on your Google Analytics or Plausible data is folly of the highest degree. Here's a subset of all the reasons your analytics package undercounts or miscounts visitors: Web analytics can only give you an approximation of what your web traffic looks like. Even when they work correctly, they paint an incomplete picture. As I said in my post about share buttons , the number one referrer for pages on this website is "Direct/none". It's impossible for Plausible to figure out where those users are coming from. Further, my server logs report three times as much traffic as my Plausible dashboard over a seven day window. Some of this might be bot traffic and thus irrelevant, but I know for a fact that a large chunk of this traffic comes from RSS readers. Plausible will never have insight into these users. My point is, if you rely on your analytics dashboard to make product decisions, you're excluding a large chunk of potential users who simply don't show up in your graphs. You might be missing out on serving thousands of potential users because you can't see them in your data. These are users who want to sign up for your newsletter, buy your app, subscribe to your service. These are human beings you could help, whose lives you could improve. I'm not saying that analytics are completely useless. They can and should have a place in your decision-making process. Just don't treat analytics data as gospel, because there will always be massive blind spots in what it tells you. To get a real understanding of how users experience your products, test them on real devices under real conditions as much as possible. And as always, get out there and talk to your users. Network errors prevent your analytics script from loading. Ad-blockers and tracking prevention block your script from loading (enabled by default on many browsers today). A JavaScript error in an unrelated part of the page prevents the analytics script from working correctly. The user loses network connectivity before the analytics script can send data to the server. The user gets impatient and bounces off your website before the page can load fully and start collecting data. Too much JavaScript on the page causes the browser tab to crash (a common issue on low-end devices). The analytics script is blocked by a DNS rule, corporate proxy, firewall, or VPN. The user has disabled JavaScript. The user's browser has limited or no support for JavaScript (Opera Mini still has more than half a million downloads on Android, and it's still widely-used in Africa ). The user is accessing your content using a service that strips JavaScript (e.g. an RSS reader, a web archiving tool, Telegram Instant View, AMP, a read-later service, or a bookmarking service). You only test your app in Chrome, so you don't realize that your website is entirely broken in Firefox and Safari.

Data Analysis HTML Web Development Analytics JavaScript

Jack Vanlightly 3 weeks ago

Can We Agree on a Storage/Workload Architecture Taxonomy?

The lines between transactional systems, analytical systems, hybrid systems, and shared storage architectures are getting blurry. This post proposes a small taxonomy for describing the different ways systems, workloads, storage tiers, visibility, and durable copies relate to each other. OLTP, OLAP, HTAP, and now LTAP? We can think of the first two as two types of workload which have specialized query engines and storage systems to support them. OLTP such as the RDBMS like Postgres and MySQL use row-based storage engines. OLAP, such as Clickhouse, cloud data warehouse and the lakehouse use column-based storage. HTAP is a hybrid workload system: one system -> both transactional and analytical workloads. The HTAP system therefore has specialized storage and specialized query engine to stitch together the row-based and columnar data. So far, we’re dealing with a single system. A Postgres (OLTP), a Clickhouse (OLAP), a SingleStore or TiDB (HTAP). So what is the recent Databricks’ LTAP announcement? LTAP is the two workloads (OLTP and OLAP) but also two systems (e.g. Postgres and lakehouse/Spark) and some blend of two different storage systems. As well single single vs multi-system, single vs multi-workload, there are other relevant concepts such as tiering and materialization: A single system can tier (move) data from hot to cold storage (for cost efficiency). One system, one copy, two tiers. Hot and cold might be the same storage format (both row-based or both columnar), or might be different formats (hot is row-based, cold is columnar). We can have two systems share the same storage tier. System A tiers (move) hot data to the storage of System B. Two systems, one copy, though System B doesn’t see the newest data yet which only exists on A. Materializing One system can materialize (copy) data into another system. Two systems, two copies. Note when I say “copy of the data”, I mean durable copy, so caching doesn’t count. If the number of copies really matters to you as a metric, then maybe caching does count, depending on how much cached data you need to make it work? If only life were simpler. It would be nice to have some shared vocabulary around this, so we can talk about system architecture more easily. So I defined some terms last year for this, and expanded it as seen below. Vis means Visibility (when is data available in the other workload). The broad classification scheme: Single tier, one system, one workload. Example: Postgres with SSD, single tier CockroachDB, standard Kafka cluster. Internal Tiering, one system, one workload, commonly tiers from hot to cold storage for cost efficiency, e.g. hot=SSD, cold=S3. Though tiering could also serve other purposes than cost. Example: Apache Kafka tiered storage, ClickHouse MergeTree tiered storage. Hybrid-Sync (aka HTAP), one system, two workloads, two or more storage with potentially different formats/tiers, e.g. hot row-based data on SSD, long-term columnar data on S3. Data is immediately available to both workloads (e.g. OLTP queries and OLAP queries). Example: SingleStore and TiDB (Pingcap). Hybrid-Async , one system, two workloads. Like Hybrid-Sync except hot row-based data is asynchronously tiered to long-term columnar format. OLAP queries do not see the very newest data. Example: Snowflake Hybrid tables. Materializing , two workloads, two systems, two copies. System A copies data to System B. Each system is dedicated to one workload, with specialized query engine and storage. Example: ETL in general, many Kafka-compatible services have automatic Iceberg materialization of topics e.g. Confluent Tableflow, Databricks Synced tables asynchronously materialize from lakehouse to lakebase (Postgres). Shared Tiering , two workloads, two systems. one copy across hot tier + shared colder tier (e.g. hot row-based data on SSD for System A, colder columnar data on S3 for System A + B). Example: Apache Fluss tiers hot data (Fluss servers) to lakehouse (lakehouse is a shared tier), LTAP. Potentially, a 7th and 8th category could hypothetically exist: Shared-Sync-RR and Shared-Sync-MM. Two systems, two workloads, one synchronous storage (each write is immediately visible in the other system. Read-replica (RR) variant has one master system and one read-only system (e.g. writes to Postgres are immediately visible for reads in lakehouse). Multi-master (MM) allows both systems to write (hard!!). At the time of writing the details on LTAP are scarce, but it seems like LTAP will fall into Shared Tiering. The thing that differentiates HTAP from LTAP is that HTAP is a single hybrid system which makes data visible to both transactional and analytical queries at the same time. LTAP is a way of unifying the data of two different systems (each targeting a different workload) and sharing the colder data such that there is no (durable) data copy required. It is fundamentally asynchronous: hottest data is only in System A and the remaining colder data is stored in System B but made available to System A (as it’s cold tier). Of course LTAP could potentially move towards the hypothetical category Shared-Sync-RR , given both systems exist in the same platform, then it gets murky again because its one platform, its veering towards HTAP (Hybrid-Sync). One thing that the marketing material of unified OLTP-OLAP system commonly glosses over are the different data models used in each, such as Third Normal Form (3NF) common in OLTP and Kimball (star and snowflake schema) common in analytics. This adds another dimension, on top of query engine, storage layout and storage substrate. If you want 3NF for OLTP and Kimball for analytics, then it’s probably going to be Materialization (as star schema is not viable as a cold tier for 3NF). What you you think of this broad classification scheme? Find on me social media :) ps, some thoughts on data copies… With Shared Tiering, you can think of the data-copy question as a dial: Dial it to no-copies-at-all means evicting data as soon as it has been tiered. Lower storage cost, but maybe it would be good to hang onto to the hot data a little longer for performance. Dial it to lots-of-data-overlap means aggressively tiering to System B but hanging onto the data in System A for the better performance profile, at the additional storage cost. And technically it would now count as cached data which might not count as a data copy, depending on how you define that. However, the data-copy question is also murky with Materialization. Because we have two (or more) independent systems, each can potentially use independent data expiration policies. For example, in Kafka, it might store 7 days, but in the lakehouse, it might store 7 years. In that case, while theoretically it is a two-copy system, the total duplication would only be 0.0027%. I generally dislike the whole “zero-copy” or “one-copy” thing, it’s too much marketing. Focusing on how many copies you have is just weird as a primary design point when you’re building data systems, the real world is more nuanced. Tiering A single system can tier (move) data from hot to cold storage (for cost efficiency). One system, one copy, two tiers. Hot and cold might be the same storage format (both row-based or both columnar), or might be different formats (hot is row-based, cold is columnar). We can have two systems share the same storage tier. System A tiers (move) hot data to the storage of System B. Two systems, one copy, though System B doesn’t see the newest data yet which only exists on A. Materializing One system can materialize (copy) data into another system. Two systems, two copies. Single tier, one system, one workload. Example: Postgres with SSD, single tier CockroachDB, standard Kafka cluster. Internal Tiering, one system, one workload, commonly tiers from hot to cold storage for cost efficiency, e.g. hot=SSD, cold=S3. Though tiering could also serve other purposes than cost. Example: Apache Kafka tiered storage, ClickHouse MergeTree tiered storage. Hybrid-Sync (aka HTAP), one system, two workloads, two or more storage with potentially different formats/tiers, e.g. hot row-based data on SSD, long-term columnar data on S3. Data is immediately available to both workloads (e.g. OLTP queries and OLAP queries). Example: SingleStore and TiDB (Pingcap). Hybrid-Async , one system, two workloads. Like Hybrid-Sync except hot row-based data is asynchronously tiered to long-term columnar format. OLAP queries do not see the very newest data. Example: Snowflake Hybrid tables. Materializing , two workloads, two systems, two copies. System A copies data to System B. Each system is dedicated to one workload, with specialized query engine and storage. Example: ETL in general, many Kafka-compatible services have automatic Iceberg materialization of topics e.g. Confluent Tableflow, Databricks Synced tables asynchronously materialize from lakehouse to lakebase (Postgres). Shared Tiering , two workloads, two systems. one copy across hot tier + shared colder tier (e.g. hot row-based data on SSD for System A, colder columnar data on S3 for System A + B). Example: Apache Fluss tiers hot data (Fluss servers) to lakehouse (lakehouse is a shared tier), LTAP. Dial it to no-copies-at-all means evicting data as soon as it has been tiered. Lower storage cost, but maybe it would be good to hang onto to the hot data a little longer for performance. Dial it to lots-of-data-overlap means aggressively tiering to System B but hanging onto the data in System A for the better performance profile, at the additional storage cost. And technically it would now count as cached data which might not count as a data copy, depending on how you define that.

Data Database Analytics

Ankur Sethi 1 months ago

Nobody clicks your share buttons

Link: https://derekhanson.blog/nobody-clicks-your-share-buttons/ (Via rendezvous with cassidoo .) I've always wondered if anyone actually used the social sharing buttons embedded on news sites and (some) WordPress blogs. Derek Hanson digs into the numbers : The UK government ran one of the most thorough studies on this. When GOV.UK added social sharing buttons, they tracked usage for 10 weeks across 6.8 million pageviews. The share buttons got clicked 14,078 times. That’s a 0.21% usage rate, which works out to about 1 in 476 visitors. The most telling part: the feature sat in their backlog for ages because zero end users had ever requested it. In their user testing, people just copied and pasted links. Moovweb found the same thing when they analyzed 61 million mobile sessions . Only 0.2% of mobile users interacted with social sharing at all. Visitors were twelve times more likely to click an advertisement. Luke Wroblewski, the interaction designer and author, crowdsourced data from his readers and landed on an average of 0.25% across 18 million pageviews. Different organizations, different audiences, same number. What do people do instead? They copy and paste URLs or use the share button in their browser. In 2012, Alexis Madrigal at The Atlantic noticed a huge chunk of the magazine’s web traffic showing up as “direct” in Google Analytics. Those visitors weren’t typing URLs or using bookmarks. They were clicking links that someone had pasted into a text thread, an email chain, a Slack channel. This reflects my own experience. "Direct/none" is the number one referrer on this very website.

Analytics Marketing

Christian Hollinger 1 months ago

Building confidence in geospatial data

How SkaldMaps generates a confidence score for data attributes that helps you gauge how accurate data is (or isn't).

Data Analytics

Justin Duke 1 months ago

Victory is a lossy compressor

The NBA's conference finals wrapped up this past week, and as a Heat fan and a Seattle émigré I was pleased with both results. Happy that the Cavs lost, because I still resent them for being the destination to which LeBron fled in 2014; happy that the Thunder lost, because fuck Howard Schultz. 1 For the uninitiated: Schultz, as owner, presided over the sale of the Seattle SuperSonics in 2006, which led directly to their relocation to Oklahoma City. The Thunder are, in a very real sense, the Sonics. I am not over it. Both losses were noteworthy. The Cavs were, after game three, safely assumed to be dead in the water — no NBA team has ever come back from a 3-0 deficit in the playoffs. Their head coach, JB Bickerstaff, infamously (yes, it can already be classified as infamous) claimed that from an analytics standpoint they were really up — that the underlying numbers had the series at a 2-2 coin flip they happened to be losing. The Thunder, last season's champions, were being discussed around the halfway mark as candidates for the greatest team of all time: whether they could break the all-time regular-season wins record, or, more humbly, be the team that finally snapped the league's eight-season no-repeat-champion streak. Those discussions promptly ended in December, when — in what turned out to be foreshadowing — the Spurs beat them three times in ten days. Even gods can bleed. Now the conversation has gone the other direction entirely: whether they should blow up their core three and pursue a win-now trade for Giannis, or some similarly older-but-better superstar. All of this is, in my opinion, kind of silly, by all parties involved. Bickerstaff should not have tried to invoke the spirit of Pythagorean victories to motivate his team or deflect criticism about their dire straits. And yet — it's not like he was completely incorrect. In much the same way that, if you know with absolute certainty a coin is rigged to land heads 70% of the time, you shouldn't start betting tails just because tails came up the last three times. 2 Now, if it comes up tails ten times in a row, maybe that's a sign to reevaluate your threshold for "absolute certainty." And the discourse around the Thunder's prospects glosses over the fact that the second-best player on their team — and, during some stretches, arguably the fourth-best player on their team — were both out for the entire series. Which is a boring but plausible explanation for why they lost. (It should be noted that they lost in seven, and scored more total points than the Spurs across the span.) But the narrative of Wemby, the seven-foot-seven godkiller, has taken root. Let me be clear: the Knicks and the Spurs did win, and no number of hypothetical simulations changes or devalues that fact. But as soon as we turn the page onto next season, the nuances that led up to the result will grow hazier and dimmer and more nebulous, until recovering them requires archaeology rather than recall. The NBA is a league where we talk about rings, not net rating. Which is to say: victory, whether over the course of a single game or an entire season, is not a lagging metric assembled from the true granules of play-by-play data. It is the thing itself — the reified artifact. The banner, the line in the record book, the ring. And there is nothing wrong with this, in the same way that it is no crime that both PNGs and JPEGs exist. Victory is a lossy compressor: it takes the full, lossless record of a season and squeezes it down into something small and portable, discarding almost all of the granular detail in order to do so. It is enormously useful to be able to optimize for file size — for what travels, for what gets remembered. But it is not the only use. All of this is to say: I bring it up not just because I'm waiting for Zach Lowe to publish his Tuesday episode previewing the Knicks-Spurs series, but because in America sports are metonymous with many things, and one of those things is business. And I have found it far more revealing, in trying to understand a company that has — capital M, capital I — Made It, not to read their most recent batch of press releases, but to crank up the Wayback Machine and go look at the launches they shipped in 2018 that failed. The press release is the JPEG. The failures are the bits that got thrown away. If you want to understand how something actually won, you don't get to recall it — you have to decompress it, and the only way to recover what victory discarded is to dig.

Sports Culture Analytics

James O'Claire 1 months ago

App Marketing: Free App Analytics vs all the “Free” paywall companies

When SensorTower acquired AppMagic earlier this week it got me thinking about why. AppGoblin and many other tools offer many free and open resources for what SensorTower and AppMagic charge thousands for. Take a look at the paid vs free vs free (but limited) of the various ASO and app marketing services out there. None of them are anywhere near as expensive as SensorTower. I think that SensorTower sees this coming and wanted to acquire their biggest competitor to try keeping it’s moat as “the” destination for app analytics.

Marketing Analytics Business

James O'Claire 1 months ago

AppGoblin App Ecosystem Report 2026 Q1

The 2026 Q1 App Ecosystem Report is here with a special section for those attending MAU in Vegas this week. Ad Networks were led by Verve once again after its strong Q4 2025, with other notable breakouts from Snap Inc. , TaurusX , adjoe , and Moloco . Business Tools were led by small but super fast growing Luciq . PayPal also posted strong mobile growth, while emerging companies like AppHarbr stood out. In attribution analytics, growth was broadly healthy across the category and was led by Tenjin . Open source product analytics platform Matomo also looked great heading into 2026. One notable absence from the growth list was AppsFlyer , which has historically been one of the category’s largest and most consistent performers but saw a small down tick in tracked market share. For Development Tools, Divkit posted solid growth. The framework launched in 2025 and is backed by Yandex . Report is totally free and the raw data is available as a free dataset download for the top 1000 app companies / ad domains to see their quarter-over-quarter 2026 Q1 growth: https://appgoblin.info/reports/app-ecosystem-report-Q1-2026

Analytics Mobile Business

Entropic Thoughts 3 months ago

Readership maths skills

Statistics Data Analysis Analytics

James Stanley 4 months ago

Bot Forensics

Most threat intelligence bots are easy to fingerprint. And trying to be stealthy often makes it worse because imperfect anti-detection methods have extra fingerprint surface area of their own. We run an instrumented honeypot site that collects data on what these bots do, and we've just released an Instant Bot Test so you can see whether we flag your bot without even having to talk to us first. You may want to see my previous post on this topic for more context on what we're doing. Since that post we've sold a handful of reports, including to a couple of big names. And we now have a website at botforensics.com to advertise our services. Anti-detection detection One of the most interesting things we've learnt is that anti-detection techniques are very rarely successful in preventing your bot from being detected. Our collector site sees only an extreme minority (<0.1%) of sessions that could plausibly be real human users. Far from preventing a bot from being detected, anti-detection measures more often provide specific fingerprints about which bot it is based on which measures are in use. Some of these measures take us from "we think this is probably a bot" to "this is bot XYZ operated by Foocorp", which is kind of an own goal. If you're going to run a bot with anti-detection measures in place (and you should, otherwise you'll trivially look like Headless Chrome), then you should definitely get a Bot Audit to make sure you aren't leaking any extra signals. The Puppeteer stealth evasions are a great example of this. Lots of bots are browsing with these evasions applied (we even see bots using them outside Puppeteer), but we can detect the evasions themselves, which often leak more signal than we would expect to see absent the evasions. We do take a canvas fingerprint because why not, but it turns out to be quite hard to definitively say that a given canvas is a bot unless you have enough data on real user sessions to rule out the possibility that it is a real user. While some people are very worried about canvas fingerprinting, a much stronger bot signal than the canvas fingerprint itself is if we read the pixel data out and it has random pixels in the wrong colour where it should be the same colour all over. And, worse, if we do the same thing twice in a row and get a different answer each time! We noticed a bot operated by Microsoft that had some very specific identifying features, including references to some of their developers' real names. Microsoft have a fairly reputable bug bounty programme, so I tested the waters by reporting it on MSRC . But after sitting on it for 2 weeks they classified it as "not important" and declined to pay a bounty, so I won't make this mistake again. To Microsoft's credit, they have still not fixed it, which is consistent with considering it not important. We are in some cases able to detect when bots are running on Kubernetes (thanks Feroz for the idea), and this also reveals some fingerprints that are unique to each Kubernetes cluster. This is a great signal because a.) hardly any real human users are browsing from inside Kubernetes, and b.) if 2 bots are running on the same Kubernetes cluster then it's a fair bet that they're operated by the same company. So far we have seen bots from 3 distinct Kubernetes clusters. We've been surprised by how few threat intelligence vendors are running their own fetching. There are 94 vendors listed on VirusTotal, but fewer than 50 genuinely distinct bots fetch our collector pages, so at most only a bit over half of those vendors are actually fetching the sites themselves. The others may outsource their fetching to a common third-party, or else they are simply consulting other threat intelligence vendors and not even doing classification themselves. If you looked at enough VirusTotal results pages you could probably work out which ones always share the same classification, maybe we should do that. One of our domains is now blocked on VirusTotal by 7 different vendors: This is kind of a poor show. You can't classify a site as phishing just because it has "bank" in the domain and the page has a login form. The litmus test for whether a site is phishing is whether you can name the site it is impersonating, and our collector site doesn't impersonate any real site. Vexatious takedowns We received our first takedown notices last week. To be honest, I expected this to happen sooner. The whole project is running on "disposable" infrastructure so that if it gets taken down it won't impact any of our other projects. But it would still be very inconvenient to have it taken down. The takedown notices were sent to our hosting provider, who forwarded them to us. It's possible they were also sent to our domain registrar, who did not forward them to us but also did not act on them. Here's the text from the first one: Hello, We have discovered a Phishing attack on your network. URL: hxxps[:]// REDACTED / IP's: REDACTED Threat Type: Phishing Threat Description: Banking credential harvesting page detected at REDACTED . The page presents a fake bank login form with a header that references BotForensics Collector Page and botforensics .com, which indicates branding inconsistent with any legitimate bank . The site is hosted on REDACTED infrastructure (IP REDACTED ) and registered recently on 2026-02-17 via REDACTED , with privacy-protected WHOIS data . The HTML shows a typical login card for username and password, a Sign In” [sic] button, and scripted UI enhancements, including external scripts and images, plus a dynamic header bar . This combination is characteristic of a phishing attempt intended to harvest user credentials . The domain age is only about 0 .01 years, and the presence of a login form on a brand-tampering page hosted on a known hosting provider strongly suggests malicious intent . Registrar abuse contact is abuse[@] REDACTED and hosting provider abuse contact is abuse[@] REDACTED . Because high confidence phishing has been detected, the page should be reported to abuse contacts and blocked; while there can be legitimate educational use of such content, the page as presented is designed to harvest credentials rather than serve legitimate banking functionality . Domain Registrar: REDACTED ASN: REDACTED This email was sent automatically by QuariShield Automated Analysis. Reports are sometimes verified using AI, while this means reports are mostly valid, there may be some false positives. For more info: REDACTED We are well aware that you may not be able to take abuse reports sent to this email address, therefore if you could forward this email to the correct team who can handle abuse reports, it would be much appreciated. Please note, replies to this email are logged, but aren't always seen, we don't usually monitor this email for replies. To contact us if you have any questions or concerns, please email [email protected] stating your Issue ID REDACTED Kind regards, QuariShield Cyber Security. (Redactions mine, but yes the text is all run into one like that with no linebreaks). A few highlights stand out: The page presents a fake bank login form with a header that references BotForensics Collector Page and botforensics .com, which indicates branding inconsistent with any legitimate bank . One would think that having branding "inconsistent with any legitimate bank" is evidence that you're not phishing? A phishing site would copy the bank's branding. The HTML shows a typical login card for username and password, a Sign In” button, and scripted UI enhancements, including external scripts and images, plus a dynamic header bar . This combination is characteristic of a phishing attempt intended to harvest user credentials Is it really? hosted on a known hosting provider What are the chances? This email was sent automatically by QuariShield Automated Analysis. Reports are sometimes verified using AI Very interesting. The takedown notices were sent by QuariShield . I emailed the QuariShield contact address and got a reply from the person operating it, and he seems friendly, and has whitelisted my collector page, which is helpful but in my opinion only part of the solution. How many other false positive takedown notices is he going to send for other websites? From what I have been able to gather, QuariShield grabs URLs from public sources, and uses an LLM agent to classify them and automatically send takedowns. On the one hand, yeah, it's not working very well yet and has a lot of false positives. On the other hand, just look at how far we've come. If you're running a traditional takedown provider: this is what's coming for you. People are spinning up (presumed) vibe-coded projects that now do fully-automated takedowns for sites that aren't even paying customers . Your anti-detection techniques may not be as effective as you think. Try our Instant Bot Test to see if we flag your bot (and please let us know how we did). And the lesson from QuariShield is: AI is coming for you.

Analytics Security AI HTML

Manuel Moreale 5 months ago

Step aside, phone: week 1

First weekly recap for this fun life experiment. To remind you what this is all about : in order to help Kevin get back to a more sane use of his time in front of his phone, we decided to publicly share 4 weeks of screen time statistics from our phones and write roundups every Sunday. Yes, we’re essentially trying to shame ourselves into being more mindful about our phone usage. Let me tell you, it definitely works. Every time I do one of these experiments, I use the first week to prove to myself that this whole phone usage situation is mostly a matter of being mindful about it, and that if I decide that I don’t want to use the phone, well, I will not use it. And it’s not very hard. Monday to Wednesday, I basically almost never picked up my phone from my desk. It was fully charged on Sunday afternoon, and I didn’t plug it in again till Thursday. I did use it when I was outside for a couple of minor things, but as you can see from the image below, screen time is reporting 9 minutes of total usage for the first 3 days of the week. Thursday and Friday, I logged a bit more screen time (had to do a few things that required the use of apps), but also because I started listening to a few podcasts while I was driving. I said I started because one thing I did this week was delete any app that’s related to content consumption from the phone. I think my personal goal for this month-long experiment is going to be to get back to a use of my phone that’s utility-driven and not consumption-focused. The phone should be a tool to do things and not a passive consumption device. Friday usage spiked, and that’s because I was out on a date, so most of the time spent with the screen on was Google Maps being open while I was in the car. I still tried to be mindful of that, though. I drove about 5 hours back and forth, but I only used Google Maps for a bit more than 1 hour. I also used the browser for the first time this week to purchase a couple of tickets for a museum, and I took a few pictures. So this is how the first week went. Not included here is last Sunday—I told Kevin we were going to start this experiment on Monday—but I clocked 11 minutes on that day. Not bad. Now, one consideration about this first week: in order to push my phone usage this low, I had to move some of my normal phone usage over to my Mac, which is how I managed to basically never touch chat apps on my phone. I know this is pretty much cheating, but it was intentional and something I was planning to do only in this first week, and I will move that screen time back on my phone starting next week. The goal is to find the right balance after all, and I like the process of pushing it all the way down to the extreme and then bringing it back up to some more sane levels. If you have decided to take part in this experiment, email me a link to your post, and I’ll include it below. Thank you for keeping RSS alive. You're awesome. Email me :: Sign my guestbook :: Support for 1$/month :: See my generous supporters :: Subscribe to People and Blogs Read Kevin's week one recap Read Thomas' week one recap Read Steve's week one recap Read John's week one and two recaps

Statistics Analytics

Manuel Moreale 6 months ago

How You Read My Content

A week ago, after chatting with Kev about his own findings , I created a similar survey (which is still open if you want to answer it) to collect a second set of data because why the heck not. Kev’s data showed that 84.5% of responses picked RSS, Fediverse was second at 7.6%, direct visits to the site were third at 5.4%, and email was last at 2.4%. My survey has a slightly different set of options and allows for multiple choices—which is why the % don’t add up to 100—but the results are very similar: This is the bulk of the data, but then there’s a bunch of custom, random answers, some of which were very entertaining to read: So the takeaway is: people still love and use RSS. Which makes sense, RSS is fucking awesome, and more people should use it. Since we’re talking data, I’m gonna share some more information about the numbers I have available, related to this blog and how people follow it. I don’t have analytics, and these numbers are very rough, so my advice is not to give them too much weight. 31 people in the survey said they read content in their inbox, but there are currently 103 people who are subscribed to my blog-to-inbox automated newsletter. RSS is a black box for the most part, and finding out how many people are subscribed to a feed is basically impossible. That said, some services do expose the number of people who are subscribed, and so there are ways to get at least an estimate of how big that number is. I just grabbed the latest log from my server, cleaned the data as best as I could in order to eliminate duplicates and also entries that feel like duplicates, for example: In this case, it’s obvious that those two are the same service, and at some point, one more person has signed up for the RSS. But how about these: All those IDs are different, but what should I do here? Do I keep them all? Who knows. Anyway, after cleaning up everything, keeping only requests for the main RSS feed, I’m left with 1975 subscribers, whatever that means. Are these actual people? Who knows. Running the exact same log file (it’s the NGINX access log from Jan 10th to Jan 13th at ~10AM) through Goaccess, with all the RSS entries removed, tells me the server received ~50k requests from ~8000 unique IPs. 33% of those hits are from tools whose UA is marked as “Unknown” by Goaccess. Same story when it comes to reported OS: 35% is marked as “Unknown”. Another 15% on both of those tables is “Crawlers”, which to me suggests that at least half of the traffic hitting the website directly is bots. In conclusion, is it still worth serving content via RSS? Yes. Is the web overrun by bots? Also yes. Is somebody watching me type these words? Maybe. If you have a site and are going to run a similar experiment, let me know about it, and I’ll be happy to link it here. Also, if you want some more data from my logs, let me know. Thank you for keeping RSS alive. You're awesome. Email me :: Sign my guestbook :: Support for 1$/month :: See my generous supporters :: Subscribe to People and Blogs 80.1% reads the content inside their RSS apps 23.8% uses RSS to get notified, but then read in the browser 10.7% visits the site directly 4.9% reads in their inbox. 1 person said they follow on Mastodon, and I am not on Mastodon, so 🤷‍♂️ 1 person left a very useful message in German, a language I don’t speak, which was quite amusing 1 person lives in my house and looks over my shoulder when I write A couple of people mentioned that they read on RSS but check the site every now and again because they like the website

Analytics Rss Data Analysis

Kev Quirk 6 months ago

How Do You Read My Content?

I'm trying to get an idea on how people consume the waffle I put out, it should only take 5 seconds to respond, and I'd be very grateful. It’s well publicised that I don’t run any kind of analytics on this site . For me, engagement is far more important. But I’m trying to better understand how you fine people consume the waffle I spit out into the world. The only reason I want to do this is that I think it will be interesting to know. I could temporarily add tracking to the site, but that feels icky to me; I’d rather have something that’s opt in. So I’ve created a really simple form that you can fill in. It only has 1 question, so should take no more than a few seconds to complete. If you’re a regular reader, I’d be very grateful if you could take a few seconds out of your day to cast a vote please. The form is embedded below, but it may not embed properly in some places (like on the RSS feed), so just in case here’s a direct link to the form too . Thanks for reading this post via RSS. RSS is great, and you're great for using it. ❤️ You can reply to this post by email , or leave a comment .

Karboosx 7 months ago

Homemade tracking system without use of third-party libs like Google Analytics

Tired of sending your analytics data to Google? Build with me a simple, self-hosted tracking system from scratch that respects user privacy, detects bots, and keeps everything on your own servers.

Analytics Web Development Programming

Alex White's Blog 7 months ago

Babbling About Solutions

I've been in so many meeting where people will discuss a tiny point to death over the course of hours/days. For example, at numerous companies there have been debates around time axis graphs and the current date. There's always someone who thinks users will be confused that the visual for the current day is less than the previous days (because the day is in progress). Additionally, the topic of timezones will come up. "Our data is in X timezone, but the user is in Y, the graph will be confusing". I was reminded of this example as I built out a "daily visitors" graph for my analytics page. I chuckled as I just implemented a solution, without hours of debate and Y salaries * Z hours of money wasted. My solution to the two problems was this: Nothing is ever perfect and every UI will confuse someone, somewhere, somehow. The key is to see if there's a significant amount of data indicating the UI is confusing, not to debate the tiny details to death before even releasing something. Show visitor count as "X visitors in 24 hours". You're not commiting to a day in a timezone, instead it's relative by hours. For the graph, use the wording "Until Now" to represent the partial nature of the value. Give users some credit and know your audience.

Analytics Data Visualization Design

Alex White's Blog 7 months ago

Privacy Focused Analytics in Under 200 Lines of Code

When I launched this blog, I told myself I wouldn't succumb to monitoring analytics. But, curiosity killed the cat and here we are! I've built and deployed a privacy focused analytics "platform" for this blog. Best of all, it's under 200 lines of code and requires a single PHP file! My analytics script (dubbed 1Script Analytics) works by recording a hash of the visitor's IP and date (inspired by Herman's analytics on Bear Blog). This allows me to count unique visitors in a privacy friendly way. The script itself is a single PHP file that does two jobs. When called directly (/analytics.php) it displays a dashboard with traffic data. When used in an a simple JS function with the query parameter, it records the visit to a SQLite database. That's it, super simple analytics. No cookies, JavaScript frameworks or dependencies. Throw it on your server, migrate the database and put a image tag in your template file. Wanna see my live analytics? Click here for the analytics dashboard. Okay I fixed a few things, guess I'm a bit sleep deprived! To properly get the referrer, I switched to JavaScript to call the analytics PHP script rather than the image method. I'm using a POST request via to pass current page and referrer to PHP. Also updated the styling slightly on the dashboard to use a grid layout. Finally, moved my sqlite file into a non-web directory on the server, updated config, and bundled the analytics script with my 11ty deployment process. Planning to layer in some simple graphs in the future, but so far pretty happy with how things are working!

Analytics JavaScript Web Development Php

Jim Nielsen 7 months ago

My Number One “Resource Not Found”

The data is in. The number one requested resource on my blog which doesn’t exist is: According to Netlify’s analytics, that resources was requested 15,553 times over the last thirty days. Same story for other personal projects I manage: “That many requests and it serves a 404? Damn Jim, you better fix that quick!” Nah, I’m good. Why fix it? I have very little faith that the people who I want most to respect what’s in that file are not going to do so . So for now, I’m good serving a 404 for . Change my mind. Reply via: Email · Mastodon · Bluesky iOS Icon Gallery : 18,531 requests. macOS Icon Gallery 10,565 requests.

Web Development Analytics

Evan Schwartz 8 months ago

Scour - October Update

Hi friends, In October, Scour ingested 1,042,894 new posts from 14,140 sources . I was also training for the NYC Marathon (which is why this email comes a few days into November)! Last month was all about Interests: Your weekly email digest now includes a couple of topic recommendations at the end. And, if you use an RSS reader to consume your Scour feed, you’ll also find interest recommendations in that feed as well. When you add a new interest on the Interests page, you’ll now see a menu of similar topics that you can click to quickly add. You can browse the new Popular Interests page to find other topics you might want to add. Infinite scrolling is now optional. You can disable it and switch back to explicit pages on your Settings page. Thanks Tomáš Burkert for this suggestion! Earlier, Scour’s topic recommendations were a little too broad. I tried to fix that and now, as you might have noticed, they’re often too specific. I’m still working on solving this “Goldilocks problem”, so more on this to come! Finally, here were a couple of my favorite posts that I found on Scour in October: Happy Scouring! - Evan Introducing RTEB: A New Standard for Retrieval Evaluation Everything About Transformers Turn off Cursor, turn on your mind

Analytics Open Source Data

Jack Vanlightly 8 months ago

How Would You Like Your Iceberg Sir? Stream or Batch Ordered?

Today I want to talk about stream analytics, batch analytics and Apache Iceberg. Stream and batch analytics work differently but both can be built on top of Iceberg, but due to their differences there can be a tug-of-war over the Iceberg table itself. In this post I am going to use two real-world systems, Apache Fluss (streaming tabular storage) and Confluent Tableflow (Kafka-to-Iceberg), as a case study for these tensions between stream and batch analytics. Apache Fluss uses zero-copy tiering to Iceberg . Recent data is stored on Fluss servers (using Kafka replication protocol for high availability and durability) but is then moved to Iceberg for long-term storage. This results in one copy of the data. Confluent Kora and Tableflow uses internal topic tiering and Iceberg materialization , copying Kafka topic data to Iceberg, such that we have two copies (one in Kora, one in Iceberg). This post will explain why both have chosen different approaches and why both are totally sane, defensible decisions. First we should understand the concepts of stream-order and batch-order . A streaming Flink job typically assumes its sources come with stream-order . For example, a simple SELECT * Flink query assumes the source is (loosely) temporally ordered, as if it were a live stream. It might be historical data, such as starting at the earliest offset of a Kafka topic, but it is still loaded in a temporal order. Windows and temporal joins also depend on the source being stream-ordered to some degree, to avoid needing large/infinite window sizes which blow up the state. A Spark batch job typically hopes that the data layout of the Iceberg table is batch-ordered , say, partitioned and sorted by business values like region, customer etc), thus allowing it to efficiently prune data files that are not relevant, and to minimize costly shuffles. If Flink is just reading a Kafka topic from start to end, it’s nothing special. But we can also get fancy by reading from two data sources: one historical and one real-time. The idea is that we can unify historical data from Iceberg (or another table format) and real-time data from some kind of event stream. We call the reading from the historical source, bootstrapping . Streaming bootstrap refers to running a continuous query that reads historical data first and then seamlessly switches to live streaming input. In order to do the switch from historical to real-time source, we need to do that switch on a given offset. The notion of a “last tiered offset” is a correctness boundary that ensures that the bootstrap and the live stream blend seamlessly without duplication or gaps. This offset can be mapped to an Iceberg snapshot. Fig 1. Bootstrap a streaming Flink job from historical then switch to real-time. However, if the historical Iceberg data is laid out with a batch-order (partitioned and sorted by business values like region, customer etc) then the bootstrap portion of a SELECT * will appear completely out-of-order relative to stream-order. This breaks the expectations of the user, who wants to see data in the order it arrived (i.e., stream-order), not a seemingly random one.  We could sort the data first from batch-order back to stream-order in the Flink source before it reaches the Flink operator level, but this can get really inefficient. Fig 2. Sort batch-ordered historical data in the Flink source task. If the table has been partitioned by region and sorted by customer, but we want to sort it by the time it arrived (such as by timestamp or Kafka offset), this will require a huge amount of work and data shuffling (in a large table). The result is not only a very expensive bootstrap, but also a very slow one (afterall, we expect fast results with a streaming query). So we hit a wall: Flink wants data ordered temporally for efficient streaming bootstrap. Batch workloads want data ordered by value (e.g., columns) for effective pruning and scan efficiency. These two data layouts are orthogonal. Temporal order preserves ingest locality; value order preserves query locality. You can’t have both in a single physical layout. Fluss is a streaming tabular storage layer built for real-time analytics which can serve as the real-time data layer for lakehouse architectures. I did a comprehensive deep dive into Apache Fluss recently, diving right into the internals if you are interested. Apache Fluss takes a clear stance. It’s designed as a streaming storage layer for data lakehouses, so it optimizes Iceberg for streaming bootstrap efficiency. It does this by maintaining stream-order in the Iceberg table. Fig 3. Fluss stores real-time and historical data in stream-order. Internally, Fluss uses its own offset (akin to the Kafka offset) as the Iceberg sort order. This ensures that when Flink reads from Iceberg, it sees a temporally ordered sequence. The Flink source can literally stream data from Iceberg without a costly data shuffle.  Let’s take look at a Fluss log table. A log table can define: Optional partitioning keys (based on one or more columns). Without them, a table is one large partition. The number of buckets per partition . The bucket is the smallest logical subdivision of a Fluss partition. Optional bucketing key for hash-bucketing. Else rows are added to random buckets, or round-robin. The partitioning and buckets are both converted to an Iceberg partition spec. Fig 4. An example of the Iceberg partition spec and sort order Within each of these Iceberg partitions, the sort order is the Fluss offset. For example, we could partition by a date field, then spread the data randomly across the buckets within each partition. Fig 5. The partitions of an Iceberg table visualized. Inside Flink, the source will generate one “split” per table bucket, routing them by bucket id to split readers. Due to the offset sort order, each Parquet file should contain contiguous blocks of offsets after compaction. Therefore each split reader naturally reads Iceberg data in offset order until it switches to the Fluss servers for real-time data (also in offset order). Fig 6. Flink source bootstraps from Iceberg visualized Once the lake splits have been read, the readers start reading from the Fluss servers for real-time data. This is great for Flink streaming bootstrap (it is just scanning the data files as a cheap sequential scan). Primary key tables are similar but have additional limitations on the partitioning and bucketing keys (as they must be subsets of the primary key). A primary key, such as device_id , is not a good partition column as it’s too fine grained, leading us to use an unpartitioned table. Fig 7. Unpartitioned primary key table with 6 buckets. If we want Iceberg partitioning, we’ll need to add another column (such as a date) to the primary key and then use the date column for the partitioning key (and device_id as a bucket key for hash-bucketing) . This makes the device_id non-unique though. In short, Fluss is a streaming storage abstraction for tabular data in lakehouses and stores both real-time and historical data in stream-order. This layout is designed for streaming Flink jobs. But if you have a Spark job trying to query that same Iceberg table, pruning is almost useless as it does not use a batch-optimized layout. Fluss may well decide to support Iceberg custom partitioning and sorting (batch-order) in the future, but it will then face the same challenges of supporting streaming bootstrap from batch-ordered Iceberg. Confluent’s Tableflow (the Kafka-to-Iceberg materialization layer) took the opposite approach. It stores two copies of the data: one stream-ordered and one optionally batch-ordered. Kafka/Kora internally tiers log segments to object storage, which is a historical data source in stream-order (good for streaming bootstrap). Iceberg is a copy, which allows for stream-order or batch-order, it’s up to the customer. Custom partitioning and sort order is not yet available at the time of writing, but it’s coming. Fig 8. Tableflow continuously materializes a copy of a Kafka topic as an Iceberg table. I already wrote why I think zero-copy Iceberg tiering is a bad fit for Kafka specifically. Much also applies to Kora, which is why Tableflow is a separate distributed component from Kora brokers. So if we’re going to materialize a copy of the data for analytics, we have the freedom to allow customers to optimize their tables for their use case, which is often batch-based analytics. Fig 9. Copy 1 (original): Kora maintains stream-ordered live and historical Kafka data. Copy 2 (derived): Tableflow continuously materializes Kafka topics as Iceberg tables. If the Iceberg table is also stored in stream-order then Flink could do an Iceberg streaming bootstrap and then switch to Kafka. This is not available right now in Confluent, but it could be built. There are also improvements that could be made to historical data stored by Kora/Kafka, such as using a columnar format for log segments (something that Fluss does today). Either way, the materialization design provides the flexibility to execute a streaming bootstrap using a stream-order historical data source, allowing the customer to optimize the Iceberg table according to their needs. Batch jobs want value locality (data clustered by common predicates), aka batch-order. Streaming jobs want temporal locality (data ordered by ingestion), aka stream-order. With a single Iceberg table, once you commit to one, the other becomes inefficient. Given this constraint, we can understand the two different approaches: Fluss chose stream-order in its Iceberg tables to support stream analytics constraints and avoid a second copy of the data. That’s a valid design decision as after all, Fluss is a streaming tabular storage layer for real-time analytics that fronts the lakehouse. But it does mean giving up the ability to use Iceberg’s layout levers of partitioning and sorting to tune batch query performance. Confluent chose a stream-order in Kora and one optionally batch-ordered Iceberg copy (via Tableflow materialization), letting the customer decide the optimum Iceberg layout. That’s also a valid design decision as Confluent wants to connect systems of all kinds, be they real-time or not. Flexibility to handle diverse systems and diverse customer requirements wins out. But it does require a second copy of the data (causing higher storage costs). As the saying goes, the opposite of a good idea can be a good idea. It all depends on what you are building and what you want to prioritize. The only losing move is pretending you can have both (stream-optimized and batch-optimized workloads) in one Iceberg table without a cost. Once you factor in the compute cost of using one format for both workloads, the storage savings disappear. If you really need both, build two physical views and keep them in sync. Some related blog posts that are relevant this one: Beyond Indexes: How Open Table Formats Optimize Query Performance Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg Understanding Apache Fluss Apache Fluss uses zero-copy tiering to Iceberg . Recent data is stored on Fluss servers (using Kafka replication protocol for high availability and durability) but is then moved to Iceberg for long-term storage. This results in one copy of the data. Confluent Kora and Tableflow uses internal topic tiering and Iceberg materialization , copying Kafka topic data to Iceberg, such that we have two copies (one in Kora, one in Iceberg). Flink wants data ordered temporally for efficient streaming bootstrap. Batch workloads want data ordered by value (e.g., columns) for effective pruning and scan efficiency. Optional partitioning keys (based on one or more columns). Without them, a table is one large partition. The number of buckets per partition . The bucket is the smallest logical subdivision of a Fluss partition. Optional bucketing key for hash-bucketing. Else rows are added to random buckets, or round-robin. Fluss chose stream-order in its Iceberg tables to support stream analytics constraints and avoid a second copy of the data. That’s a valid design decision as after all, Fluss is a streaming tabular storage layer for real-time analytics that fronts the lakehouse. But it does mean giving up the ability to use Iceberg’s layout levers of partitioning and sorting to tune batch query performance. Confluent chose a stream-order in Kora and one optionally batch-ordered Iceberg copy (via Tableflow materialization), letting the customer decide the optimum Iceberg layout. That’s also a valid design decision as Confluent wants to connect systems of all kinds, be they real-time or not. Flexibility to handle diverse systems and diverse customer requirements wins out. But it does require a second copy of the data (causing higher storage costs). Beyond Indexes: How Open Table Formats Optimize Query Performance Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg Understanding Apache Fluss

Analytics Database

Jeff Geerling 10 months ago

Digging deeper into YouTube's view count discrepancy

For a great many tech YouTube channels, views have been markedly down from desktop ("computer") users since August 10th (or so). This month-long event has kicked up some dust—enough that two British YouTubers, Spiffing Brit and Josh Strife Hayes are having a very British argument 1 over who's right about the root cause. Spiffing Brit argued it's a mix of YouTube's seasonality (it's back to school season) and channels falling off, or as TechLinked puts it, " git gud ", while Josh Strife Hayes points out the massive number of channels which identified a historic shift down in desktop views (compared to mobile, tablet, and TV) starting after August 10. This data was corroborated by this Moist Critical video as well.

Data Media Analytics

Martin Fowler 11 months ago

Actions to improve impact intelligence

Sriram Narayan continues his article on impact intelligence by outlining five actions that can be done to improve impact intelligence: introduce robust demand management, pay down measurement debt introduce impact validation, offer your CFO/COO an alternative to ROI, equip your teams.

Business Analytics