An update on a pre-registered result about the coronavirus
By March 22nd, I strongly suspected there was a widespread coronavirus epidemic in Japan. This was not widely believed at the time. I, working with others, conducted an independent research project. By March 25th we had sufficient certainty to act. We projected that the default course of the epidemic would lead to a public health crisis. We attempted to disseminate the results to appropriate parties, out of a sense of civic duty. We initially did this privately attached to our identities and publicly but anonymously to maximize the likelihood of being effective and minimize risks to the response effort and to the team. We were successful in accelerating the work of others. The situation is, as of this writing, still very serious. In retrospect, our pre-registered results were largely correct. I am coming forward with them because the methods we used, and the fact that they arrived at a result correct enough to act upon prior to formal confirmation, may accelerate future work and future responses here and elsewhere. I am an American. I speak Japanese and live in Tokyo. I have spent my entire adult life in Japan. I have no medical nor epidemiology background. My professional background is as a software engineer and entrepreneur. I presently work in technology. This project was on my own initiative and in my personal capacity. What follows is the observed history of the last 30 days, an explanation of how the independent research project came to be, an explanation of how we chose to distribute our work product and why, and reflections on what this experience suggests for others’ efforts against the coronavirus. During mid-March, the public consensus of domestic experts, international researchers, authorities worldwide, and the international news media is that Japan is weathering the coronavirus situation well. The narrative centers on uncertainty surrounding the timing of a sporting event. As of March 19th, the official statistics show less than 1,000 coronavirus infections in Japan. This compares extremely favorably with the experience of peer nations such as Italy and the United States, despite Japan’s first infections having occurred earlier. The public mood is that prompt early action has been successful and normalcy has been mostly restored. Some believe widespread mask use, early social distancing measures, and implementation of a cluster-based management strategy has proven sufficient to contain the coronavirus situation. Doubts have been raised. Japan’s experience to this point has been extraordinary. At this point they are mostly idle speculation. Theories exist that suggest all is not what it may seem, but many of them allege implausible conspiracies. The consensus was not correct. There were actually widely geographically distributed outbreaks of coronavirus. The situation was worsening rapidly throughout March. That fact was discoverable but not widely known. Here is a brief recap of an extremely fluid situation, for the convenience of readers who have not kept up with it. The March 22nd and March 25th entries are not widely known. Thursday March 19th : The NHK panel of experts outlines the national consensus on the coronavirus situation, what the plan is, and what the risk factors to the plan are. Archived in Japanese and in English . (This is an unofficial translation commissioned from an unaffiliated commercial translation agency.) Saturday March 21st : Governor Yoshimura (of Osaka) distributes, on live television, an official assessment of the situation in Osaka and Hyogo. Few note that this happened or understand the significance of the assessment until later in the week. Sunday March 22nd : I tweet a hash at 11:24 AM, discussed below. Monday March 23rd : Governor Koike (of Tokyo) gives a press conference, at which she claims that it may be necessary to impose a lockdown on Tokyo if there is a rapid rise in infections. She immediately clarifies that this must be avoided at all costs. Tuesday March 24th : A sporting event is postponed. Wednesday March 25th : I tweet a hash at 1:31 PM, discussed below. Later in the day, Governor Koike (of Tokyo) urges the populace to start taking voluntary social distancing measures beginning on Saturday. Thursday March 26th : The New York Times publishes Japan’s virus success has puzzled the world. Has its luck run out? . A short time later, economist Tyler Cowen at George Mason University published The coronavirus situation in Japan is probably much worse than you think , covering claims made by a “working group.” Those claims are discussed in more detail below. In other news: the government’s panel of experts formally assesses that coronavirus is likely “rampant” throughout Japan. Friday March 27th : NHK, Japan’s national broadcaster, does a segment on why “a powerful American newspaper” has such questions about Japan’s well-contained coronavirus issue. Many reporters domestically begin to ask hard questions. Saturday March 28th : The Prime Minister addresses the nation, explains there is no present need to declare a state of emergency, but explains that Japan is in “a state of national hardship (国難) such that it has not experienced since the war.” [ Correction : After re-checking the transcript of this press conference and my notes, I realized this remark was actually made on April 1st. The press conference did mark a turning point, though, both for official recognition of the crisis and for the content of reporters’ questions about it.] Sunday March 29th through Sunday April 5th: There are increasingly urgent calls from prefectural governments, the Japanese Medical Association, researchers, and others to declare a state of emergency. Monday April 6th: The Prime Minister announces a state of emergency will be declared in seven prefectures, including Osaka and Tokyo. Tuesday April 7th: The New York Times publishes Japan declared a coronavirus emergency. Is it too late? Thursday April 9th: Aichi requests that the government broaden the scope of the state of emergency to include Aichi. Thursday April 16th : Japan broadens the state of emergency to include the entire nation, and designates 13 prefectures as being of particular concern, including Tokyo, Osaka, and Aichi. There are published reports in Tokyo of hospitals turning away suspected coronavirus patients. Friday April 17th : The Prime Minister addresses the nation, imploring it to reduce human-to-human contact by 70~80%. Monday April 20th : Increasingly specific and repeated calls to strengthen social distancing measures are beginning to show some objective improvement, though the situation is incredibly complex and in different phases throughout the nation. The number of infections continues to increase rapidly. There are increasing reports of difficulties in accessing medical care. The number of acknowledged coronavirus cases exceeds 11,000. The number of acknowledged cases requiring the highest level of care is approximately 230. The National Policy Agency has reclassified several suspicious deaths, including a man who passed away on a city street in Tokyo, as having been caused by coronavirus. Tuesday April 21st : There are increasingly common warnings by experts that, given the degree of spread of the infection, it is implausible that it will be rolled back within a year. We will likely see new outbreaks and new waves. Experts warn we will likely need to institute far stricter measures than seemed reasonable a month ago, to avoid a collapse of the medical system. The situation was much worse than we thought. On March 26th at 11:13 PM JST, economist Tyler Cowen published an essay on Marginal Revolution titled “The situation in Japan is probably much worse than you think.” The essay recounts of his correspondence, over the prior several days, with an anonymous “working group”, operating in Tokyo. The Working Group (proper noun hereafter) concluded that Japan had a geographically distributed coronavirus epidemic and predicted a public health crisis in April. The Working Group had been quietly circulating its evidence with appropriate parties in policy circles. As there had been pronouncements by experts officially which were equivalent to their research’s core result, that Japan factually had a coronavirus epidemic, the Working Group published the white paper anonymously to allow the public to prepare. I instigated the Working Group and I was the primary author of its white paper. Dr. Cowen linked to a copy of the white paper. We had published it anonymously via a third-party website moments before. Our choice to be anonymous was a considered one and is discussed in more detail below. Reception of the white paper was mixed. Some people viewed it with understandable concern, given the prevailing consensus that Japan was weathering the situation well. Some organizations were moved by its claims and took high-quality actions in response to it. Some people questioned the motives of the authors and suggested it was extremely unlikely to be correct, because credible science doesn’t get done by amateurs and then anonymously pastebinned. We had circulated an earlier version of the white paper privately to several parties to check results, obtain feedback, and inform their own investigations. Dr. Cowen was one of those parties. Here is the circulated version . The circulated version matches this hash from March 25th at 1:31 JST . It is substantially identical in conclusions to the published version, with minor wording differences. The final published version of the white paper is presented below, unedited from when it was published for distribution via Marginal Revolution. I’ve lightly visually distinguished it from the rest of this essay. Commentary continues afterwards . (If you need to reference the white paper directly, you can use this link . I do not control any other published copy and cannot vouch for their integrity or availability in the future.) The governmental and media consensus is that Japan is weathering covid-19 well. This consensus is wrong. Japan’s true count of covid-19 cases is understated. It may be understated by a factor of 5X or more. Japan is likely seeing transmission rates similar to that experienced in peer nations, not the rates implied by the published infection counts. The cluster containment strategy has already failed. Japan is not presently materially intervening at a social level. Accordingly, Japan will face a national-scale public health crisis within a month, absent immediate and aggressive policy interventions. Update as of the afternoon of March 26th : The government’s panel of experts has said that infections are highly likely to be “rampant.” We concur with that assessment. We are presently unaware of published official projections consistent with the projections discussed in this white paper, with the exception of the one from Osaka, discussed below. The consensus As of March 24th at 3 PM JST, Japan has reported 1,135 cases (excluding those aboard the Diamond Princess, charter flights, and officers attending to them), of which 859 are active, 54 are critical, and 41 have resulted in the death of the patient. The government has messaged a wait-and-see attitude (検討する) with respect to proactive containment measures outside of diagnosed clusters. There do not exist official reports of material community spread outside of surveilled clusters. The current figure is likely a massive undercount. If it is an undercount, it is highly unlikely that Japan’s official position that it is correctly identifying most clusters quickly and preventing spread from them is accurate. As Japan’s strategy is built around aggressive treatment and containment of clusters, breakout from clusters is an emergency. It threatens rapid uncontrolled increase in infections, which will cause a breakdown in care (which Japan refers to as an “overshoot”) when the hospital system is overwhelmed, leading to a sharp increase in deaths. We have statistical evidence suggesting that containment failure and community spread has already happened . Reasons to doubt the consensus Japan is undercounting asymptomatic individuals, who can spread the virus. Japan has had a public policy of refusing to test asymptomatic individuals except for those having a deep degree of direct contact with an infected individual (濃厚接触者), and initially only tested individuals with contact if they also had a fever or difficulty breathing. (「濃厚接触者」については、発熱や呼吸器症状が現れた場合、検査対象者として扱う) Accordingly, we should expect this testing policy to underdetect asymptomatic infections, and indeed we have evidence suggestive of this. The Ministry of Health, Labor, and Welfare has released national statistics for patients who were diagnosed with covid-19 as a result of a PCR test on 3/10, 3/11, 3/12, 3/13, 3/14, and 3/23. These statistics include breakdowns by whether a patient was symptomatic when diagnosed or asymptomatic. Asymptomatic patients are consistently roughly 10% of all diagnoses (ranging from a low of 10.24% to a high of 11.29%). This is far less than we should expect for covid-19. Japan’s experience on the Diamond Princess is instructive of what we should see if everyone in a population were extensively tested. All passengers were tested, repeatedly, before being allowed to disembark. The National Institute of Infectious Diseases reports that 48% of infected patients were asymptomatic at the time of sample collection. If the true rate of asymptomatic infection is higher than the observed rate of asymptomatic infection, and we make the generous assumption that 100% of symptomatic patients are successfully identified, then there must be a large population of asymptomatic infectious carriers who are not counted in official statistics. They are not subject, unless they are in direct contact with an infected individual, to any restrictions, monitoring, contact tracing, or medical care. They do not consider themselves ill. They do not know they are capable of spreading the disease to others. To estimate how many asymptomatic infections are being missed, we divide the number of cases of diagnosed symptomatic infections by the estimate of the true rate of symptomatic infections (which is lower than the observed rate). This gives us an estimate for the true number of infections. The difference between this estimate and the current official count are estimated asymptomatic infections which have not been diagnosed. E.g. Using the March 22nd Ministry of Health, Labor, and Welfare data: 907 / (1 - 0.48) =~ 1,750 true infections This is approximately 70% higher than the number of detected infections on March 22nd. This also magnifies the impact of any underdetection or misclassification of symptomatic infections. For every symptomatic infection missed by the current regime, we are also likely missing an asymptomatic infection. Iceland’s chief epidemiologist, who has testing capacity to cover almost the entire population of Iceland, reports that 50% of their infections are asymptomatic. If this ratio held in Japan, it would suggest Japan has ~80% more infections than are publicly reported. A pre-print from the Journal of Infectious Diseases , by a Japan-based team, uses Japanese citizens’ experience to calculate the asymptomatic rate. Japan evacuated citizens from Wuhan and exhaustively tested them on returning to the country, catching more asymptomatic infections than the testing strategy Japan generally employs. The researchers arrive at an asymptomatic ratio of 30.8%. This suggests that Japan has ~30% more domestic cases than are currently reported, again assuming perfect identification of all symptomatic cases. A pre-print from Eurosurveillance estimates the true asymptomatic rate on the Diamond Princess at 17.9%, using more sophisticated modeling than the calculation we performed above. This again implies that Japan is undercounting cases, though not as dramatically as either of the above estimates would suggest. Japan’s reported deaths are likely caused by a larger infection count than it is reporting. A pre-print from the Center for Mathematical Modeling of Infections diseases, Using a delay-adjusted case fatality ratio to estimate under-reporting , attempts to calculate the true number of cases by reasoning from predicted case fatality ratios (CFRs). Each death observed in a country at a particular date corresponds to, probabilistically, approximately 1 / CFR infections occurring over an interval of several days prior to the observed date of death. This research estimates Japan’s detection rate to be between 15% and 35%. Accordingly, Japan is undercounting cases by a factor of approximately 3X to 6X . Countries that test people who have been in Japan find more infections than Japan does. Singapore has traced four coronavirus cases to individuals who had just been in Japan. To be conservative we’ll assume three of these infections, not four, are Japan-related, as we have no knowledge of travel of the Japanese Singaporean resident. Due to the usual course of the disease, Singaporean health authorities concluded it is likely that these individuals contracted it while they were in Japan. Singapore’s sole airport keeps statistics of inbound passengers by country . The average passengers per month over February and March in 2018 and 2019 (most recent data available) was 128,000. Over a roughly 2 week period, we would expect approximately 64,000 passengers to fly from Japan to Singapore. 3 infections in that population is a rate of approximately 47 basis points, which is 5X the 9 basis points rate of infection in Japan. If one believes the government, the rate with surveilled clusters backed out be a tiny fraction of 9 basis points. This implies either that tourists are exceptionally unlucky at stumbling into exactly the wrong music shows or medical facilities or, in the alternative, that there is unsurveilled community transmission bringing Japan’s true case count to many, many times the admitted count. Why does Singapore’s airport detect more infections in people who have been in Japan than Japan detects in people who have been in Japan? It is likely because Singapore’s airport tests aggressively and Japanese medical offices do not. If we believe it is credible that tourists passing through Japan encounter clusters at the same rate as people who stay in Japan, which is unlikely, this implies the official count is understated by 5X . If tourists are only getting affected by community transmission, this implies the official count is understated by 25X or more . Where are the missing asymptomatic cases? Japan’s official position, as stated at the March 19th Panel of Experts on NHK ( 「感染拡大地域では自粛検討を」専門家会議が提言 ) as well as elsewhere, is that Japan is focusing on a cluster-based containment strategy. Japan’s testing capacity is underused, as a matter of policy. Local health offices control access to testing. The Japanese Medical Association has alleged, as reported by NHK, that doctors were denied permission to test more than 290 patients where the doctors felt the tests were medically necessary. Testing capacity (where it is used) is allocated to symptomatic individuals (preferentially to those with contact with diagnosed individuals or travel histories to epidemic-afflicted regions) and people with high degrees of contact with diagnosed individuals. The inference is thus that missing asymptomatic cases are outside of identified clusters, in the general population, potentially causing community transmission. The above statistics are strong circumstantial evidence of a very material number of non-clustered cases, which risk community transmission. We have direct evidence of community transmission as well: local Japanese governments are beginning to report it. Japan’s containment strategy is failing Japan official sources at the prefectural level are beginning to acknowledge that containment efforts locally are failing. For example, Osaka Prefecture and Hyogo Prefecture are currently epicenters of the outbreak in Japan. In a document which Gov. Yoshimura showed on TV on March 21st and is dated as having been prepared on March 16th, experts acknowledged that cluster containment was failing, as evidenced by new infections without a detected link to an existing cluster (community transmission). Quote: “It is believed that infections without surveilled chain to a cluster continue to increase and that therefore a rapid increase in infections has already begun.” (見えないクラスター連鎖が増加しつつあり、感染の急激な増加がすでに始まっていると考えられる。) Aichi Prefecture (Nagoya) has reported and unreported shortages of hospital beds suitable for high-grade infectious disease treatment (like covid-19) and generally. Asahi Shinbun reported on March 11th that over half of their capacity (161 beds) was already used. It was not generally reported at that time that they had begun triage. According to the prefecture on the 22nd , there are currently 103 hospitalized patients. 27 of them have not been tied to either of the prefecture’s known clusters. Nagoya is likely in a state of uncontrolled outbreak and medical care will likely suffer there, within days . As of March 19th, the government’s declared containment strategy remained rapid cluster identification, surging medical attention on diagnosed patients, and asking for voluntary changes in behavior from the populace. The cluster containment strategy has failed. Medical care in epicenters will be modified to adjust to a worsening reality, within days. The voluntary behavioral changes have been modest. They have been insufficient to maintain viability of the first two prongs of the strategy. We should not expect this level of behavioral change to prevent the situation from worsening. Travel within Japan is routine, central to commerce and leisure, and almost entirely unimpeded. In normal times, there are 500,000 passengers on the Tokkaido Shinkansen between Tokyo and Osaka per day . In February, usage of it was down by only 8% , per the Nikkei Shinbun. An outbreak in Osaka, Nagoya, Tokyo, or similar major metropolitan areas is extremely likely to metastasize throughout Japan absent aggressive restrictions on movement, especially public transportation. This outbreak and subsequent spread has almost certainly already happened. Japan is not materially preparing at a societal level It has been widely reported domestically and internationally that the Japanese populace is extremely cooperative, hygiene-focused, and has a culture of donning masks to prevent infecting others and/or as a precaution against seasonal hay fever. This is not a strategy . We are already observing breakout infections. We should assume, until we see persuasive evidence otherwise, that infection spread in Japan resembles that of peer nations taking minimal precautions. This requires us to believe the evidence of our eyes and our instruments, not the evidence of our hopes. The concrete action taken by Japan was suspending substantially all schooling nationwide on February 27th, two weeks before the spring holiday. Japan has discouraged large events, such as the live music event which generated Osaka’s first surveilled cluster. Aside from these measures, and individual citizens and organizations adopting very modest levels of caution, it is business as usual. By casual observation, mask wearing in central Tokyo is below 30%, including in well-attended outdoor events such as hanami (cherry blossom watching) parties. Reporters have not observed social distancing or universal mask use at press conferences about the epidemic . The government has recently released guidance that recommends masks most strongly in environments which are enclosed, with high density of people, with conversations or vocalization. This recommendation has not succeeded in closing bars or restaurants, and appears to carve out mass transit and hanami, which are economically and socially significant. It is uncertain the degree to which this carveout is warranted by medical science. Japan will face a national health crisis within a month Osaka forecasts a likelihood of 3,374 infections (including 227 severe cases) before April 3rd, compounding a rate of more than 6X per week . (This estimate was included in the document shared by the governor.) New York, with aggressive measures to slow the spread of disease, shows compounding of only approximately 2X per week. If government infection counts were accurate, but containment has failed, and we use optimistic doubling rates observed in peer nations taking aggressive measures, we would expect to see on the order of 3,000 cases, including more than 200 severe cases, in each of Nagoya, Osaka, Tokyo by the end of April . If we do not use peer nations’ experience for doubling rates, and instead rely upon the estimates of the cluster identification working group who prepared the report for Osaka, and we do not implement aggressive measures to slow the spread of disease, we could see more than ten times to one hundred times that number. These scenarios both will likely lead to a breakdown in provision of care, which Japan refers to as an “overshoot.” The experience of peer nations suggests that that would lead to patients dying in the wake of care being impeded, at a sharply increased rate, and in patients of other conditions dying as other forms of medical care are also impeded. Tokyo on March 23rd announced that it had 118 beds appropriate to safely treat high-level infectious disease patients, with plans of adding 700 beds for severely affected patients and 3,300 for those with moderate symptoms. Additionally, the bottleneck is likely not beds but rather skilled medical personnel; Japan faces an ongoing shortage of them at the best of times. Japan is likely unable to surge them from unaffected regions to epicenters. Many medical personnel already live in epicenters. Those that don’t will be equally needed to treat the likely coming uncontrolled outbreaks at home. The above scenarios apply estimates of growth rates to currently reported numbers of infections. The currently reported numbers are very likely not accurate . They are, as described above, a gross undercount. Reality is very likely worse than current guidance from official sources, and therefore the April we encounter will be worse than observers who rely on those numbers suppose. We project a true count of over 500,000 infections, including more than 5,000 severe cases, and a breakdown in provision of care (“overshoot”) in Nagoya, Osaka, and Tokyo, before the end of April. There will almost certainly be other breakdowns nationally. April is extremely unlikely to be the worst month. Accordingly, by the end of April, we will have an undeniable national public health emergency. Japan must act now As of March 19th, the government’s declared policy was that it would wait-and-see about infection spread and consider asking the public to engage in voluntary (自粛) social distancing in regions with outbreaks. Jake Adelstein reported “However, a Japanese official who gave an off-the-record briefing to Asia Times suggested that a “don’t ask, don’t tell” strategy, based on minimal testing and buttressed by information massage, has been quietly emplaced.” Governor Koike of Tokyo floated a trial balloon on March 23rd contemplating the possible lockdown of Tokyo if there were an outbreak in Tokyo, while emphasizing that a lockdown should be avoided at all costs. Governor Koike announced, on March 25th, a voluntary stay-at-home order for Tokyo starting on March 28th. Many cities in peer nations have experienced what we will soon experience. We do not believe they would advise us to delay our response. As a nation, we believe we are in a peaceful spring. We are not. We will, within the month of April, confront a crisis worse than any since the war. We must take immediate, concerted, aggressive policy steps in light of this reality. The white paper makes a large number of claims about an extremely complicated subject. A key principle of scientific investigation is that claims should be falsifiable ; one should be able to predict evidence which, if it became available, would disprove the claim. Claims which are not falsifiable are sharply less interesting because they can sound right without being right. Most claims in the white paper are designed to be falsifiable in the light of sufficient evidence. Many of them were not widely believed when written. The core result of the white paper is that there presently existed a coronavirus epidemic widely geographically distributed in Japan, which was beyond containment efforts. If one had believed that Japan factually had no such epidemic, one would have been proven unambiguously right when April failed to demonstrate thousands of geographically distributed coronavirus cases. The core result was correct. The white paper predicts that Japan would face an acknowledged public health crisis on a scale not experienced since the war, in April 2020. This prediction was correct. There are many, many more claims. I doubt every claim is exactly correct; predictions about the future, in particular, are sensitive to the future effectiveness of the response effort. We made the best possible effort to be thorough, to source our key claims to reliable official pronouncements or data sources, and to check the logic of our arguments with experts. At the same time, we were non-experts attempting a complex analysis of a fluid situation while racing against what we knew to be a developing crisis. I accept responsibility for any errors in the analysis. That is a complicated question. Different fields have different standards for what constitutes novelty and utility. The best and earliest reporting I am aware of related to this subject was Motoko Rich in the New York Times: Japan’s virus success has puzzled the world. Has its luck run out? , published late Thursday March 26th JST. This is, as far as I am aware, the first published artifact (anywhere) which goes beyond speculating that the public consensus is wrong to reporting facts which are incompatible with the public consensus. That article lays out dots that had not been connected in print before: the implications of the strategy to intentionally limit testing, the growing shortage of beds in several cities, the fact that government-affiliated experts had concluded exponential growth in infections was happening in Osaka, etc. The article is extremely good reporting. It gets the narrative essentially right. It sources key claims to official pronouncements. It gets credible experts to make very pointed observations about what those facts mean, attached to their names and reputations. The article made waves . The NHK (Japan’s national broadcaster) ran a segment on it on Friday March 27th, after which the discourse shifted markedly . Reporters in Japan began asking questions sounding like “Are we testing adequately? Really? ”, “If we have official projections for Osaka, which are disastrous… do there exist projections for Tokyo? What. Do. They. Say?”, “Experts say that the numbers presently believed to be true would represent a miracle. Has there been a miracle? ” The Prime Minister addressed the nation on the night of Saturday March 28th, explaining that Japan faced “a state of national hardship (国難) such that it has not experienced since the war.” [ correction : this comment was made on April 1st] while Japan had not yet experienced a rapid increase in cases as Europe and the U.S. had, it could do so at any time. There was uncharacteristically aggressive questioning from reporters. The NYT followed that article with other pieces. They make for excellent reading. Newspapers are not, however, formally in the business of making predictions about the future, particularly not unsourced speculative predictions. The article does not include any speculative predictions about what might come to pass in April. There are organizations in the world which are in the business of making predictions about the future, and then taking actions consistent with those predictions. Very few of those organizations adopted public postures consistent with having high certainty in correct assessment of the situation in March or predictions about April. Many reasonably calibrated people would have a guess as to which organization was most likely to understand the shape of the near future on this issue. I will broadly refrain from comment regarding that organization, for predictable reasons. May history be understanding of everyone living in troubled times and working with imperfect information… and may history arrive at the truth. To “pre-register” a prediction or conclusion is to commit to it, prior to doing the analysis or waiting for the event which the prediction is about. I pre-registered our results via, and this is jargon, “dropping a hash on Twitter.” A security researcher understands that phrase to mean: I had a sensitive document. I have demonstrated an exact time by which I possessed it. I did not publish the document publicly at that time. I expected possibly publishing it in the future. I would be able to easily demonstrate, and they would be able to easily check, that a future publication was an exact and unedited copy of that document. If you are not a security researcher, here is a primer on how this works and why the security community uses it. This technique has also been used by scientists and mathematicians to pre-register results, including by Isaac Newton. (“At present I have thought fit to register [my results] by [use of a cypher which predates modern cryptography].” page 123 ) The Working Group had their analysis substantially completed on Wednesday March 25th in the early afternoon, though wordsmithing continued. The Working Group was formed two days earlier , on the morning of Monday March 23rd. There were four members, all professionals living and working in Tokyo, from a variety of career and national backgrounds. No member has worked in medicine or epidemiology. At the point of formation, we individually had theories. We were close to convinced that there was a widely geographically distributed epidemic in Japan. We had insufficient proof to convince experts the theories were correct. We were not sufficiently convinced that we were correct to warrant a robust response effort. I do not want to tell the stories of each member of the Working Group. Perhaps they will in their own time. But I can tell you when and how I arrived at my theory. I spent February dealing with the quotidian problems of being employed with young children and aging parents. I was not more than peripherally aware of the coronavirus problem. I help people sell software; an epidemic in China and Iran didn’t seem like obviously the most important thing in the world. I said something along those lines on Twitter, got knuckle rapped by a better calibrated friend, and got a bit more up to speed. By early March I was extremely worried about the safety of my family in the United States. I was skeptical that published descriptions of the state of affairs in Japan were fully accurate but thought the likely impact locally would be low. I know the exact moment that I got worried for Japan: when I understood, from first-hand reports out of Italy, that coronavirus had non-linear impacts not just for the supply chain (which I understood) but also for the healthcare system. On March 10th, I looked around in Tokyo and imagined healthcare substantially collapsing here as well. I didn’t do anything major, for a while. I assumed everyone in a position to act had reached this conclusion . Surely, talented epidemiologists understand exponential growth and already know the shape of things to come. Surely, there is a department of people working day and night on the national response effort. Surely, all professionals understand how to parse official communications. Surely, in a hundred thousand rooms, responsible professionals have told their leadership what would soon happen. Surely, in a hundred thousand rooms, people in positions of authority have pulled out a thick red binder, steeled their hearts, and gone to work. And so I mostly focused on the situation abroad, worrying for family and friends. I tried to distract myself with work. Then, by complete happenstance, two very smart people in one day told me that they believed the public consensus . Japan was weathering the coronavirus situation well. I was lucky to be in Tokyo. Imagine how surprised you would be if someone turned into a cat in front of you. That was how surprised I was. I immediately attempted to update them on my understanding and reasoning for it, assigning an 80% probability to Japan (and Tokyo specifically) following the experience of peer nations. I suggested they orient their organizations consistently with that. And then I looked at the behavior of people in the world, really looked, and suddenly had the terrifying thought that maybe many did not know. So I wrote a memo to myself, on Sunday March 22nd, to gather my thoughts on the matter. It started from a simple thought exercise: what would I see in the world if Japan was in the midst of an uncontrolled coronavirus epidemic? And were we seeing those things? It came to the conclusion that we were. I was 90% confident that I had reached the correct result. I was far less confident that the memo would change anyone’s views unless they had an exceptionally high regard for me. I hashed the memo and published the hash on Twitter. I did not publish the memo at the time. I considered immediately publishing the memo. I did not feel that this was likely to be instrumentally effective at saving lives. I thought that risks associated with publishing could potentially cost lives or bring strong sanctions down upon myself and people close to me. I felt, while my subjective confidence was 90%, that I was likely miscalibrated. The likelihood that one non-expert, doing casual sleuthing in his spare time, had scooped not just any expert but almost all the experts and almost all the parties in formal authority felt infinitesimally small. So perhaps my internal calibration was wrong. Then I reasoned through two worlds: If I published and was living in a universe where Japan did not have a present epidemic, there was no possible gain to claiming there was one. There was some possibility of inciting a panic and diverting efforts from orderly management of a moderate public health problem. Causing any hindrance whatsoever to the containment effort would be terrible; the world had ample evidence of what uncontained coronavirus outbreaks looked like. A distant secondary consideration, but a real one, was that choosing to publish could bring down heavy sanctions, through predictable pathways. If I published and was living in a universe where Japan did have a present epidemic, it felt unlikely that publishing speculation would rapidly recalibrate the understanding of a department of people surely already considering the issue. Indeed, plausibly it would cause them to delay better, more scientific analysis. Delaying analysis delays the consequent response effort; delaying the response effort is unthinkable. Similarly, it is plausible to me that public health authorities are sometimes parsimonious with information flow to maximize the effectiveness of policy interventions. There was a high-salience political question being debated domestically about a long-time centerpiece of economic strategy. I believed that the ultimate resolution of that question was inevitable, regardless of whether Japan had an epidemic. Even supposing I were to be right, I feared that there would need to be a scapegoat for that resolution, which was inevitably going to be acutely unpopular. This might leave me right about the epidemic, ineffective or counterproductive at accelerating the response effort, and responsible for bringing severe sanctions down upon on myself or people close to me. I didn’t know what to do. So I used a trick that I have done a handful of times before, mostly regarding predicting perfidy of Bitcoin exchanges, and tweeted a hash. I had an idea on how to use the hash purposefully . I have been writing on the Internet for almost 15 years. I am broadly respected in my community. People know that I’m a congenital optimist. There was a coded, deniable way to say not just “I have discovered an interesting fact about one of my weird hobbies; tune in later and we’ll see if I was right” but “I am convinced, in my bones, that this time the sky is actually falling and that you likely don’t know that yet. If you know what I’m saying and you trust me: get out the red binder.” In a world where I was wrong, people who trust me professionally would likely think less of me. That was an acceptable risk. And in a world where I was right, it would at least have allowed some to prepare, and created a record for future investigation. I felt that some people would read between the lines of this tweet: 1) I am materially wrong about the most consequential thing I've had to have a view on in 15 years. You should probably degrade your estimate of my ability to think through complex problems. 2) We need a data point to couner "Nobody could possibly have seen this coming." Shortly thereafter, I got an email from someone I respect enormously. He understands the idiom of dropping a hash on Twitter. He has read enough of my work to have a high opinion of me and a tight bead on my interests. He assumed that the most contentious topic I was likely to have a surprising opinion on was on covid-19. He suggested I publish immediately , with the objective of saving lives by giving people time to prepare. I told him why I thought publishing what I had could be counterproductive, but that plausibly I could work harder and get something convincing enough to accelerate conclusions in policy circles. He suggested that I leverage a news organization. News organizations are more credible to policy circles than private individuals. News organizations have resources to check conclusions and strong institutional controls to avoid publishing imprudently. A deserved reputation for this allows them the ability to distribute trustworthy conclusions across many policy makers quickly, in parallel . This might accelerate policy deliberations versus briefing policy makers serially. News organizations do not fear the social consequences of getting ahead of news stories. They consider that almost uniformly positive. This strategy had not occurred to me . The only options I was actively considering were “publish in my personal capacity” or “send a memo to… I lack sufficient insight into the decision-making process here to know who it would be addressed to, actually.” As soon as he made the suggestion, the outlines of a plan fell into place: Find like-minded people. Write a better memo. Check it with a medical researcher. Brief as many organizations as would talk to us, as close to policy apparatuses as possible. I began working. Twenty minutes later, in the dead of night, three people in Tokyo had agreed to a kickoff meeting on Monday morning. Via videoconferencing, naturally. On Monday we fleshed out the goal (“Accelerate the response effort, with the goal of saving lives”), a plan of action (“Leverage high-status organizations to surface internal conclusions we think probably exist, or generate evidence sufficient to bootstrap experts to those conclusions rapidly”), and a timeframe. We also recruited our fourth and final member. We wanted the conclusions in policy circles by Friday, because policy circles might take weekends off but coronavirus does not. We knew interesting reporting in English-language media typically takes half a day to percolate among domestic bilinguals prior to being translated. This implied a publication deadline of Thursday. We assumed media organizations would need our work by Wednesday to check it and produce reporting informed by it. Why did we not produce our work in Japanese? Partly, we thought Japanese-language media outlets were less likely to bite on this shape of story coming from this shape of sources. Partly it was simple expedience. I write ten times faster in English than in Japanese, which is not an advantage to squander when every day matters. This gave us approximately 54 hours to do our research, check it with an expert, produce a credible artifact, and provide it to appropriate parties. Monday through Wednesday passed in a blur. We digested a dozen epidemiology papers. We conducted a review of two months of reporting in Japanese and English. We came up with a hypothesis why our result wouldn’t be obvious to existing efforts. We figured out which data sources and which signals were either likely underexamined or intriguing enough to suggest follow-up. We crunched numbers. We found a medical researcher to review our conclusions. We incorporated feedback from that researcher. We wrote the memo. We decided to call it a white paper, because that sounded more credible. Then we briefed a small number of organizations about it. We landed a major break on Monday, almost by accident: we found the document which Governor Yoshimura had presented on live television on Saturday. The document was written by epidemiology experts and included the conclusion that the infections in Osaka and Hyogo had escaped containment and that exponential growth had already begun. After reading this document, we moved to ~100% confidence that Japan factually did have geographically distributed outbreaks. This official assessment also excluded the possibility that we were the only team which had reached the core result. That had always felt likely, but came as an immense relief. It meant the response effort would arrive sooner. It also meant we only had to assist others in understanding the import of experts’ results, rather than convincing others that our result was equivalent. We had our research. We had checked it as well as we were capable of within our networks. We had official confirmation of the core results about the status quo. We started having conversations with appropriate parties. Some desire to remain unnamed; others may be happy to share that they spoke with us. Our goal was accelerating their ability to take appropriate actions. Some organizations quickly made high-quality decisions after being shown the white paper. In some cases, they told us that they were able to use portions of our work to inform their own work. At some point in the future, stakeholders in substantially every organization worldwide will probably inquire as to how their organization performed during the coronavirus pandemic. May all judgments be just and merciful. Some have advanced a theory which starts from the observed timeline and assumes the behavior of very many people was motivated by a sporting event. Beware of simple, narratively compelling explanations for the behaviors of complicated systems. I do not believe there was a decision made to prioritize a sporting event over public safety. I can believe that the fact of the sporting event impacted the operation of a very complicated system. These are two very different claims. Consider myself as a very small cog in a very complicated system. No one has ever told me that I should prioritize a sporting event. No one who has ever known me would accuse me of sharing that preference. Sontaku (忖度) means, roughly, to intuit the preferences of other parts of the system and act to facilitate them without ever having been explicitly instructed to do so by formal authorities within the system. Some people believe that sontaku is a uniquely Japanese phenomenon. I tend to believe that it is a useful word which describes an extremely common human behavior. A regulatory lawyer in the U.S. advising their client on the risk level of a particular course of action is likely sontaku-ing using their mental model of the regulator rather than directly asking the regulator what to do. Being good at sontaku-ing the regulator is core to the lawyer’s job. The client is plausibly sontaku-ing the lawyer at the same time! Their lawyer doesn’t have to order them to abandon an unwise course of action. Their lawyer has no authority to do that. All the lawyer has to do is say “It is my opinion as your attorney…” Sophisticated clients will understand that sentence, worded in that fashion, as a non-order. Sophisticated clients will likely follow that non-order, because getting the benefit of high-quality non-orders is why you hire regulatory attorneys. We could conclude from this that American regulatory lawyers are inscrutable. Or we could conclude that they are humans, doing something humans do in systems much bigger than themselves. This little cog sontaku-ed its way into understanding that the fact of the sporting event materially impacted parts of the calculus. There are few conspiracies in the world. There are many systems with complicated decision-making processes, internal data flows, and incentive structures for actors within them. Sometimes those systems do not produce the results the system would most profess to want or that actors within them would want. I have some speculative hypotheses. I am substantially less confident in them than I was confident in our earlier results. Call it 60% that these represent a major factor. These are offered in the hopes that future researchers can ask the right questions. Japan had, speaking inexactly, a Plan A and a Plan B. They’re described in detail by the panel of experts report in Japanese and in English (this is an unofficial translation commissioned from a commercial translation agency). Plan A revolved around aggressively containing clusters with a goal of preventing community transmission. Plan A allocated almost all testing capacity (and responder efforts! Tests don’t ask you about your social graph themselves! That’s hard, manual work!) to quickly defining the boundaries of clusters. Plan A surged medical attention on diagnosed cases, hospitalizing (and thereby sequestering) all diagnosed individuals. Plan A asked for relatively minor social distancing measures from the populace, principally avoiding holding large events in areas with known cases. Plan B is the playbook arrived upon by most Western nations. Plan B is known to be extremely unpalatable for a variety of reasons. You can call it a lockdown, you can call it putting the economy into stasis, you can call it strongly-suggested extraordinary social distancing, you can call it whatever you want, but nobody in the world is enthusiastic about instituting Plan B unless they need to . There was a designed transition from Plan A to Plan B. If one observes an explosive growth in infections, then Plan A is no longer viable, so move to Plan B. There are, similarly, various intensities available within Plan B, with predefined guidelines as to when they should be employed in an area. Given Plan A appeared to work , substantially everyone preferred it to Plan B. There were likely departments upon departments of people who were implementing Plan A, collecting data from the testing regime specified by Plan A, reporting on the success of Plan A, etc. I have no reason to believe, and do not believe, anything but that they carried out their duties to the best of their abilities. My hypothesis: it was many peoples’ job to carry out Plan A. It was in many peoples’ interests for Plan A to work. The daily updates about Plan A went to many people. They monitored Plan A’s progress under its own terms. It may not have been anyone’s job to wake up every morning, assume Plan A was no longer viable, and search for proof that that had happened. Plan A’s specified testing regime may be accidentally incapable of telling you Plan A has failed . How is that possible? Because both Plan A and Plan B were substantially written years ago , before the world understood the biology of covid-19. They may not have adapted quickly enough to what we learned early in 2020. In particular, this non-expert believes they do not sufficiently make an allowance for transmission of the infection by asymptomatic patients, and they assume that the supermajority of infection spread happens within clusters. And so they concentrate testing capacity “on and around” clusters, on symptomatic patients likely to have worse clinical experiences. By design, with the best of intentions, Plan A does not “waste” testing capacity or medical resources on asymptomatic patients or people outside of clusters. Plan A assumes most of these patients need no help and some will just need the ordinary attention of the medical system. If they happen to cause a cluster, then that cluster will be quickly detected and contained. Plan A assumes it will quickly learn if it is becoming untenable, because it will see clusters without a surveilled chain to other clusters. If you have a cluster, and you rigorously test people “close to it”, you will find approximately the correct number of symptomatic patients. You will quickly hospitalize them and give them the best medical care. Everyone involved does their job. Everyone gets feedback that they are doing their job correctly. Most patients get better. Plan A looks like it is succeeding, and considers itself succeeding, until it detects undeniable evidence of sustained community spread. The perfect storm for this strategy is: what if coronavirus is capable of causing clusters but also capable of exponential growth just through community transmission by asymptomatic patients? What if you successfully reduce the incidence of clusters such that they don’t become undeniably common before you have tens of thousands of infected patients very well-distributed throughout your social graph? In that hypothetical world, you end up with… something which looks not too unlike the experience of Japan in March 2020. That is one hypothesis. Experts will thoroughly study the experience of the last few months. Many people doubted that there was a coronavirus epidemic because evidence of it should be extremely obvious . Here are three doubts one might have, and some hypotheses as to why those doubts did not point in the correct direction. Experts will eventually have much better data and more reasoned conclusions than these. “An uncontrolled coronavirus epidemic should, within a month, result in many thousands of patients. Where are the patients?” The asymptomatic ones are exactly where they would otherwise be on any given day in spring. The symptomatic ones are likely receiving care for a viral infection or shrugging it off because their symptoms are mild. If they seek care, they will likely to be told to get some rest and drink liquids. Most will do that and get better. Some will have a few very difficult days, but then get better. A small number will be hospitalized. They will get aggressive high-quality care for pneumonia, but pneumonia is much more common than coronavirus and does not trigger a test in Plan A. The testing criteria require both severe symptoms and a link to a cluster (or recent trip to a nation widely believed to have an epidemic). “You can’t have an uncontrolled coronavirus epidemic without a sharp increase in mortality. Where do we see that?” An average of approximately 3,500 people pass away in Japan per day. Who is the first person to produce an Excel analysis about this? What is their job title? At what cadence do they refresh this analysis? How many days does it take the data to flow through various organizations and stabilize, such that the number in that Excel file is complete and reliable? What is the analyst’s threshold to notice they are surprised? How quickly can they get that message to the departments in charge of Plan A? How quickly can they overcome those departments’ insistence that they’re running all the tests Plan A calls for and gotten results consistent with a well-managed public health issue? I don’t know if that stylized analyst is a particular person, a department, a process, or similar. But I am aware of one system which functions not dissimilarly to this stylized analyst. The National Institute of Infectious Diseases collects per-large-city stats on flu and pneumonia deaths. It takes weeks for this data to be available to them, and the quality and freshness of the data is not uniform over the 21 large cities they cover. Here are Tokyo’s stats . As of this writing, they show excess deaths in late February relative to the National Institute of Infected Diseases’ baseline. This was analyzed as above their model’s threshold for whether the number of excess deaths is unexpected by statistical variance. The number of excess deaths in Tokyo was approximately 50 per week for two weeks. Data for March is not yet available. The other cities in their data set either do not have any reports available or do not consistently show excess deaths above the threshold, as of the most recent statistics available, all of which are on a delay of several weeks. Tokyo is a large metropolis and one might need to put that number in perspective. One perspective: Tokyo has reported approximately 75 coronavirus deaths total through this writing. The March numbers for flu and pneumonia deaths are modeled to decrease as the flu season winds down. You may have a high-quality hypothesis as to what I believe those numbers will show once they are available. If these numbers reflected primarily seasonal flu and typical cases of pneumonia, April’s numbers would decline again from March’s. It will be very observable, in approximately six weeks, if this indeed happens. “Why did it appear to work for so long?” My leading hypothesis, again as a non-expert, would be substantial path dependence in how coronavirus spreads over social networks. We’ve seen this in cities in many countries: “bad luck” with one early patient at a well-attended wedding or funeral can quickly spiral into hundreds of patients, each of whom starts their own infection chain. Early patients may have happened to not be central to the social graph of Japan. Most people who came into contact with them got lucky. It took a few more generations for the virus to get into the densely connected part of Japan’s social graph than it did in peer nations. Coronavirus’ biology means the difference of a few generations makes a massive difference in the scale, and therefore detectability, of the epidemic. For a few days. Combine this with Plan A probably legitimately succeeding with regards to almost all the patients from almost all the early clusters. Against covid-19’s biology, “almost” buys you some number of days but does not suffice to blunt the epidemic. We spend a day every day. I think that plausibly gets you pretty close to the observed experience of March without requiring notable malfeasance, theories that Japan is dissimilar from other Western nations, a unique unobserved environmental X-factor, etc. We, humanity, are still in the early days of our response to coronavirus. We are even earlier in understanding what the last few months should tell us to do regarding future challenges. We, the Working Group, produced a novel result and we produced it relatively early. Coming forward and showing the result was early and correct enough to act upon will, hopefully, allow others to produce better results faster and give themselves permission to make correct decisions sooner. Good science and good actions informed by science will save lives. A long time ago I did a bit of work on disaster alert systems. They’re not dissimilar to fire alarms. The engineered purpose of a fire alarm is not merely to let people know there is a fire. Many will have already perceived the fire. The alarm, buttressing training delivered far before the alarm rings, gives you unquestionable and immediate permission to evacuate . We know that otherwise some people, smelling smoke and feeling uneasy, would look around the room, see other people not moving, and conclude “Who am I, to disrupt everything going on by shouting ‘Fire! Fire! Fire!’?” The history of humanity has seen far too many rooms where no one shouted ‘Fire!’ early enough. Too many people and too many organizations have passed precious days this year waiting for permission to take actions reasonable given evidence they already had . You, yes you, in your individual capacity, are not done with consequential decisions you will have to make about coronavirus. You had a decision point every day in February on whether to buy more food than usual. Perhaps you bought extra, expecting the likelihood of a lockdown or disruption in the supply chain. Perhaps not. Either way: you will be faced with that same decision again. You will have other ones, too. We will not be done with coronavirus in the next few months. You have likely not seen the final iteration of extraordinary measures in your city. You, yes you, upon whose considered judgment others rely for their safety, need to understand what has happened this year. You need to urgently adapt your internal decision-making process on the basis of available evidence. You do not have the luxury of waiting until the papers are accepted for peer review, the inquiry is finished, and the history books are written. The virus will not wait. Humanity must act faster than the disaster. We hope, in our small way, to accelerate your understanding and actions. In that spirit, some additional reflections, representing things I now believe that I would have been surprised by in January. Perhaps you can consider them more carefully and faster than I did. I think epistemic humility is invoked perhaps a bit too much. There is something of a meme going around that experts need space to do their work and that non-experts intruding onto their terrain only wastes their time. I have heard it called “epistemic trespassing.” You can see above where believing the truth of this narrative cost days of precious time. Similar concerns have delayed response efforts elsewhere. I have a great appreciation for science but am pretty ambivalent about opinions as to whether the white paper constitutes it. If one needs to call it “just lucky guesses”, fine; how do we make radically more lucky guesses this year? No professional courtesy, no personal discomfort, no preservation of modesty, no fear of repercussions should delay the response. Get a right enough answer. Make a right enough decision. Be prudent, check your work, be appropriately confident in it, but then act and act boldly. I have a renewed appreciation for speed. Covid-19 does not sleep. The disease course and doubling times absent intervention are facts which the universe offers to us. They happen to be very short. This is inconvenient for many of our institutions. The biological reality of the virus cares nothing for our convenience. Some organizations are have difficulty making decisions under uncertainty and quickly iterating on them. Some organizations are built around this feature. The second type of organization will outperform the first on a problem moving as fast as a coronavirus epidemic. We should think, very carefully, on how we design organizations, particularly organizations which are systemic chokepoints during prompt crises. They need to move faster than seems currently reasonable to hope for. Twitter is having its finest hour. If the Working Group had a fifth member it would be Twitter. Twitter put coronavirus on our radars weeks before more traditional sources. The experience of peer nations, delivered in real-time over Twitter, taught what signals would flip locally before danger was obvious. Twitter connected four professionals in a large city. This made us ambiently aware of each other sufficiently such that, in the dead of night and with no prior discussion, we knew who locally was steeped in the problem, trustworthy, and skilled. Twitter accelerated our ability to identify and contact responsible individuals at several organizations. Twitter enabled the hash-as-signal-flare tactic. This enabled the right person to offer the right advice very quickly. This advice lead to a better plan days faster than a Twitterless world. Software really, really matters. We successfully executed a fast, novel research project without ever being in the same room (for obvious reasons). Messaging apps, videoconferencing software, office suites, and similar were crucial to the effort. Very little of it existed even ten years ago , certainly not in a form usable enough for us to have hit our timeframe. Most of them will run on the phone in your pocket. Some are frequently used for frivolous ends and therefore too often considered frivolous themselves. They are not marketed or understood as tools for the advancement of human knowledge. They work. They work. They work. Because of free or cheap software (mostly already installed on our computers and phones), we spent less than 30 minutes total on booting up infrastructure for four researchers and every minute after that on achieving results. The price of infrastructure matters, too. The total spent on the project was less than $2,000. Our grant-making agency was a credit card. There is a difference between the truth of a thing, and the acknowledgment of the truth of a thing. Japan faced a coronavirus epidemic no later than early March. That is a truth; it was not an acknowledged truth until later. I would hope that, if any truth about my minor part in this is acknowledged, it is this: I am a responsible professional. I have no relevant expertise or authority. I have avoided unproductive criticism. I might have made some guesses, during a year when many people were guessing on many topics. I quietly told some people about them. Some guesses may, perhaps by happenstance, align with official guidance and credible published reporting. And should that be the truth acknowledged, and less convenient truths be quickly forgotten, that would be fine by me. If I am wrong, then I will accept any consequences. My actions were my own actions. This is a very different essay than my typical work. It may be judged to very different standards in very different quarters than usual. Please excuse my need to write more reservedly and participate less in subsequent commentary than I usually would. Some truths have social consequences, acknowledged or not. Humanity’s response to coronavirus implicates almost everyone. We live in imperfect, divided, fractious, and hurting world, as we always have. We will comfort the afflicted. We will mourn those who pass. We will learn. We will beat this thing , with high science and with hand soap. The Working Group hopes that we have been of some small service to the response effort. I’m presently sheltering at home with my family. The immediate future is unlikely to be easy, for anyone. I hope to see you here in happier times. Tokyo truly is the greatest city in the world. I wish you health, safety, and success in your endeavors.
Business
Journalism
JavaScript
Gaming
Data
HTML
JSON
Career
Go
Marketing
Data Analysis
Security
Statistics
Programming
Shell