Text analytics hints at how volumes of writing get\u00a0created<\/h4>\n
In the broadest of strokes, Natural Language Processing transforms language into constructs that can be usefully manipulated. Since deep-learning embeddings have proven so powerful, they\u2019ve also become the default: pick a model, embed your data, pick a metric, do some RAG. To add new value, it helps to have a different take on crunching language.
The one I\u2019ll share today started years ago, with a single\u00a0book.<\/p>\n
The Orchid Thief <\/em>is both non-fiction and full of mischief. I had first read it in my 20s, skipping most of the historical anecdata, itching for its first-person accounts. At the time, I laughed out loud but turned the pages in quiet fury, that someone could live so deeply and write so well. I wasn\u2019t all that sure these were different things.<\/p>\n Within a year I had moved to London to start anew. Lots being the operative word.<\/p>\n Behind the modern fa\u00e7ade of professional services, British industry is alive to its old factories and shipyards. It employs Alice to do a thing, and then hand it over to Bob; he turns some screws, and it\u2019s on to Charlie. One month on, we all do it again. As a newcomer, I noticed habits weren\u2019t so much a ditch to fall into, but a mound to\u00a0stake.<\/p>\n I was also reading lots. Okay, I was reading the New Yorker<\/em>. My most favourite thing was to flip a fresh one on its cover, open it from the back, and read the opening sentences of one, Anthony Lane, who writes film reviews. Years and years, not once did I go see a\u00a0movie.<\/p>\n Every now and again, a flicker would catch me off-guard. A barely-there thread between the New Yorker <\/em>corpus and my non-Pulitzer outputs. In both corpora, each piece was different to its siblings, but also\u2026not quite. <\/em>Similarities echoed. And I knew the ones in my work had arisen out of a repetitive process.<\/p>\n In 2017 I began meditating on the threshold separating writing that feels formulaic<\/em> from one that can be explicitly written out as a\u00a0formula<\/em>.<\/p>\n The argument goes like this: volume of repetition hints at a (typically tacit) form of algorithmic decision-making. But procedural repetition leaves fingerprints. Trace the fingerprints to surface the procedure; suss out the algorithm; and the software practically writes\u00a0itself.<\/p>\n In my last job, I was no longer writing lots. My software\u00a0was.<\/p>\n Companies can, in principle, learn enough about their own flows to reap enormous gains, but few bother. Folks seem far more enthralled with what somebody else<\/em> is\u00a0doing.<\/p>\n For example, my bosses, and later my clients, kept wishing their staff could mimic the Economist<\/em>\u2019s house style. But how would you find which steps the Economist<\/em> takes to end up sounding the way it\u00a0does?<\/p>\n Read a single Economist<\/em> article, and it feels breezy and confident. Read lots of them, and they sound kind of alike. A full printed magazine comes out once a week. Yeah, I was betting on\u00a0process.<\/p>\n For fun, let\u2019s apply a readability function (measured in years of education) to several hundred Economist<\/em> articles. Let\u2019s also do the same to hundreds of articles published by a frustrated European asset\u00a0manager.<\/p>\n Then, let\u2019s get ourselves a histogram to see how those readability scores are distributed.<\/p>\n Just two functions, and look at the insights we\u00a0get!<\/p>\n Notice how separated the curves are; this asset manager is not<\/em> sounding like the Economist<\/em>. We could drill further to see what\u2019s causing this disparity. (For a start, it\u2019s often crazy-long sentences<\/a>.)<\/p>\n But also, notice how the Economist<\/em> puts a hard limit on the readability score they allow. The curve is inorganic, betraying they apply a strict readability check in their editing\u00a0process.<\/p>\n Finally\u200a\u2014\u200aand many of my clients struggled with this\u200a\u2014\u200athe Economist<\/em> vows to write plainly enough that an average highschooler could take it\u00a0in.<\/p>\n I had expected these charts. I had scribbled them on paper. But when a real one first lit up my screen, it was as though language herself had\u00a0giggled.<\/p>\n Now, I wasn\u2019t exactly the first on the scene. In 1964, statisticians Frederick Mosteller and David Wallace landed on the cover of Time<\/em> magazine, their forensic literary analysis settling a 140-year old debate<\/a> over the authorship of a famed dozen of anonymously-written essays.<\/p>\n But forensic analytics always looks at the single item in relation to two corpora: the one created by the suspected author, and the null hypothesis. Comparative analytics only cares about comparing bodies of\u00a0text.<\/p>\n Let\u2019s retrace our steps: given a corpus, we applied the same function on each of the texts (the readability function). This mapped the corpus onto a set (in this case, numbers). On this set we applied another function (the histogram). Finally, we did it to two different corpora\u200a\u2014\u200aand compared the\u00a0results.<\/p>\n If you squint, you\u2019ll see I\u2019ve just described Excel.<\/p>\n What looks like a table is actually a pipeline<\/em><\/strong>, <\/em>crunching columns sequentially. First along the column, followed by functions on the results, followed by comparative analysis functions.<\/p>\n Well, I wanted Excel, but for\u00a0text.<\/p>\n Not strings\u200a\u2014\u200atext. I wanted to apply functions like Count Verbs or First Paragraph Subjector First Important Sentence. And it had to be flexible enough so I could ask any question<\/em>; who knows what would end up mattering?<\/p>\n In 2020 this kind of solution did not exist, so I built it. And boy did this software not \u2018practically write itself\u2019! Making it possible to ask any question needed some good architecture decisions, which I got wrong twice before ironing out the\u00a0kinks.<\/p>\n In the end, functions are defined once, by what they do to a single input text. Then, you pick and choose the pipeline steps, and the corpora on which they\u00a0act.<\/p>\n With that, I started a writing-tech consulting company, FinText<\/a>. I planned to build while working with clients, and see what\u00a0sticks.<\/p>\n The first commercial use case I came up with was social listening<\/a>. Market research and polling are big business. It\u2019s now the height of the pandemic, everyone\u2019s at home. I figured that processing active chatter on dedicated online communities could be a new way to access client thinking.<\/p>\n Any first software client would have felt special, but this one<\/a> was thrilling, because my concoction actually helped real people get out of a tight\u00a0spot:<\/p>\n Working towards a big event, they had planned to launch a flagship report, with data from a paid YouGov survey. But its results were tepid. So, with their remaining budget, they bought a FinText study. It was our findings that they put front and centre in their final\u00a0report<\/a>.<\/p>\n But social listening did not take off. Investment land is quirky because pools of money will always need a home; the only question is who\u2019s the landlord. Industry people I talked to mostly wanted to know what their competitors were up\u00a0to.<\/p>\n So the second use case\u200a\u2014\u200acompetitive content analytics\u200a\u2014\u200awas met with warmer response. I sold about half a dozen companies on this solution (including, for example, Aviva Investors<\/a>).<\/p>\n All along, our engine was collecting data no one else had. Such was my savvy, it wasn\u2019t even my idea to run training sessions, a client first asked for one. That\u2019s how I learned companies like buying training.<\/p>\n Otherwise, my steampunk take on writing was proving tricky to sell. It was all too abstract. What I needed was a dashboard: pretty charts, with real numbers, crunched from live data. A pipeline did the crunching, and I hired a small team to do the pretty\u00a0charts.<\/p>\n
I went into financial services, which is like a theme park for nerds. And, for the ensuing decade, would only take jobs with lots of\u00a0writing.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A82IO4SAXlPwtsyX8ih9bPw.png?ssl=1\")
Enter Text Analytics<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A8LrA79XdxEheUQRW_243BQ.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Adbtq5FGuzlI6hWF5p29rpQ.png?ssl=1\")
Building A Text Analytics Engine<\/strong><\/h4>\n
What the Market\u00a0Said<\/strong><\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AW_q__FqPYGsMQYcd7VuiDg.png?ssl=1\")