{"id":3026,"date":"2025-04-11T07:02:26","date_gmt":"2025-04-11T07:02:26","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/04\/11\/ivory-tower-notes-the-problem\/"},"modified":"2025-04-11T07:02:26","modified_gmt":"2025-04-11T07:02:26","slug":"ivory-tower-notes-the-problem","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/04\/11\/ivory-tower-notes-the-problem\/","title":{"rendered":"Ivory Tower Notes: The\u00a0Problem"},"content":{"rendered":"

Ivory Tower Notes: The\u00a0Problem
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Did you ever spend<\/mdspan> months on a Machine Learning<\/a> project, only to discover you never defined the \u201ccorrect\u201d problem at the start? If so, or even if not, and you are only starting with the data science or AI field, welcome to my first Ivory Tower Note, where I will address this topic.\u00a0<\/p>\n

\n

The term <\/em>\u201cIvory Tower\u201d<\/em><\/strong> is a metaphor for a situation in which someone is isolated from the practical realities of everyday life. <\/em>In academia<\/em><\/strong>, the term often refers to researchers who engage deeply in theoretical pursuits and remain distant from the realities that practitioners face outside academia.<\/em><\/p>\n

As a former researcher, I wrote a <\/em>short series of posts from my old Ivory Tower notes\u200a\u2014\u200athe notes before the LLM era.<\/em><\/strong><\/p>\n

Scary, I know. I am writing this to manage expectations and the question, \u201cWhy ever did you do things this way?\u201d\u200a\u2014\u200a\u201cBecause no LLM told me how to do otherwise 10+ years ago.\u201d<\/em><\/p>\n

That\u2019s why my notes contain \u201clegacy\u201d topics such as <\/em>data mining, machine learning, multi-criteria decision-making, and (sometimes) human interactions, airplanes<\/em><\/strong> \"\u2708\" <\/em>and<\/em><\/strong> <\/em>art.<\/em><\/strong><\/p>\n

Nonetheless, whenever there is an opportunity, I will map my \u201cold\u201d knowledge to <\/em>generative AI advances <\/em><\/strong>and<\/em> <\/em><\/strong>explain how I applied it to datasets beyond the Ivory Tower.<\/em><\/p>\n

Welcome to post #1\u2026<\/strong><\/p>\n

\n

How every Machine Learning and AI journey\u00a0starts<\/h3>\n

\u200a\u2014\u200aIt starts with a problem.\u00a0<\/strong><\/p>\n

For you, this is usually \u201cthe\u201d problem because you need to live with it for months or, in the case of research, years<\/em>.\u00a0<\/p>\n

With \u201cthe\u201d problem, I am addressing the business problem you don\u2019t fully understand or know how to solve at first.\u00a0<\/p>\n

An even worse scenario is when you think you fully understand and know how to solve it quickly<\/em>. This then creates only<\/em> more problems that are again only<\/em> yours to solve. But more about this in the upcoming sections.\u00a0<\/p>\n

So, what\u2019s \u201cthe\u201d problem\u00a0about?<\/h3>\n

Causa:<\/strong> It\u2019s mostly about not managing or leveraging resources properly\u200a\u2014\u200a workforce, equipment, money, or time.\u00a0<\/p>\n

Ratio: <\/strong>It\u2019s usually about generating business value, which can span from improved accuracy, increased productivity, cost savings, revenue gains, faster reaction, decision, planning, delivery or turnaround times.\u00a0<\/p>\n

Veritas:<\/strong> It\u2019s always about finding a solution that relies and is hidden somewhere in the existing dataset.\u00a0<\/p>\n

Or, more than one dataset that someone labelled as \u201cthe one\u201d, and that\u2019s waiting for you to solve the<\/em> problem. Because datasets follow and are created from technical or business process logs, \u201cthere has to be a solution lying somewhere within them.<\/em>\u201d<\/p>\n

Ah, if only it were so easy.<\/p>\n

Avoiding a different chain of thought again, the point is you will need to:<\/p>\n

\n

1\u200a\u2014\u200a<\/strong>Understand the problem fully,
2\u200a\u2014\u200a<\/strong>If not given, find the dataset \u201cbehind\u201d it, and\u00a0
3\u200a\u2014\u200a<\/strong>Create a methodology to get to the solution that will generate business value from\u00a0it.\u00a0<\/p>\n<\/blockquote>\n

On this path, you will be tracked and measured, and time will not be on your side to deliver the solution that will solve \u201cthe universe equation.\u201d\u00a0<\/p>\n

That\u2019s why you will need to approach the problem methodologically, drill down to smaller problems first, and focus entirely on them because they are the root cause of the overall problem.\u00a0<\/p>\n

That\u2019s why it\u2019s good to learn how to\u2026<\/p>\n

Think like a Data Scientist.<\/a><\/h3>\n

Returning to the problem itself, let\u2019s imagine that you are a tourist lost somewhere in the big museum, and you want to figure out where you are. What you do next is walk to the closest info map on the floor, which will show your current location.\u00a0<\/p>\n

At this moment, in front of you, you see something like this:\u00a0<\/p>\n

\"\"
Data Science<\/mdspan> Process. Image by Author, inspired by Microsoft Learn<\/a><\/figcaption><\/figure>\n

The next thing you might tell yourself is, \u201cI want to get to Frida Kahlo\u2019s painting.\u201d <\/em>(Note<\/em>: These are the insights you want to get.)<\/p>\n

Because your goal is to see this one painting that brought you miles away from your home and now sits two floors below, you head straight to the second floor. Beforehand, you memorized the shortest path to reach your goal. (Note<\/em>: This is the initial data collection and discovery phase.)<\/p>\n

However, along the way, you stumble upon some obstacles\u200a\u2014\u200athe elevator is shut down for renovation, so you have to use the stairs. The museum paintings were reordered just two days ago, and the info plans didn\u2019t reflect the changes, so the path you had in mind to get to the painting is not accurate.\u00a0<\/p>\n

Then you find yourself wandering around the third floor already, asking quietly again, \u201cHow do I get out of this labyrinth and get to my painting faster?<\/em>\u201d<\/p>\n

While you don\u2019t know the answer, you ask the museum staff on the third floor to help you out, and you start collecting the new data to get the correct route to your painting. (Note<\/em>: This is a new data collection and discovery phase.)<\/p>\n

Nonetheless, once you get to the second floor, you get lost again, but what you do next is start noticing a pattern in how the paintings have been ordered chronologically and thematically to group the artists whose styles overlap, thus giving you an indication of where to go to find your painting. (Note<\/em>: This is a modelling phase overlapped with the enrichment phase from the dataset you collected during school days\u200a\u2014\u200ayour art knowledge.)<\/p>\n

Finally, after adapting the pattern analysis and recalling the collected inputs on the museum route, you arrive in front of the <\/em>painting you had been planning to see since booking your flight a few months ago.\u00a0<\/p>\n

What I described now is how you approach data science and, nowadays, generative AI problems. You always start with the end goal in mind<\/strong> and ask yourself:<\/p>\n

\n

\u201cWhat is the expected outcome I want or need to get from\u00a0this?\u201d<\/p>\n<\/blockquote>\n

Then you start planning from this question backwards<\/strong>. The example above started with requesting holidays, booking flights, arranging accommodation, traveling to a destination, buying museum tickets, wandering around in a museum, and then seeing the painting you\u2019ve been reading about for ages.\u00a0<\/p>\n

Of course, there is more to it, and this process should be approached differently if you need to solve someone else\u2019s problem, which is a bit more complex than locating the painting in the museum.\u00a0<\/p>\n

In this case, you have to\u2026<\/p>\n

Ask the \u201cgood\u201d questions.<\/h3>\n

To do this, let\u2019s define what a good<\/em> question means<\/a> [1<\/a>]:\u00a0<\/p>\n

\n

A good<\/strong> data science question must be concrete<\/strong>, tractable<\/strong>, and answerable<\/strong>. Your question works well if it naturally points to a feasible approach<\/strong> for your project. If your question is too vague<\/strong> to suggest what data you need, it won\u2019t effectively guide<\/strong> your work.<\/p>\n<\/blockquote>\n

Formulating good<\/em> questions keeps you on track so you don\u2019t get lost in the data that should be used to get to the specific problem solution, or you don\u2019t end up solving the wrong problem.<\/p>\n

Going into more detail, good<\/em> questions will help identify gaps in reasoning, avoid faulty premises, and create alternative scenarios in case things do<\/em> go south (which almost always happens)\"\ud83d\udc47\ud83c\udffc\".<\/p>\n

\"\"
Image created by Author after analyzing \u201cChapter 2. Setting goals by asking good questions\u201d from \u201cThink Like a Data Scientist\u201d book\u00a0[2]<\/strong><\/figcaption><\/figure>\n

From the above-presented diagram, you understand how good<\/em> questions, first and foremost, need to support concrete assumptions.<\/strong> This means they need to be formulated in a way that your<\/em> premises are clear and ensure they can be tested without mixing up facts with opinions.<\/p>\n

Good<\/em> questions produce answers<\/strong> that move you closer to your goal, whether through confirming hypotheses, providing new insights, or eliminating wrong paths. They are measurable, <\/strong>and with this, they connect to project goals<\/strong> because they are formulated with consideration of what\u2019s possible, valuable, and efficient [2].<\/p>\n

Good<\/em> questions are answerable with available data<\/strong>, considering current data relevance and limitations.\u00a0<\/p>\n

Last but not least, good<\/em> questions anticipate obstacles. <\/strong>If something is certain in data science, this is the uncertainty<\/em>, so having backup plans when things don\u2019t work as expected is important to produce results for your project.<\/p>\n

Let\u2019s exemplify this with one use case of an airline company that has a challenge with increasing its fleet availability<\/strong> due to unplanned technical groundings (UTG).<\/p>\n

These unexpected maintenance events disrupt flights and cost the company significant money. Because of this, executives decided to react to the problem and call in a data scientist (you) to help them improve aircraft availability.<\/p>\n

Now, if this would be the first data science task you ever got, you would maybe start an investigation by asking:<\/p>\n

\n

\u201cHow can we eliminate all unplanned maintenance events?\u201d<\/em><\/p>\n<\/blockquote>\n

You understand how this question is an example of the wrong or \u201cpoor\u201d one because:<\/p>\n