Introduction<\/h2>\n
In the \u201cever rapidly changing landscape of Data and AI<\/em>\u201d (!), understanding data and AI architecture has never been more critical. However something many leaders overlook is the importance of data team structure.<\/p>\n While many of you reading this probably identify as\u00a0the\u00a0<\/em>data team, something most don\u2019t realise is how limiting that mindset can be.<\/p>\n Indeed, different team structures and skill requirements significantly impact an organisation\u2019s ability to actually use Data and AI to drive meaningful results. To understand this, it\u2019s helpful to think of an analogy.<\/p>\n Imagine a two-person household. John works from home and Jane goes to the office. There\u2019s a bunch of house admin Jane relies on John to do, which is a lot easier since he\u2019s the one at home most of the time.<\/p>\n Jane and John have kids and after they\u2019re grown up a bit John has twice as much admin to do! Thankfully, the kids are trained to do the basics; they can wash up, tidy and even occasionally do a bit of hoovering with some coercion.<\/p>\n As the kids grow up, John\u2019s parents move in. They\u2019re pretty old, so John looks after them, but fortunately, the kids are basically self-sufficient at this point. Over time John\u2019s role has changed quite a bit! But he\u2019s always made it one happy, nuclear family \u2014 thanks to John and Jane.<\/p>\n Back to data \u2014 John is a bit like the data team, and everyone else is a domain expert. They rely on John, but in different ways. This has changed a lot over time, and if it hadn\u2019t it could have been a disaster.<\/p>\n In the rest of this article, we\u2019ll explore John\u2019s journey from a Centralised, through Hub-and-spoke to a Platform mesh-style data team.<\/p>\n A central team is responsible for a lot of things that will be familiar to you:<\/p>\n This is a lot of work for a few people! In fact, it\u2019s practically impossible to nail all of this at once. It\u2019s best to keep things small and manageable, focusing on a few key use cases and leveraging powerful tooling to get a head start early.<\/p>\n You might even get a nanny or au Pair to help with the work (in this case \u2014 consultants).<\/p>\n But this pattern has flaws. It is easy to fall into the\u00a0silo trap<\/a>, a scenario where the central team become a huge bottleneck for Data and AI requests. Data Teams also need to acquire domain knowledge from domain experts to effectively answer requests, which is also time-consuming and hard.<\/p>\n One way out is to expand the team. More people means more output. However, there are better more modern approaches that can make things go even faster.<\/p>\n But there is only one John. So what can he do?<\/p>\n The partially decentralised setup is an attractive model for medium-sized organisations or small, tech-first ones where there are technical skills outside of the data team<\/a>.<\/p>\n The simplest form has the data team maintaining BI infrastructure, but not the content itself. This is left to \u2018power users\u2019 that take this into their own hands and build the BI themselves.<\/p>\n This, of course, runs into all kinds of issues, such as the silo trap,\u00a0data discovery<\/a>, governance, and\u00a0confusion<\/a>. Confusion is especially painful when people who are told to self-serve try and fail due to a lack of understanding of the data.<\/p>\n An increasingly popular approach is for additional layers of the stack to be opened up. There is the\u00a0rise of the analytics engineer<\/a>\u00a0and data analysts are increasingly taking on more responsibility. This includes using tools, doing data modelling, building end-to-end pipelines, and advocating to the business.<\/p>\n This has led to enormous problems when implemented incorrectly. You wouldn\u2019t let your five-year-old son look after the care of your elders and take care of the house unattended.<\/p>\n Specifically, a lack of basic data modelling principles and data warehouse engines leads to model sprawl and spiralling costs. There are two classic examples.<\/p>\n One is when multiple people try to define the same thing, such as revenue. marketing, finance, and product all have a different version. This leads to inevitable arguments at quarterly business reviews when every department reports with a different number \u2014 analysis paralysis.<\/p>\n The other is rolling counts. Let\u2019s say finance wants revenue for the month, but product wants to know what it is on a rolling seven-day basis. \u201cThat\u2019s easy,\u201d says the analyst. \u201cI\u2019ll just create some materialised views with these metrics in them\u201d.<\/p>\n As any data engineer knows, this rolling counts operation is pretty expensive, especially if the granularity needs to be by day or hour, since then you need a calendar to \u2018fan out\u2019 the model. Before you know it there are\u00a0 Simply asking for the lowest granularity required (daily), materialising that, and creating views downstream can solve this problem but would require some central resource.<\/p>\n An early Hub and Spoke model must have a clear delineation of responsibility if the knowledge outside the data team is young or juvenile.<\/p>\n As teams grow, legacy, code-only frameworks like Apache Airflow also give rise to a problem: a lack of visibility. People outside the data team seeking to understand what is going will be reliant on additional tools to understand what happens end-to-end, since legacy UIs do not aggregate metadata from different sources.<\/p>\n It is imperative to surface this information to domain experts. How many times have you been told the \u2018data doesn\u2019t look right\u2019, only to realise after tracing everything manually that it was an issue on the data producer side?<\/p>\n By increasing visibility, domain experts are connected directly to owners of source data or processes, which allows fixes to be faster. This removes unnecessary load, context switching, and tickets for the data team.<\/p>\n A pure hub and spoke is a bit like delegating your teenage children with specific responsibilities within clear guardrails. You don\u2019t just give them tasks to do like taking the bins out and cleaning their room \u2014 you ask for what you want, like a \u201cclean and tidy room,\u201d and you trust them to do it. Incentives work well here.<\/p>\n In a pure hub and spoke approach, the data team administers the platform and lets others use it. They build the frameworks for building and deploying AI and Data pipelines, and manage access control.<\/p>\n Domain experts can build stuff end-to-end if they need to. This means they can move data, model it, orchestrate the pipeline, and activate it with AI or dashboards as they see fit.<\/p>\n Often, the central team will also do a bit of this. Where data models across domains are complex and overlapping, they should almost always take ownership of delivering core data models. The tail should not wag the dog.<\/p>\n This starts to resemble a data product mindset \u2014 while a finance team could take ownership for investing and cleaning ERP data, the central team would own an important data products like the customers table or invoices table.<\/p>\n This structure is very powerful as it is very collaborative. It often works only if domain teams have a reasonably high degree of technical proficiency.<\/p>\n Platforms that allow use of code and no-code together are recommended here, otherwise a hard technical dependency on the central team will always exist.<\/p>\n Another characteristic of this pattern is training and support. The central team or hub will spend some time supporting and upskilling the spokes to build AI and Data workflows efficiently within guardrails.<\/p>\n Again, providing visibility here is hard with legacy orchestration frameworks. Central teams will be burdened with keeping metadata stores up-to-date, like Data Catalogs, so business users can understand what is going on.<\/p>\n The alternative \u2014 upskilling domain experts to have deep python expertise learning frameworks with steep learning curves, is even harder to pull off.<\/p>\n The natural endpoint in our theoretical household journey takes us to the much-criticised Data Mesh<\/a> or Platform Mesh approach.<\/p>\n In this household, everyone is expected to know what their responsibilities are. Children are all grown up and can be relied on to keep the house in order and look after its inhabitants. There is close collaboration and everyone works together seamlessly.\u00a0<\/p>\n Sounds pretty idealistic, don\u2019t you think!?<\/p>\n In practice, it\u2019s rarely this easy. Allowing satellite teams to use their own infrastructure and build whatever they want is a surefire way to lose control and slow things down.<\/p>\n Even if you were to standardise tooling across teams, best practices would still suffer.<\/p>\n I\u2019ve spoken to countless teams in massive organisations such as retail chains or airlines, and avoiding a mesh is not an option because multiple business divisions depend on each other.<\/p>\n These teams use different tools. Some leverage Airflow instances and legacy frameworks built by consultants years ago. Others use the latest tech and a full, bloated, Modern Data Stack.<\/p>\n They all struggle with the same problem; collaboration, communication, and orchestrating flows across different teams.<\/p>\n Implementing a single overarching platform for building Data and AI workflows here can help. A\u00a0unified control plane<\/a>\u00a0is almost like an orchestrator of orchestrators, that aggregates metadata across different places and shows end to end lineage across domains.<\/a><\/p>\n Naturally it makes for an effective control plane where anyone can gather to debug failed pipelines, communicate, and recover \u2014 all without relying on a central Data Engineering Team who would otherwise be a bottleneck.<\/p>\n There are clear analogies for this in software engineering. Often, code results in logs that are collated by a single tool such as DataDog. These platforms provide a single place to see everything happening (or not happening), alerts, and collaboration for incident resolution.<\/p>\n Organisations are like families. As much as we like the idea of one, big, happy, self-sufficient family, there are often responsibilities we need to bear to make things work out initially.<\/p>\n As they mature, members get closer to independence, like John\u2019s kids. Others find their place as dependent but loyal stakeholders, like John\u2019s parents.<\/p>\n Organisations are no different. Data Teams<\/a> are maturing away from do-ers in Centralised Teams to Enablers in Hub and Spoke architectures. Eventually, most organisations will have dozens if not hundreds of people who are pioneering Data and AI workflows in their own spokes.<\/p>\n Once this happens, it\u2019s likely that how Data and AI is used in small, agile organisations will resemble the complexity of much larger enterprises where collaboration and orchestration across different teams is inevitable.<\/p>\n Understanding where organisations are in relation to these patterns is imperative. Trying to force a Data-as-Product mindset on an immature company, or sticking to a large central team in a large and mature organisation will result in disaster.<\/p>\n Good luck The post Platform-Mesh, Hub and Spoke, and Centralised | 3 Types of data team<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nCentralised teams<\/h2>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Data-team-1.png?resize=895%2C955&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Data-team-2-1024x646.png?resize=1024%2C646&ssl=1\")
Partially decentralised or hub and spoke<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Data-team-3-1024x599.png?resize=1024%2C599&ssl=1\")
rolling_30_day_sales\u00a0<\/code>,\u00a0rolling_7_day_sales<\/code>\u00a0,\u00a0rolling_45_day_sales<\/code>\u00a0 and so on. These models cost an order of magnitude more than was required.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Data-team-4-1024x772.png?resize=1024%2C772&ssl=1\")
Hub and spoke (pure)<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/03\/Data-team-5-1024x843.png?resize=1024%2C843&ssl=1\")
Platform mesh\/data product<\/h2>\n
Summary<\/h2>\n
![\"\ud83c\udf40\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f340.png?ssl=1\")
<\/p>\n