{"id":356,"date":"2024-12-04T07:02:38","date_gmt":"2024-12-04T07:02:38","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2024\/12\/04\/query-optimization-for-mere-humans-in-postgresql-875ab864390a\/"},"modified":"2024-12-04T07:02:38","modified_gmt":"2024-12-04T07:02:38","slug":"query-optimization-for-mere-humans-in-postgresql-875ab864390a","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2024\/12\/04\/query-optimization-for-mere-humans-in-postgresql-875ab864390a\/","title":{"rendered":"Query Optimization for Mere Humans in PostgreSQL"},"content":{"rendered":"<p>    Query Optimization for Mere Humans in PostgreSQL<br \/>\n \t<BR><br \/>\n<BR><\/BR><br \/>\n    <!-- no image --><br \/>\n \t<BR><br \/>\n<BR><\/BR><\/p>\n<div>\n<h3>PostgreSQL: Query Optimization for Mere\u00a0Humans<\/h3>\n<h4>Understanding a PostgreSQL execution plan with practical examples<\/h4>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1024\/0*l-3On57NJoJ7maR3\"><figcaption>Photo by <a href=\"https:\/\/unsplash.com\/@grakozy?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash\">Greg Rakozy<\/a> on\u00a0<a href=\"https:\/\/unsplash.com\/photos\/silhouette-photography-of-person-oMpAz-DN-9I?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash\">Unsplash<\/a><\/figcaption><\/figure>\n<p>Today, users have high expectations for the programs they use. Users expect programs to have amazing features, to be fast, and to consume a reasonable amount of resources.<\/p>\n<p>As developers, we should thrive to give our users the best experience possible. It\u2019s pretty common that the database becomes the bottleneck, and optimizing queries and eliminating the bottlenecks is not an easy task. Unfortunately, as programs become more and more complex, and as the data become bigger, it becomes harder to write flawless SQL\u00a0queries.<\/p>\n<p>Today, I am going to focus on a technique to find those bottlenecks, using the <em>Explain<\/em> clause. My goal today is to show you that finding and eliminating those bottlenecks is not rocket science. Everyone can find their bottlenecks without breaking a\u00a0sweat.<\/p>\n<p>The code for this article can be found on\u00a0<a href=\"https:\/\/github.com\/eyaltrabelsi\/my-notebooks\/blob\/master\/Lectures\/query_optimization_for_mere_humans\/Query%20Optimization%20for%20Mere%20Humans.ipynb\">GitHub<\/a>.<\/p>\n<p>Note: All images, unless otherwise noted, are by the\u00a0author.<\/p>\n<h3>Let\u2019s explain Explain\u00a0\ud83d\udcdc<\/h3>\n<p>Interactions with databases are done using declarative languages, where SQL is the most common one. The database decides how and what to do behind the scenes and the only glimpse it provides is the execution plan.<\/p>\n<p>This limitation makes implementing proper debugging tools, and profilers almost impossible in practice. <strong>So we are kind of stuck with execution plans.<\/strong><\/p>\n<h4>Buzzword alert \ud83d\udc1d!! my goal is to democratize execution plans.<\/h4>\n<p>In PostgreSQL in order to get the execution plan one should use <em>Explain<\/em>\/<em>Explain analyze\u00a0<\/em>clauses:<\/p>\n<ul>\n<li>\n<em>EXPLAIN<\/em> shows what the planner planned to\u00a0do.<\/li>\n<li>\n<em>EXPLAIN ANALYZE<\/em> what the planner plans to do, <strong>execute the query,<\/strong> and also show how it did\u00a0it.<\/li>\n<\/ul>\n<p><strong>Pro Tip #1<\/strong>\ud83d\udc83: go over an execution plan at least once in your career. It&#8217;s similar across databases, and it is a rare skill in companies.<\/p>\n<p><strong>Pro Tip #2 <\/strong>\ud83d\udc83: prefer <em>EXPLAIN ANALYZE<\/em> as it holds more information for most\u00a0cases.<\/p>\n<p><strong>Warning #1<\/strong> \u26a0\ufe0f don\u2019t use <em>EXPLAIN ANALYZE <\/em>on destructive operations like <em>DELETE\/UPDATE, EXPLAIN <\/em>will suffice and it doesn\u2019t run the\u00a0query.<\/p>\n<p><strong>Warning #2<\/strong> \u26a0\ufe0f don\u2019t use <em>EXPLAIN ANALYZE <\/em>when resources are scarce like production monitoring, and when a query never finishes<em>, EXPLAIN <\/em>will suffice and it doesn\u2019t run the\u00a0query.<\/p>\n<p><em>Explain<\/em> is an awesome tool as it can imply reasons why a query was slow including:<\/p>\n<ul>\n<li>Missing\/Overused indices\/partitions.<\/li>\n<li>Unoptimized database configurations.<\/li>\n<li>Redundant Operations.<\/li>\n<li>Stale statistics.<\/li>\n<li>Too much\u00a0I\/O.<\/li>\n<\/ul>\n<p>For the more thorough people you can see the <em>Explain<\/em> clause syntax in the next\u00a0figure:<\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/820\/1%2AO55DqK52i4rQy8kWvO7REg.png?ssl=1\"><figcaption>PostgreSQL Explain Clause\u00a0Syntax<\/figcaption><\/figure>\n<h3>Understanding Explain Anatomy\ud83e\udec0<\/h3>\n<p>We will use it as an example of a simple query: we want to count the number of users that don\u2019t have Twitter\u00a0handles.<\/p>\n<pre>EXPLAIN ANALYZE<br>SELECT COUNT(*) FROM users WHERE twitter != '';<\/pre>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AdfsAx-mjU3KhZSbsKR9yfg.png?ssl=1\"><figcaption>We can see the execution plan returned from the EXPLAIN ANALYZE\u00a0clause<\/figcaption><\/figure>\n<p>It looks cryptic at first, and It\u2019s even longer than our query, and that on a small example of real-world execution plans can be overwhelming if you don&#8217;t focus\u00a0\ud83d\ude2d.<\/p>\n<p><strong>But it does provide useful information. <\/strong>We can see that the query execution took 1.27 seconds, while the query planning took only 0.4 milli-seconds (negligible time).<\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AtcijQ_GFp8iq--nnxbtmAg.png?ssl=1\"><figcaption>We can see the time the query planning and execution took<\/figcaption><\/figure>\n<p>The execution plan is structured as an inverse tree. In the next figure, you can see the execution plan is divided into different nodes each one of which represents a different operation whether it&#8217;s an <em>Aggregation<\/em> or a\u00a0<em>Scan.<\/em><\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2APOmK5sHID4H5APlfbmaL-w.png?ssl=1\"><figcaption>We can see the time the query planning and execution took<\/figcaption><\/figure>\n<p>There are many kinds of nodes operations, from Scan related (<em>\u2018Seq Scan\u2019, \u2018Index Only Scan\u2019, etc\u2026<\/em>), Join related(<em> \u2018Hash Join\u2019, \u2019Nested Loop\u2019, etc\u2026), <\/em>Aggregation related<em> (\u2018GroupAggregate\u2019, \u2019Aggregate\u2019, etc\u2026) <\/em>and others<em> ( \u2018Limit\u2019, \u2018Sort\u2019, \u2018materialize\u2019, etc..). <\/em><strong>Fortunately you need to remember any of\u00a0this.<\/strong><\/p>\n<p><strong>Pro Tip #3 <\/strong>\ud83d\udc83: Focus is key, look only on nodes that are problematic.<\/p>\n<p><strong>Pro Tip #4 <\/strong>\ud83d\udc83: Cheat\u00a0! on the problematic nodes search what they mean in the <a href=\"https:\/\/www.pgmustard.com\/docs\/explain\">explain glossary<\/a>.<\/p>\n<p>Now, let&#8217;s drill down into how we know which node is the problematic one.<\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AjXnnH_uAybaahQ1WzsXdRw.png?ssl=1\"><figcaption>There is a lot of information we can see on each\u00a0node<\/figcaption><\/figure>\n<p>Let&#8217;s drill down to what those metrics actually\u00a0mean.<\/p>\n<ul>\n<li>\n<strong>Actual Loops<\/strong>: the number of loops the same node executed is 1. To get the total time and rows, the actual time and rows need to be multiplied by loops\u00a0values.<\/li>\n<li>\n<strong>Actual Rows<\/strong>: the actual number of produced rows of the <em>Aggregate<\/em> node is 1 (per-loop average and we have loops is\u00a01).<\/li>\n<li>\n<strong>Plan Rows<\/strong>: the estimated number of produced rows of the <em>Aggregate<\/em> node is 1. The estimated number of rows can be off depending on statistics.<\/li>\n<li>\n<strong>Actual Startup Time<\/strong>: the time it took to return the first row in milliseconds of the <em>Aggregate<\/em> node is 1271.157 (aggregated and includes previous operations).<\/li>\n<li>\n<strong>Startup Cost<\/strong>: arbitrary units that represent the estimated time to return the first row of the <em>Aggregate<\/em> node is 845110(aggregated and includes previous operations).<\/li>\n<li>\n<strong>Actual Total Time<\/strong>: the time it took to return all the rows in ms of the <em>Aggregate<\/em> node is 1271.158 (per-loop average and we have loops is 1 and aggregated and include previous operations).<\/li>\n<li>\n<strong>Total Cost<\/strong>: arbitrary units that represent the estimated time to return all the rows of <em>Aggregate<\/em> node is 845110 (aggregated).<\/li>\n<li>\n<strong>Plan Width<\/strong>: the estimated average size of rows of the <em>Aggregate<\/em> node is 8\u00a0bytes.<\/li>\n<\/ul>\n<p><strong>Pro Tip #5 <\/strong>\ud83d\udc83: be wary of loops, remember to multiply loops when you care about <em>Actual Rows <\/em>and<em> Actual Total\u00a0Time.<\/em><\/p>\n<p>We will drill in the next section on a practical example.<\/p>\n<h3>Example: Performance Optimization\ud83d\udc06<\/h3>\n<p>We will use the same query as\u00a0before.<\/p>\n<pre>EXPLAIN ANALYZE<br>SELECT COUNT(*) FROM users WHERE twitter != '';<\/pre>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AsWHKqRRwZ3zCr6M3vZFK7Q.png?ssl=1\"><figcaption>We will focus on the Seq Scan node which has the longest actual\u00a0time<\/figcaption><\/figure>\n<p>We focus on the longest operation which is the <a href=\"https:\/\/www.pgmustard.com\/docs\/explain\/sequential-scan\">sequential scan<\/a> on the users&#8217; table. The scan filters out 2,487,813 rows and takes us 1.27 seconds out of\u00a01.271.<\/p>\n<p><strong>But we are mere humans that don\u2019t tell us anything. <\/strong>Let&#8217;s google it (you can use ChatGPT as well)\u00a0!!!.<\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AjMQGhTdhk76sSVyrHOZBhQ.png?ssl=1\"><figcaption>Googling how to make seq scan faster in PostgreSQL<\/figcaption><\/figure>\n<pre>CREATE INDEX twitter_test ON users (twitter)<\/pre>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2ABwvC6JwEIGIVnJRf9j3c_Q.png?ssl=1\"><figcaption>We are doing much better, but the Scan node is still the longest in terms of actual\u00a0time<\/figcaption><\/figure>\n<p>We can see that now we perform an <a href=\"https:\/\/www.pgmustard.com\/docs\/explain\/index-only-scan\">index only scan<\/a> on the users&#8217; table. It takes us 0.29 seconds instead of 1.27 seconds, which is awesome but not enough for\u00a0us.<\/p>\n<p><strong>Pro Tip #6<\/strong>\ud83d\udc83: optimize your queries one baby step at a\u00a0time.<\/p>\n<p>To understand how much data is passed to the scan. We could use the buffers parameter as you can see down\u00a0below.<\/p>\n<p><strong>Pro Tip #7<\/strong>\ud83d\udc83: When comparing execution plans, look at several\u00a0metrics.<\/p>\n<pre>EXPLAIN (ANALYZE, BUFFERS)<br>SELECT COUNT(*) FROM users WHERE twitter != ''<\/pre>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/proxy\/1%2AZYn0VC5d7Gc8oX0w3dPAMw.png?ssl=1\"><figcaption>We can see quite a lot of information about how much data hit the cache or\u00a0disk<\/figcaption><\/figure>\n<p>We have 51,854 pages to read all from the cache (400 MB), so improving configurations probably won\u2019t change things drastically.<\/p>\n<p>But, we are not out of options. Since the scan filters out 2,487,813 rows, we can change the index into a partial index but it doesn\u2019t come for free. It will cause writes to take longer, and it will take additional storage, which is quite impactful on systems that scale vertically.<\/p>\n<p><strong>Pro Tip #8 <\/strong>\ud83d\udc83: there is no free\u00a0lunch.<\/p>\n<h3>Good optimization options\ud83e\udd1e\ud83c\udffb<\/h3>\n<p>I won\u2019t delve into too many details as this blog is already quite long. These are the first things one might want to tackle when he has slow\u00a0queries:<\/p>\n<ul>\n<li>Picking the right scan\u00a0method.<\/li>\n<li>Picking the right join\u00a0method.<\/li>\n<li>Picking the right join\u00a0order.<\/li>\n<li>Push Filters as soon as possible.<\/li>\n<li>Reducing disk IO operations when\u00a0needed.<\/li>\n<\/ul>\n<p>In order to manually check specific optimization one can enable\/disable settings.<\/p>\n<pre>SET enable_seqscan TO off;<br>EXPLAIN (ANALYZE) SELECT * FROM foo WHERE c1 &gt; 500;<br>SET enable_seqscan TO on;<\/pre>\n<p><strong>Warning #3<\/strong> \u26a0\ufe0f: enable\/disable settings only after you tried the most basic optimizations as most of the time, PostgreSQL knows what it is\u00a0doing.<\/p>\n<h3>Aren\u2019t there easier\u00a0ways?!\ud83d\ude4f<\/h3>\n<p>Unfortunately, <em>Explain <\/em>is not perfect and there are reasons why it\u2019s not in every developer toolbox:<\/p>\n<ul>\n<li>You don\u2019t have a history of all the execution plans in production.<\/li>\n<li>Not trivial to tune complex queries<br \/>\u200a\u2014\u200aIt\u2019s long, and not that easy to read.<br \/>\u200a\u2014\u200aIt doesn\u2019t tell you why a particular optimization is not used.<br \/>\u200a\u2014\u200aIt doesn\u2019t tell you how to rewrite your\u00a0queries.<\/li>\n<\/ul>\n<p>We can overcome the lack of history by using tools like <a href=\"https:\/\/www.postgresql.org\/docs\/current\/auto-explain.html\">auto_explain<\/a> and <a href=\"https:\/\/github.com\/2ndQuadrant\/pg_stat_plans\">pg_stat_plans<\/a> to record the execution plans on certain conditions such that they won\u2019t have a major effect on production. Another way is to record what queries run at what time and try to reproduce it, but it&#8217;s more complicated than it\u00a0looks.<\/p>\n<p>We can overcome complex tuning with some very opinionated tools. These focus your attention on what works for most use cases. Some of the most prominent tools\u00a0are:<\/p>\n<ul>\n<li>\n<a href=\"https:\/\/www.eversql.com\/\">eversql<\/a>\u200a\u2014\u200aa mature solution that aims to suggest changes to your <em>PostgreSQL <\/em>queries.<\/li>\n<li>\n<a href=\"https:\/\/www.metisdata.io\/\">metis<\/a>\u200a\u2014\u200aaims to suggest changes as a guarding mechanism as part of your development and CI\/CD processes for <em>PostgreSQL<\/em> databases.<\/li>\n<li>\n<a href=\"https:\/\/github.com\/eyaltrabelsi\/query-flow\">QueryFlow<\/a>\u200a\u2014\u200aan open-source tool that allows identifying bugs and performance tuning multiple queries (as the hardest queries to debug behave nicely in isolation).<\/li>\n<\/ul>\n<p><strong>Pro Tip #9 <\/strong>\ud83d\udc83: use tools to make your life\u00a0easy.<\/p>\n<p>I will give you a taste of how convenient it is to use tools like QueryFlow (For more details you can read the following).<\/p>\n<figure><img data-recalc-dims=\"1\" decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AiVwp45epASb6E79l1ugdzw.png?ssl=1\"><figcaption>QueryFlow visualization of execution plan focusing on\u00a0duration<\/figcaption><\/figure>\n<p>It should be extremely easy to see that the <em>Index Only Scan <\/em>width is much bigger than the aggregation and indicate this is where we should focus. On multiple complex queries, other tools tend to\u00a0lack<\/p>\n<h3>Last words<\/h3>\n<p>In this article, we reviewed some of the most common reasons that can cause otherwise perfectly good SQL to be too slow for any time-sensitive applications, and <strong>walk through a mythological way to identify those and avoid\u00a0them<\/strong>.<\/p>\n<p>Due to the extent of the topic, there are many optimizations I haven\u2019t covered. For this reason, I have added additional resources in the end if you want to go the extra\u00a0mile.<\/p>\n<p>I am optimistic about the future. I believe these kinds of tools will be as easy as <a href=\"https:\/\/www.w3schools.com\/python\/python_file_open.asp\"><em>opening files in python<\/em><\/a>, either by integrating into IDEs, and clients, or providing SAS solutions. This will enable us to become proactive instead of reactive.<\/p>\n<p>I hope I was able to share my enthusiasm for this fascinating topic and that you find it useful, and as always I am open to any kind of constructive feedback.<\/p>\n<h3>Additional Resources \ud83d\udcda<\/h3>\n<ul>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=Ls-uE1V31lE&amp;list=WL&amp;index=5&amp;ab_channel=PostgresConference\">Deeper Understanding of PostgreSQL Execution Plan\u00a0(video)<\/a><\/li>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=mCwwFAl1pBU&amp;ab_channel=SouthernCaliforniaLinuxExpo\">EXPLAIN Explained (video)<\/a><\/li>\n<li><a href=\"https:\/\/public.dalibo.com\/exports\/conferences\/_archives\/_2012\/201211_explain\/understanding_explain.pdf\">Understanding Explain<\/a><\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=875ab864390a\" width=\"1\" height=\"1\" alt=\"\"><\/p>\n<hr>\n<p><a href=\"https:\/\/towardsdatascience.com\/query-optimization-for-mere-humans-in-postgresql-875ab864390a\">Query Optimization for Mere Humans in PostgreSQL<\/a> was originally published in <a href=\"https:\/\/towardsdatascience.com\/\">Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n<p> \t<BR><br \/>\n <BR><\/BR><br \/>\n    Eyal Trabelsi<br \/>\n \t<BR><br \/>\n<BR><\/BR><br \/>\n<a href=\"https:\/\/medium.com\/m\/global-identity-2?redirectUrl=https%3A%2F%2Ftowardsdatascience.com%2Fquery-optimization-for-mere-humans-in-postgresql-875ab864390a\">Go to original source<\/a><br \/>\n \t<BR><br \/>\n <BR><\/BR><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Query Optimization for Mere Humans in PostgreSQL PostgreSQL: Query Optimization for Mere\u00a0Humans Understanding a PostgreSQL execution plan with practical examples Photo by Greg Rakozy on\u00a0Unsplash Today, users have high expectations for the programs they use. Users expect programs to have amazing features, to be fast, and to consume a reasonable amount of resources. As developers, [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[62,401,403,402,400,404],"tags":[406,405,407],"class_list":["post-356","post","type-post","status-publish","format-standard","hentry","category-aimldsaimlds","category-data-engineering","category-database","category-optimization","category-postgresql","category-sql","tag-execution","tag-explain","tag-postgresql"],"_links":{"self":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts\/356"}],"collection":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/comments?post=356"}],"version-history":[{"count":0,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/posts\/356\/revisions"}],"wp:attachment":[{"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/media?parent=356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/categories?post=356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mailitics.com\/index.php\/wp-json\/wp\/v2\/tags?post=356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}