Sebastian Raschka has helped demystify deep learning for thousands through his books, tutorials and teachings<\/h4>\n
Sebastian Raschka has helped shape how thousands of data scientists and machine learning engineers learn their craft. As a passionate coder and proponent of open-source software, a contributor to scikit-learn and the creator of the mlxtend library, his code runs in production systems worldwide. But his greatest impact is through his teachings\u200a\u2014\u200ahis books Machine Learning with PyTorch and Scikit-Learn<\/a>, Machine Learning Q and AI and Build a Large Language Model (From Scratch)<\/a> have become essential guides for practitioners navigating the complex landscape of modern\u00a0AI.<\/p>\n Drawing from over a decade of experience building AI systems and teaching at the intersection of academia and industry, Sebastian offers a unique perspective on mastering machine learning fundamentals while staying adaptable in this rapidly evolving field. As Senior Staff Research Engineer at Lighting AI, he continues to bridge the gap between cutting-edge research and practical implementation. In our in-depth discussion in this installment of Learning from Machine Learning<\/a>, he shared concrete strategies for everything from building reliable production systems to thinking critically about the future of Artificial General Intelligence (AGI).<\/p>\n Our wide-ranging discussion yielded many insights, which are summarized into 13 key\u00a0lessons:<\/p>\n Approach machine learning with patience, taking concepts step-by-step, in order to build a solid foundation. \u201cYou should, make sure you understand the bigger picture and intuition.\u201d Grasp the high-level concepts before getting bogged down in implementation details. Sebastian explains, \u201cI would start with a book or a course and just work through that, almost with a blindness on not getting distracted by other resources.\u201d<\/p>\n \u201cI would start with a book or a course and just work through that, almost with a blindness on not getting distracted by other resources.\u201d<\/p><\/blockquote>\n Borrowing from Andrew Ng, Sebastian shares, \u201cIf we don\u2019t understand a certain thing, maybe let\u2019s not worry about it. Just yet.\u201d Getting stuck on unclear details can slow you down. Move forward when needed rather than obsessing over gaps. Sebastian expands, \u201cIt happens to me all the time. I get distracted by something else, I look it up and then it\u2019s like a rabbit role and you feel, \u2018wow, there\u2019s so much to learn\u2019 and then you\u2019re frustrated and overwhelmed because the day only has twenty four hours, you can\u2019t possibly learn it\u00a0all.\u201d<\/p>\n Remember it\u2019s about \u201cdoing one thing at a time, step by step. It\u2019s a marathon, not a sprint.\u201d For early data scientists, he stresses building strong fundamentals before diving into the specifics of advanced techniques.<\/p>\n \u201cFinding a project you\u2019re interested in is the best way to get involved in machine learning and to learn new skills.\u201d He recalled getting hooked while building a fantasy sports predictor, combining his soccer fandom with honing his data abilities. Sebastian explains, \u201cThat\u2019s how I taught myself pandas.\u201d Tackling hands-on projects and solving real problems that you feel passionate about accelerates learning.<\/p>\n My first project in machine learning\u2026 was a fun one\u2026 I was working on fantasy sports predictions back then. I was a big soccer fan. Based on that I built machine learning classifiers with scikit-learn, very simple ones, to basically predict [who] the promising players were, and that was very interesting as an exercise because that\u2019s how I taught myself pandas\u2026 I tried to automate as much as possible, so I was also trying to do some simple NLP, going through news articles, basically predicting the sentiment and extracting names from players who were injured and these types of things. It was very challenging, but it was a very good exercise to learn data processing and implementing simple\u00a0things.\u201d<\/p><\/blockquote>\n When beginning a new ML project you should always find some baseline performance. For example when starting a text classification project, Sebastian says, \u201cEven if you know more sophisticated techniques, even if it makes sense to use a Large Language Model\u2026 Start with a simple logistic regression, maybe a bag of words to get a baseline.\u201d<\/p>\n By building a baseline before trying more advanced techniques you can get a better understanding of the problem and the data. If you run into issues when implementing more advanced techniques, having a baseline model where you already read and processed the data can help debug more complex models. If an advanced model underperforms the baseline, it may be an indicator that there are data issues rather than model limitations.<\/p>\n \u201cI would say always start with [simple techniques] even if you know more sophisticated techniques if we go back to what we talked about with large language models even if it makes more sense for a classification problem to fine-tune a large language model for that, I would start\u2026 with a simple logistic regression classifier, maybe bag-of-words model to just get a baseline. Use something where you are confident, it\u2019s very simple and it works, let\u2019s say using scikit-learn before trying the more complicated things. It\u2019s not only because we don\u2019t want to use the complicated things because the simple ones are efficient, it\u2019s more about also even checking our solutions like if our fine-tuned model or let\u2019s say BERT or LLM performs worse than the logistic regression classifier maybe we have a bug in our code, maybe we didn\u2019t process the input correctly, [maybe we didn\u2019t] tokenize it correctly – it\u2019s always a good idea to really start simple and then increasingly get complicated or improve – let\u2019s say improve by adding things instead of starting complicated and then trying to debug the complicated solution to find out where the error is essentially.\u201d<\/p><\/blockquote>\n The field is changing quickly. While it\u2019s important to start slow and take things step by step it is equally important to stay flexible and open to adopting new methods and ideas. Techniques and approaches in machine learning have a tendency to come in and out of\u00a0style.<\/p>\n Sebastian stresses the importance of adaptability amid relentless change. \u201cThings change completely. We were using [Generative Adversarial Networks] GANs [a few years ago] and now we\u2019re using diffusion models\u2026 [be] open to change.\u201d Machine learning rewards the nimble. He emphasizes being open to new experiences both in machine learning and\u00a0life.<\/p>\n The pursuit of Artificial General Intelligence (AGI) is a worthy goal but specialized systems often provide better results. Depending on the use case, a specialized system may be more appropriate than a one-size-fits-all approach. Sebastian discusses how systems may be a combination of smaller models where the first model is used to determine which specialized model the task should be directed\u00a0to.<\/p>\n Regardless, the pursuit for AGI is an incredible motivator and has led to many breakthroughs. As Sebastian explains, the quest for AGI pushed breakthroughs like DeepMind\u2019s AlphaGo beating the best humans at Go. And while AlphaGo itself may not be directly useful, \u201cit ultimately led to AlphaFold, the first version, for protein structure prediction.\u201d<\/p>\n The dream of AGI serves as inspiration, but specialized systems focused on narrow domains currently provide the most value. Still, the race towards AGI has led to advances that found practical application.<\/p>\n \u201cI think no one knows how far we are from AGI\u2026 I think there\u2019s a lot more hype around AGI it appears closer than before of course because we have these models. There are people though who say okay this is the totally wrong approach we need something completely different if we want to get AGI no one knows what that approach looks like so it\u2019s really hard to\u00a0say\u2026<\/p><\/blockquote>\n \u2026The thing though what I always find interesting is do we need AGI more like a philosophical question\u2026 AGI is useful as the motivation<\/strong>. I think it motivates a lot of people to work on AI to make that progress. I think without AGI we wouldn\u2019t have things like AlphaGo where they had the breakthrough they basically beat the best player at go\u2026 how is that useful – I would say maybe go and chess engines are not useful but I think it ultimately led to AlphaFold the first version for protein structure prediction and then AlphaFold 2 which is not based on large language models but uses large language models. So in that case I think without large language models and without the desire maybe to develop AGI we wouldn\u2019t have all these very useful things in the Natural Sciences<\/strong> and so my question is do we need AGI or do we really just need good models for special purposes\u2026<\/strong>\u201d<\/p><\/blockquote>\n Coding algorithms without depending on external libraries (e.g., using just Python) helps build a better understanding of the underlying concepts. Sebastian explains, \u201cImplementing algorithms from scratch helps build intuition and peel back the layers to make things more understandable.\u201d<\/p>\n \u201cImplementing algorithms from scratch helps build intuition and peel back the layers to make things more understandable.\u201d<\/p><\/blockquote>\n Fortunately, Sebastian shares many of these educational implementations through posts and tutorials. We dove into Sebastian\u2019s breakdown of Self-Attention of LLMs from Scratch<\/a> where he breaks down the importance of the \u201cself-attention\u201d mechanism which is a cornerstone of both transformers and stable-diffusion.<\/p>\n In real world applications, you don\u2019t have to reinvent the wheel. Sebastian expands for things that already exist, \u201cI think that is a lot of work and also risky.\u201d While building from scratch is enlightening, production-ready applications rely on proven, battle-tested libraries.<\/p>\n \u201cwhat I did was for education\u2026 let\u2019s implement a principal component analysis from scratch or let\u2019s implement a self-attention mechanism from scratch and writing the code but not necessarily as a library because I think there are already a lot of efficient implementations out there so it doesn\u2019t really make sense to reinvent the wheel<\/strong> but it\u2019s more about let\u2019s peel back a few layers make a very simple implementation of that so that people can read them because that\u2019s one thing\u200a\u2014\u200adeep learning libraries are becoming more powerful. If we look at PyTorch for example but they are also becoming much much harder to read\u200a\u2014\u200aso if I would ask you to take a look at the convolution operation in PyTorch I wouldn\u2019t even understand\u2026 I wouldn\u2019t even know where to look\u2026 to start with it\u2026 I mean for good reason because they implemented it very efficiently and then there\u2019s cuda on top of that\u2026 but as a user if I want to customize or even understand things it\u2019s very hard to look at the code so in that case I think there\u2019s value in peeling back the layers making a simple implementation for educational purposes to understand how things\u00a0work.<\/p><\/blockquote>\n Getting a model to relatively high performance is much easier than squeezing out the last few percentage points to reach extremely high performance. But that final push is vital\u200a\u2014\u200ait\u2019s the difference between an impressive prototype and a production-ready system. Even if rapid progress was made initially, the final seemingly marginal gains to reach \u201cperfection\u201d are very challenging.<\/p>\n Even if rapid progress was made initially, the final seemingly marginal gains to reach \u201cperfection\u201d are very challenging.<\/p><\/blockquote>\n Sebastian uses self-driving cars to drive this point across. \u201cFive years ago, they already had pretty impressive demos\u2026 but I do think it\u2019s the last few percent that are crucial.\u201d He continues, \u201cFive years ago, it was almost let\u2019s say 95% there, almost ready. Now five years later, we are maybe 97\u201398%, but can we get the last remaining percent points to really nail it and have them on the road reliably.\u201d<\/p>\n Sebastian draws a comparison between ChatGPT and Self-Driving cars. While astounding demos of both technologies exist, getting those last few percentage points of performance to reach full reliability has proven difficult and\u00a0vital.<\/p>\n Sebastian cautions against forcing ML everywhere, stating \u201cIf you have a hammer, everything looks like a nail\u2026 the question becomes when to use AI and when not to use AI.\u201d The trick is often knowing when to use rules, ML, or other tools. Sebastian shares, \u201cRight now, we are using AI for a lot of things because it is exciting, and we want to see how far we can push it until it breaks or doesn\u2019t work\u2026 sometimes we have nonsensical applications of AI because of\u00a0that.\u201d<\/p>\n Automation has limits. Sometimes rules and human expertise outperform AI. It\u2019s important to pick the best approach for each task. Just because we can use AI\/ML as a solution doesn\u2019t mean we should for every\u00a0problem.<\/p>\n \u201c[There\u2019s] a saying if you have a hammer everything looks like a nail<\/strong>, and I think this is right now a little bit true with ChatGPT because we just have fun with it\u2026 let me see if it can do this and that but it doesn\u2019t mean we should be using it for everything\u2026 now the question is basically the next level\u2026 when to use AI and when not to use AI\u2026 <\/strong>because right now we are using AI for a lot of things because it\u2019s exciting and we want to see how far we can push it until it let\u2019s say breaks so it doesn\u2019t work but sometimes we have nonsensical applications of AI because of that.\u00a0\u2026like training a neural network that can do calculation\u2026 but we wouldn\u2019t let it do the math matrix multiplication itself because you know it\u2019s non-deterministic in a sense so you don\u2019t know if it\u2019s going to be correct or not depending on your inputs and there are definite rules that we can use so why approximate when we can have it accurate\u201d<\/p><\/blockquote>\n Ensemble methods like model stacking can improve prediction robustness, but diversity is key\u200a\u2014\u200acombining correlated models that make similar types of errors won\u2019t provide much\u00a0upside.<\/p>\n As Sebastian explains, \u201cBuilding an ensemble of different methods is usually something to make [models] more robust and [produce] accurate predictions. And ensemble methods usually work best if you have an ensemble of different methods. If there\u2019s no correlation in terms of how they work. So they are not redundant, basically.\u201d<\/p>\n The goal is to have a diverse set of complementary models. For example, you might ensemble a random forest model with a neural network, or a gradient boosting machine with a k-nearest neighbors model. Stacking models that have high diversity improves the ensemble\u2019s ability to correct errors made by individual models.<\/p>\n So when building ensembles, seek diversity\u200a\u2014\u200ause different algorithms, different feature representations, different hyperparameters, etc. Correlation analysis of predictions can help identify which models provide unique signal vs redundancy. The key is having a complementary set of models in the ensemble, not just combining slight variations of the same approach.<\/p>\n \u201c\u2026building an ensemble of different methods is usually something to improve how you can make more robust and accurate predictions and ensemble methods usually work best if you have an ensemble of different methods\u200a\u2014\u200aif there\u2019s no correlation in terms of how they work. So they are not redundant<\/strong> basically. That is also one argument why it makes sense to maybe approach the problem from different angles to produce totally different systems that we can then combine.\u201d<\/p><\/blockquote>\n Models with varying strengths and weaknesses can effectively counterbalance each other\u2019s shortcomings, leading to more reliable overall performance.<\/p>\n \u201cThere\u2019s a whole branch of research on [how] neural networks are typically overconfident on out of distribution data.\u201d ML predictions can be misleadingly overconfident on unusual data. Sebastian describes, \u201cSo what happens is if you have data that is slightly different from your training data or let\u2019s say out of the distribution, the network will if you program it to give a confidence score as part of the output, this score for the data where it\u2019s especially wrong is usually over confident\u2026 which makes it even more dangerous.\u201d Validate reliability before deployment, rather than blindly trusting in confidence scores. Confidence scores can often be high for wrong predictions making them misleading for unfamiliar data.<\/p>\n Validate reliability before deployment, rather than blindly trusting in confidence scores.<\/p><\/blockquote>\n To combat overconfidence in practice, start by establishing multiple validation sets that include both edge cases and known out-of-distribution examples, keeping a separate test set for final verification. A robust monitoring system is equally crucial\u200a\u2014\u200atrack confidence scores over time, monitor the rate of high-confidence errors, set up alerts for unusual confidence patterns, and maintain comprehensive logs of all predictions and their associated confidence scores.<\/p>\n For production systems, implement fallback mechanisms including simpler backup models, clear business rules for low-confidence cases, and human review processes for highly uncertain predictions. Regular maintenance is essential: as new data becomes available it may be worthwhile to retrain models, adjust confidence thresholds based on real-world performance, fine-tune out-of-distribution detection parameters, and continuously validate model calibration. These practices help ensure your models remain reliable and self-aware of their limitations, rather than falling into the trap of overconfidence.<\/p>\n ChatGPT (and other generative models) are good brainstorming partners and can be used for ideation when \u201cit doesn\u2019t need to be 100% correct.\u201d Sebastian warns that the model\u2019s output should not be used as the final output. Large language models can generate text to accelerate drafting but require human refinement. It\u2019s important to be fully aware of the limitations of the\u00a0LLMs.<\/p>\n \u201cMake sure you have fun. Try not to do all at once.\u201d Learning is most effective and sustainable when it\u2019s enjoyable. Passion for the process itself, not just outcomes, leads to mastery. Sebastian emphasizes to remember to recharge and connect with others who inspire you. Sebastian shares, \u201cWhatever you do, have fun, enjoy, share the joy\u2026 things are sometimes complicated and work can be intense. We want to get things done, but don\u2019t forget\u2026 to stop and enjoy sometimes.\u201d<\/p>\n While the field\u2019s rapid advancement and complexity can be overwhelming, Sebastian offers a clear path forward: build rock-solid fundamentals, always start with baseline models, and maintain systematic approaches to combat common pitfalls. He advocates for implementing algorithms from scratch before using high-level, optimized libraries to ensure deep understanding. He offers practical strategies\u200a\u2014\u200asuch as including diversity in ensemble models, critically assessing model confidence, and recognizing the difficulty of the \u201clast mile\u201d\u200a\u2014\u200afor developing reliable and dependable production-quality AI\u00a0systems.<\/p>\n Sebastian stresses that mastering machine learning isn\u2019t about chasing every new development<\/strong>. Instead, it\u2019s about building a strong foundation that enables you to evaluate and adapt to meaningful advances. By focusing on core principles while remaining open to new techniques, we can build the confidence to face increasingly complex challenges. Whether you\u2019re implementing your first machine learning project or architecting enterprise-scale AI systems, the key is to embrace the learning process: start simple, evaluate thoroughly, and never stop questioning your assumptions. In a field that seems to reinvent itself almost daily, these timeless principles are our most reliable\u00a0guides.<\/p>\n Enjoy these lessons and check out the full Learning from Machine Learning<\/a> interview here:<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AXaprQataxU2vdJ_kIJIOpg.png?ssl=1\")
\n
Lessons from a\u00a0Pioneer<\/h3>\n
1. Start simple and be\u00a0patient<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/500\/0%2AOCFPM_KDjFyKrGuS.jpg?ssl=1\")
2. Learn by\u00a0doing<\/h4>\n
3. Always get a\u00a0baseline<\/h4>\n
4. Embrace\u00a0change<\/h4>\n
5. Find balance between specialized and general\u00a0systems<\/h4>\n
6. When learning, implement from\u00a0scratch<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ADidT8VniVRFn6p6aWOlBVA.jpeg?ssl=1\")
7. In production, don\u2019t reinvent the\u00a0wheel!<\/h4>\n
8. It\u2019s the last mile that\u00a0counts<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2ADqYdNPnIggvoN9ajmUMB8A.png?ssl=1\")
9. Use the right tool for the\u00a0job<\/h4>\n
10. Seek Diversity in Model Ensembles<\/h4>\n
11. Beware of overconfidence<\/h4>\n
12. Leverage Large Language Models responsibly<\/h4>\n
13. Remember to have\u00a0fun!<\/h4>\n
Conclusion<\/h3>\n
Check out the Full Interview<\/h3>\n