# Table of Contents
- [Welcome to DataTalks.Club](#welcome-to-datatalks-club)
- [15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide – DataTalks.Club](#15-free-data-engineering-courses-5-paid-courses-complete-guide-datatalks-club)
- [Open Source and Free AI Agent Evaluation Tools – DataTalks.Club](#open-source-and-free-ai-agent-evaluation-tools-datatalks-club)
- [Join our Slack – DataTalks.Club](#join-our-slack-datatalks-club)
- [Data Science Manager vs Expert: Which Role Does Your Company Need? – DataTalks.Club](#data-science-manager-vs-expert-which-role-does-your-company-need-datatalks-club)
- [20+ Best Data Science Slack Communities to Join in 2025 – DataTalks.Club](#20-best-data-science-slack-communities-to-join-in-2025-datatalks-club)
- [Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities – DataTalks.Club](#data-team-roles-explained-alexey-grigorev-olx-on-skills-and-responsibilities-datatalks-club)
- [How Do Data Professionals Use MLOps Tools and Frameworks? – DataTalks.Club](#how-do-data-professionals-use-mlops-tools-and-frameworks-datatalks-club)
- [Building Discipline in Machine Learning with ML Zoomcamp – DataTalks.Club](#building-discipline-in-machine-learning-with-ml-zoomcamp-datatalks-club)
- [Building an AI Agent that Thrives in the Real World – DataTalks.Club](#building-an-ai-agent-that-thrives-in-the-real-world-datatalks-club)
- [How Do Professionals Use AI Tools for Personal Productivity? – DataTalks.Club](#how-do-professionals-use-ai-tools-for-personal-productivity-datatalks-club)
- [Winning Solutions from the LLM Zoomcamp 2024 Competition – DataTalks.Club](#winning-solutions-from-the-llm-zoomcamp-2024-competition-datatalks-club)
- [How Do Data Professionals Use Data Engineering Tools and Practices? – DataTalks.Club](#how-do-data-professionals-use-data-engineering-tools-and-practices-datatalks-club)
- [How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp – DataTalks.Club](#how-to-build-a-blood-cell-classifier-for-cancer-prediction-a-case-study-from-ml-zoomcamp-datatalks-club)
- [How Do Professionals Use LLM Tools and Frameworks? – DataTalks.Club](#how-do-professionals-use-llm-tools-and-frameworks-datatalks-club)
- [Rahul Jain – DataTalks.Club](#rahul-jain-datatalks-club)
- [Aashish Nair – DataTalks.Club](#-aashish-nair-datatalks-club)
- [How to Build a Waste Classifier: A Case Study from ML Zoomcamp – DataTalks.Club](#how-to-build-a-waste-classifier-a-case-study-from-ml-zoomcamp-datatalks-club)
- [Prepare for (Better) Structured Data Extraction – DataTalks.Club](#prepare-for-better-structured-data-extraction-datatalks-club)
- [Aaron Wishnick – DataTalks.Club](#aaron-wishnick-datatalks-club)
- [Aaisha Muhammad – DataTalks.Club](#aaisha-muhammad-datatalks-club)
- [8 Newsletters for Data Science, AI, and ML Enthusiasts – DataTalks.Club](#8-newsletters-for-data-science-ai-and-ml-enthusiasts-datatalks-club)
- [A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won! – DataTalks.Club](#a-summary-of-the-kaggle-kitchenware-classification-competition-find-out-who-won-datatalks-club)
- [Naming Variables in Machine Learning – DataTalks.Club](#naming-variables-in-machine-learning-datatalks-club)
- [DataOps: Similarities and Differences with Data Engineering and Data Science – DataTalks.Club](#dataops-similarities-and-differences-with-data-engineering-and-data-science-datatalks-club)
- [Important SQL Fact That Everyone Should Know – DataTalks.Club](#important-sql-fact-that-everyone-should-know-datatalks-club)
- [Interview with Ken Wu – DataTalks.Club](#interview-with-ken-wu-datatalks-club)
- [Abouzar Abbaspour – DataTalks.Club](#abouzar-abbaspour-datatalks-club)
- [Adam Sroka – DataTalks.Club](#adam-sroka-datatalks-club)
- [Aditya Seshaditya – DataTalks.Club](#aditya-seshaditya-datatalks-club)
- [Interview with Valerii Chetvertakov – DataTalks.Club](#interview-with-valerii-chetvertakov-datatalks-club)
- [Data Engineers Aren't Plumbers – DataTalks.Club](#data-engineers-aren-t-plumbers-datatalks-club)
- [Regularization in Regression – DataTalks.Club](#regularization-in-regression-datatalks-club)
- [Agita Jaunzeme – DataTalks.Club](#agita-jaunzeme-datatalks-club)
- [Admond Lee Kin Lim – DataTalks.Club](#admond-lee-kin-lim-datatalks-club)
- [Guidelines to Get a Data Engineer Job Against the Odds – DataTalks.Club](#guidelines-to-get-a-data-engineer-job-against-the-odds-datatalks-club)
- [Do You Know the Golden Rules While Working With Data? – DataTalks.Club](#do-you-know-the-golden-rules-while-working-with-data-datatalks-club)
- [What is DataOps exactly? – DataTalks.Club](#what-is-dataops-exactly-datatalks-club)
- [Agnes van Belle – DataTalks.Club](#agnes-van-belle-datatalks-club)
- [Aishwarya Jadhav – DataTalks.Club](#aishwarya-jadhav-datatalks-club)
- [Aditya Gautam – DataTalks.Club](#aditya-gautam-datatalks-club)
- [Agostino Calamia – DataTalks.Club](#agostino-calamia-datatalks-club)
- [Agnieszka Mikołajczyk – DataTalks.Club](#agnieszka-miko-ajczyk-datatalks-club)
- [Alena Astrakhantseva – DataTalks.Club](#alena-astrakhantseva-datatalks-club)
- [How to run PostgreSQL and PgAdmin with Docker – DataTalks.Club](#how-to-run-postgresql-and-pgadmin-with-docker-datatalks-club)
- [How I Landed a Job As a Product Analyst – DataTalks.Club](#how-i-landed-a-job-as-a-product-analyst-datatalks-club)
- [Aleksander Molak – DataTalks.Club](#aleksander-molak-datatalks-club)
- [Starting a Career in Data Science at 45 – DataTalks.Club](#starting-a-career-in-data-science-at-45-datatalks-club)
- [Key Lessons from ML Zoomcamp: Serena Haidar – DataTalks.Club](#key-lessons-from-ml-zoomcamp-serena-haidar-datatalks-club)
- [Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences – DataTalks.Club](#simplify-technical-concepts-a-3-step-framework-for-non-technical-audiences-datatalks-club)
- [MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices – DataTalks.Club](#mlops-in-10-minutes-design-train-and-operate-with-proven-practices-datatalks-club)
- [Aleksander Kruszelnicki – DataTalks.Club](#aleksander-kruszelnicki-datatalks-club)
- [Alexander Guschin – DataTalks.Club](#alexander-guschin-datatalks-club)
- [DataTalks.Club Community Demographics – DataTalks.Club](#datatalks-club-community-demographics-datatalks-club)
- [What Open Source Can Do For Your Data Career – DataTalks.Club](#what-open-source-can-do-for-your-data-career-datatalks-club)
- [Adrian Brudaru – DataTalks.Club](#adrian-brudaru-datatalks-club)
- [Alex Chung – DataTalks.Club](#alex-chung-datatalks-club)
- [Alex Ioannides – DataTalks.Club](#alex-ioannides-datatalks-club)
- [Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals – DataTalks.Club](#data-storytelling-characters-conflict-and-conclusion-for-data-professionals-datatalks-club)
- [The Hiring Process for Data Professionals – DataTalks.Club](#the-hiring-process-for-data-professionals-datatalks-club)
- [Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data – DataTalks.Club](#customer-segmentation-with-rfm-and-k-means-7-segments-from-gaming-data-datatalks-club)
- [Akela Drissner – DataTalks.Club](#akela-drissner-datatalks-club)
- [Alicja Notowska – DataTalks.Club](#alicja-notowska-datatalks-club)
- [Alex Kim – DataTalks.Club](#alex-kim-datatalks-club)
- [Alexander Hendorf – DataTalks.Club](#alexander-hendorf-datatalks-club)
- [Alex Litvinov – DataTalks.Club](#alex-litvinov-datatalks-club)
- [How to Build a Data Science Team from Scratch: Complete Hiring Guide – DataTalks.Club](#how-to-build-a-data-science-team-from-scratch-complete-hiring-guide-datatalks-club)
- [LLM Zoomcamp: Free LLM Engineering Course and Certification – DataTalks.Club](#llm-zoomcamp-free-llm-engineering-course-and-certification-datatalks-club)
- [How to Setup a Lightweight Local Version for Airflow – DataTalks.Club](#how-to-setup-a-lightweight-local-version-for-airflow-datatalks-club)
- [Anahita Pakiman – DataTalks.Club](#anahita-pakiman-datatalks-club)
- [Alvaro Navas Peire – DataTalks.Club](#alvaro-navas-peire-datatalks-club)
- [Andreas Kretz – DataTalks.Club](#andreas-kretz-datatalks-club)
- [Andreas Syrén – DataTalks.Club](#andreas-syr-n-datatalks-club)
- [Anastasia Karavdina – DataTalks.Club](#anastasia-karavdina-datatalks-club)
- [Andrada Olteanu – DataTalks.Club](#andrada-olteanu-datatalks-club)
- [MLOps Zoomcamp: Free MLOps Course and Certification – DataTalks.Club](#mlops-zoomcamp-free-mlops-course-and-certification-datatalks-club)
- [Alexander Daniel Rios – DataTalks.Club](#alexander-daniel-rios-datatalks-club)
- [Alexia Audevart – DataTalks.Club](#alexia-audevart-datatalks-club)
- [Alex Petrov – DataTalks.Club](#alex-petrov-datatalks-club)
- [Andrey Cheptsov – DataTalks.Club](#andrey-cheptsov-datatalks-club)
- [Andrei Tserakhau – DataTalks.Club](#andrei-tserakhau-datatalks-club)
- [Andre Schumacher – DataTalks.Club](#andre-schumacher-datatalks-club)
- [Andreea Munteanu – DataTalks.Club](#andreea-munteanu-datatalks-club)
- [Amber Roberts – DataTalks.Club](#amber-roberts-datatalks-club)
- [Angela Ramirez – DataTalks.Club](#angela-ramirez-datatalks-club)
- [Anish Shah – DataTalks.Club](#anish-shah-datatalks-club)
- [Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide – DataTalks.Club](#python-ci-cd-with-github-actions-pre-commit-linters-and-pytest-guide-datatalks-club)
- [Andrey Shtylenko – DataTalks.Club](#andrey-shtylenko-datatalks-club)
- [Andrew McMahon – DataTalks.Club](#andrew-mcmahon-datatalks-club)
- [Andy Petrella – DataTalks.Club](#andy-petrella-datatalks-club)
- [Andrew Jones – DataTalks.Club](#andrew-jones-datatalks-club)
- [NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset – DataTalks.Club](#ner-with-reformer-in-trax-end-to-end-tutorial-on-a-kaggle-dataset-datatalks-club)
- [Anusha Akkina – DataTalks.Club](#anusha-akkina-datatalks-club)
- [20+ Best Free Machine Learning Courses: Learn from Stanford, MIT and Google Without Paying Tuition – DataTalks.Club](#20-best-free-machine-learning-courses-learn-from-stanford-mit-and-google-without-paying-tuition-datatalks-club)
- [Apurva Misra – DataTalks.Club](#apurva-misra-datatalks-club)
- [Antonis Stellas – DataTalks.Club](#antonis-stellas-datatalks-club)
- [Anna Hannemann – DataTalks.Club](#anna-hannemann-datatalks-club)
- [Ankur A. Patel – DataTalks.Club](#ankur-a-patel-datatalks-club)
- [Aparna Dhinakaran – DataTalks.Club](#aparna-dhinakaran-datatalks-club)
- [Deploy ML Models on AWS Lambda with Docker Containers and SAM – DataTalks.Club](#deploy-ml-models-on-aws-lambda-with-docker-containers-and-sam-datatalks-club)
- [Arman Jabbari – DataTalks.Club](#arman-jabbari-datatalks-club)
- [Arseny Kravchenko – DataTalks.Club](#arseny-kravchenko-datatalks-club)
- [Arpit Choudhury – DataTalks.Club](#arpit-choudhury-datatalks-club)
- [Artemii Frolov – DataTalks.Club](#artemii-frolov-datatalks-club)
- [Ba Linh Le – DataTalks.Club](#ba-linh-le-datatalks-club)
- [Atita Arora – DataTalks.Club](#atita-arora-datatalks-club)
- [Barbara Sobkowiak – DataTalks.Club](#barbara-sobkowiak-datatalks-club)
- [Barr Moses – DataTalks.Club](#barr-moses-datatalks-club)
- [Antje Barth – DataTalks.Club](#antje-barth-datatalks-club)
- [Anthony Virtuoso – DataTalks.Club](#anthony-virtuoso-datatalks-club)
- [Bartosz Mikulski – DataTalks.Club](#bartosz-mikulski-datatalks-club)
- [Ashish Patel – DataTalks.Club](#ashish-patel-datatalks-club)
- [Ben Taylor – DataTalks.Club](#ben-taylor-datatalks-club)
- [Bart Vandekerckhove – DataTalks.Club](#bart-vandekerckhove-datatalks-club)
- [Bastien Boutonnet – DataTalks.Club](#bastien-boutonnet-datatalks-club)
- [Bela Wiertz – DataTalks.Club](#bela-wiertz-datatalks-club)
- [Bhavani Ravi – DataTalks.Club](#bhavani-ravi-datatalks-club)
- [Carmine Paolino – DataTalks.Club](#carmine-paolino-datatalks-club)
- [Caitlin Moorman – DataTalks.Club](#caitlin-moorman-datatalks-club)
- [Boyan Angelov – DataTalks.Club](#boyan-angelov-datatalks-club)
- [Ben Wilson – DataTalks.Club](#ben-wilson-datatalks-club)
- [Starting a Career as a Data Scientist – DataTalks.Club](#starting-a-career-as-a-data-scientist-datatalks-club)
- [The Essentials of Public Speaking for a Career in Data Science – DataTalks.Club](#the-essentials-of-public-speaking-for-a-career-in-data-science-datatalks-club)
- [DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained – DataTalks.Club](#devops-vs-mlops-workflows-monitoring-and-maturity-models-explained-datatalks-club)
- [Cathy Chen – DataTalks.Club](#cathy-chen-datatalks-club)
- [Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps – DataTalks.Club](#free-datatalks-club-courses-ml-data-engineering-mlops-llm-ai-dev-tools-zoomcamps-datatalks-club)
- [Christine Cepelak – DataTalks.Club](#christine-cepelak-datatalks-club)
- [Chris Fregly – DataTalks.Club](#chris-fregly-datatalks-club)
- [Chip Huyen – DataTalks.Club](#chip-huyen-datatalks-club)
- [Alexey Grigorev – DataTalks.Club](#alexey-grigorev-datatalks-club)
- [Christiaan Swart – DataTalks.Club](#christiaan-swart-datatalks-club)
- [Christopher Bergh – DataTalks.Club](#christopher-bergh-datatalks-club)
- [Dânia Meira – DataTalks.Club](#d-nia-meira-datatalks-club)
- [CJ Jenkins – DataTalks.Club](#cj-jenkins-datatalks-club)
- [Christian Winkler – DataTalks.Club](#christian-winkler-datatalks-club)
- [Cristian Martinez – DataTalks.Club](#cristian-martinez-datatalks-club)
- [Daliana Liu – DataTalks.Club](#daliana-liu-datatalks-club)
- [Dan Becker – DataTalks.Club](#dan-becker-datatalks-club)
- [Daniel Egbo – DataTalks.Club](#daniel-egbo-datatalks-club)
- [Daniel Svonava – DataTalks.Club](#daniel-svonava-datatalks-club)
- [AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers – DataTalks.Club](#ai-dev-tools-zoomcamp-free-course-to-master-ai-tools-for-developers-datatalks-club)
- [Christoph Molnar – DataTalks.Club](#christoph-molnar-datatalks-club)
- [Data Engineering Zoomcamp: Free Data Engineering Course and Certification – DataTalks.Club](#data-engineering-zoomcamp-free-data-engineering-course-and-certification-datatalks-club)
- [Danny Leybzon – DataTalks.Club](#danny-leybzon-datatalks-club)
- [Angelica Lo Duca – DataTalks.Club](#angelica-lo-duca-datatalks-club)
- [Danny Ma – DataTalks.Club](#danny-ma-datatalks-club)
- [Dashel Ruiz Perez – DataTalks.Club](#dashel-ruiz-perez-datatalks-club)
- [Dat Tran – DataTalks.Club](#dat-tran-datatalks-club)
- [Dave Flynn – DataTalks.Club](#dave-flynn-datatalks-club)
- [David Bader – DataTalks.Club](#david-bader-datatalks-club)
- [Dave Bechberger – DataTalks.Club](#dave-bechberger-datatalks-club)
- [David Bednar – DataTalks.Club](#david-bednar-datatalks-club)
- [David Mertz – DataTalks.Club](#david-mertz-datatalks-club)
- [David Gates – DataTalks.Club](#david-gates-datatalks-club)
- [David Stephenson – DataTalks.Club](#david-stephenson-datatalks-club)
- [Demetrios Brinkmann – DataTalks.Club](#demetrios-brinkmann-datatalks-club)
- [ML Zoomcamp: Free Machine Learning Engineering Course and Certification – DataTalks.Club](#ml-zoomcamp-free-machine-learning-engineering-course-and-certification-datatalks-club)
- [Delina Ivanova – DataTalks.Club](#delina-ivanova-datatalks-club)
- [Daynan Crull – DataTalks.Club](#daynan-crull-datatalks-club)
- [Dimitri Visnadi – DataTalks.Club](#dimitri-visnadi-datatalks-club)
- [David Sweet – DataTalks.Club](#david-sweet-datatalks-club)
- [Denise Gosnell – DataTalks.Club](#denise-gosnell-datatalks-club)
- [Denis Rothman – DataTalks.Club](#denis-rothman-datatalks-club)
- [Dmitry Muzalevskiy – DataTalks.Club](#dmitry-muzalevskiy-datatalks-club)
- [Don Jones – DataTalks.Club](#don-jones-datatalks-club)
- [Eleni Stamatelou – DataTalks.Club](#eleni-stamatelou-datatalks-club)
- [Elena Samuylova – DataTalks.Club](#elena-samuylova-datatalks-club)
- [Eddy Zulkifly – DataTalks.Club](#eddy-zulkifly-datatalks-club)
- [Duygu Altinok – DataTalks.Club](#duygu-altinok-datatalks-club)
- [Douglas Gray – DataTalks.Club](#douglas-gray-datatalks-club)
- [Elias Nema – DataTalks.Club](#elias-nema-datatalks-club)
- [Eleni Tzirita Zacharatou – DataTalks.Club](#eleni-tzirita-zacharatou-datatalks-club)
- [Doug Turnbull – DataTalks.Club](#doug-turnbull-datatalks-club)
- [Elle O'Brien – DataTalks.Club](#elle-o-brien-datatalks-club)
- [Emeli Dral – DataTalks.Club](#emeli-dral-datatalks-club)
- [Emil Bogomolov – DataTalks.Club](#emil-bogomolov-datatalks-club)
- [Ellen König – DataTalks.Club](#ellen-k-nig-datatalks-club)
- [Ella (Wati) Sahnan – DataTalks.Club](#ella-wati-sahnan-datatalks-club)
- [Eric Sims – DataTalks.Club](#eric-sims-datatalks-club)
- [Emmanuel Ameisen – DataTalks.Club](#emmanuel-ameisen-datatalks-club)
- [Emmanuel Raj – DataTalks.Club](#emmanuel-raj-datatalks-club)
- [Engin Yöyen – DataTalks.Club](#engin-y-yen-datatalks-club)
- [Ertugrul Mutlu – DataTalks.Club](#ertugrul-mutlu-datatalks-club)
- [Fabiana Clemente – DataTalks.Club](#fabiana-clemente-datatalks-club)
- [Eugene Yan – DataTalks.Club](#eugene-yan-datatalks-club)
- [Erum Afzal – DataTalks.Club](#erum-afzal-datatalks-club)
- [Evan Shellshear – DataTalks.Club](#evan-shellshear-datatalks-club)
- [Faisal Masood – DataTalks.Club](#faisal-masood-datatalks-club)
- [Filipa Castro – DataTalks.Club](#filipa-castro-datatalks-club)
- [Florian Hoenicke – DataTalks.Club](#florian-hoenicke-datatalks-club)
- [Fernando Doglio – DataTalks.Club](#fernando-doglio-datatalks-club)
- [Gant Laborde – DataTalks.Club](#gant-laborde-datatalks-club)
- [Geo Jolly – DataTalks.Club](#geo-jolly-datatalks-club)
- [Gloria Quiceno – DataTalks.Club](#gloria-quiceno-datatalks-club)
- [Gonçalo Sequeira – DataTalks.Club](#gon-alo-sequeira-datatalks-club)
- [Giuseppe Bonaccorso – DataTalks.Club](#giuseppe-bonaccorso-datatalks-club)
- [Gráinne McKnight – DataTalks.Club](#gr-inne-mcknight-datatalks-club)
- [Greg Coquillo – DataTalks.Club](#greg-coquillo-datatalks-club)
- [Guillaume Lemaître – DataTalks.Club](#guillaume-lema-tre-datatalks-club)
- [Hagop Dippel – DataTalks.Club](#hagop-dippel-datatalks-club)
- [Guy Adams – DataTalks.Club](#guy-adams-datatalks-club)
- [Hannes Hapke – DataTalks.Club](#hannes-hapke-datatalks-club)
- [Hayden Liu – DataTalks.Club](#hayden-liu-datatalks-club)
- [Haziqa Sajid – DataTalks.Club](#haziqa-sajid-datatalks-club)
- [Hélder Russa – DataTalks.Club](#h-lder-russa-datatalks-club)
- [Hiba Jamal – DataTalks.Club](#hiba-jamal-datatalks-club)
- [Himanshu Upreti – DataTalks.Club](#himanshu-upreti-datatalks-club)
- [Hugo Bowne-Anderson – DataTalks.Club](#hugo-bowne-anderson-datatalks-club)
- [Igor Demidov – DataTalks.Club](#igor-demidov-datatalks-club)
- [Igor Susmelj – DataTalks.Club](#igor-susmelj-datatalks-club)
- [Ilia Ivanov – DataTalks.Club](#ilia-ivanov-datatalks-club)
- [Illia Todor – DataTalks.Club](#illia-todor-datatalks-club)
- [Ilya Boytsov – DataTalks.Club](#ilya-boytsov-datatalks-club)
- [Articles – DataTalks.Club](#articles-datatalks-club)
- [Irina Brudaru – DataTalks.Club](#irina-brudaru-datatalks-club)
- [Ioannis Mesionis – DataTalks.Club](#ioannis-mesionis-datatalks-club)
- [Isabella Bicalho – DataTalks.Club](#isabella-bicalho-datatalks-club)
- [Itai Admi – DataTalks.Club](#itai-admi-datatalks-club)
- [Ivan Bilan – DataTalks.Club](#ivan-bilan-datatalks-club)
- [Ivan Brigida – DataTalks.Club](#ivan-brigida-datatalks-club)
- [Ivan Potapov – DataTalks.Club](#ivan-potapov-datatalks-club)
- [Jack Blandin – DataTalks.Club](#jack-blandin-datatalks-club)
- [Jacques Peeters – DataTalks.Club](#jacques-peeters-datatalks-club)
- [Jakob Graff – DataTalks.Club](#jakob-graff-datatalks-club)
- [James Phoenix – DataTalks.Club](#james-phoenix-datatalks-club)
- [Jamie Broomall – DataTalks.Club](#jamie-broomall-datatalks-club)
- [Janna Lipenkova – DataTalks.Club](#janna-lipenkova-datatalks-club)
- [Jan Schlicht – DataTalks.Club](#jan-schlicht-datatalks-club)
- [Jan Zawadzki – DataTalks.Club](#jan-zawadzki-datatalks-club)
- [Jeanine Harb – DataTalks.Club](#jeanine-harb-datatalks-club)
- [Jeff Katz – DataTalks.Club](#jeff-katz-datatalks-club)
- [Jekaterina Kokatjuhha – DataTalks.Club](#jekaterina-kokatjuhha-datatalks-club)
- [Jens Albrecht – DataTalks.Club](#jens-albrecht-datatalks-club)
- [Jesse Anderson – DataTalks.Club](#jesse-anderson-datatalks-club)
- [Jessi Ashdown – DataTalks.Club](#jessi-ashdown-datatalks-club)
- [Jessica Greene – DataTalks.Club](#jessica-greene-datatalks-club)
- [Jessie Yaros – DataTalks.Club](#jessie-yaros-datatalks-club)
- [Joe Reis – DataTalks.Club](#joe-reis-datatalks-club)
- [Johanna Bayer – DataTalks.Club](#johanna-bayer-datatalks-club)
- [Book of the Week – DataTalks.Club](#book-of-the-week-datatalks-club)
- [Johannes Hötter – DataTalks.Club](#johannes-h-tter-datatalks-club)
- [Jonas Christensen – DataTalks.Club](#jonas-christensen-datatalks-club)
- [Jonathan Rioux – DataTalks.Club](#jonathan-rioux-datatalks-club)
- [José María Sánchez Salas – DataTalks.Club](#jos-mar-a-s-nchez-salas-datatalks-club)
- [Jon Skeet – DataTalks.Club](#jon-skeet-datatalks-club)
- [Josh Fischer – DataTalks.Club](#josh-fischer-datatalks-club)
- [Josh Tobin – DataTalks.Club](#josh-tobin-datatalks-club)
- [Joyce Kay Avila – DataTalks.Club](#joyce-kay-avila-datatalks-club)
- [Juan Manuel Perafan – DataTalks.Club](#juan-manuel-perafan-datatalks-club)
- [Juan Orduz – DataTalks.Club](#juan-orduz-datatalks-club)
- [Juan Pablo – DataTalks.Club](#juan-pablo-datatalks-club)
- [Julia Ostheimer – DataTalks.Club](#julia-ostheimer-datatalks-club)
- [Justin Ryan – DataTalks.Club](#justin-ryan-datatalks-club)
- [Events – DataTalks.Club](#events-datatalks-club)
- [Katarzyna Foremniak – DataTalks.Club](#katarzyna-foremniak-datatalks-club)
- [Kate Ogochukwu Nwankwo – DataTalks.Club](#kate-ogochukwu-nwankwo-datatalks-club)
- [Katharine Jarmul – DataTalks.Club](#katharine-jarmul-datatalks-club)
- [Katie Bauer – DataTalks.Club](#katie-bauer-datatalks-club)
- [Ken Youens-Clark – DataTalks.Club](#ken-youens-clark-datatalks-club)
- [Kevin Huo – DataTalks.Club](#kevin-huo-datatalks-club)
- [Khuyen Tran – DataTalks.Club](#khuyen-tran-datatalks-club)
- [Kim Falk – DataTalks.Club](#kim-falk-datatalks-club)
- [Kishan Manani – DataTalks.Club](#kishan-manani-datatalks-club)
- [Konrad Banachewicz – DataTalks.Club](#konrad-banachewicz-datatalks-club)
- [Kranti K. Parisa – DataTalks.Club](#kranti-k-parisa-datatalks-club)
- [Krzysztof Ograbek – DataTalks.Club](#krzysztof-ograbek-datatalks-club)
- [DataTalks.Club Podcast – DataTalks.Club](#datatalks-club-podcast-datatalks-club)
- [Krzysztof Szafanek – DataTalks.Club](#krzysztof-szafanek-datatalks-club)
- [People of DataTalks.Club – DataTalks.Club](#people-of-datatalks-club-datatalks-club)
- [Ksenia Legostay – DataTalks.Club](#ksenia-legostay-datatalks-club)
- [Kyle Shannon – DataTalks.Club](#kyle-shannon-datatalks-club)
- [Lalit Pagaria – DataTalks.Club](#lalit-pagaria-datatalks-club)
- [Lars Albertsson – DataTalks.Club](#lars-albertsson-datatalks-club)
- [Larysa Visengeriyeva – DataTalks.Club](#larysa-visengeriyeva-datatalks-club)
- [Laurence Moroney – DataTalks.Club](#laurence-moroney-datatalks-club)
- [Lavanya Gupta – DataTalks.Club](#lavanya-gupta-datatalks-club)
- [Leandro von Werra – DataTalks.Club](#leandro-von-werra-datatalks-club)
- [Leonard Püttmann – DataTalks.Club](#leonard-p-ttmann-datatalks-club)
- [Leon Wei – DataTalks.Club](#leon-wei-datatalks-club)
- [Lera Kaimashnіkova – DataTalks.Club](#lera-kaimashn-kova-datatalks-club)
- [Lewis Tunstall – DataTalks.Club](#lewis-tunstall-datatalks-club)
- [Liesbeth Dingemans – DataTalks.Club](#liesbeth-dingemans-datatalks-club)
- [Lina Weichbrodt – DataTalks.Club](#lina-weichbrodt-datatalks-club)
- [Lindsay McQuade – DataTalks.Club](#lindsay-mcquade-datatalks-club)
- [Loïc Magnien – DataTalks.Club](#lo-c-magnien-datatalks-club)
- [Lisa Cohen – DataTalks.Club](#lisa-cohen-datatalks-club)
- [Lior Barak – DataTalks.Club](#lior-barak-datatalks-club)
- [Loris Marini – DataTalks.Club](#loris-marini-datatalks-club)
- [Luca Massaron – DataTalks.Club](#luca-massaron-datatalks-club)
- [Luis Serrano – DataTalks.Club](#luis-serrano-datatalks-club)
- [Luís Oliveira – DataTalks.Club](#lu-s-oliveira-datatalks-club)
- [Luke Whipps – DataTalks.Club](#luke-whipps-datatalks-club)
- [Madiha Khalid – DataTalks.Club](#madiha-khalid-datatalks-club)
- [Magdalena Konkiewicz – DataTalks.Club](#magdalena-konkiewicz-datatalks-club)
- [Magdalena Kuhn – DataTalks.Club](#magdalena-kuhn-datatalks-club)
- [Mahmoud AbdelAziz – DataTalks.Club](#mahmoud-abdelaziz-datatalks-club)
- [Manmohan Gosada – DataTalks.Club](#manmohan-gosada-datatalks-club)
- [Marco De Sa – DataTalks.Club](#marco-de-sa-datatalks-club)
- [Marcello La Rocca – DataTalks.Club](#marcello-la-rocca-datatalks-club)
- [Maria Bruckert – DataTalks.Club](#maria-bruckert-datatalks-club)
- [Manoj Kukreja – DataTalks.Club](#manoj-kukreja-datatalks-club)
- [Marianna Diachuk – DataTalks.Club](#marianna-diachuk-datatalks-club)
- [Mariano Semelman – DataTalks.Club](#mariano-semelman-datatalks-club)
- [Maria Sukhareva – DataTalks.Club](#maria-sukhareva-datatalks-club)
- [Maria Vechtomova – DataTalks.Club](#maria-vechtomova-datatalks-club)
- [Marijn Markus – DataTalks.Club](#marijn-markus-datatalks-club)
- [Mario Lazo – DataTalks.Club](#mario-lazo-datatalks-club)
- [Mark Ryan – DataTalks.Club](#mark-ryan-datatalks-club)
- [Martin Kleppmann – DataTalks.Club](#martin-kleppmann-datatalks-club)
- [Martin Potančok – DataTalks.Club](#martin-potan-ok-datatalks-club)
- [Marysia Winkels – DataTalks.Club](#marysia-winkels-datatalks-club)
- [Mary Jane Dykeman – DataTalks.Club](#mary-jane-dykeman-datatalks-club)
- [Matt Harrison – DataTalks.Club](#matt-harrison-datatalks-club)
- [Matthew Housley – DataTalks.Club](#matthew-housley-datatalks-club)
- [Matt Palmer – DataTalks.Club](#matt-palmer-datatalks-club)
- [Maxime Labonne – DataTalks.Club](#maxime-labonne-datatalks-club)
- [Maxim Lukichev – DataTalks.Club](#maxim-lukichev-datatalks-club)
- [Max Schultze – DataTalks.Club](#max-schultze-datatalks-club)
- [Mehdi OUAZZA – DataTalks.Club](#mehdi-ouazza-datatalks-club)
- [Meor Amer – DataTalks.Club](#meor-amer-datatalks-club)
- [Merel Theisen – DataTalks.Club](#merel-theisen-datatalks-club)
- [Merve Noyan – DataTalks.Club](#merve-noyan-datatalks-club)
- [Meryem Arik – DataTalks.Club](#meryem-arik-datatalks-club)
- [Michael Munn – DataTalks.Club](#michael-munn-datatalks-club)
- [Michael Taylor – DataTalks.Club](#michael-taylor-datatalks-club)
- [Micheal Lanham – DataTalks.Club](#micheal-lanham-datatalks-club)
- [Meysam Asgari-Chenaghlu – DataTalks.Club](#meysam-asgari-chenaghlu-datatalks-club)
- [Miguel Morales – DataTalks.Club](#miguel-morales-datatalks-club)
- [Mihail Eric – DataTalks.Club](#mihail-eric-datatalks-club)
- [Mikhail Sveshnikov – DataTalks.Club](#mikhail-sveshnikov-datatalks-club)
- [Mikio Braun – DataTalks.Club](#mikio-braun-datatalks-club)
- [Mısra Turp – DataTalks.Club](#m-sra-turp-datatalks-club)
- [Moein Foroughi – DataTalks.Club](#moein-foroughi-datatalks-club)
- [Nadia Nahar – DataTalks.Club](#nadia-nahar-datatalks-club)
- [Nakul Bajaj – DataTalks.Club](#nakul-bajaj-datatalks-club)
- [Naomi Nguyen – DataTalks.Club](#naomi-nguyen-datatalks-club)
- [Natalie Kwong – DataTalks.Club](#natalie-kwong-datatalks-club)
- [Nataliya Portman – DataTalks.Club](#nataliya-portman-datatalks-club)
- [Nathan Wang – DataTalks.Club](#nathan-wang-datatalks-club)
- [Nastasia Saby – DataTalks.Club](#nastasia-saby-datatalks-club)
- [Neal Lathia – DataTalks.Club](#neal-lathia-datatalks-club)
- [Nemanja Radojkovic – DataTalks.Club](#nemanja-radojkovic-datatalks-club)
- [Niall Murphy – DataTalks.Club](#niall-murphy-datatalks-club)
- [Nick Bilozerov – DataTalks.Club](#nick-bilozerov-datatalks-club)
- [Nick Singh – DataTalks.Club](#nick-singh-datatalks-club)
- [Nicolas Rassam – DataTalks.Club](#nicolas-rassam-datatalks-club)
- [Nielsen Aileen – DataTalks.Club](#nielsen-aileen-datatalks-club)
- [Nikita Iserson – DataTalks.Club](#nikita-iserson-datatalks-club)
- [Nikita Kozodoi – DataTalks.Club](#nikita-kozodoi-datatalks-club)
- [Nikola Maksimovic – DataTalks.Club](#nikola-maksimovic-datatalks-club)
- [Nikolay Smorchkov – DataTalks.Club](#nikolay-smorchkov-datatalks-club)
- [Ning Wang – DataTalks.Club](#ning-wang-datatalks-club)
- [Nishant Mohan – DataTalks.Club](#nishant-mohan-datatalks-club)
- [Noah Gift – DataTalks.Club](#noah-gift-datatalks-club)
- [Noel Kwan – DataTalks.Club](#noel-kwan-datatalks-club)
- [Nour Karessli – DataTalks.Club](#nour-karessli-datatalks-club)
- [Oleg Novikov – DataTalks.Club](#oleg-novikov-datatalks-club)
- [Oleg Polivin – DataTalks.Club](#oleg-polivin-datatalks-club)
- [Olga Ivina – DataTalks.Club](#olga-ivina-datatalks-club)
- [Olga Petrova – DataTalks.Club](#olga-petrova-datatalks-club)
- [Ondřej Bothe – DataTalks.Club](#ond-ej-bothe-datatalks-club)
- [Ondřej Kubera – DataTalks.Club](#ond-ej-kubera-datatalks-club)
- [Padma Chitturi – DataTalks.Club](#padma-chitturi-datatalks-club)
- [Orlando Hohmeier – DataTalks.Club](#orlando-hohmeier-datatalks-club)
- [Orell Garten – DataTalks.Club](#orell-garten-datatalks-club)
- [Parul Pandey – DataTalks.Club](#parul-pandey-datatalks-club)
- [Parvathy Krishnan – DataTalks.Club](#parvathy-krishnan-datatalks-club)
- [Pastor Soto – DataTalks.Club](#pastor-soto-datatalks-club)
- [Patricio Cerda Mardini – DataTalks.Club](#patricio-cerda-mardini-datatalks-club)
- [Pauline Clavelloux – DataTalks.Club](#pauline-clavelloux-datatalks-club)
- [Paul Iusztin – DataTalks.Club](#paul-iusztin-datatalks-club)
- [Pavel Chernetsov – DataTalks.Club](#pavel-chernetsov-datatalks-club)
- [Paul Orland – DataTalks.Club](#paul-orland-datatalks-club)
- [Philippe Saadé – DataTalks.Club](#philippe-saad-datatalks-club)
- [Phil Winder – DataTalks.Club](#phil-winder-datatalks-club)
- [Prasoon Shukla – DataTalks.Club](#prasoon-shukla-datatalks-club)
- [Pier Paolo Ippolito – DataTalks.Club](#pier-paolo-ippolito-datatalks-club)
- [Polina Mosolova – DataTalks.Club](#polina-mosolova-datatalks-club)
- [Prateek Joshi – DataTalks.Club](#prateek-joshi-datatalks-club)
- [Rachael Tatman – DataTalks.Club](#rachael-tatman-datatalks-club)
- [Rachel Lim – DataTalks.Club](#rachel-lim-datatalks-club)
- [Raghav Bali – DataTalks.Club](#raghav-bali-datatalks-club)
- [Rahul Jain – DataTalks.Club](#rahul-jain-datatalks-club)
- [Ramiro Aznar – DataTalks.Club](#ramiro-aznar-datatalks-club)
- [Ranjitha Kulkarni – DataTalks.Club](#ranjitha-kulkarni-datatalks-club)
- [Raphaël Hoogvliets – DataTalks.Club](#rapha-l-hoogvliets-datatalks-club)
- [Reem Mahmoud – DataTalks.Club](#reem-mahmoud-datatalks-club)
- [Rileen Sinha – DataTalks.Club](#rileen-sinha-datatalks-club)
- [Rishabh Bhargava – DataTalks.Club](#rishabh-bhargava-datatalks-club)
- [Rob De Wit – DataTalks.Club](#rob-de-wit-datatalks-club)
- [Rob Zinkov – DataTalks.Club](#rob-zinkov-datatalks-club)
- [Roksolana Diachuk – DataTalks.Club](#roksolana-diachuk-datatalks-club)
- [Roman Grebennikov – DataTalks.Club](#roman-grebennikov-datatalks-club)
- [Rosona Eldred – DataTalks.Club](#rosona-eldred-datatalks-club)
- [Ross Brigoli – DataTalks.Club](#ross-brigoli-datatalks-club)
- [Roy Jafari – DataTalks.Club](#roy-jafari-datatalks-club)
- [Rui Machado – DataTalks.Club](#rui-machado-datatalks-club)
- [Ruslan Shchuchkin – DataTalks.Club](#ruslan-shchuchkin-datatalks-club)
- [Rustem Feyzkhanov – DataTalks.Club](#rustem-feyzkhanov-datatalks-club)
- [Sabina Firtala – DataTalks.Club](#sabina-firtala-datatalks-club)
- [Sadat Anwar – DataTalks.Club](#sadat-anwar-datatalks-club)
- [Sadik Bakiu – DataTalks.Club](#sadik-bakiu-datatalks-club)
- [Sally-Ann DeLucia – DataTalks.Club](#sally-ann-delucia-datatalks-club)
- [Sandra Kublik – DataTalks.Club](#sandra-kublik-datatalks-club)
- [Sage Elliott – DataTalks.Club](#sage-elliott-datatalks-club)
- [Santona Tuli – DataTalks.Club](#santona-tuli-datatalks-club)
- [Sara EL-ATEIF – DataTalks.Club](#sara-el-ateif-datatalks-club)
- [Sarah Mestiri – DataTalks.Club](#sarah-mestiri-datatalks-club)
- [Sara Menefee – DataTalks.Club](#sara-menefee-datatalks-club)
- [Sara Robinson – DataTalks.Club](#sara-robinson-datatalks-club)
- [Saurav Maheshkar – DataTalks.Club](#saurav-maheshkar-datatalks-club)
- [Savaş Yıldırım – DataTalks.Club](#sava-y-ld-r-m-datatalks-club)
- [Sean Sheng – DataTalks.Club](#sean-sheng-datatalks-club)
- [Sebastian Ayala Ruano – DataTalks.Club](#sebastian-ayala-ruano-datatalks-club)
- [Sebastian Raschka – DataTalks.Club](#sebastian-raschka-datatalks-club)
- [Sedat Kapanoglu – DataTalks.Club](#sedat-kapanoglu-datatalks-club)
- [Sejal Vaidya – DataTalks.Club](#sejal-vaidya-datatalks-club)
- [Sergei Boitsov – DataTalks.Club](#sergei-boitsov-datatalks-club)
- [Serena Haidar – DataTalks.Club](#serena-haidar-datatalks-club)
- [Sergei Shaikin – DataTalks.Club](#sergei-shaikin-datatalks-club)
- [Serg Masis – DataTalks.Club](#serg-masis-datatalks-club)
- [Shachar Meir – DataTalks.Club](#shachar-meir-datatalks-club)
- [Shir Meir Lador – DataTalks.Club](#shir-meir-lador-datatalks-club)
- [Shubham Saboo – DataTalks.Club](#shubham-saboo-datatalks-club)
- [Sidharth Ramachandran – DataTalks.Club](#sidharth-ramachandran-datatalks-club)
- [Simon Stiebellehner – DataTalks.Club](#simon-stiebellehner-datatalks-club)
- [Simon Thompson – DataTalks.Club](#simon-thompson-datatalks-club)
- [Sivan Biham – DataTalks.Club](#sivan-biham-datatalks-club)
- [Sofya Yulpatova – DataTalks.Club](#sofya-yulpatova-datatalks-club)
- [Sonal Goyal – DataTalks.Club](#sonal-goyal-datatalks-club)
- [Soumik Rakshit – DataTalks.Club](#soumik-rakshit-datatalks-club)
- [Soledad Galli – DataTalks.Club](#soledad-galli-datatalks-club)
- [Srivathsan Canchi – DataTalks.Club](#srivathsan-canchi-datatalks-club)
- [Stefan Gudmundsson – DataTalks.Club](#stefan-gudmundsson-datatalks-club)
- [Stefanie Molin – DataTalks.Club](#stefanie-molin-datatalks-club)
- [Stefan Jansen – DataTalks.Club](#stefan-jansen-datatalks-club)
- [Supreet Kaur – DataTalks.Club](#supreet-kaur-datatalks-club)
- [Susan Walsh – DataTalks.Club](#susan-walsh-datatalks-club)
- [Santiago Valdarrama – DataTalks.Club](#santiago-valdarrama-datatalks-club)
- [Shawn Swyx Wang – DataTalks.Club](#shawn-swyx-wang-datatalks-club)
- [Tammy Liang – DataTalks.Club](#tammy-liang-datatalks-club)
- [Tanya Berger-Wolf – DataTalks.Club](#tanya-berger-wolf-datatalks-club)
- [Tamara Atanasoska – DataTalks.Club](#tamara-atanasoska-datatalks-club)
- [Tatiana Gabruseva – DataTalks.Club](#tatiana-gabruseva-datatalks-club)
- [Tatyjana Ankudo – DataTalks.Club](#tatyjana-ankudo-datatalks-club)
- [Theofilos Papapanagiotou – DataTalks.Club](#theofilos-papapanagiotou-datatalks-club)
- [Tereza Iofciu – DataTalks.Club](#tereza-iofciu-datatalks-club)
- [Thomas Nield – DataTalks.Club](#thomas-nield-datatalks-club)
- [Thomas Wolf – DataTalks.Club](#thomas-wolf-datatalks-club)
- [Thom Ives – DataTalks.Club](#thom-ives-datatalks-club)
- [Timothy Davis – DataTalks.Club](#timothy-davis-datatalks-club)
- [Tobias Zwingmann – DataTalks.Club](#tobias-zwingmann-datatalks-club)
- [Todd Underwood – DataTalks.Club](#todd-underwood-datatalks-club)
- [Tomasz Hinc – DataTalks.Club](#tomasz-hinc-datatalks-club)
- [Tomasz Lelek – DataTalks.Club](#tomasz-lelek-datatalks-club)
- [Tomaz Bratanic – DataTalks.Club](#tomaz-bratanic-datatalks-club)
- [Tomek Jamiński – DataTalks.Club](#tomek-jami-ski-datatalks-club)
- [Tommy Dang – DataTalks.Club](#tommy-dang-datatalks-club)
- [Uri Gilad – DataTalks.Club](#uri-gilad-datatalks-club)
- [Vadim Smolyakov – DataTalks.Club](#vadim-smolyakov-datatalks-club)
- [Valeriia Kuka – DataTalks.Club](#valeriia-kuka-datatalks-club)
- [Valerii Babushkin – DataTalks.Club](#valerii-babushkin-datatalks-club)
- [Vanessa Aguilar – DataTalks.Club](#vanessa-aguilar-datatalks-club)
- [Verena Weber – DataTalks.Club](#verena-weber-datatalks-club)
- [Victoria Perez Mola – DataTalks.Club](#victoria-perez-mola-datatalks-club)
- [Vijay Kiran – DataTalks.Club](#vijay-kiran-datatalks-club)
- [Ville Tuulos – DataTalks.Club](#ville-tuulos-datatalks-club)
- [Vincent Tatan – DataTalks.Club](#vincent-tatan-datatalks-club)
- [Vincent Warmerdam – DataTalks.Club](#vincent-warmerdam-datatalks-club)
- [Vin Vashishta – DataTalks.Club](#vin-vashishta-datatalks-club)
- [Violetta Mishechkina – DataTalks.Club](#violetta-mishechkina-datatalks-club)
- [Vishwas BV – DataTalks.Club](#vishwas-bv-datatalks-club)
- [Vladimir Haltakov – DataTalks.Club](#vladimir-haltakov-datatalks-club)
- [Wendy Mak – DataTalks.Club](#wendy-mak-datatalks-club)
- [Willem Pienaar – DataTalks.Club](#willem-pienaar-datatalks-club)
- [Will McGugan – DataTalks.Club](#will-mcgugan-datatalks-club)
- [Will Russell – DataTalks.Club](#will-russell-datatalks-club)
- [Xia He-Bleinagel – DataTalks.Club](#xia-he-bleinagel-datatalks-club)
- [Yuan Tang – DataTalks.Club](#yuan-tang-datatalks-club)
- [Yulia Pavlova – DataTalks.Club](#yulia-pavlova-datatalks-club)
- [Yury Kashnitsky – DataTalks.Club](#yury-kashnitsky-datatalks-club)
- [Zhamak Dehghani – DataTalks.Club](#zhamak-dehghani-datatalks-club)
- [Machine Learning Bookcamp – DataTalks.Club](#machine-learning-bookcamp-datatalks-club)
- [Mastering Machine Learning Algorithms - Second Edition – DataTalks.Club](#mastering-machine-learning-algorithms-second-edition-datatalks-club)
- [Reinforcement Learning – DataTalks.Club](#reinforcement-learning-datatalks-club)
- [Deep Learning with Structured Data – DataTalks.Club](#deep-learning-with-structured-data-datatalks-club)
- [Data Teams – DataTalks.Club](#data-teams-datatalks-club)
- [Machine Learning Design Patterns – DataTalks.Club](#machine-learning-design-patterns-datatalks-club)
- [Math for Programmers – DataTalks.Club](#math-for-programmers-datatalks-club)
- [Machine Learning for Algorithmic Trading – DataTalks.Club](#machine-learning-for-algorithmic-trading-datatalks-club)
- [Machine Learning Engineering in Action – DataTalks.Club](#machine-learning-engineering-in-action-datatalks-club)
- [Designing Data-Intensive Applications – DataTalks.Club](#designing-data-intensive-applications-datatalks-club)
- [Database Internals – DataTalks.Club](#database-internals-datatalks-club)
- [Street Coder – DataTalks.Club](#street-coder-datatalks-club)
- [Learning Tensorflow.js – DataTalks.Club](#learning-tensorflow-js-datatalks-club)
- [AI and Machine Learning for Coders – DataTalks.Club](#ai-and-machine-learning-for-coders-datatalks-club)
- [The Practitioner's Guide to Graph Data – DataTalks.Club](#the-practitioner-s-guide-to-graph-data-datatalks-club)
- [Transformers for Natural Language Processing – DataTalks.Club](#transformers-for-natural-language-processing-datatalks-club)
- [Tiny Python Projects – DataTalks.Club](#tiny-python-projects-datatalks-club)
- [Machine Learning Using TensorFlow Cookbook – DataTalks.Club](#machine-learning-using-tensorflow-cookbook-datatalks-club)
- [The Coding Career Handbook – DataTalks.Club](#the-coding-career-handbook-datatalks-club)
- [Grokking Deep Reinforcement Learning – DataTalks.Club](#grokking-deep-reinforcement-learning-datatalks-club)
- [Data Governance: The Definitive Guide – DataTalks.Club](#data-governance-the-definitive-guide-datatalks-club)
- [Building Machine Learning Pipelines – DataTalks.Club](#building-machine-learning-pipelines-datatalks-club)
- [Advanced Algorithms and Data Structures – DataTalks.Club](#advanced-algorithms-and-data-structures-datatalks-club)
- [Graph Databases in Action – DataTalks.Club](#graph-databases-in-action-datatalks-club)
- [Cleaning Data for Effective Data Science – DataTalks.Club](#cleaning-data-for-effective-data-science-datatalks-club)
- [Data Science on AWS – DataTalks.Club](#data-science-on-aws-datatalks-club)
- [Engineering MLOps – DataTalks.Club](#engineering-mlops-datatalks-club)
- [Relevant Search – DataTalks.Club](#relevant-search-datatalks-club)
- [Interpretable Machine Learning with Python – DataTalks.Club](#interpretable-machine-learning-with-python-datatalks-club)
- [Applied Natural Language Processing in the Enterprise – DataTalks.Club](#applied-natural-language-processing-in-the-enterprise-datatalks-club)
- [Practical Recommender Systems – DataTalks.Club](#practical-recommender-systems-datatalks-club)
- [Tuning Up – DataTalks.Club](#tuning-up-datatalks-club)
- [Grokking Machine Learning – DataTalks.Club](#grokking-machine-learning-datatalks-club)
- [Practical MLOps – DataTalks.Club](#practical-mlops-datatalks-club)
- [Business Skills for Data Scientists – DataTalks.Club](#business-skills-for-data-scientists-datatalks-club)
- [Software Mistakes and Tradeoffs – DataTalks.Club](#software-mistakes-and-tradeoffs-datatalks-club)
- [DataOps for Dummies – DataTalks.Club](#dataops-for-dummies-datatalks-club)
- [Python Feature Engineering Cookbook – DataTalks.Club](#python-feature-engineering-cookbook-datatalks-club)
- [Effective Data Science Infrastructure – DataTalks.Club](#effective-data-science-infrastructure-datatalks-club)
- [Transfer Learning in Action – DataTalks.Club](#transfer-learning-in-action-datatalks-club)
- [Mastering Transformers – DataTalks.Club](#mastering-transformers-datatalks-club)
- [AI-Powered Search – DataTalks.Club](#ai-powered-search-datatalks-club)
- [Blueprints for Text Analytics Using Python – DataTalks.Club](#blueprints-for-text-analytics-using-python-datatalks-club)
- [Data Analysis with Python and PySpark – DataTalks.Club](#data-analysis-with-python-and-pyspark-datatalks-club)
- [Generative AI with Python and TensorFlow 2 – DataTalks.Club](#generative-ai-with-python-and-tensorflow-2-datatalks-club)
- [Ace The Data Science Interview – DataTalks.Club](#ace-the-data-science-interview-datatalks-club)
- [Building Machine Learning Powered Applications – DataTalks.Club](#building-machine-learning-powered-applications-datatalks-club)
- [Own Your Tech Career – DataTalks.Club](#own-your-tech-career-datatalks-club)
- [Deep Learning with fastai Cookbook – DataTalks.Club](#deep-learning-with-fastai-cookbook-datatalks-club)
- [Mastering spaCy – DataTalks.Club](#mastering-spacy-datatalks-club)
- [Machine Learning Engineering with Python – DataTalks.Club](#machine-learning-engineering-with-python-datatalks-club)
- [Effective Pandas – DataTalks.Club](#effective-pandas-datatalks-club)
- [A Visual Introduction to Deep Learning – DataTalks.Club](#a-visual-introduction-to-deep-learning-datatalks-club)
- [Hands-On Data Preprocessing in Python – DataTalks.Club](#hands-on-data-preprocessing-in-python-datatalks-club)
- [Data Engineering with Apache Spark, Delta Lake, and Lakehouse – DataTalks.Club](#data-engineering-with-apache-spark-delta-lake-and-lakehouse-datatalks-club)
- [Serverless Analytics with Amazon Athena – DataTalks.Club](#serverless-analytics-with-amazon-athena-datatalks-club)
- [Interpretable Machine Learning – DataTalks.Club](#interpretable-machine-learning-datatalks-club)
- [Artificial Intelligence with Python – DataTalks.Club](#artificial-intelligence-with-python-datatalks-club)
- [Natural Language Processing with Transformers – DataTalks.Club](#natural-language-processing-with-transformers-datatalks-club)
- [Everyday ML Questions – DataTalks.Club](#everyday-ml-questions-datatalks-club)
- [Practical Fairness – DataTalks.Club](#practical-fairness-datatalks-club)
- [AI-Powered Business Intelligence – DataTalks.Club](#ai-powered-business-intelligence-datatalks-club)
- [Python Machine Learning By Example – DataTalks.Club](#python-machine-learning-by-example-datatalks-club)
- [Designing Machine Learning Systems – DataTalks.Club](#designing-machine-learning-systems-datatalks-club)
- [Grokking Streaming Systems – DataTalks.Club](#grokking-streaming-systems-datatalks-club)
- [Hands-On Data Analysis with Pandas - Second Edition – DataTalks.Club](#hands-on-data-analysis-with-pandas-second-edition-datatalks-club)
- [Data Analytics Initiatives – DataTalks.Club](#data-analytics-initiatives-datatalks-club)
- [Essential Math for Data Science – DataTalks.Club](#essential-math-for-data-science-datatalks-club)
- [Fundamentals of Data Engineering – DataTalks.Club](#fundamentals-of-data-engineering-datatalks-club)
- [Skills of a Successful Software Engineer – DataTalks.Club](#skills-of-a-successful-software-engineer-datatalks-club)
---
# Welcome to DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
* * *
The place to talk about data
============================
Global online community of data science professionals, ML engineers, and AI practitioners
-----------------------------------------------------------------------------------------
Subscribe to our weekly newsletter and join our Slack.
We'll keep you informed about everything happening in the Club.
Email
Join
You'll get an invite within 3 minutes
#### Talk about data, machine
learning, and engineering
#### Attend weekly events
and learn from free courses
#### Ask career questions and
discuss career options
* * *
#### Upcoming events
* [Fact-Checking with Wikidata](https://luma.com/7fs5v7os)
on 20 Jan 2026 by [Philippe Saadé](https://datatalks.club/people/philippesaade.html)
* [AI Engineering: Skill Stack, Agents, LLMOps, and How to Ship AI Products](https://luma.com/fx3wplwn)
on 26 Jan 2026 by [Paul Iusztin](https://datatalks.club/people/pauliusztin.html)
* [Analytics Engineering with dbt Workshop](https://luma.com/c8c55og5)
on 27 Jan 2026 by [Juan Manuel Perafan](https://datatalks.club/people/juanmanuelperafan.html)
* [The Future of AI Agents](https://luma.com/0s0dcpdl)
on 10 Feb 2026 by [Aditya Gautam](https://datatalks.club/people/adityagautam.html)
* [From APIs to Warehouses: AI-Assisted Data Ingestion with dlt](https://luma.com/hzis1yzp)
on 17 Feb 2026 by [Aashish Nair](https://datatalks.club/people/aashishnair.html)
Check [events](https://datatalks.club/events.html)
for all past events. You can also subscribe to [our Google calendar](https://calendar.google.com/calendar/?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
to get notified about all our events.
#### Latest podcast episodes
* [Building Pet Health Tech: ML, Sensors, and Dog Behavior Data](https://datatalks.club/podcast/s22e08-building-pet-health-tech-ml-sensors-and-dog-behavior-data.html)
with [Sofya Yulpatova](https://datatalks.club/people/sofyayulpatova.html)
* [Reinventing a Career in Tech](https://datatalks.club/podcast/s22e07-reinventing-career-in-tech.html)
with [Xia He-Bleinagel](https://datatalks.club/people/xiahebleinagel.html)
* [From Black-Box Systems to Augmented Decision-Making](https://datatalks.club/podcast/s22e06-from-black-box-systems-to-augmented-decision-making.html)
with [Anusha Akkina](https://datatalks.club/people/anushaakkina.html)
* [Practical LLM Engineering and RAG: Prompting, Evaluation and Real-World Workflows](https://datatalks.club/podcast/practical-llm-engineering-and-rag.html)
with [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html)
* [Bioinformatics Workflows in Practice: Sequencing, Metagenomics, and Open-Source Tools](https://datatalks.club/podcast/bioinformatics-worflows-tools-and-data-science.html)
with [Sebastian Ayala Ruano](https://datatalks.club/people/sebastianayalaruano.html)
Check the [podcast](https://datatalks.club/podcast.html)
page for all past podcast episodes.
#### Our Sponsors
[](https://dlthub.com/)
[](https://www.astronomer.io/)
[](https://kestra.io/)
#### Book of the week
Check the [book of the week](https://datatalks.club/books.html)
page for more books!
#### Latest articles
* [Open Source and Free AI Agent Evaluation Tools](https://datatalks.club/blog/open-source-free-ai-agent-evaluation-tools.html)
by [Haziqa Sajid](https://datatalks.club/people/haziqasajid.html)
* [15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide](https://datatalks.club/blog/free-data-engineering-courses.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [LLM Zoomcamp: Free LLM Engineering Course and Certification](https://datatalks.club/blog/llm-zoomcamp.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# 15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide
15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide
=================================================================
### Find the best data engineering course for you: compare 15 free and 5 paid options with certificates, platforms, and tech stacks
10 Dec 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Finding the right data engineering course can be overwhelming with hundreds of options across different platforms, price points, and skill levels.
This blog post compares 15 free data engineering courses and 5 paid options, detailing platforms, certificates, tech stacks, prerequisites, and durations so you can choose the course that fits your goals and budget.
Data Engineering Courses Summary Table
--------------------------------------
### Free & Free-to-Audit Courses
| # | Course | Platform | Level | Format/Certificate | Duration |
| --- | --- | --- | --- | --- | --- |
| 1 | [Data Engineering Zoomcamp](https://datatalks.club/blog/free-data-engineering-courses.html#1-data-engineering-zoomcamp) | DataTalks.Club | Intermediate | Free + certificate | 9 weeks |
| 2 | [IBM Data Engineering Professional Certificate](https://datatalks.club/blog/free-data-engineering-courses.html#2-ibm-data-engineering-professional-certificate) | Coursera | Beginner | Free audit / paid cert | ~6 months |
| 3 | [DeepLearning.AI Data Engineering Professional Certificate](https://datatalks.club/blog/free-data-engineering-courses.html#3-deeplearningai-and-aws-data-engineering-professional-certificate) | Coursera | Intermediate | Free audit / paid cert | ~3 months |
| 4 | [Snowflake Data Engineering Professional Certificate](https://datatalks.club/blog/free-data-engineering-courses.html#4-snowflake-data-engineering-professional-certificate) | Coursera | Beginner | Free audit / paid cert | ~4 weeks |
| 5 | [Data Engineering Foundations Specialization](https://datatalks.club/blog/free-data-engineering-courses.html#5-ibm-data-engineering-foundations-specialization) | Coursera | Beginner | Free audit / paid cert | ~2 months |
| 6 | [Introduction to Data Engineering](https://datatalks.club/blog/free-data-engineering-courses.html#6-ibm-introduction-to-data-engineering) | Coursera | Beginner | Free audit / paid cert | 1 week |
| 7 | [Python, Bash and SQL Essentials for Data Engineering](https://datatalks.club/blog/free-data-engineering-courses.html#7-python-bash-and-sql-essentials-for-data-engineering-specialization-by-duke-university) | Coursera | Beginner | Free audit / paid cert | ~4 months |
| 8 | [Applied Python Data Engineering Specialization](https://datatalks.club/blog/free-data-engineering-courses.html#8-applied-python-data-engineering-specialization-by-duke-university) | Coursera | Intermediate | Free audit / paid cert | ~5 months |
| 9 | [Generative AI for Data Engineers Specialization](https://datatalks.club/blog/free-data-engineering-courses.html#9-generative-ai-for-data-engineers-specialization-by-ibm) | Coursera | Intermediate | Free audit / paid cert | 8 weeks |
| 10 | [Preparing for Google Cloud Certification: Cloud Data Engr Professional Certificate](https://datatalks.club/blog/free-data-engineering-courses.html#10-preparing-for-google-cloud-certification-cloud-data-engr-professional-certificate-by-google-cloud) | Coursera | Intermediate | Free audit / paid cert | ~4 weeks |
| 11 | [Meta Database Engineer Professional Certificate](https://datatalks.club/blog/free-data-engineering-courses.html#11-meta-database-engineer-professional-certificate) | Coursera | Beginner | Free audit / paid cert | ~6 months |
| 12 | [AI: Advanced Data Engineering](https://datatalks.club/blog/free-data-engineering-courses.html#12-ai-advanced-data-engineering-by-pragmatic-ai-labs) | edX | Advanced | Free audit / paid cert | 4 weeks |
| 13 | [DelftX: AI Skills for Engineers - Data Engineering and Data Pipelines](https://datatalks.club/blog/free-data-engineering-courses.html#13-delftx-ai-skills-for-engineers---data-engineering-and-data-pipelines-by-delft-university-of-technology-tu-delft) | edX | Introductory | Free audit / paid cert | 6 weeks |
| 14 | [AI: Spark, Hadoop, and Snowflake for Data Engineering](https://datatalks.club/blog/free-data-engineering-courses.html#14-ai-spark-hadoop-and-snowflake-for-data-engineering-by-pragmatic-ai-labs) | edX | Introductory | Free audit / paid cert | 4 weeks |
| 15 | [Understanding Data Engineering](https://datatalks.club/blog/free-data-engineering-courses.html#15-understanding-data-engineering-by-datacamp) | DataCamp | Beginner | Free start / paid subscription | 2 hours |
### Paid Courses & Certifications
| # | Course | Platform | Level | Format/Certificate | Duration |
| --- | --- | --- | --- | --- | --- |
| P1 | [Data Engineer in Python Career Track](https://datatalks.club/blog/free-data-engineering-courses.html#1-data-engineer-in-python-career-track-by-datacamp) | DataCamp | Intermediate | Paid subscription | ~40 hours |
| P2 | [Data Engineering with AWS Nanodegree](https://datatalks.club/blog/free-data-engineering-courses.html#2-data-engineering-with-aws-nanodegree) | Udacity | Intermediate | Paid nanodegree | 39 hours |
| P3 | [Data Engineering with Microsoft Azure Nanodegree](https://datatalks.club/blog/free-data-engineering-courses.html#3-data-engineering-with-microsoft-azure-nanodegree) | Udacity | Advanced | Paid nanodegree | 56 hours |
| P4 | [Data Engineer Career Path](https://datatalks.club/blog/free-data-engineering-courses.html#4-data-engineer-career-path-by-dataquest) | Dataquest | Beginner | Paid subscription | ~4 months |
| P5 | [AWS Certified Data Engineer - Associate (DEA-C01) Exam](https://datatalks.club/blog/free-data-engineering-courses.html#5-aws-certified-data-engineer---associate) | AWS Certification | Associate | Paid exam | 130 minutes (self-paced prep) |
How We Collected This List
--------------------------
Fully free data‑engineering courses that grant both complete access to materials **and** a free certificate are rare. Many courses below allow auditing of lecture videos and readings for free, but graded assignments, peer feedback and certificates usually require payment. We have therefore included both truly free courses and free‑to‑audit courses in this list so you can choose the option that best fits your budget and learning goals.
1\. Data Engineering Zoomcamp
-----------------------------
Data Engineering Zoomcamp: A community‑driven, hands‑on bootcamp for building production‑grade data pipelines
1. **Platform:** DataTalks.Club (GitHub/Slack)
2. **Provider:** DataTalks.Club community
3. **Level:** Intermediate (beginner‑friendly)
4. **Prerequisites:** Comfort with the command line and basic SQL; Python experience helpful but not mandatory.
5. **Key topics covered:** Infrastructure & prerequisites; workflow orchestration; data warehousing; analytics engineering; batch & stream processing; capstone project.
6. **Tools/tech stack:** Docker, PostgreSQL, GCP & Terraform, Mage.AI, Google Cloud Storage, BigQuery, dbt, BI tools, Apache Spark & Spark SQL, Kafka, KSQL, Faust.
7. **Format:** Free (open materials & certificate)
8. **Duration:** 9‑week structured program
9. **Certificate:** Free certificate after completion
[Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
is a free, community‑driven, hands‑on course that teaches how to build production‑grade data pipelines. Learners work through weekly modules, collaborate via Slack and complete a capstone project; the course emphasizes practical skills and portfolio building.
> The next cohort of the Data Engineering Zoomcamp begins January 12, 2026. Learn necessary skills to become a data engineer in 9 weeks and build production-grade data pipelines. [Register now](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
> to join the course and stay updated.
2\. IBM Data Engineering Professional Certificate
-------------------------------------------------
IBM Data Engineering Professional Certificate: A comprehensive program equipping beginners with job‑ready data‑engineering skills
1. **Platform:** Coursera
2. **Provider:** IBM
3. **Level:** Beginner
4. **Prerequisites:** Basic computer literacy; no programming experience required.
5. **Key topics covered:** Relational databases (MySQL, PostgreSQL, Db2); NoSQL databases; big‑data tools (MongoDB, Cassandra, Hadoop, Spark); ETL and data pipelines with Apache Airflow and Kafka; generative‑AI basics.
6. **Tools/tech stack:** SQL, MongoDB, Cassandra, Hadoop, Spark, Apache Airflow, Kafka, Python
7. **Format:** Free to audit; paid certificate
8. **Duration:** About 6 months at 10 hrs/week (16 courses)
9. **Certificate:** IBM Professional Certificate
[IBM Data Engineering Professional Certificate](https://www.coursera.org/professional-certificates/ibm-data-engineer)
is a comprehensive program equips beginners with job‑ready data‑engineering skills. Participants learn relational and NoSQL database management, big‑data processing and building ETL pipelines using Airflow and Kafka; the series culminates in applied projects.
3\. DeepLearning.AI and AWS Data Engineering Professional Certificate
---------------------------------------------------------------------
DeepLearning.AI and AWS Data Engineering Professional Certificate: A practitioner‑oriented program teaching data engineering mental models and practical pipeline techniques
1. **Platform:** Coursera
2. **Provider:** DeepLearning.AI
3. **Level:** Intermediate
4. **Prerequisites:** Intermediate Python programming and familiarity with data structures
5. **Key topics covered:** Data‑engineering lifecycle; designing data models; building scalable pipelines; using AWS, Hadoop, Spark and Kinesis
6. **Tools/tech stack:** AWS, Hadoop, Apache Spark, Kinesis, SQL, Python
7. **Format:** Free to audit; paid certificate
8. **Duration:** ~3 months, 3‑course series
9. **Certificate:** Professional certificate from DeepLearning.AI
[DeepLearning.AI Data Engineering Professional Certificate](https://www.coursera.org/professional-certificates/data-engineering)
is a practitioner‑oriented program that teaches the mental model of data engineering and practical techniques for building pipelines on modern big‑data platforms.
4\. Snowflake Data Engineering Professional Certificate
-------------------------------------------------------
Snowflake Data Engineering Professional Certificate: Learn Snowflake's architecture and build scalable pipelines with SQL and Python
1. **Platform:** Coursera
2. **Provider:** Snowflake
3. **Level:** Beginner
4. **Prerequisites:** Basic SQL and Python recommended
5. **Key topics covered:** Ingesting data at scale; performing transformations with SQL/Python; orchestration; DevOps and observability for data pipelines
6. **Tools/tech stack:** Snowflake platform, SQL, Python, DevOps tooling
7. **Format:** Free to audit; paid certificate
8. **Duration:** About 4 weeks at ~10 hrs/week
9. **Certificate:** Snowflake professional certificate
[Snowflake Data Engineering Professional Certificate](https://www.coursera.org/professional-certificates/snowflake-data-engineering)
is a series that introduces learners to Snowflake’s architecture and teaches how to build and monitor scalable pipelines using SQL and Python, plus best practices in DevOps and observability.
5\. IBM Data Engineering Foundations Specialization
---------------------------------------------------
IBM Data Engineering Foundations Specialization: A five‑course series introducing key concepts and tools for data engineers
1. **Platform:** Coursera
2. **Provider:** IBM
3. **Level:** Beginner
4. **Prerequisites:** Basic computer literacy; no prior data‑engineering experience
5. **Key topics covered:** Python fundamentals; relational databases & SQL; database design; data‑engineering lifecycle; data architecture, pipelines and ETL
6. **Tools/tech stack:** Python, SQL, IBM Db2, ETL tools
7. **Format:** Free to audit; paid certificate
8. **Duration:** 2 months (5 courses)
9. **Certificate:** IBM specialization certificate
[IBM Data Engineering Foundations Specialization](https://www.coursera.org/specializations/data-engineering-foundations)
is a five‑course series that introduces key concepts and tools used by data engineers, including programming in Python and designing databases and pipelines.
6\. IBM Introduction to Data Engineering
----------------------------------------
IBM Introduction to Data Engineering: A short course providing a high‑level overview of data engineering and technologies
1. **Platform:** Coursera
2. **Provider:** IBM
3. **Level:** Beginner
4. **Prerequisites:** None
5. **Key topics covered:** Data‑engineering roles; data‑platform architectures; relational vs NoSQL databases; big‑data engines (Hadoop, Spark); security & governance
6. **Tools/tech stack:** SQL, NoSQL, Hadoop, Spark
7. **Format:** Free to audit; paid certificate
8. **Duration:** 1‑week course
9. **Certificate:** Course certificate
[IBM Introduction to Data Engineering](https://www.coursera.org/learn/introduction-to-data-engineering)
is a short course that provides a high‑level overview of data engineering, the technologies involved and where data engineers fit within a data ecosystem.
7\. Python, Bash and SQL Essentials for Data Engineering Specialization by Duke University
------------------------------------------------------------------------------------------
Python, Bash and SQL Essentials for Data Engineering Specialization: Build foundational skills in scripting and database interaction
1. **Platform:** Coursera
2. **Provider:** Duke University
3. **Level:** Beginner
4. **Prerequisites:** Beginner‑level Linux skills; no Python experience required
5. **Key topics covered:** Python, Bash & Linux for data‑engineering tasks; connecting and querying SQL databases; web scraping; data manipulation and version control
6. **Tools/tech stack:** Python, Bash, Linux CLI, pandas, Jupyter, Git, MySQL, AWS SageMaker
7. **Format:** Free to audit; paid certificate
8. **Duration:** 4 months at 5 hrs/week (4 courses)
9. **Certificate:** Duke specialization certificate
[Python, Bash and SQL Essentials for Data Engineering Specialization](https://www.coursera.org/specializations/python-bash-sql-data-engineering-duke)
builds foundational skills in scripting and database interaction, using hands‑on labs in Jupyter notebooks to practice Python, Bash and SQL.
8\. Applied Python Data Engineering Specialization by Duke University
---------------------------------------------------------------------
Applied Python Data Engineering Specialization: Learn to leverage big‑data platforms and DevOps practices for robust pipelines and ML workflows
1. **Platform:** Coursera
2. **Provider:** Duke University
3. **Level:** Intermediate
4. **Prerequisites:** Experience with Python, Git, Docker and Kubernetes; background in linear algebra and statistics
5. **Key topics covered:** Building scalable big‑data pipelines (Hadoop, Spark, Snowflake, Databricks); ML workflows with PySpark & MLFlow; DataOps/DevOps; data visualization and storytelling
6. **Tools/tech stack:** Hadoop, Spark, Snowflake, Databricks, PySpark, MLFlow, Python visualization libraries, Kubernetes, Docker
7. **Format:** Free to audit; paid certificate
8. **Duration:** 5 months at 10 hrs/week (3 courses)
9. **Certificate:** Duke specialization certificate
[Applied Python Data Engineering Specialization](https://www.coursera.org/specializations/python-data-engineering)
is designed for data‑focused software engineers and researchers, this specialization teaches how to leverage big‑data platforms and DevOps practices to build robust pipelines and ML workflows.
9\. Generative AI for Data Engineers Specialization by IBM
----------------------------------------------------------
Generative AI for Data Engineers Specialization: Explore how generative AI enhances data‑engineering tasks and ETL pipelines
1. **Platform:** Coursera
2. **Provider:** IBM
3. **Level:** Intermediate
4. **Prerequisites:** No experience required, though prior data‑engineering knowledge is helpful
5. **Key topics covered:** Generative‑AI models and tools (text/code/image/audio/video); prompt‑engineering techniques; using generative AI for data warehouse schema design, data generation, augmentation and anonymization; case studies in ETL and data repositories
6. **Tools/tech stack:** Generative‑AI models, prompt‑engineering tools (ChatGPT, IBM Watsonx, Prompt Lab, Spellbook, Dust)
7. **Format:** Free to audit; paid certificate
8. **Duration:** 8 weeks at 2 hrs/week (3 courses)
9. **Certificate:** IBM specialization certificate
[Generative AI for Data Engineers Specialization](https://www.coursera.org/specializations/generative-ai-for-data-engineers)
explores how generative AI enhances data‑engineering tasks. Students learn prompt engineering and apply generative‑AI tools to ETL pipelines and data management.
10\. Preparing for Google Cloud Certification: Cloud Data Engr Professional Certificate by Google Cloud
-------------------------------------------------------------------------------------------------------
Preparing for Google Cloud Certification: Cloud Data Engr Professional Certificate: Prepare for Google Cloud's Professional Data Engineer certification
1. **Platform:** Coursera
2. **Provider:** Google Cloud
3. **Level:** Intermediate
4. **Prerequisites:** Proficiency with SQL and experience with programming languages such as Python
5. **Key topics covered:** Identifying and using Google Cloud Big Data and ML products; interactive analysis with BigQuery; migrating MySQL and Hadoop workloads using Cloud SQL and Dataproc; choosing appropriate data‑processing services
6. **Tools/tech stack:** BigQuery, Cloud SQL, Dataproc, Dataflow, Data Lakes, TensorFlow, Apache Hadoop & Spark
7. **Format:** Free to audit; paid certificate; includes Qwiklabs hands‑on labs
8. **Duration:** 4 weeks at about 10 hrs/week (5‑course series)
9. **Certificate:** Google Cloud professional certificate
[Preparing for Google Cloud Certification: Cloud Data Engr Professional Certificate](https://www.coursera.org/professional-certificates/gcp-data-engineering)
prepares learners for Google Cloud’s Professional Data Engineer certification by teaching how to leverage Google’s data‑processing services and includes hands‑on labs via Qwiklabs.
11\. Meta Database Engineer Professional Certificate
----------------------------------------------------
Meta Database Engineer Professional Certificate: A beginner‑friendly program teaching database engineering from the ground up
1. **Platform:** Coursera
2. **Provider:** Meta
3. **Level:** Beginner
4. **Prerequisites:** None
5. **Key topics covered:** SQL proficiency; database creation, management and optimization; building database‑driven Python applications; advanced data‑modeling concepts; interview preparation
6. **Tools/tech stack:** MySQL, SQL, Python, Django, Linux CLI, Git
7. **Format:** Free to audit; paid certificate
8. **Duration:** 6 months at 6 hrs/week (9 courses)
9. **Certificate:** Meta professional certificate
[Meta Database Engineer Professional Certificate](https://www.coursera.org/professional-certificates/meta-database-engineer)
is a beginner‑friendly program that teaches database engineering from the ground up, covering relational databases, Python applications and data modeling, with five applied projects to build real‑world skills.
12\. AI: Advanced Data Engineering by Pragmatic AI Labs
-------------------------------------------------------
AI: Advanced Data Engineering: Learn to design and optimize data pipelines at enterprise scale using modern technologies
1. **Platform:** edX
2. **Provider:** Pragmatic AI Labs
3. **Level:** Advanced
4. **Prerequisites:** Previous data‑engineering experience (self‑paced course)
5. **Key topics covered:** Scaling data systems; working with Celery and RabbitMQ for message queues; optimizing workflows with Apache Airflow; using vector and graph databases for scalable data management
6. **Tools/tech stack:** Celery, RabbitMQ, Apache Airflow, vector databases, graph databases
7. **Format:** Self‑paced; free to audit; paid certificate
8. **Duration:** 4 weeks (3-6 hrs/week)
9. **Certificate:** Verified certificate available
[AI: Advanced Data Engineering](https://www.edx.org/learn/computer-science/pragmatic-ai-labs-open-source-llmops-2)
by Pragmatic AI Labs is a advanced course that teaches how to design and optimize data pipelines at enterprise scale using modern message‑queuing and database technologies, with hands‑on labs for practicing each concept.
13\. DelftX: AI Skills for Engineers - Data Engineering and Data Pipelines by Delft University of Technology (TU Delft)
-----------------------------------------------------------------------------------------------------------------------
DelftX: AI Skills for Engineers - Data Engineering and Data Pipelines: A foundational course introducing data‑engineering principles for AI applications
1. **Platform:** edX
2. **Provider:** Delft University of Technology (TU Delft)
3. **Level:** Introductory
4. **Prerequisites:** None
5. **Key topics covered:** Importance of data management for AI; data requirements; obtaining data; extracting and querying data from databases with SQL; setting up Python notebooks; using pandas for tabular data; visualizing data with seaborn
6. **Tools/tech stack:** Python, pandas, Jupyter notebooks, seaborn, SQL
7. **Format:** Self‑paced; free to audit; paid certificate
8. **Duration:** 6 weeks, 5-7 hrs/week
9. **Certificate:** Verified certificate available
[DelftX: AI Skills for Engineers - Data Engineering and Data Pipelines](https://www.edx.org/learn/artificial-intelligence/delft-university-of-technology-ai-skills-for-engineers-data-engineering-and-data-pipelines)
by Delft University of Technology (TU Delft) is a foundational course that introduces engineers to data‑engineering principles for AI applications, including SQL data extraction and Python‑based data handling and visualization.
14\. AI: Spark, Hadoop, and Snowflake for Data Engineering by Pragmatic AI Labs
-------------------------------------------------------------------------------
AI: Spark, Hadoop, and Snowflake for Data Engineering: Explore essential data‑engineering platforms and optimize them using PySpark and Python
1. **Platform:** edX
2. **Provider:** Pragmatic AI Labs
3. **Level:** Introductory
4. **Prerequisites:** No prior experience required
5. **Key topics covered:** Managing & optimizing Hadoop, Spark and Snowflake; performing analytics and ML tasks in Databricks; using PySpark for Python data science; managing ML lifecycle with MLflow; applying Kaizen, DevOps & DataOps methodologies
6. **Tools/tech stack:** Hadoop, Spark, Snowflake, Databricks, PySpark, MLflow
7. **Format:** Self‑paced; free to audit; paid certificate
8. **Duration:** 4 weeks (3-6 hrs/week)
9. **Certificate:** Verified certificate available
[AI: Spark, Hadoop, and Snowflake for Data Engineering](https://www.edx.org/learn/computer-science/pragmatic-ai-labs-spark-hadoop-and-snowflake-for-data-engineering)
by Pragmatic AI Labs is an introductory course that explores essential data‑engineering platforms and teaches how to optimize them using PySpark and Python while integrating DevOps/DataOps practices.
15\. Understanding Data Engineering by DataCamp
-----------------------------------------------
Understanding Data Engineering: A short, no‑code course explaining data engineering concepts and workflows
1. **Platform:** DataCamp
2. **Provider:** DataCamp
3. **Level:** Beginner
4. **Prerequisites:** None
5. **Key topics covered:** Definition of data engineering; data workflows (ingestion, storage, transformation, serving); differences between batch and streaming processing; responsibilities of data engineers versus other roles
6. **Tools/tech stack:** Conceptual; introduces SQL and cloud concepts
7. **Format:** Free to start (audit) but part of a paid subscription
8. **Duration:** 2‑hour course with 11 videos & 32 exercises
9. **Certificate:** DataCamp certificate
[Understanding Data Engineering](https://www.datacamp.com/courses/understanding-data-engineering)
by DataCamp is a short, no‑code course that explains what data engineering is, outlines typical workflows and clarifies how the role differs from data science and analytics.
Paid Courses
------------
### 1\. Data Engineer in Python Career Track by DataCamp
1. **Platform:** DataCamp
2. **Provider:** DataCamp
3. **Level:** Intermediate
4. **Prerequisites:** Prior Python knowledge and familiarity with cloud concepts
5. **Key topics covered:** Data manipulation and automation; building ETL/ELT pipelines; using Apache Airflow for workflow orchestration; efficient coding practices; version control with Git
6. **Tools/tech stack:** Python, pandas, SQL, Apache Airflow, dbt, Git, command line, cloud services
7. **Format:** Subscription‑based (paid)
8. **Duration:** ~40 hours across multiple courses and projects
9. **Certificate:** DataCamp track certificate
[Data Engineer in Python Career Track](https://www.datacamp.com/tracks/data-engineer-in-python)
by DataCamp is a professional track that deepens Python‑centric data‑engineering skills, teaching learners to design and automate data pipelines and orchestrate workflows using Airflow and industry best practices.
### 2\. Data Engineering with AWS Nanodegree
1. **Platform:** Udacity
2. **Provider:** Udacity
3. **Level:** Intermediate
4. **Prerequisites:** Knowledge of relational data models, command‑line basics, intermediate Python and basic Git
5. **Key topics covered:** Data modeling (relational & NoSQL); building ETL pipelines with PostgreSQL and Apache Cassandra; designing and implementing cloud data warehouses on AWS; working with Spark and data lakes; automating pipelines with Airflow
6. **Tools/tech stack:** PostgreSQL, Apache Cassandra, Amazon S3, Redshift, IAM, VPC, EC2, RDS, Apache Spark, Airflow
7. **Format:** Paid nanodegree program
8. **Duration:** 39 hours; 4 courses & 4 projects
9. **Certificate:** Udacity Nanodegree
[Data Engineering with AWS Nanodegree](https://www.udacity.com/course/data-engineer-nanodegree--nd027)
by Udacity is a project‑based program that trains learners to design data models, build data warehouses and lakes and automate pipelines on AWS using Airflow and Spark.
### 3\. Data Engineering with Microsoft Azure Nanodegree
1. **Platform:** Udacity
2. **Provider:** Udacity
3. **Level:** Advanced
4. **Prerequisites:** Command‑line basics, relational data models, intermediate Python and basic Git
5. **Key topics covered:** Creating relational & NoSQL data models; designing and implementing cloud data warehouses with Azure Synapse; building data lakes and lakehouse architectures using Spark & Azure Databricks; orchestrating pipelines with Azure Data Factory & Synapse Analytics
6. **Tools/tech stack:** PostgreSQL, Apache Cassandra, Azure Synapse Analytics, Azure Databricks, Azure Data Factory, Apache Spark
7. **Format:** Paid nanodegree program
8. **Duration:** 56 hours; 6 courses & 4 projects
9. **Certificate:** Udacity Nanodegree
[Data Engineering with Microsoft Azure Nanodegree](https://www.udacity.com/course/data-engineering-with-microsoft-azure-nanodegree--nd0277)
by Udacity is a program that prepares learners to become cloud data engineers on Azure by teaching scalable data architectures, data warehouses/lakes and orchestrated pipelines.
### 4\. Data Engineer Career Path by Dataquest
1. **Platform:** Dataquest
2. **Provider:** Dataquest
3. **Level:** Beginner‑friendly
4. **Prerequisites:** None (learning by doing)
5. **Key topics covered:** Python programming; data architecture & pipelines; data wrangling & cleaning; SQL & multi‑table databases; algorithms; command‑line & Git; workflow automation
6. **Tools/tech stack:** Python, pandas, NumPy, SQLite, PostgreSQL, MapReduce, Jupyter notebooks, command line, Git
7. **Format:** Subscription‑based (paid); self‑paced
8. **Duration:** 4 months at 5 hrs/week (22 courses & 12 projects)
9. **Certificate:** Dataquest career‑path certificate
[Data Engineer Career Path](https://www.dataquest.io/path/data-engineering/)
by Dataquest is a structured, project‑based path that takes learners from Python basics through building data pipelines and processing large datasets, with guided projects to build a portfolio.
### 5\. AWS Certified Data Engineer - Associate
1. **Platform:** AWS Certification
2. **Provider:** Amazon Web Services
3. **Level:** Associate-level certification
4. **Prerequisites:** Recommended 2-3 years of data‑engineering experience and 1-2 years of hands‑on AWS experience
5. **Key topics covered:** Implementing data pipelines; ingesting and transforming data; selecting optimal data stores; designing data models and managing data lifecycles; operationalizing and monitoring pipelines; ensuring authentication, encryption & governance
6. **Tools/tech stack:** AWS services such as Amazon Kinesis, Amazon MSK, DynamoDB Streams, AWS DMS, AWS Glue, Amazon Redshift and other AWS data services
7. **Format:** Certification exam; 65 multiple-choice/multiple‑response questions; 130‑minute duration
8. **Duration:** Self‑study; exam is 130 minutes
9. **Certificate:** AWS Certified Data Engineer - Associate credential
[AWS Certified Data Engineer - Associate](https://aws.amazon.com/certification/certified-data-engineer-associate/)
by Amazon Web Services is a certification that validates competence in designing, building and maintaining data pipelines on AWS, including data ingestion, storage, transformation and governance. Candidates should have several years of data‑engineering experience.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Open Source and Free AI Agent Evaluation Tools – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Open Source and Free AI Agent Evaluation Tools
Open Source and Free AI Agent Evaluation Tools
==============================================
### 7 free and open-source AI agent evaluation tools compared: Arize Phoenix, LangSmith, DeepEval, Ragas, Promptfoo, OpenAI Evals, and Comet Opik for LLM agent testing
25 Dec 2025 by [Haziqa Sajid](https://datatalks.club/people/haziqasajid.html)
Evaluating LLM-based agents is essential for building reliable AI applications. Many assume model quality drives agent quality. In reality, it’s the guidance and evaluation behind it.
For example, OpenAI’s internal [frontier evals](https://openai.com/index/evals-drive-next-chapter-of-ai)
combine automated scoring and human grading to benchmark models across safety, factuality, and alignment. Their [Safety Evaluations Hub](https://openai.com/safety/evaluations-hub)
also shares public insights into model reliability and alignment. But that’s one part of the story.
Even strong models can face major risks in agentic setups, where errors can spread across multiple tools and reasoning can drift. Multi-step interactions often reveal unexpected behaviors that single-model tests miss, making evaluation tricky.
Open-source and free AI agent evaluation tools become invaluable here. They offer trace-level observability, flexible metrics, and experiment logging frameworks to capture outputs, tool usage, and decision paths. This article examines free and open-source tools for evaluating AI agents and demonstrates how they can be used to evaluate safety, dependability, and performance.
What is AI Agent Evaluation
---------------------------
AI agent evaluation is the process of assessing how effective and reliable LLM-powered agents are at tasks such as question answering, reasoning, and tool use. The main challenge is evaluating the decision-making LLM, which usually requires a mix of qualitative and quantitative metrics.
Qualitative metrics judge the quality of the content. These can use automated checks against ground truth or an [LLM-as-a-judge](https://arize.com/llm-as-a-judge/)
that rates responses using a defined rubric.
Quantitative metrics provide numerical, consistent measurements. Common examples include perplexity, accuracy, precision, recall, and F1.
Agents also rely on function-calling models or sub-agents, so evaluation must check whether the system chooses the correct tool or delegates to the right component. As more parts are added, a structured evaluation framework becomes essential.
Why AI Agent Evaluation Matters
-------------------------------
AI agents are nondeterministic, so the same prompt can produce different outputs each time. This makes simple test-case checking unreliable. Proper evaluation helps developers measure how well agents perform across key areas, from tool selection to response quality.
For instance, specialized metrics like [Tool Calling Accuracy](https://arize.com/ai-agents/agent-evaluation/)
check whether an agent selects and uses the correct tool with the appropriate arguments. Other metrics might track an agent’s planning ability, path convergence (efficiency of reaching a goal), or parameter extraction correctness.
Many such metrics exist, and the need for them becomes clear when you look at the architecture below:
Simple Architecture of an Agentic Application
The LLM hands off tasks based on what it can and cannot do. Generative tasks are handled directly by the model, while non-generative tasks depend on external tools, which may trigger a sequence of tool calls. Some tasks are delegated to specialized agents, and that chain of delegation can continue further.
Even though newer models show lower hallucination rates, an agentic system introduces many interconnected LLM calls. A small chance of failure can compound and increase overall risk. This makes it essential to evaluate agents before deployment.
[OpenAI evaluations](https://openai.com/safety/evaluations-hub)
against hallucinations on the PersonQA dataset
Moreover, deployed [AI agent frameworks](https://arize.com/ai-agents/agent-frameworks/)
need continuous monitoring for failures. In production, new inputs or multi-agent workflows can trigger unexpected behavior such as cascading tool errors, rising token usage, or context overflow. Ongoing evaluation, including automatic scoring of a sample of live prompts, helps detect spikes in errors or hallucinations before they affect users.
Open-source evaluation tools provide transparency and control because teams can inspect and adjust evaluators, avoid lock-in, and integrate into agent workflows. Strong evaluation practices connect development and deployment and help ensure agents remain reliable and safe.
How to Choose an Evaluation Tool
--------------------------------
When selecting from AI agent evaluation tools, focus on the following:
* **Capabilities and Scope:** Check if the tool handles single-call tasks or full agent trajectories, and whether it supports built-in or custom evaluation metrics. It largely depends on the project use case and expected outcomes from the AI agent.
* **Integration and Compatibility:** Verify available SDKs/APIs, framework support (e.g., LangChain, LlamaIndex, OpenAI Python libraries), and preferred programming language.
* **Accessibility and Support:** Decide between open-source libraries or hosted platforms. Since AI evolves at lightning speed, your evaluation tooling must be able to keep pace.
The selection criteria above provide teams with a solid starting point. When requirements change, adapt and pivot accordingly.
Comparison
----------
Below, we compare the top open-source and free evaluation tools.
#### Arize Phoenix
Arize Phoenix is an open-source observability and evaluation SDK for LLM applications. It supports automatic [OpenTelemetry tracing](https://arize.com/docs/phoenix/tracing/llm-traces)
through the OpenInference framework.
With a single call, instrumentation can be enabled for LLM calls, tool executions, retriever outputs, and more. The library includes auto-instrumentors for frameworks such as LangChain, LlamaIndex, DSPy, and OpenAI Agents, so function calls and tool spans are captured without manual setup.
For example, you can install the Python SDK and start tracing like this:
from arize.otel import register
tracer_provider = register(
space_key="YOUR_SPACE_KEY",
api_key="YOUR_API_KEY",
project_name="my_agent_project"
)
# Auto-instrument OpenAI Agents (LangChain, etc.)
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor
OpenAIAgentsInstrumentor().instrument(tracer_provider=tracer_provider)
This code will capture every LLM call, tool invocation, and retriever call made by your agent. Arize Phoenix then logs these traces and lets you run evaluations against them.
Arize Phoenix records detailed agent traces (spans), capturing every AI component
Phoenix includes [pre-built evaluation templates](https://arize.com/docs/phoenix/evaluation/running-pre-tested-evals)
for common tasks. Examples include hallucination detection for Q&A or RAG, summarization quality scoring, toxicity checks, and agent tool selection by using the [agent function calling eval](https://arize.com/docs/phoenix/evaluation/running-pre-tested-evals/tool-calling-eval)
. Custom LLM-based evaluators can also be created by defining a rubric and using any model from providers such as OpenAI, Anthropic, Azure, AWS Bedrock, or Google Gemini.
Phoenix organizes testing into [datasets and experiments](https://arize.com/docs/phoenix/datasets-and-experiments/overview-datasets)
. Inputs, with or without expected outputs, are uploaded as datasets, and experiments compare different agent versions. Results are stored to enable tracking changes from updated prompts, model swaps, or tooling adjustments. Phoenix also uses semantic clustering on inputs and outputs, grouping similar failures with embeddings to help surface root causes.
Phoenix is [open-source](https://github.com/Arize-ai/phoenix/)
and can run in local, cloud, or on-prem environments. It supports Python and TypeScript via the OpenInference instrumentation libraries.
#### Arize AX Free (Hosted)
Arize AX is a hosted AI observability and evaluation platform that builds on Phoenix’s tracing and evaluation capabilities. It provides a managed interface for monitoring and analysis that is accessible both to developers, product managers, and SMEs. Arize AX provides an evaluation from simple LLM calls to complex agent behaviour.
Arize AX Free (Hosted) provides LLM Calls Traces along with Latency, Tokens, and Status
AX tracks agent behavior automatically through its one of many [tracing integrations](https://arize.com/docs/ax/observe/tracing/setup/integrations)
or through [manual instrumentation](https://arize.com/docs/ax/observe/tracing/setup/manual-instrumentation)
when deeper control is needed.. Each run is recorded with all LLM calls, tool executions, and retrieval steps, giving you a complete picture of how the agent operated.
An [example of agent tracking behavior](https://arize.com/docs/ax/cookbooks/agents/tracing-and-evaluating-agents)
with LLM call and tool execution
AX includes [online evaluations for production monitoring](https://arize.com/resource/online-llm-evaluations/)
. Evaluators can be attached to incoming traces so each live agent execution is scored as it runs. This produces real-time dashboards for metrics such as accuracy, cost, and latency, and makes it possible to trigger alerts when patterns change.
AX also supports [offline evals](https://arize.com/docs/ax/evaluate/offline-evals)
for pre-deployment testing. You upload a CSV/JSONL of test prompts, run your agent against them, and score the outputs with built-in or custom evaluators. Experiments let you compare runs side-by-side. The entire process is no-code. You configure evaluators (LLM-judge or code-based) and see results in AX’s UI. AX also offers ready-made evaluators for hallucinations, bias, and other common issues.
from phoenix.evals.llm import LLM
from phoenix.evals.metrics import HallucinationEvaluator
llm = LLM(model="gpt-4o", provider="openai")
hallucination = HallucinationEvaluator(llm=llm)
hallucination.bind({"input": "query", "output": "response", "context": "reference"})
scores = hallucination.evaluate(df.to_dict())
[Annotations and justifications](https://arize.com/docs/ax/evaluate/evaluators/llm-as-a-judge/arize-evaluators-llm-as-a-judge)
for the AI model's responses
AX includes a [Prompt Playground](https://arize.com/docs/ax/prompts/prompt-playground)
, a web interface for A/B-testing prompts. It supports tool calls, so tool usage can be viewed directly. Prompt versions can be saved as experiments for later comparison. An in-app assistant named [Alyx](https://arize.com/docs/ax/alyx/arize-copilot/ai-powered-eval-analysis)
can generate evaluation templates and explain results.
For code-oriented workflows, AX allows [code-based evaluators](https://arize.com/docs/ax/evaluate/evaluators/code-evaluations)
. Python functions can be written for deterministic checks such as regex matching, JSON schema validation, keyword checks, or other logic, and can be combined with LLM-based scoring. AX can evaluate both individual traces and full sessions, enabling scoring of multi-turn interactions, such as coherence or goal completion, in addition to single responses.
#### LangSmith (Community)
LangSmith is LangChain’s official observability and evaluation suite. It offers a free [Developer plan](https://www.langchain.com/pricing)
(up to ~5k traces/month) and integrates tightly with LangChain agents. Like the other tools, LangSmith logs agent traces (chat histories, tool calls, function outputs) and provides dashboards for metrics like router accuracy, token cost, and latency. Here’s the simplest example that logs a Langsmith trace:
import openai
from langsmith import traceable
client = openai.Client()
@traceable(run_type="llm", name="Minimal Trace", project_name="My Project")
def call_openai(msgs):
return client.chat.completions.create(
model="gpt-4o-mini",
messages=msgs,
).choices[0].message.content
messages = [\
{"role": "system", "content": "You are a helpful assistant."},\
{"role": "user", "content": "Hello!"}\
]
print(call_openai(messages))
LangSmith supports [online](https://docs.langchain.com/langsmith/evaluation-concepts#online-evaluation)
and [offline](https://docs.langchain.com/langsmith/evaluation-concepts#offline-evaluation)
evaluation. Logged traces can be loaded for pre-deployment testing, or evaluators can be attached to new traces for live monitoring. The platform includes a [prompt hub](https://smith.langchain.com/hub)
and playground, a managed repository of prompt templates. Prompt variants can be run on sample inputs to observe differences in performance.
Built-in evaluation methods in LangSmith include [scoring](https://docs.langchain.com/langsmith/llm-as-judge)
and “gold standard” tests. You can write your own checks (regex asserts, value comparisons) or use a queued annotation workflow to collect human labels. There’s even a CI/CD integration, which allows evaluations to run automatically on each code commit to track changes before merging.
**Note:** LangSmith is a hosted service (free tier) rather than an open-source tool, but it is included here because the free plan provides substantial functionality. It can also be relevant for teams already working with LangChain due to its integration with that ecosystem.
#### OpenAI Evals
[OpenAI’s Evals](https://github.com/openai/evals#:~:text=,Get%20started%20%E2%86%92)
is an open-source framework for LLM evaluation and a registry of common benchmark tasks. It provides a large library of pre-built evals (defined in YAML) covering QA, reasoning puzzles, code generation, content filtering, and more.
For example, you can find YAML evals for multiple-choice QA, fact checking, reading comprehension, etc. Each eval is defined by a YAML spec pointing to a dataset and a scoring method (accuracy, multiple choice scoring, pass/fail, etc.). You run evals via the `oaieval` CLI or API, which submits each prompt to the chosen model and automatically computes scores.
Scoring supports both ground-truth and LLM-graded approaches. Tasks with known answers allow exact output comparison, while others use an LLM judge guided by a YAML rubric, with models like GPT-4 or Claude evaluating responses. Users can also create [custom evals](https://platform.openai.com/docs/guides/evals?api-mode=responses#create-an-eval-for-a-task)
without coding by preparing a JSONL of inputs and optional expected outputs and editing a YAML template to specify the model and scoring scheme.
#### DeepEval
[DeepEval](https://deepeval.com/docs/getting-started)
(by Confident AI) is an open-source Python framework for LLM testing, inspired by unit-test frameworks like pytest. It includes 30+ built-in metrics for generation quality, many based on recent research.
For example, [DeepEval](https://deepeval.com/docs/getting-started)
implements LLM-judge metrics like [G-Eval](https://deepeval.com/docs/metrics-llm-evals)
and [DAG](https://deepeval.com/docs/metrics-dag)
, as well as classic metrics (ROUGE, BLEU) and specialized scores for RAG and agent tasks (relevance, faithfulness, factuality, tool correctness).
Below is a dashboard of an RAG application tested on hallucinations using DeepEval.
[Testing RAG hallucinations](https://developers.llamaindex.ai/python/framework/community/integrations/deepeval/)
using DeepEval
You write test cases in code. A DeepEval test case includes inputs and expected outputs (if available). You then attach one or more metrics to the test case. DeepEval’s `assert_test()` API will run the LLM on the input, compute the metric(s), and throw an assertion error if thresholds aren’t met. This makes it feel like writing a pytest unit test.
For example:
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval
def test_medical_advice():
correctness = GEval(
name="Correctness",
criteria="Is the actual output medically accurate given the expected?",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=0.5
)
case = LLMTestCase(
input="I have a persistent cough and fever. Should I be worried?",
actual_output="A persistent cough and fever could be a viral infection or something more serious...",
expected_output="A persistent cough and fever could indicate a range of illnesses from mild to serious.",
test_metadatas={}
)
assert_test(case, correctness)
This example asserts that the model’s response meets a 0.5-threshold on “Correctness” as defined by an LLM judge. If it fails, the test is flagged. DeepEval also supports **multi-turn** tests (full conversations) and can auto-generate synthetic test data by mutating contexts or deriving Q&A from them.
For security, it includes optional [red-teaming tools](https://deepeval.com/docs/red-teaming-introduction)
(e.g., toxicity or bias detectors) to proactively find adversarial failures.
#### Ragas
Ragas is an [open-source evaluation](https://docs.ragas.io/en/stable/)
toolkit that emphasizes an **experiments-first** workflow. You create [_datasets_](https://docs.ragas.io/en/stable/concepts/datasets/)
(collections of inputs with expected outputs) and _experiments_ that run your agent on a dataset. Each experiment applies one or more metrics to the results. Ragas provides both traditional scores (BLEU, ROUGE, etc.) and LLM-driven metrics, and you can easily define custom metrics with Python decorators.
For agent use cases, Ragas includes specialized metrics. Ragas offers _Tool Call Accuracy_ (did the agent choose and use the correct tool?), _Agent Goal Accuracy_ (did the agent reach the intended goal?), among others. Here’s a [list](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics)
of other metrics other than that you can create your own [custom metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_write_your_own_metric/)
.
The library handles dataset versioning and result tracking internally, and it integrates smoothly with frameworks like LangChain and LlamaIndex for loading data or logs. Ragas even has test-case generation utilities. You can automatically create new test examples by prompting an LLM to mutate existing prompts or contexts.
from ragas import Experiment, Dataset
# Load a JSONL dataset where each item has 'input' and 'expected' fields
dataset = Dataset("test_data.jsonl", input_key="input", expected_key="expected")
# Define an experiment with a specific agent model and metrics
exp = Experiment(
name="agent_eval",
model_name="gpt-4",
metrics=["tool_call_accuracy", "agent_goal_accuracy"]
)
exp.run(dataset)
exp.show_results()
This snippet (schematic) shows how to define a Ragas `Experiment` on a test set. It will compute the listed metrics for each example and summarize them.
#### Promptfoo
Promptfoo is an open-source CLI (and library) designed for testing and red-teaming LLM applications in a developer-friendly way. You write declarative test specifications (YAML or JSON) that define prompts, variables, and expected behaviors, then run them via the `promptfoo` tool.
Promptfoo runs [entirely locally by default](https://github.com/promptfoo/promptfoo)
, so data is not sent to external servers. It is cross-platform and includes live-reload and caching features for faster iteration. Multiple model providers can be tested in the same suite, including OpenAI GPT, Claude, Google Gemini, and HuggingFace transformers.
For example, you might specify a list of prompts and input variables in a YAML file and then run:
# Example YAML config (pseudo)
prompts:
- prompt1.txt
providers:
- openai:gpt-4
tests:
- vars: {question: "What is 2+2?"}
assert:
- type: contains
value: "4"
Example YAML Config for Promptfoo
$ promptfoo eval config.yaml
Running `promptfoo eval config.yaml` would execute the prompt on GPT-4 and check that the answer contains “4”.
[Promptfoo's CLI-driven evaluation](https://www.promptfoo.dev/docs/intro)
showing a matrix of model outputs and PASS/FAIL checks (left), and a security report of adversarial tests (right)
Promptfoo will execute each prompt (on specified model providers, e.g., OpenAI, Anthropic, Google, HuggingFace, or custom endpoints) and produce a side-by-side matrix of outputs. You can [attach assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/)
(pass/fail conditions) to each test. These assertions can be code snippets (JavaScript or Python) or semantic checks (like “output should contain JSON” or “should be similar to X”). Promptfoo will mark a test PASS only if all assertions hold.
Promptfoo includes built-in [red-teaming and security tests](https://www.promptfoo.dev/docs/red-team/)
. It provides adversarial test suites that cover common jailbreak or bias scenarios and can be run on prompts and models. The tool generates a high-level risk report that summarizes any failures or leaks.
#### Comet Opik
[Comet’s](https://www.comet.com/site/products/opik/)
Opik is an open-source platform for LLM evaluation and observability, similar in spirit to Arize. It lets you instrument and trace your agent just like the others (each LLM call, tool use, and RAG retrieval can be logged). You write to Opik’s Python SDK or CLI to record traces and then apply evaluation metrics.
Opik comes with a suite of [automated scorers](https://www.comet.com/docs/opik/evaluation/metrics/overview)
(both LLM-based and rule-based) for common quality dimensions. For instance, it has built-in evaluators for factual accuracy, hallucination, relevance, and even bias or toxicity. Comet has built-in _LLM judges_ for complex issues like hallucination and factuality. You can also define your own metrics via an SDK. Here’s an example:
from opik import Opik, track
from opik.evaluation import evaluate
from opik.evaluation.metrics import Equals, Hallucination
from opik.integrations.openai import track_openai
import openai
# Define the task to evaluate
openai_client = track_openai(openai.OpenAI())
MODEL = "gpt-3.5-turbo"
@track
def your_llm_application(input: str) -> str:
response = openai_client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": input}],
)
return response.choices[0].message.content
# Define the evaluation task
def evaluation_task(x):
return {
"output": your_llm_application(x['input'])
}
# Create a simple dataset
client = Opik()
dataset = client.get_or_create_dataset(name="Example dataset")
dataset.insert([\
{"input": "What is the capital of France?"},\
{"input": "What is the capital of Germany?"},\
])
# Define the metrics
hallucination_metric = Hallucination()
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
experiment_config={
"model": MODEL
}
)
Moreover, Opik supports LLM [unit testing with pytest](https://www.comet.com/docs/opik/testing/pytest_integration)
. Its CI integration is built around pytest. You can write tests that compare an agent’s output to criteria, and Opik will mark builds as failed if any test fails. This lets you include model evaluation as part of your normal software tests.
Opik also provides [guardrails](https://www.comet.com/docs/opik/production/guardrails)
and [optimization tools](https://www.comet.com/docs/opik/agent_optimization/algorithms/tool_optimization)
. For privacy, it can automatically redact PII or other sensitive content in logs. For performance, it has an optimizer that can tune your prompts or agent parameters to improve evaluation scores. In the dashboard, you can see interactive dashboards and [alerts](https://www.comet.com/docs/opik/production/alerts)
for evaluation metrics. For example, trending graphs of accuracy or real-time alerts if something drops sharply.
Takeaways
---------
AI agents are powerful but complex to deploy because their probabilistic, multi-step behavior introduces many points of failure. Different parts of an agent, such as LLMs, tools, retrievers, and workflows, each need their own evaluation approach.
A range of open-source tools helps cover this space.
FAQs
----
What's the difference between open-source and free AI agent evaluation tools?
Open-source tools like Arize Phoenix, DeepEval, and Ragas are fully public and self-hosted. You can inspect or modify code, use them at no cost, and run them locally or in your cloud environment. Free hosted services such as LangSmith’s free tier or Arize AX’s free tier provide a managed platform with UIs and storage, but may have usage limits.
Do I need to pay to use these AI agent evaluation tools?
Most tools have free, open-source versions. OpenAI Evals, DeepEval, Ragas, Promptfoo, and Comet Opik, are completely free. Arize Phoenix is open-source; Arize AX has a free tier. LangSmith offers a free developer plan with 5k traces/month.
Can these AI agent evaluation tools evaluate multi-step agents?
Yes. Tools like Arize Phoenix and Arize AX, LangSmith, DeepEval, Ragas, and Opik are built for multi-turn, multi-tool agents. They can trace an entire agent trajectory and compute metrics on full sessions.
Can I use multiple AI agent evaluation tools together?
Yes. They serve complementary needs. For example, OpenAI Evals or DeepEval can handle offline benchmarking, while Arize or Opik can monitor live traffic. Promptfoo can be used in CI pipelines for security checks.
Do these tools support custom evaluation metrics?
Yes. Nearly all tools allow user-defined metrics. Arize Phoenix and Arize AX both support LLM-judge rubrics or Python or TypeScript code. Ragas supports custom Python metrics via callbacks or decorators.
Which AI agent evaluation tool has the best TypeScript support?
Arize Phoenix and Arize AX provide dedicated TypeScript (Node) SDKs for instrumentation and evaluation. Promptfoo and LangSmith are primarily CLI/web tools but integrate with any codebase.
Are these AI agent evaluation tools truly free?
All listed tools are free to use; open-source ones have permissive licenses. Some, like Arize AX or LangSmith, have paid tiers but offer generous free plans. Scoring may involve LLM calls, which can incur API costs, but the frameworks themselves don’t charge.
What's the difference between observability and evaluation in AI agent tools?
Observability captures what the agent is doing in real time through tracing, logging, and dashboards. Evaluation measures quality against criteria or ground truth. Tools like Arize and Opik combine both by logging traces and attaching evaluators.
_This post is sponsored by [Arize](https://arize.com/)
. We thank the [Arize](https://arize.com/)
team for their ongoing support of the community._
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Join our Slack – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Slack
=====
Join our vibrant community of data science professionals, machine learning engineers, and AI practitioners on Slack.
How to Join
-----------
To join our Slack community, write your email and click on the "Join" button.
Email
Join
1. You will receive an invite link via email
2. Click on the invite link to finish registration in Slack
3. Start participating in our community discussions!
**Need help?** If you haven't received an invite, or the invite link doesn't work, [fill in this form](https://airtable.com/shrhUBN51Jy10fjJq)
and we'll add you to Slack manually. We check the form once per day.
When participating in discussions, please follow [our community guidelines](https://datatalks.club/slack/guidelines.html)
.
Channels
--------
Available Slack Channels
------------------------
* `#book-of-the-week` – to talk about books with book authors (check the [books page](https://datatalks.club/books.html)
for more information)
* `#career-questions` – for career discussions (switching from one role to something data-related, being better at work, etc)
* `#datascience` – for talking about data science, machine learning, algorithms, training process, and ml-related libraries
* `#engineering` – for discussing the engineering aspects of data science: data engineering, ML engineering, MLOps, and so on
* `#events` – to talk about events (not just our events, but events in other communities as well)
* `#jobs` – for jobs
* `#memes` – for pictures and memes
* `#random` – for everything else
**How to find and join channels:** [Slack's guide to joining channels](https://slack.com/intl/en-de/help/articles/205239967-Join-a-channel)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Science Manager vs Expert: Which Role Does Your Company Need? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Science Manager vs Expert: Which Role Does Your Company Need?
Data Science Manager vs Expert: Which Role Does Your Company Need?
==================================================================
### Complete comparison of skills, responsibilities, and team roles. Learn when to hire a data science manager vs technical expert and which profile your company needs.
03 Aug 2025 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
Photo by [krakenimages](https://unsplash.com/@krakenimages?utm_source=medium&utm_medium=referral)
on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)
There’s a lot of debate in the tech world about whether data science managers or data science experts are more valuable to organizations. Some believe that managers are better able to develop and implement strategy, while others argue that experts are more skilled at working with data.
So, which is it? In this article, we’ll explore the pros and cons of each option to help you make a decision for your organization.
The article is organized as follows:
* The roles of manager and expert
* Skills comparison
* Responsibilities
* Working in a team
* When to hire a manager or an expert
The roles of manager and expert
-------------------------------
The expert role is different from the manager’s because **experts need a very deep understanding of algorithms and technologies from the area in which they have many years of experience, while managers only have a general knowledge of the topic.** Very often, an expert doesn’t only have a good knowledge of technology, but also domain knowledge.
Data science managers are responsible for a team of data scientists and ensuring that projects are completed on time and within budget. They also work with clients to understand their needs and ensure that the team is meeting their expectations.
Data science experts are responsible for providing insights and recommendations based on data analysis. They also develop and test models to improve decision-making and help businesses solve complex problems.
Key differences between Data Science Manager and Data Science Expert roles: managers focus on team leadership and strategy, while experts focus on deep technical knowledge and execution.
Skills
------
### Data Science Manager Skills
Data science managers need a blend of technical and non-technical skills:
**Technical Skills:**
* High-level understanding of data science tools and techniques
* Ability to interpret experiment results and provide feedback
* Sufficient technical depth to build credibility with the team
**Non-Technical Skills:**
* Strong communication and interpersonal abilities
* Project management experience
* Ability to work with different teams and manage multiple projects
* Strategic thinking and planning
While managers don’t need the same depth as experts, technical understanding helps them guide their teams effectively and troubleshoot issues.
### Data Science Expert Skills
Data science experts are highly technical individuals with deep knowledge in specific areas:
* Deep expertise in algorithms and statistical methods
* Advanced machine learning and modeling techniques
* Programming proficiency (Python, R, SQL)
* Research and experimentation capabilities
* Domain-specific knowledge
This specialized expertise is invaluable for solving complex problems, though general management skills and understanding of data science principles are sufficient for managerial roles.
Skill comparison: Data Science Managers need strong communication, project management, and high-level technical understanding, while Data Science Experts require deep technical expertise in algorithms, statistics, and machine learning.
Responsibilities
----------------
### Data Science Manager Responsibilities
The responsibilities of a data science manager involve:
* Leading and coordinating a team of data scientists
* Developing data science strategies
* Overseeing projects from start to finish
* Directing the personal development of each team member
Managers have a **holistic approach** to data science projects. They communicate with stakeholders and business users, understand machine learning model requirements, and identify new opportunities to leverage data science across business areas. In addition to team management, they’re often involved in business development and sales, requiring strong communication, presentation, and negotiation skills.
### Data Science Expert Responsibilities
Data science experts are responsible for executing data science projects. They possess in-depth knowledge of:
* Algorithms and statistical techniques
* Large-scale data wrangling and processing
* Research and development of new analytical methods
* Implementation of advanced techniques into production systems
While their focus is technical depth, experts also need strong communication skills for collaborating with team members and explaining complex findings.
Working in a team
-----------------
As a data science manager, one of your primary responsibilities is to develop and manage a team of data scientists. This includes hiring talented individuals, providing them with the resources they need to do their job effectively, and setting clear expectations. Additionally, you need to ensure that your team is working together harmoniously and effectively towards common goals.
As a data science manager, it’s important to review your team’s work and ensure they’re on the right track. This requires understanding the data and analysis they’re performing. Without this capability, it becomes difficult to effectively manage the team’s productivity. Managers should provide clear direction to each team member and communicate goals for the year or month.
Data science experts are often able to provide insights and suggestions that can help improve the work of the data science team. In addition, data science experts are often able to build strong teams because of their deep understanding of the field. They know what skills and knowledge are necessary for success and can identify talented individuals. Data science managers, on the other hand, may not have the same level of expertise. While they may be able to identify potential team members, they may not have the ability to fully assess their skills and knowledge.
Do you need a manager or an expert?
-----------------------------------
The answer to this question may depend on the size and scope of your project.
If you have a large project with many moving parts, then you might need a data science manager to keep everything organized and on track. A manager can also be helpful in coordinating team members with different skill sets.
On the other hand, if your project is more focused and you have a clear idea of what you want to achieve, then you might be better off working with a data science expert. An expert can help you fine-tune your project and ensure success.
Ultimately, the decision of whether to work with a manager or an expert depends on your specific needs and goals. If you are unsure, it might be helpful to consult with both types of professionals to get a sense of what would work best for your project.
You should keep in mind that **if you hire a data science manager first, they can then build and scale the data science team** according to your organization’s needs.
Summary
-------
Congratulations! You have just learned the difference between data science manager and data science expert!
**Data Science Managers:**
* Lead teams and ensure projects are completed on time and within budget
* Need general technical skills and strong communication abilities
* Focus on strategy, team development, and stakeholder management
**Data Science Experts:**
* Do the actual work of data analysis and modeling
* Need deep expertise in statistics, machine learning, and algorithms
* Focus on building and implementing advanced analytical solutions
So, which role is right for you? If you feel like you have the leadership skills necessary to manage a team of data scientists, then a data science manager might be the right choice. If you’d rather focus on the actual work of data science, then a data science expert might be a better fit.
Ultimately, it’s up to you to decide which role best suits your needs! The decision depends on your organization’s current stage, project complexity, and whether you need strategic leadership or specialized technical execution.
* * *
_This article is based on the podcast episode [Data Science Manager vs Data Science Expert](https://datatalks.club/podcast/s06e03-manager-vs-expert.html#data-science-managers-in-startups)
with Barbara Sobkowiak at DataTalks.Club._
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# 20+ Best Data Science Slack Communities to Join in 2025 – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
20+ Best Data Science Slack Communities to Join in 2025
20+ Best Data Science Slack Communities
=======================================
### A guide to the most active Slack groups for data, AI, and machine learning professionals.
19 Oct 2025 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Finding the right [data science Slack community](https://datatalks.club/slack.html)
can make a big difference in how you learn, work, and grow in the field. Slack groups bring together data scientists, machine learning engineers, and AI enthusiasts from around the world. They’re places where you can ask technical questions, share projects, discover learning resources, and connect with peers working on similar challenges.
In this guide, we’ve collected some of the most active and useful data science Slack communities to join in 2025.
Why You Can Trust This List
---------------------------
When you search for data science Slack communities, many of the lists you’ll find online are outdated. Some links point to broken Heroku pages, some communities no longer exist, and others have gone inactive.
Since it’s 2025, we decided to collect an updated list of active communities. Every Slack group included here is still running, with open links where possible, and organized into categories so you can quickly find what’s most relevant to you.
| Category | Communities |
| --- | --- |
| 1\. General Data Science Communities | [DataTalks.Club](https://datatalks.club/blog/slack-communities.html#datatalksclub)
, [Open Data Science Community](https://datatalks.club/blog/slack-communities.html#open-data-science-community)
, [Data Science Salon](https://datatalks.club/blog/slack-communities.html#data-science-salon)
, [Data Science Learning Community](https://datatalks.club/blog/slack-communities.html#data-science-learning-community) |
| 2\. Machine Learning and AI Communities | [MLOps Community](https://datatalks.club/blog/slack-communities.html#mlops-community)
, [TWIML Community](https://datatalks.club/blog/slack-communities.html#twiml-community)
, [AI Accelerator Institute Slack](https://datatalks.club/blog/slack-communities.html#ai-accelerator-institute-slack)
, [Convergence by Comet ML](https://datatalks.club/blog/slack-communities.html#convergence-by-comet-ml) |
| 3\. Specialized Tools and Practices | [dbt Community](https://datatalks.club/blog/slack-communities.html#dbt-community)
, [Data Visualization Society](https://datatalks.club/blog/slack-communities.html#data-visualization-society)
, [Spark NLP](https://datatalks.club/blog/slack-communities.html#spark-nlp)
, [Locally Optimistic](https://datatalks.club/blog/slack-communities.html#locally-optimistic)
, [Data Reliability Engineering Community](https://datatalks.club/blog/slack-communities.html#data-reliability-engineering-community) |
| 4\. Programming Language Communities | [PySlackers](https://datatalks.club/blog/slack-communities.html#pyslackers)
, [PyLadies](https://datatalks.club/blog/slack-communities.html#pyladies)
, [R-Team for Data Analysis](https://datatalks.club/blog/slack-communities.html#r-team-for-data-analysis)
, [R-Ladies](https://datatalks.club/blog/slack-communities.html#r-ladies) |
| 5\. Diversity and Inclusion Communities | [PyLadies](https://datatalks.club/blog/slack-communities.html#pyladies)
, [R-Ladies](https://datatalks.club/blog/slack-communities.html#r-ladies)
, [Data Angels](https://datatalks.club/blog/slack-communities.html#data-angels) |
| 6\. Invite-Only Communities | [AI Researchers and Enthusiasts](https://datatalks.club/blog/slack-communities.html#ai-researchers-and-enthusiasts)
, [DSNet – Data Science Network](https://datatalks.club/blog/slack-communities.html#dsnet--data-science-network)
, [Great Expectations (GX Core)](https://datatalks.club/blog/slack-communities.html#great-expectations-gx-core)
, [Academic Data Science Alliance (ADSA)](https://datatalks.club/blog/slack-communities.html#academic-data-science-alliance-adsa)
, [datascientists](https://datatalks.club/blog/slack-communities.html#datascientists)
, [AI-ML-Data Science Lovers](https://datatalks.club/blog/slack-communities.html#ai-ml-data-science-lovers)
, [KaggleNoobs](https://datatalks.club/blog/slack-communities.html#kagglenoobs) |
1\. General Data Science Communities
------------------------------------
### DataTalks.Club
One of the largest Slack communities dedicated to all things data, from analytics and visualization to machine learning and data engineering. Among its 80,000+ members, you’ll find data scientists, ML engineers, and enthusiasts who use the space to share knowledge, ask career questions, and join in discussions. You’ll find channels ranging from `#career` and `#datascience` to `#book-of-the-week`.
[Join here](https://datatalks.club/slack.html)
### Open Data Science Community
A global community connecting data scientists, engineers, and researchers around open collaboration. You can share articles, tutorials, code, and advice, as well as create projects, events, and courses. Their Slack emphasizes peer learning and advancing open data science practices.
[Join here](https://docs.google.com/forms/d/e/1FAIpQLSdjQB90EdZGV7Eelwo20WFC1ziP884FR_mtrsrdXPhEKIB3Ow/viewform)
### Data Science Salon
The official Slack space for the Data Science Salon and DSSelevate community. It’s a platform where you can connect with data science managers and practitioners through hosted chats, networking, industry news, and updates on upcoming events. The group grew out of the Data Science Salon conference and includes data scientists, ML engineers, and tech leads.
[Join here](https://info.datascience.salon/apply-to-dss-slack-workspace)
### Data Science Learning Community
An open community focused on accessible learning in data science. It provides tools, resources, and peer support for you at all stages, whether you’re looking for your first data job or you’re an experienced professional continuing your learning. Their Slack is designed as a collaborative space to help you stay motivated without the high costs of traditional courses.
[Join here](https://dslc.io/)
2\. Machine Learning and AI Communities
---------------------------------------
### MLOps Community
One of the largest global groups, with over 27,900 members, for practitioners working on machine learning in production. MLOps Community’s Slack hosts discussions on MLOps best practices, jobs, industry news, and events. You’ll find members ranging from engineers to researchers, sharing real-world challenges and solutions for operating ML systems at scale.
[Join here](https://gatewaze.mlops.community/?mode=slack)
### TWIML Community
A global Slack group for machine learning, deep learning, and AI practitioners. The community runs study groups for courses such as _fast.ai Deep Learning_, _Stanford CS224N_, and _Deeplearning.ai_, and also organizes special interest groups on topics like Swift for TensorFlow and Kaggle competitions. You can use the Slack channels to share tips, resources, and get support while working through these programs.
[Join here](https://twimlai.com/community/)
### AI Accelerator Institute Slack
A community of over 9,300 members run by the AI Accelerator Institute. It’s a place where you can connect with peers, exchange ideas, and get feedback from others working in AI. You’ll find members ranging from practitioners to enthusiasts, and the channels cover networking, discussions, and support if you’re exploring applied AI.
[Join here](https://www.aiacceleratorinstitute.com/join-the-aiai-slack-community/)
### Convergence by Comet ML
This Slack group is linked to Comet ML’s Convergence conference. The community focuses on topics around large language models, including evaluation methods, agentic AI, and responsible use of generative AI. It’s a space for you if you’re interested in the technical and practical aspects of building and deploying LLM-based applications.
[Join here](https://join.slack.com/t/convergenceml/shared_invite/zt-11q0wlne8-oMgAiaOsHo8~2VdNqoO7fQ?utm_source=chatgpt.com)
3\. Specialized Tools and Practices
-----------------------------------
### dbt Community
A Slack group for data professionals who use or are interested in dbt. You can connect to share knowledge, improve your skills, and discuss data transformation practices. The community includes both newcomers and experienced users, with channels covering technical help, events, and best practices.
[Join here](https://www.getdbt.com/community/join-the-community)
### Data Visualization Society
A global community with over 14,000 members for those working with or interested in data visualization. Their Slack community has channels for introductions, specific visualization topics, and professional development. You’ll find members ranging from beginners to experienced practitioners, sharing resources, projects, and advice.
[Join here](https://www.datavisualizationsociety.org/slack-community)
### Spark NLP
A Slack community focused on natural language processing with Spark NLP, an open-source library built on Apache Spark. You can discuss NLP techniques, share resources, and get help with model integration from frameworks such as TensorFlow, ONNX, OpenVINO, and Llama.cpp. It’s a mix of developers, researchers, and practitioners working with NLP at scale.
[Join here](https://spark-nlp.slack.com/join/shared_invite/zt-1zotzpe9e-dcIAs9I6jcsW92k5Y0rvVA#/shared-invite/email)
### Locally Optimistic
A Slack group for current and aspiring data analytics leaders. Started in New York in 2018, it has grown into an active community where you can exchange experiences, advice, and lessons learned with other analytics professionals. The group also supports local meetups alongside the online discussions.
[Join here](https://locallyoptimistic.com/community/)
### Data Reliability Engineering Community
A Slack community focused on data reliability challenges. It brings together data engineers and scientists where you can share experiences, discuss common issues, and exchange best practices for building more reliable data systems.
[Join here](https://join.slack.com/t/datareliabili-h4y1326/shared_invite/zt-16md9v2dw-DjHdb_zSg7UD6i5iayYqyg?utm_campaign=Dre-Con+2022&utm_source=chatgpt.com&utm_medium=email&_hsenc=p2ANqtz--nK749i8gE0-bASdaC7I9SmNAYxJGFIMfIR0yrJ0EkJ9OoxH0KEjjCwbYRoN63kEIGl5nT)
4\. Programming Language Communities
------------------------------------
### PySlackers
An inclusive Slack community with over 38,600 members for Python enthusiasts, ranging from beginners to professionals who have built their careers around the language. The group provides resources, support, and community projects, creating a space where you can learn, share knowledge, and collaborate.
[Join here](https://pyslackers.com/web/slack)
### PyLadies
An international mentorship group for women who use and love Python. 13,700+ members include full-time developers, hobbyists, and contributors to open-source projects. This Slack community provides a space where you can learn, network, and get support to become an active participant and leader in the Python community.
[Join here](https://slackin.pyladies.com/)
### R-Team for Data Analysis
A global Slack group for those learning and working with R. The community encompasses a wide range of topics, from beginner questions to advanced subjects like time series analysis. You’ll find popular channels like `#r-help` for coding questions, `#resources` for learning materials, `#jobs` for opportunities, and `#mooc_courses` for course discussions.
[Join here](https://slofile.com/slack/r-data-team?utm_source=chatgpt.com)
### R-Ladies
A global Slack community supporting gender diversity in the R programming ecosystem. You can use the space to share news, discuss packages, and exchange ideas in a safe and welcoming environment. This Slack community connects R-Ladies chapters worldwide and serves as a hub for networking, learning, and collaboration.
[Join here](https://guide.rladies.org/comm/slack/)
5\. Diversity and Inclusion Communities
---------------------------------------
### PyLadies
An international mentorship group for women who use and love Python. 13,700+ members include full-time developers, hobbyists, and contributors to open-source projects. The Slack provides a space where you can learn, network, and get support to become an active participant and leader in the Python community.
[Join here](https://slackin.pyladies.com/)
### R-Ladies
A global Slack community supporting gender diversity in the R programming ecosystem. You can use the space to share news, discuss packages, and exchange ideas in a safe and welcoming environment. This Slack community connects R-Ladies chapters worldwide and serves as a hub for networking, learning, and collaboration.
[Join here](https://guide.rladies.org/comm/slack/)
### Data Angels
A community founded in 2020 to support women across all areas of data. The Slack with more than 2,600 members hosts discussions, mentorship programs, and panels, as well as local meetups. It’s a space where you can collaborate, grow your career, and network within the wider data ecosystem.
[Join here](https://www.dataangels.org/home)
6\. Invite-Only Communities
---------------------------
Some communities have Slack groups but don’t provide a public link to join. To access these, you’ll need either an existing account or an invitation from an administrator or an active member. We’ve listed these separately below.
### AI Researchers and Enthusiasts
A Slack group for researchers and hobbyists interested in artificial intelligence. The community is open to all levels, from early learners to advanced researchers, and provides a space where you can exchange theories, concepts, and perspectives on AI.
[Sign up here](https://ai-researchers.slack.com/)
### DSNet – Data Science Network
A growing Slack community supported by Jovian.ml, with open channels, events, and shared resources for those in data science. It serves as a space where you can learn from others, exchange insights, and stay connected with developments across the field.
[Sign up here](https://dsnetorg.slack.com/)
### Great Expectations (GX Core)
The Slack community around Great Expectations, an open-source tool for data quality. Thousands of members contribute by sharing deployment experiences, discussing best practices, and helping teams improve reliability in their data workflows. You can join to share your own experiences and learn from others.
[Sign up here](https://greatexpectationstalk.slack.com/)
### Academic Data Science Alliance (ADSA)
A Slack group connected to ADSA, a professional association focused on advancing responsible and inclusive use of data science and AI in academic research, education, and training. You’ll find academics, educators, and practitioners collaborating on the future of data science in higher education.
[Sign up here](https://academicdatascience.slack.com/)
### datascientists
A Slack group focused on data science and related fields such as data warehouses, business intelligence, and analytics. You can discuss technical challenges, share tools and resources, and exchange advice from your own professional experiences. The community is aimed at both newcomers and experienced practitioners who want to learn from each other and expand their professional network.
[Sign up here](https://datascientists.slack.com/)
### AI-ML-Data Science Lovers
A laid-back community for those interested in AI, machine learning, and data science. Unlike some of the more specialized or technical groups, this Slack is more conversational and opinion-driven. You can share thoughts on industry developments, discuss general concepts, and exchange knowledge in a relaxed environment. It’s a good place to keep in touch with a broad range of perspectives across the field.
[Sign up here](https://datasciencelovers.slack.com/)
### KaggleNoobs
A beginner-friendly Slack community centered on Kaggle competitions and projects. The group provides guidance if you’re new to data science and want to improve your coding and modeling skills through practice. You can ask and answer questions, share tutorials, and discuss strategies for specific challenges. It’s particularly useful if you’re just getting started with Kaggle as a way to learn data science through real-world problems.
[Sign up here](https://kagglenoobs.slack.com/)
Conclusion
----------
Joining a Slack community can be one of the easiest ways for you to stay connected in the fast-moving worlds of data science, machine learning, and AI. These groups give you access to peers who are facing similar challenges, channels full of resources and tutorials, and opportunities to network or even find your next project or job.
Did we miss a community you find valuable? Let us know in [our Slack](https://datatalks.club/slack.html)
so we can keep this list up to date.
Frequently Searched Questions
-----------------------------
How do I join these Slack communities?
Most communities have a public invitation link that you can click to join directly. For invite-only communities, you’ll need to sign up on their website or request an invitation from an existing member. Some require approval before you can join.
How do I join the MLOps Community Slack?
Go to the [MLOps Community](https://datatalks.club/blog/slack-communities.html#mlops-community)
section and click “Join here.” That link takes you to the community’s official signup page for instant Slack access.
How do I join the dbt Community Slack?
Go to the [dbt Community](https://datatalks.club/blog/slack-communities.html#dbt-community)
section and click “Join here.” That link takes you to the community’s official signup page for instant Slack access.
How do I join the Locally Optimistic Slack?
Go to the [Locally Optimistic](https://datatalks.club/blog/slack-communities.html#locally-optimistic)
section and click “Join here.” That link takes you to the community’s official signup page for instant Slack access.
How do I join the Open Data Science (ODS.ai) Slack?
Go to the [Open Data Science Community](https://datatalks.club/blog/slack-communities.html#open-data-science-community)
section and click “Join here.” That link takes you to the community’s official signup page for instant Slack access.
Is there a list of public Slack communities I can join?
Yes, this page. Browse the table at the top for categories, then jump to any section for a description and a direct join link.
How do I join Open Data Science (ODS.ai) Slack?
Go to the [Open Data Science Community](https://datatalks.club/blog/slack-communities.html#open-data-science-community)
section and click “Join here.” That link takes you to the community’s official signup page for instant Slack access.
Are these communities free to join?
Yes, all the Slack communities listed here are free to join. However, some may have premium tiers or paid events associated with their broader organizations.
What should I expect when I first join?
Most communities have an introduction channel where you can introduce yourself as a new member. Take time to read the community guidelines, explore different channels, and observe the conversation style before jumping in. Many have specific channels for beginners or newcomers.
How active are these communities?
Activity levels vary significantly. Large communities like MLOps Community (27,900+ members) and PySlackers (38,600+ members) have constant activity, while smaller, specialized groups may have more focused but less frequent discussions. We’ve only included communities that are currently active as of 2025.
Can I join multiple communities?
Absolutely! You can be a member of several communities to get diverse perspectives and access different types of expertise. Just be mindful of managing notifications across multiple Slack workspaces.
What if I'm a complete beginner?
Several communities are explicitly beginner-friendly, including Data Science Learning Community, KaggleNoobs, and the general channels in DataTalks.Club. Look for channels labeled “beginners,” “newbies,” or “help” in any community you join.
How do I find job opportunities in these communities?
Many communities have dedicated job boards or career channels. MLOps Community, DataTalks.Club, and Data Science Salon are particularly good for job postings. Always read the community guidelines about job posting etiquette before you share opportunities.
Can I promote my own projects or content?
Most communities allow self-promotion in moderation, but always check the specific rules first. Many have dedicated channels for sharing projects, blog posts, or resources. The key is for you to be genuinely helpful to the community, not just promotional.
What if a community becomes inactive or the link doesn't work?
Since the Slack ecosystem changes frequently, some links may break over time. We’ve verified all links as of September 2025, but if you find an issue, try searching for the community’s main website or reaching out to current members on other platforms like LinkedIn or Twitter.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities
Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities
================================================================================
### Essential guide to PM, Data Analyst, Data Scientist, Data Engineer, ML Engineer, and MLOps/SRE: skills, responsibilities, and how teams ship ML products
05 Aug 2025 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
In this article, we’ll talk about different roles in a data team and discuss their responsibilities.
In particular, we will cover:
* The types of roles in a data team;
* The responsibilities of each role;
* The skills and knowledge each role needs to have.
> Want to listen to it as a podcast? Go to [Anchor.fm](https://anchor.fm/datatalksclub/episodes/Roles-in-a-data-team---Alexey-Grigorev-emqcft)
> or your favorite podcasting platfrom.
This is not a comprehensive list and the majority of what you will read in this article is my opinion, which comes out of my experience from working as a data scientist.
You can interpret the following information as “the description of data roles from the perspective of a data scientist”. For example, my views on the role of a data engineer may be a bit simplified because I don’t see all the complexities of their work firsthand. I do hope you will find this information useful nonetheless.
Roles in a Team
---------------
A typical data team consists of the following roles:
* Product managers,
* Data analysts,
* Data scientists,
* Data engineers,
* Machine learning engineers, and
* Site reliability engineers / MLOps engineers.
The data team: key roles collaborating to deliver ML-powered products
All these people work to create a data product.
To explain the core responsibilities of each role, we will use a case scenario:
Suppose we work at an online classifieds company. It’s a platform where users can go to sell things they don’t need (like OLX, where I work). If a user has an iPhone they want to sell — they go to this website, create a listing and sell their phone.
On this platform, sellers sometimes have problems with identifying the correct category for the items they are selling. To help them, we want to build a service that suggests the best category. To sell their iPhone, the user creates a listing and the site needs to automatically understand that this iPhone has to go in the “mobile phones” category.
Use case: service suggests the correct listing category automatically
Let’s start with the first role: product manager.
Product Manager
---------------
A product manager is someone responsible for developing products. Their goal is to make sure that the team is building the right thing. They are typically less technical than the rest of the team: they don’t focus on the implementation aspects of a problem, but rather the problem itself.
PM ensures the team builds the right product for users
Product managers need to ensure that the product is actually used by the end-users. This is a common problem: in many companies, engineers create something that doesn’t solve real problems. Therefore, the product manager is somebody who speaks to the team on behalf of the users.
The primary skills a PM needs to have are communication skills. For data scientists, communication is a soft skill, but for a product manager — it’s a hard skill. They have to have it to perform their work.
Product managers also do a lot of planning: they need to understand the problem, come up with a solution, and make sure the solution is implemented in a timely manner. To accomplish this, PMs need to know what’s important and plan the work accordingly.
When somebody has a problem, they approach the PM with it. Then the task of the PM is to figure out if users actually need this feature, how important this feature is, and if the team has the capacity to implement it.
Let’s come back to our example. Suppose somebody comes to the PM and says:
“We want to build a feature to automatically suggest the category for a listing. Somebody’s selling an iPhone, and we want to create a service that predicts that the item goes in the mobile phones category.”
Product managers need to answer these questions:
* “Is this feature that important to the user?”
* “Is it an important problem to solve in the product at all?”
To answer these questions, PMs ask data analysts to help them figure out what to do next.
Data Analyst
------------
Data analysts know how to analyze the data available in the company. They discover insights in the data and then explain their findings to others.
Analysts discover insights and define metrics for decisions
So, analysts need to know:
* What kind of data the company has;
* How to get the data;
* How to interpret the results;
* How to explain their findings to colleagues and management.
Data analysts are also often responsible for defining key metrics and building different dashboards. This includes things like showing the company’s profits, displaying the number of listings, or how many contacts buyers made with sellers. Thus, data analysts should know how to calculate all the important business metrics, and how to present them in a way that is understandable to others.
When it comes to skills, data analysts should know:
* SQL — this is the main tool that they work with;
* Programming languages such as Python or R;
* Tableau or similar tools for building dashboards;
* Basics of statistics;
* How to run experiments;
* A bit of machine learning, such as regression analysis, and time series modeling.
For our example, product managers turn to data analysts to help them quantify the extent of the problem. Together with the PM, the data analyst tries to answer questions like:
* “How many users are affected by this problem?”
* “How many users don’t finish creating their listing because of this problem?”
* “How many listings are there on the platform that don’t have the right category selected?”
PM partners with analyst to quantify problem scope and impact
After the analyst gets the data, analyzes it and answers these questions, they may conclude: “Yes, this is actually a problem”. Then the PM and the team discuss the repost and agree: “Indeed, this problem is actually worth solving”.
Now the data team will go ahead and start solving this problem.
After the model for the service is created, it’s necessary to understand if the service is effective: whether this model helps people and solves the problem. For that, data analysts usually run experiments — usually, A/B tests.
When running an experiment, we can see if more users successfully finish posting an item for sale or if there are fewer ads that end up in the wrong category.
Data Scientist
--------------
The roles of a data scientist and data analyst are pretty similar. In some companies, it’s the same person who does both jobs. However, data scientists typically focus more on predicting rather than explaining.
A data analyst fetches the data, looks at it, explains what’s going on to the team, and gives some recommendations on what to do about it. A data scientist, on the other hand, focuses more on creating machine learning services. For example, one of the questions that a data scientist would want to answer is “How can we use this data to build a machine learning model for predicting something?”
Data scientists build predictive models and ML services
In other words, data scientists incorporate the data into the product. Their focus is more on engineering than analysis. Data scientists work more closely with engineers on integrating data solutions into the product.
The skills of data scientists include:
* Machine learning — the main tool for building predictive services;
* Python — the primary programming language;
* SQL — necessary to fetch the data for training their models;
* Flask, Docker, and similar — to create simple web services for serving the models.
For our example, the data scientists are the people who develop the model used for predicting the category. Once they have a model, they can develop a simple web service for hosting this model.
Data Engineers
--------------
Data engineers do all the heavy lifting when it comes to data. A lot of work needs to happen before data analysts can go to a database, fetch the data, perform their analysis, and come up with a report. This is precisely the focus of data engineers — they make sure this is possible. Their responsibility is to prepare all the necessary data in a form that is consumable for their colleagues.
Data engineers build reliable pipelines and make data accessible
To accomplish this, data engineers create “a data lake”. All the data that users generate needs to be captured properly and saved in a separate database. This way, analysts can run their analysis, and data scientists can use this data for training models.
Another thing data engineers often need to do, especially at larger companies, is to ensure that the people who look at the data have the necessary clearance to do so. Some user data is sensitive and people can’t just go looking around at personal information (such as emails or phone numbers) unless they have a really good reason to do so. Therefore, data engineers need to set up a system that doesn’t let people just access all the data at once.
The skills needed for data engineers usually include:
* AWS or Google Cloud — popular cloud providers;
* Kubernetes and Terraform — infrastructure tools;
* Kafka or RabbitMQ — tools for capturing and processing the data;
* Databases — to save the data in such a way that it’s accessible for data analysts;
* Airflow or Luigi — data orchestration tools for building complex data pipelines.
In our example, a data engineer prepares all the required data. First, they make sure the analyst has the data to perform the analysis. Then they also work with the data scientist to prepare the information that we’ll need for training the model. That includes the title of the listing, its description, the category, and so on.
A data engineer isn’t the only type of engineer that a data team has. There are also machine learning engineers.
Machine Learning Engineer
-------------------------
Machine learning engineers take whatever data scientists build and help them scale it up. They also ensure that the service is maintainable and that the team follows the best engineering practices. Their focus is more on engineering than on modeling.
ML engineers productionize models and ensure reliability at scale
The skills ML engineers have are similar to that of data engineers:
* AWS or Google Cloud;
* Infrastructure tools like Kubernetes and Terraform;
* Python and other programming languages;
* Flask, Docker, and other tools for creating web services.
Additionally, ML engineers work closely with more “traditional” engineers, like backend engineers, frontend engineers, or mobile engineers, to ensure that the services from the data team are included in the final product.
For our example, ML engineers work together with data scientists on productionizing the category suggestion services. They make sure it’s stable once it’s rolled out to all the users. They must also ensure that it’s maintainable and it’s possible to make changes to the service in the future.
There’s another kind of engineer that can be pretty important in a data team — site reliability engineers.
DevOps / Site Reliability Engineer
----------------------------------
The role of SREs is similar to the ML engineer, but the focus is more on the availability and reliability of the services.
SREs aren’t strictly limited to working with data. Their role is more general: they tend to focus less on business logic and more on infrastructure, which includes things like networking and provisioning infrastructure.
SREs/MLOps ensure reliability, monitoring, and automation of services
Therefore, SREs look after the servers where the services are running and take care of collecting all the operational metrics like CPU usage, how many requests per second there are, the services’ processes, and so on.
As the name suggests, site reliability engineers have to make sure that everything runs reliably. They set up alerts and are constantly on call to make sure that the services are up and running without any interruptions. If something breaks, SREs quickly diagnose the problem and fix it, or involve an engineer to help find the solution.
The skills needed for site reliability engineers:
* Cloud infrastructure tools;
* Programming languages like Python,
* Unix/Linux;
* Networking;
* Best DevOps practices like automation, CI/CD, and the like.
Of course, ML engineers and data engineers should also know these best practices, but the focus of DevOps engineers/SREs is to establish them and make sure that they are followed.
There is a special type of DevOps engineer, called “MLOps engineer”.
MLOps Engineer
--------------
An MLOps engineer is a DevOps engineer who also knows the basics of machine learning. Similar to an SRE, the responsibility of an MLOps Engineer is to make sure that the services, developed by data scientists, ML engineers, and data engineers, are up and running all the time.
MLOps engineers know the lifecycle of a machine learning model: the training phase, serving phase, and so on.
Despite having this knowledge, MLOps Engineers are still focused more on operational support than on anything else. This means that they need to know and follow all the DevOps practices and make sure that the rest of the team is following them as well. They accomplish this by setting up things like continuous retraining, and CI/CD pipelines.
Even though everyone in the team has a different focus, they all work together on achieving the same goal: solve the problems of the users.
Summary
-------
To summarize, the roles in the data team and their responsibilities are:
* Product managers — make sure that the team is building the right thing, act as a gateway to all the requests and speak on behalf of the users.
* Data analysts — analyze data, define key metrics, and create dashboards.
* Data scientists — build models and incorporate them into the product.
* Data engineers — prepare the data for analysts and data scientists.
* ML engineers — productionize machine learning services and establish the best engineering practices.
* Site reliability engineers — focus on availability, reliability, enforce the best DevOps practices.
This list is not comprehensive, but it should be a good starting point if you are just getting into the industry, or if you just want to know how the lines between different roles are defined in the industry.
This article is based on the podcast “Roles in a data team”. You can watch it on YouTube:
Or listen to audio on Anchor.fm (or your favorite podcast platform):
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How Do Data Professionals Use MLOps Tools and Frameworks? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How Do Data Professionals Use MLOps Tools and Frameworks?
How Do Data Professionals Use MLOps Tools and Frameworks?
=========================================================
### Results of our DataTalks.Club Survey
28 Apr 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
We [surveyed](https://docs.google.com/forms/d/e/1FAIpQLScdx1FAIp2GDGgiMf7xu-I1PfhsQBJDvFstGmWmWbpP4S69Zg/viewform)
over 200 data professionals to understand the tools they use for deploying, monitoring, versioning, and managing machine learning workflows.
In this article, we highlight key trends, from model deployment and monitoring to CI/CD practices and team structures, revealing how organizations are operationalizing their ML models in production.
Let’s explore the data.
Model Deployment Tools
----------------------
Deployment practices vary widely.
A substantial portion (nearly 40%) does not deploy models. Among those who do, Kubernetes, SageMaker, and Azure ML are the top tools.
* **Kubernetes** is used by 27.1% of respondents.
* **AWS SageMaker** and **Google AI Platform** appear frequently (27.1% and 21.6%, respectively), while **Azure Machine Learning** is used by 18.3%.
* **TensorFlow Serving** is deployed by 9.6% of teams.
* Notably, 38.5% of respondents indicated that they do not deploy models at all.
Some respondents mentioned alternatives (e.g., TorchServe, OpenVino, MLFlow Serving, and homegrown solutions), suggesting a diverse ecosystem with no single solution dominating all use cases.
Distribution of model deployment tools usage among data professionals.
Monitoring ML Models in Production
----------------------------------
Monitoring practices are still in the early stages. A majority (57.9%) reported that they do not monitor models in production. Open-source and general-purpose monitoring tools (Grafana, ELK) are more common than specialized ML tools.
Among specific tools:
* **Prometheus and Grafana** are the most common monitoring tools (20.8%).
* Other solutions include the **ELK Stack** (9.1%), **Evidently AI** (5.6%), and custom monitoring scripts (11.2%).
Distribution of ML model monitoring tools usage.
CI/CD for ML Workflows
----------------------
CI/CD adoption for ML shows mixed usage. Almost half of respondents don’t use CI/CD for ML. Among those who do, GitLab CI/CD and MLflow are leading choices. Traditional DevOps tools dominate over ML-native pipelines.
**GitLab CI/CD** leads with 26.6%, followed by **Jenkins** (14.5%).
Distribution of CI/CD tools usage for ML workflows.
Model Versioning
----------------
A significant 58.1% of respondents do not use any model versioning tools.
Among those who do:
* **MLflow** is the most popular tool for versioning models (32.3%).
* **DVC (Data Version Control)** and **Weights & Biases (W&B)** are used by 10.6% and 11.1% of respondents, respectively.
Distribution of model versioning tools usage.
Data Versioning Tools
---------------------
Notably, 75.8% of respondents don’t use data versioning tools. DVC leads among those that do. There’s a notable tooling gap in reproducible data pipelines.
Among those who do:
* **DVC** is used by 16.5% of participants.
* Other tools such as **Quilt**, **LakeFS**, and **Pachyderm** see minimal adoption.
Distribution of data versioning tools usage.
Feature Store Adoption
----------------------
Feature stores are not widely adopted. A dominant 74.9% reported not using any feature store, indicating that many teams rely on custom or ad hoc solutions for feature management.
Only a minority use dedicated feature stores like **AWS SageMaker Feature Store** (12.3%), **Databricks Feature Store** (10.8%), or **Vertex AI Feature Store** (7.7%).
Distribution of feature store usage among data professionals.
Model Training and Experimentation Tools
----------------------------------------
More than half (55.2%) of respondents do not use dedicated experimentation tools, suggesting that many teams might leverage native frameworks or simple scripts.
Among those who do:
* **MLflow** leads again with 34.3% adoption.
* **Weights & Biases (W&B)** is used by 13.4%, while **TensorBoard** and **Neptune.ai** have lower adoption (10% and 3.5%, respectively).
Distribution of model training and experimentation tools usage.
Workflow Orchestration for ML Pipelines
---------------------------------------
Orchestrating ML pipelines is not yet widespread. Over half (53.6%) of respondents do not employ any workflow orchestration tools, which may impact scalability and automation.
Among those who do:
* **Apache Airflow** is the most used orchestration tool (34%).
* Other tools such as **Prefect** (6.2%), **Kubeflow** (7.2%), and **AWS Step Functions** (7.7%) are also noted.
Distribution of workflow orchestration tools usage.
Model Retraining Frequency
--------------------------
Retraining remains reactive or infrequent for most teams.
* A substantial 43.9% do not retrain their models once deployed.
* Among those that do, 28.6% retrain models as needed based on performance degradation, and 23% retrain periodically.
* Only a very small percentage (around 3.1%) use continuous online learning.
Distribution of model retraining frequency among data professionals.
Infrastructure for ML Workloads
-------------------------------
Regarding where ML workloads run, the results show a mix of diverse deployment environments, with many organizations leveraging cloud scalability.
* **Cloud platforms** dominate with AWS (39.9%), Azure (28.7%), and GCP (20.8%).
* **On-premise infrastructures** are used by 21.3% of respondents, while 11.2% report a hybrid approach.
Distribution of ML infrastructure usage among organizations.
Team Structure and Centralization
---------------------------------
Most teams are small or solo. This aligns with the widespread absence of orchestration, monitoring, and versioning practices.
Only 18.8% reported having a centralized MLOps team, while the vast majority (81.2%) do not, indicating that ML operations are often managed in a more distributed manner.
* The largest segment (45.6%) have small ML teams of 1–5 members.
* About 34.8% report teams of 6–10 members.
Distribution of ML team sizes across organizations.
Production ML Models
--------------------
While the survey captured data on the number of models in production, many respondents reported having few or no models deployed. This suggests that many organizations are still in early stages of scaling ML operations.
Distribution of the number of ML models in production across organizations.
Conclusion
----------
The survey reveals a diverse and evolving ML and MLOps landscape.
Key takeaways include:
* **Deployment:** Many tools are in use, from Kubernetes to cloud-native platforms, but many teams have not yet deployed models.
* **Monitoring:** Over half of respondents do not monitor their models, underscoring an opportunity for improved observability.
* **Versioning & Experimentation:** Although tools like MLflow and DVC are gaining traction, many teams are not using dedicated versioning solutions.
* **Infrastructure & Retraining:** Cloud platforms are widely used, and most models are not retrained frequently.
* **Challenges:** Data quality, deployment, monitoring, and integration remain critical pain points.
Overall, while many teams are adopting modern tools for ML workflows, there remains ample opportunity for standardizing practices and addressing operational challenges as organizations mature their ML initiatives.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Building Discipline in Machine Learning with ML Zoomcamp – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Building Discipline in Machine Learning with ML Zoomcamp
Building Discipline in Machine Learning with ML Zoomcamp
========================================================
### A disciplined path to data prep, validation, and deployment, plus how public sharing accelerated my skills, connections, and offers.
11 Aug 2025 by [Alexander Daniel Rios](https://datatalks.club/people/alexanderdanielrios.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
In our previous article, we wrote about an [end-to-end blood cell classifier for cancer prediction](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
, a final project for [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
created by graduate Alexander Daniel Rios. In this follow-up, Alexander reflects on how the course shifted his approach from ad-hoc experimentation to a structured, professional workflow. He walks through the skills he developed, the two final projects he completed, and why working in public and with a community made a difference for his professional growth.
> [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
> is a free, four-month online course on machine learning fundamentals and deploying models to production.
What were your projects like before ML Zoomcamp?
------------------------------------------------
Alexander: Chaotic. I learned many concepts on my own in a disorganized way. My notebooks were hard to maintain, I didn’t follow a clear structure, and I often jumped from one model to another without proper validation or traceability.
What changed during the course?
-------------------------------
Alexander: I learned to articulate my thoughts more clearly and work in a professional, end-to-end manner. The recorded lessons, hands-on assignments, and final projects helped me structure my workflow and effectively utilize the correct tools, from development to deployment.
Which skills did you develop?
-----------------------------
Alexander: I learned a lot of practical skills! Some of them:
* Systematic data preparation
* Applying regression and classification with proper validation
* Selecting appropriate evaluation metrics
* Importantly, understanding when and why to use a specific model.
I also gained confidence with neural networks, transfer learning, and making inferences with models deployed via Flask or AWS Lambda.
What projects did you work on?
------------------------------
Alexander: We were encouraged to do two final projects and participate in a Kaggle competition. My first project was about [creating a classifier to predict cancer in blood cells](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
. I talked about it in the recent article on the DataTalksClub blog.
A screenshot from Alexander’s article on the DataTalksClub blog on his final project about [creating a classifier to predict cancer in blood cells](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
My second project focuses on classifying disaster-related tweets. I also [wrote an article](https://www.notion.so/Natural-Language-Processing-using-spaCy-TensorFlow-and-BERT-model-architecture-1895067176b380d09484d4b0338b0c5e?pvs=21)
about this project, in which I explore how I utilized spaCy, TensorFlow, and BERT-based architectures for NLP.
I found both projects engaging because they address real-world challenges. Their complexity let me apply what I learned and push further with concepts beyond the core curriculum.
And what about your Kaggle competition experience?
--------------------------------------------------
Alexander: As part of the course, I joined the [ML Zoomcamp 2024 Retail Forecasting Competition on Kaggle](https://www.kaggle.com/competitions/ml-zoomcamp-2024-competition)
. The task was to predict next-month product demand using 25 months of historical data across multiple stores. I applied feature engineering, time series analysis, model evaluation, and deployment. The pipeline included temporal feature extraction, economic indicators, and robust validation.
[ML Zoomcamp 2024 Retail Forecasting Competition on Kaggle](https://www.kaggle.com/competitions/ml-zoomcamp-2024-competition)
The final model achieved 4th place globally, among dozens of participants. This strengthened my skills in real-world forecasting, time-series modeling, efficient preprocessing, and competition-style thinking.
How did the course contribute to your capstone work?
----------------------------------------------------
Alexander: The [ML Zoomcamp course](https://datatalks.club/blog/machine-learning-zoomcamp.html)
provided the theoretical and practical foundations I needed. It offered a structured path from ML fundamentals to pipeline design and deployment. I gained hands-on experience preparing and validating datasets, selecting models, and evaluating them properly. I also worked with production tools such as Docker, Flask, and TensorFlow Serving, which were key to transforming the project into a functional product.
Complete ML Zoomcamp curriculum: from machine learning fundamentals to production deployment
What kept you motivated?
------------------------
Alexander: The active community, practical challenges, and guided projects encouraged me to take the work beyond an academic setting. Although the final project was the starting point, the methods and mindset I learned helped me scale the solution, add components like segmentation and multitask learning, and build a robust deployment.
How did sharing your work affect your growth?
---------------------------------------------
Alexander: A lot. Each week, I posted on LinkedIn about the tasks I was working on, the challenges I faced, and what I was discovering. That habit led to connections, feedback, and visibility, resulting in conversations with people from diverse backgrounds, suggestions to improve my work, and even job offers and freelance requests. As a natural extension, I created [my first personal blog](https://www.notion.so/Machine-Learning-Notes-10d5067176b380cba9adff35f4a997a1?pvs=21)
focused on ML Zoomcamp, where I wrote weekly reflections, explained my decisions, and shared what I learned.
Alexander’s personal blog with notes from ML Zoomcamp
This experience helped me grow technically and as a communicator, and it gave me the confidence to share my work beyond the classroom. It showed the value of learning in public and being part of a community.
Thanks for sharing your experience with us, Alexander!
Conclusion
----------
Alexander’s journey started with scattered notebooks and unclear validation and ended with a reproducible workflow, well-chosen metrics, and deployable models. ML Zoomcamp provided the structure and the practical stack to make that shift possible. Along the way, community and public sharing turned weekly progress into feedback, opportunities, and confidence.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Building an AI Agent that Thrives in the Real World – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Building an AI Agent that Thrives in the Real World
Building an AI Agent that Thrives in the Real World
===================================================
### A Guide to Development, Testing, and Monitoring
26 Feb 2025 by [Sally-Ann DeLucia](https://datatalks.club/people/sallyanndelucia.html)
Building an AI agent that thrives
Building an AI agent and keeping it running smoothly in production can feel like a daunting task. When it comes to working with LLMs, it’s still a bit of uncharted territory—most of us are figuring things out through trial and error. Even after you’ve launched in production, things can break or perform unexpectedly, forcing you to go back, iterate, and try again. The line between development and production often feels blurry, with constant back-and-forth.
That’s where a tool like Arize becomes essential. But with such a robust toolkit, it can be overwhelming to know which features to use and when. We’ve experienced the pain of building, so we’re here to help you navigate these complexities.
In this blog, we’ve broken down how we develop, iterate, and improve our AI Assistant–Copilot–using Arize and Phoenix. We’ll guide you through the key features and workflows that make it all work.
From development to production: a continuous AI improvement loop
AI Agent Testing and Iterating
------------------------------
When it comes to developing new skills, or testing and iterating on features, Phoenix is our go-to tool. Its traces provide us with invaluable insights during development. When we build a new skill, we usually start with a proof of concept—a basic skeleton that performs the desired task. From there, we dive into testing and iterating.
We’ve created a testing framework that integrates Copilot components directly into a notebook, making it easy to run test queries and the traces are immediately available in Phoenix for review. This lets us assess if data functions are fetching the correct data from Arize, if routers are properly calling functions, and whether functions are receiving the right arguments—all while checking if the responses align with our expectations. In short, Phoenix traces help us understand what’s working and what’s not, enabling us to iterate quickly and confidently. For us, Phoenix traces are one of our most frequently used development tools.
Once a skill is live in production, Arize becomes our primary tool. We rely on it daily, especially the traces, to see how users interact with Copilot and whether it’s performing as expected. Filtering data is crucial here. For example, we often filter by user email to differentiate internal use from customer interactions. We’ll scroll through traces, inspecting the steps the application took to ensure they align with our expectations.
Tracing AI performance in production
Recently, a change to our tracing endpoint broke several generative skills, causing widespread issues. We identified the problem using the traces page by filtering for error traces. Once we reviewed them, it became clear what was wrong, and we were able to resolve the issue quickly. However, to avoid a similar situation in the future, we set up error monitors that alert us when more than five errors occur, allowing us to fix issues even faster. We also monitor eval labels for potential issues like jailbreaking attempts.
Proactive AI monitoring with automated alerts
Daily Flows: Dashboards and Monitoring
--------------------------------------
Another key part of our daily workflow is checking out Arize dashboards to monitor patterns and usage. We track high-level metrics such as the number of Copilot requests, average query costs, and token counts. These metrics help us gauge traffic and cost. Error rates are also a critical metric we follow closely on our dashboards.
One of the most insightful parts of the dashboard is tracking the number of requests by skill and user. Being able to visualize this helps us see which users are the most active—making it easier to gather feedback—and which skills are gaining the most traction. Early on, this data helped us identify where to invest in development, allowing us to double down on our most powerful skills.
Visualizing usage trends
Evaluating Agent Performance with Online Evals
----------------------------------------------
To keep a pulse on Copilot’s performance in production, we set up online jobs to run automatic evals on new data. We started with a simple QA correctness eval, using Phoenix’s eval library, to check whether Copilot was answering user queries correctly. This helped us filter and drill into traces, refining performance over time. With tools like annotations, we can now review eval performance and mark incorrect evaluations for future iteration.
Not all skills fit under a generic QA correctness eval, so we’ve since built skill-specific evals with additional layers of evaluation. By refining our eval sets, we ensure more comprehensive monitoring.
Harnessing Datasets and Experiments
-----------------------------------
Daily trace reviews help us form hypotheses and build intuition about user behavior. One of our favorite features for this is datasets. As we review traces, we often identify patterns—whether it’s problematic traces, buggy behavior, or less-than-ideal results—and save them to datasets for future testing. One way we like to do this is to collect unsupported user queries in a dataset, as it helps pinpoint missing functionality in Copilot. Datasets are versatile, serving as collections of examples for testing or as active components in our development workflow with experiments.
Experiments are our go-to when testing changes, such as model updates or A/B tests. By combining datasets and experiments, we can systematically test changes in our application. We define tasks that replicate parts of the application and use evaluators to measure success. Whether troubleshooting issues or validating improvements, experiments give us a structured way to measure the impact of our changes and ensure things stay on track.
Handling Model Switches with Experiments
----------------------------------------
Back in May, when OpenAI released GPT-4-o, we were excited to take advantage of its improved performance, speed, and security. What we didn’t anticipate was the significant impact switching models would have. Many of our skills began to fail—some produced odd responses, others ignored instructions, and some even stopped working altogether. It quickly became clear that we needed to manually test every single function, document our findings, and iterate until everything was fixed. It was a painful process.
Now, with experiments, we can take a far more systematic approach when handling changes like model switches. We create a golden dataset, set up tasks that mirror our real-world setup, and evaluate whether the new model produces the correct results across all golden inputs. This allows us to predict exactly where things will break and how we need to adjust.
We also use experiments in ongoing development. For example, after reviewing our dashboards and determining that AI Search was the most promising skill to invest in, we saved a collection of traces where users attempted queries that weren’t fully supported. We noticed three clear categories of user expectations: text-to-filter, table search, and analytics. So, we developed three new skills and updated our router to handle them.
Instead of releasing these updates blindly, we leveraged datasets and experiments to thoroughly test the changes.
We built experiments to verify that our new router was selecting the correct function and passing the right arguments. We also ran experiments to ensure each new function was performing as expected. This data-driven, iterative approach has been game-changing for us, allowing us to refine our product based on real insights and confidently validate changes before release.
Comparing experiments for making data-driven decisions
Automating with CI/CD Pipelines
-------------------------------
To streamline this process further, we automated our evaluation workflow using a CI/CD pipeline with [Git actions](https://docs.arize.com/arize/llm-experiments-and-testing/how-to-experiments/ci-cd-for-automated-experiments)
. We wrote experiment scripts for both the router and skills tests and added a workflow file to our repo. Now, every time a push is made to the AI Search code directory, these experiments run automatically. Just like any other Git check, the experiment will either pass or fail. If the mean score of the evaluators is below 0.7, the check fails.
This automated setup gives us confidence that any change we make can be tested thoroughly and with minimal manual effort. We plan to implement experiments across all our skills to ensure we understand the impact of every change we make.
Continuous Monitoring and Troubleshooting
-----------------------------------------
The evaluators we built for our experiments are also used as evals running in production via online jobs. This means we have continuous coverage—both during iteration and once a skill is live. If a skill starts to underperform, we typically notice an increase in failed evals. When that happens, we filter by the skill or trace type, dig into the traces, and begin troubleshooting.
The process is straightforward: once we identify an issue—whether with the prompt or function—we go directly from a trace to the Prompt Playground. The Playground is automatically loaded with the prompt template and variables, allowing us to quickly make edits and test if we can resolve the problem. If we have a dataset of problematic traces, we can load it into the Playground to test the new prompt across all examples, ensuring we’ve addressed the issue.
If the problem is with function calling, the Playground helps here too. We can load our router prompt and tool definitions, adjust them as needed, and test the router’s ability to call the correct function with the right arguments. Being able to replicate our exact environment within the Playground gives us confidence in the changes we make before pushing them to production.
Bringing It All Together: Code, Commit, Iterate
-----------------------------------------------
Once we’ve run our tests and are satisfied with the results, we commit the code, kicking off the entire process we’ve outlined. From Phoenix traces and Arize dashboards to experiments and the Prompt Playground, these tools allow us to continuously iterate, improve, and refine Copilot. Without them, we couldn’t operate with the same level of confidence or velocity.
Development isn’t a linear process. We use these workflows as essential parts of both development and production, constantly looking for ways to improve the user experience. By sharing these workflows, we hope to inspire you to take a similar approach in your own development processes.
Want hands-on experience rigorously evaluating AI agents? Take the **Evaluating AI Agents** short course from DeepLearning.AI and Arize [here](https://www.deeplearning.ai/short-courses/evaluating-ai-agents/?utm_campaign=arize-launch&utm_medium=partner&utm_source=arize)
and level up your skills today!
_This post is sponsored by [Arize](https://arize.com/)
. We thank the [Arize](https://arize.com/)
team for their ongoing support of the community._
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How Do Professionals Use AI Tools for Personal Productivity? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How Do Professionals Use AI Tools for Personal Productivity?
How Do Professionals Use AI Tools for Personal Productivity?
============================================================
### Results of our DataTalks.Club Survey
10 Apr 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
We surveyed 300 DataTalks.Club community members, primarily professionals in data, machine learning, and software engineering, to understand how AI tools are integrated into daily workflows and impact personal productivity.
Our survey form
In this article, we present key findings on usage patterns, application areas, and emerging trends among technical professionals.
Introduction
------------
AI tools are increasingly integral to both personal and professional activities. While many professionals enjoy the efficiency gains these tools offer, a subset of respondents also expressed concerns about potential overreliance.
This survey sheds light on how technical professionals use AI, which tools are most prevalent, and the tangible impacts on productivity.
Let’s explore what we’ve found out!
AI Tools Integration
--------------------
Our survey shows that AI is now a routine part of daily life.
Most community members engage with AI tools daily
Key findings include:
* **Daily usage:** About 70% of respondents use AI tools every day, both at work and at home.
* **User maturity:** A majority (70%) have been using AI for over a year, with roughly 40.2% using it for 1–2 years and 30.3% for more than 2 years. This indicates a mature user base that has incorporated AI into routine tasks. These users feel at ease with AI tools and rely on their capabilities.
70% use AI for a year or more
This sustained usage reflects increasing market maturity and familiarity, moving beyond early adoption to broader, long-term integration.
Primary Use Cases for AI
------------------------
AI is most commonly used for coding and research assistance
Given the technical focus of our community, the AI applications include:
* **Coding assistance (87.7%)**: AI tools are extensively used to generate code, debug, and improve overall efficiency.
* **Research assistance (72.3%)**: Many professionals rely on AI to quickly gather and summarize information.
* **Brainstorming and personal productivity:** Interestingly, brainstorming (68.8%) and personal productivity tasks (58.9%) are the next largest use cases.
* **Content generation (46.6%):** Nearly half of the respondents use AI to streamline content creation.
* **Data analysis (39.9%):** Data analysis is less popular, with only 40% of people using AI tools for this task, which likely highlights that the reasoning capabilities of AI tools are still developing.
Tools
-----
We see that ChatGPT dominates the market, but we also use other tools such as Claude or Gemini.
### Chat-Based Tools
Most respondents use ChatGPT by a wide margin
While the market for chat-based AI tools is diversifying, a few key players continue to dominate it.
* **ChatGPT:** Leads the market with 92.1% usage among respondents.
* **Complementary tools:** **Google Gemini** and **Anthropic Claude** are used by smaller segments, often as complementary tools **alongside ChatGPT** rather than as stand-alone solutions.
* Other platforms, like Perplexity or Copilot, trail behind.
### AI Integration into IDEs
AI-driven coding assistance is becoming standard practice in the tech community. People use AI tools in **technical workflows** rather than just for general productivity tasks.
GitHub Copilot is the most popular AI development tool
Among developer-focused tools:
* **GitHub Copilot:** It is the most popular, with **77.9% of respondents utilizing it.** This popularity likely stems from its development by the widely recognized platform GitHub, which is used by nearly everyone in the tech community.
* At the same time, **newer and less popular applications like Cursor** still maintain a user base of 20% and are likely to grow.
### Additional AI Tools
When asked about additional AI tools, respondents mentioned using a **wide variety of niche tools** for tasks such as image generation, voice synthesis, and even custom frameworks.
Notable mentions include:
* Advanced code generation and debugging beyond standard IDE plugins.
* Image generation (e.g., DALL-E) and voice synthesis.
* Specialized platforms for search, summarization, and home automation.
These free-form responses illustrate that professionals are experimenting with a diverse ecosystem to meet specific needs instead of relying on popular AI applications for all tasks.
Impact on Productivity
----------------------
In general, how has AI impacted the lives of our community members?
According to their responses, AI integration has been beneficial for nearly everyone, with the main impacts being:
* **Efficiency, time-saving, and productivity:** AI reduces the time required for routine tasks, leading to faster work completion.
* **Better focus:** Outsourcing routine tasks to concentrate on higher-value activities.
* **Improved communication and documentation:** AI assists with drafting emails, technical documents, and specifications.
Here’s a summary table of the main insights regarding the impact of AI and some quotes from our respondents:
| | | |
| --- | --- | --- |
| Significant increase in productivity | AI dramatically increases productivity, often by reducing task completion times. | "10x lol"
"Saved at least 1 week of work"
"Doubled my productivity" |
| Time savings & efficiency gains | AI reduces the time spent on routine tasks, research, and debugging, allowing focus on higher-value activities. | "It has sped things up substantially"
"I have recorded tremendous improvement in my delivery speed" |
| Better coding & technical workflows | AI assists with code generation, debugging, and learning new languages, streamlining the development process. | "Efficient coding"
"Helps to refactor code and find solutions"
"Faster code suggestions. Debugging made fast" |
| Learning & ideation support | AI accelerates learning and ideation by providing quick insights, creative suggestions, and educational support. | "Learn much faster"
"It has allowed me to come up with test cases for my code"
"Helps me understand concepts rapidly and improve my communication" |
| Improved communication & documentation | AI improves the quality and speed of drafting emails, technical specifications, and other documentation tasks. | "Craft better emails"
"Write the first drafts of technical specifications"
"Improves documentation" |
Main Challenges and Future Opportunities
----------------------------------------
1. **Adoption barriers:** Many professionals use AI for coding and research. However, embedding these tools into broader workflows remains challenging. The wide variety of available tools complicates interoperability. This could be an opportunity for developers to improve integration and user experience.
2. **Quality concerns:** Many users still struggle to obtain high-quality, contextually relevant outputs. As AI becomes more critical in decision-making and high-stakes environments, achieving reliable performance is essential.
Conclusion
----------
Our survey shows that AI tools are a key part of the professional toolkit for technical experts, driving significant gains in efficiency and productivity. Although integration challenges and quality issues persist, the benefits are clear. Continued innovation in these areas will likely lead to even broader and more effective use of AI in professional settings.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Winning Solutions from the LLM Zoomcamp 2024 Competition – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Winning Solutions from the LLM Zoomcamp 2024 Competition
Winning Solutions from the LLM Zoomcamp 2024 Competition
========================================================
### Solving High School Mathematics Problems with LLMs
21 Oct 2024 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
In October 2024, we announced the winners of the [LLM Zoomcamp 2024 Competition](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/overview)
, where participants tackled high school math problems using large language models (LLMs). This challenge was part of our [LLM Zoomcamp](https://github.com/DataTalksClub/llm-zoomcamp)
, a free course focused on real-world LLM applications.
LLM Zoomcamp 2024 Competition [(source)](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/overview)
.
In this article, we’ll share insights from the competition and spotlight some of the top solutions. Here’s what we’ll cover:
* [Overview of the LLM Zoomcamp 2024 Competition](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#overview-of-the-llm-zoomcamp-2024-competition)
* [Dataset Description](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#dataset-description)
* [Artur G’s Solution: Combining Code Generation with Chain-of-Thought Reasoning](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#artur-gs-solution-combining-code-generation-with-chain-of-thought-reasoning)
* [Blaqadonis’ Solution: Agent-based Approach Leveraging LangGraph](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#blaqadonis-solution-agent-based-approach-leveraging-langgraph)
* [Vladyslav Khaitov’s Hybrid Model Approach](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#vladyslav-khaitovs-hybrid-model-approach)
* [Slava Shen’s Solution: Blending Results with Data-Driven Logic](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html#slava-shens-solution-blending-results-with-data-driven-logic)
### Overview of the LLM Zoomcamp 2024 Competition
A sample Russian ЕГЭ exam answer sheet [(source)](https://en.wikipedia.org/wiki/Unified_State_Exam)
.
The competition tasked participants with solving high school-level math problems from the [Russian ЕГЭ exam](https://en.wikipedia.org/wiki/Unified_State_Exam)
using LLMs. The ЕГЭ exam is a standardized test required for admission to Russian universities and professional colleges.
We provided participants with the original problem statements in Russian and their English translations, generated by GPT-4. The goal was to solve these problems using LLMs without manual intervention.
### Dataset Description
A screenshot of the train data, source: train.csv
The dataset for the competition consisted of several CSV files containing training and test data. Key components of the dataset included:
* Training Data: Contained problem statements in English and Russian, answers, and hints. The data also included unchecked entries that might contain errors.
* Test Data: Required participants to predict the correct answers based on the problem statements in both languages.
* Sample Submission File: Offered guidance on the expected format for final submissions.
Now, let’s take a closer look at the winning solutions and the approaches behind them.
### Artur G’s Solution: Combining Code Generation with Chain-of-Thought Reasoning
LLM Zoomcamp leaderboard [(source)](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/leaderboard)
.
Artur G’s [solution](https://github.com/ArturGR3/LLM-kaggle-competition)
used Claude-3.5 Sonnet’s natural language understanding and Python code generation capabilities. He applied the Zero-shot Chain-of-Thought technique to prompt Claude-3.5 Sonnet to reason through problems and created two solutions:
* Solution 1: Reasoning without code.
* Solution 2: Reasoning supported by Python code execution.
To select the final solution, Artur compared the results from both approaches. If they matched, the answer was considered correct. If they didn’t, an additional reasoning step was performed to refine the answer.
Artur’s solution also included several safeguards:
* **[Instructor](https://github.com/jxnl/instructor)
validation** to ensure the Python code was executable.
* A **timeout mechanism** to prevent infinite loops.
* **Error handling** that allowed for retries in case of code failures.
Here are Artur’s key takeaway from this competition:
> One might wonder if LLMs are useful to solve math problems. My answer is yes; the techniques I used, such as structured outputs, chain of thought reasoning, multithreading, and error handling with retry mechanisms, can elevate most LLM-based applications to the next level.
### Blaqadonis’ Solution: Agent-based Approach Leveraging LangGraph
A screenshot of a notebook with Blaqadonis' solution [(source)](https://colab.research.google.com/drive/1EtEePcBQUQ731c4QARKcJuVyGERbxGAI?usp=sharing)
.
Blaqadonis’s [solution](https://colab.research.google.com/drive/1EtEePcBQUQ731c4QARKcJuVyGERbxGAI?usp=sharing)
used an agent-based approach, coordinating two models: OpenAI’s GPT-4 Omni and Meta’s Llama 3.1 70B through LangGraph, a low-level framework for building stateful AI applications. His solution achieved high accuracy at a cost of less than $3 per test run.
Blaqadonis used the Groq API to handle rate limits and avoid throttling. With a leaderboard score of 96.25%, his solution showcased the scalability and cost-effectiveness of agentic systems in solving complex problems. He also noted some challenges in the reliability of such systems.
Reflecting on his experience, Blaqadonis said:
> What impressed me most was the leap in reasoning capabilities from GPT-3.5 Turbo to the current models. This improvement was reflected in the leaderboard scores: from 90% to 96.25% using agents. Among the latest models, there was a significant gap between the GPT-4o-mini and the original GPT-4o.
### Vladyslav Khaitov’s Hybrid Model Approach
A screenshot of Vladyslav Khaitov's description of their solution. [(source)](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/discussion/540578)
.
Vladyslav Khaitov’s [solution](https://www.kaggle.com/competitions/llm-zoomcamp-2024-competition/discussion/540578)
combined open-source [NuminaMath-7B-TIR-GPTQ](https://huggingface.co/AI-MO/NuminaMath-7B-TIR-GPTQ)
with GPT-4 Omni to balance cost and performance. His hybrid approach used different models for different tasks, achieving high accuracy and cost efficiency.
Vladyslav initially adapted a [solution](https://www.kaggle.com/code/lewtun/numina-1st-place-solution/notebook)
from the [AI Mathematical Olympiad competition](https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize)
, which used the [NuminaMath-7B-TIR-GPTQ](https://huggingface.co/AI-MO/NuminaMath-7B-TIR-GPTQ)
math LLM and Python REPL. Then, for certain task types where this solution performed poorly, Vladyslav used OpenAI’s GPT-4o to handle those specific cases.
Key aspects of Vladyslav’s solution included:
* **Majority voting**: Running models multiple times and selecting the most common solution, a form of ensembling for LLMs. Vladyslav noted that while this required many runs, it significantly improved results.
* **Extensive post-processing**: Much of the post-processing came from the Numina solution, but Vladyslav added more to accommodate differences in data and task types.
* **Prompt experimentation**: Minimal prompts worked best for Numina, while structured prompts like “Please reason step by step, and put your final answer within `\\boxed{}`” improved results for GPT-4o.
Vladyslav shared his conclusions from the competition:
> Many open-source models (without tool use, like Mathstral) didn’t perform well enough. Even OpenAI GPT-4o-mini fell short. The difference between GPT-4o-mini and GPT-4o was significant.
### Slava Shen’s Solution: Blending Results with Data-driven Logic
A screenshot of Slava Shen’s submissions.
Slava Shen provided multiple [solutions](https://docs.google.com/document/d/1zAFJEvP3hdquuR6emN55qYnzHUC1mLY68JrxM8mD63Q/edit?usp=sharing)
, all based on GPT-4o-mini:
* Solution 1: Combined solutions 2 and 3, with mismatches resolved through randomized selection based on a hyperparameter.
* Solution 2: A refined version of an initial [Kaggle notebook](https://www.kaggle.com/code/vyacheslavshen/double-check-with-llms)
with adjustments to prompts and temperature settings.
* Solution 3: The most complex approach, which involved dynamically constructing few-shot prompts using Elastic Search and running Python code extracted from the model’s output.
Slava’s solutions demonstrated flexibility and robustness, combining different models and methodologies, resulting in a final score of 95%.
### Conclusion
The LLM Zoomcamp 2024 Competition was a great opportunity to test the limits of LLMs in solving structured mathematical problems. Participants explored diverse approaches, from blending open-source and closed-source models to using agentic systems and advanced prompt engineering techniques.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How Do Data Professionals Use Data Engineering Tools and Practices? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How Do Data Professionals Use Data Engineering Tools and Practices?
How Do Data Professionals Use Data Engineering Tools and Practices?
===================================================================
### Results of our DataTalks.Club Survey
29 Apr 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
We [surveyed](https://docs.google.com/forms/d/e/1FAIpQLScdx1FAIp2GDGgiMf7xu-I1PfhsQBJDvFstGmWmWbpP4S69Zg/viewform)
over 200 data professionals involved in data engineering tasks to understand which tools, platforms, and methodologies they rely on.
Our survey form
In this article, we share key findings and major trends in storage, orchestration, integration, and more.
Data Storage Solutions
----------------------
Organizations rely on a mix of established and emerging storage options. Traditional relational databases are still dominant, while newer architectures, such as data lakehouses, are rapidly gaining ground.
* **Relational Databases:** Employed by 70.9% of respondents, they remain the go-to solution for structured, transactional workloads.
* **Data Warehouses & Lakehouses:** Both are used by 53.6% of respondents, indicating a trend toward combining structured storage with modern, flexible architectures.
* **Data Lakes:** Reported by 49.8%, continuing to serve as a repository for raw data ingestion.
* **Lower Usage:** NoSQL (27.8%), vector databases (18.6%), and niche options like MinIO (0.8%) are less common, reflecting their more specialized or emerging status.
\[DATA\]
Which data storage solutions do you use? (Select all that apply)
* **Relational Databases** – 168 (70.9%)
* **Data Warehouses** – 127 (53.6%)
* **Data Lakehouses** – 127 (53.6%)
* **Data Lakes** – 118 (49.8%)
* **NoSQL Databases** – 66 (27.8%)
* **Vector Databases** – 44 (18.6%)
* **MinIO** – 2 (0.8%)
* Others (Salesforce, HDFS, “Not Sure”) – each <1%
\[DATA\]
Relational databases are a cornerstone for many organizations, given their maturity and robustness for transactional workloads. The near parity in adoption between data warehouses and data lakehouses indicates a trend toward converging traditional structured storage with more flexible, modern architectures.
While data lakes remain popular for raw, unprocessed data, the rise of lakehouses points to an industry move toward unifying the benefits of both warehouses and lakes. The relatively lower usage of niche solutions like vector databases suggests that while they receive buzz in certain AI/ML circles, they have yet to achieve widespread adoption in day-to-day operations.
Data Warehouses
---------------
Data warehouses are key for business intelligence and analytics. Major cloud vendors dominate the landscape, providing fast query performance and advanced integration.
* **Google BigQuery:** Leads with 38.6%, benefiting from its seamless integration in the Google Cloud ecosystem.
* **Snowflake:** Used by 32.1%, showing strong competition in the cloud analytics space.
* **Amazon Redshift (25%) and Azure Synapse Analytics (20.1%):** Also significant, reinforcing the presence of diversified cloud vendor solutions.
* **Others:** Smaller players (e.g., ClickHouse at 8.2%) indicate that while there’s room for specialized solutions, the market is largely consolidated around the major vendors.
\[DATA\]
Which data warehouse solutions do you use?
* **Google BigQuery** – 71 (38.6%)
* **Snowflake** – 59 (32.1%)
* **Amazon Redshift** – 46 (25%)
* **Azure Synapse Analytics** – 37 (20.1%)
* **ClickHouse** – 15 (8.2%)
* Others (Vertica, Motherduck, Databricks, Greenplum) – each <4%
\[DATA\]
The survey data shows Google BigQuery as the leading choice, which may be attributed to its integration within the broader Google Cloud ecosystem that is well-regarded for data science and machine learning workloads. Snowflake and Amazon Redshift also enjoy strong adoption, reflecting a competitive market where multiple cloud vendors address analytical needs. The presence of Azure Synapse Analytics and other smaller platforms further demonstrates that organizations weigh considerations such as integration with existing cloud services and pricing structures when choosing.
Data Lakes and Lakehouses
-------------------------
### Data Lakes
Cloud-based storage services are the primary choice for handling raw and unprocessed data, reflecting a shift towards managed, scalable solutions.
* **Amazon S3:** Dominates with 52.8%, followed by
* **Google Cloud Storage:** 34.2%, and
* **Azure Data Lake Storage:** 30.6%.
* Legacy systems like Apache Hadoop (HDFS) are present but on a smaller scale (19.2%), emphasizing an industry-wide shift to cloud-native options.
\[DATA\]
Which data lake solutions do you use? (Select all that apply)
* **Amazon S3** – 102 (52.8%)
* **Google Cloud Storage** – 66 (34.2%)
* **Azure Data Lake Storage** – 59 (30.6%)
* **Apache Hadoop (HDFS)** – 37 (19.2%)
* **MinIO** – 3 (1.6%)
* Other/None – minimal
\[DATA\]
Cloud object storage services dominate as data lakes of choice, emphasizing their scalability, durability, and ease of integration with various processing engines. The significant adoption of services like Amazon S3 and Google Cloud Storage underscores the importance of cloud infrastructure for modern data operations. The relatively lower usage of on-premise or legacy systems (like HDFS) hints at a broader industry transition toward fully managed, cloud-native environments.
### Lakehouse Architectures
While still emerging, lakehouse architectures are being trialed by a subset of respondents, with Databricks at the forefront.
* **Databricks:** Used by 31.3% of survey participants, making it the leading lakehouse solution.
* **Other Technologies:** Apache Iceberg (13%), Delta Lake (12.5%), and Apache Hudi (2.6%) have more modest adoption.
* **Adoption Status:** Over half (58.3%) report not using any lakehouse solutions, suggesting that many organizations are either in a pilot phase or see limited current need for this hybrid model.
\[DATA\]
Do you use any lakehouse architecture solutions? (Select all that apply)
* **Databricks** – 60 (31.3%)
* **Apache Iceberg** – 25 (13%)
* **Delta Lake** – 24 (12.5%)
* **Apache Hudi** – 5 (2.6%)
* **Not using any Lakehouse** – 112 (58.3%)
\[DATA\]
Although lakehouse concepts are gaining traction, over half of the survey respondents still do not utilize this architecture, which may suggest a cautious approach toward new methodologies or a limited requirement in some organizations. Databricks leads among lakehouse adopters, benefiting from its well-integrated ecosystem and community support. The relatively modest adoption rates for Apache Iceberg, Delta Lake, and Apache Hudi reflect an experimental phase where many organizations still evaluate the benefits versus the complexity of a lakehouse approach.
Workflow Orchestration
----------------------
Workflow orchestration is crucial for managing data pipelines, but its adoption varies based on team complexity and project scale.
* **Apache Airflow:** The dominant player with 48.3%, favored for its mature ecosystem and community support.
* **Other Tools:** AWS Step Functions (12%), Mage (7.2%), Prefect (6.7%), and Dagster (4.8%) serve niche or emerging needs.
* **No Orchestration:** Notably, 35.9% of respondents do not utilize any orchestration tool, which might reflect simpler workflows or resource constraints in smaller teams.
\[DATA\]
Which workflow orchestration tools do you use to manage data pipelines? (Select all that apply)
* **Apache Airflow** – 101 (48.3%)
* **AWS Step Functions** – 25 (12%)
* **Prefect** – 14 (6.7%)
* **Mage** – 15 (7.2%)
* **Dagster** – 10 (4.8%)
* **No orchestration used** – 75 (35.9%)
* Others (Beam, GitHub Actions, Jenkins) – each <2%
Apache Airflow is the dominant player, likely because of its mature ecosystem and widespread community support. However, the notable fraction of respondents not using any orchestration tools suggests that smaller teams or straightforward workflows may not yet require the overhead of dedicated scheduling systems. The varied adoption of alternatives like Prefect and Dagster indicates ongoing experimentation with newer and sometimes more streamlined orchestration solutions.
Data Integration (ETL/ELT)
--------------------------
In data integration, modular and SQL-driven approaches are gaining favor, although many organizations still rely on custom or manual solutions.
* **dbt:** Leads at 33.5%, indicating strong momentum behind this modern transformation tool.
* **Other Tools:** Airbyte (8.4%), Fivetran (7.9%), and dlt (6.9%) are present in a smaller share, while Talend registers 5.4%.
* **Manual/Custom Approaches:** Almost half (46.3%) do not use formal ETL tools, highlighting both the persistence of legacy practices and opportunities for greater automation.
\[DATA\]
Which data integration or ETL/ELT tools do you use? (Select all that apply)
* **dbt** – 68 (33.5%)
* **Airbyte** – 17 (8.4%)
* **dlt** – 14 (6.9%)
* **Fivetran** – 16 (7.9%)
* **Talend** – 11 (5.4%)
* **No ETL tools used** – 94 (46.3%)
* Others (custom Python scripts, self-built) – many mentioned
\[DATA\]
The adoption of dbt reflects its growing popularity and ease of use in transforming data at scale, reinforcing the trend towards modular, SQL-driven approaches to data engineering. Yet, almost half of the respondents still depend on custom or manual solutions, which implies a clear maturity gap in tooling. This reliance on bespoke methods may also signal opportunities for improved automation and integration frameworks, especially in organizations lacking dedicated data teams.
Data Processing Frameworks
--------------------------
Data processing frameworks vary widely in scale. Many professionals favor robust, well-known libraries for their flexibility and ease of use.
* **Pandas:** The leading tool for data manipulation at 69.5%, reflecting its simplicity for handling small- to medium-sized tasks.
* **Apache Spark:** Adopted by 46.6%, especially where scalable, distributed processing is required.
* **Other Options:** Apache Flink (7.6%), Dask (4.5%), and Apache Beam (4.9%) have smaller footprints. Notably, 14.8% report not using any formal processing framework, which may indicate reliance on ad hoc methods or simpler workflows.
\[DATA\]
Which frameworks do you use for data processing? (Select all that apply)
* **Pandas** – 155 (69.5%)
* **Apache Spark** – 104 (46.6%)
* **Apache Flink** – 17 (7.6%)
* **Dask** – 10 (4.5%)
* **Apache Beam** – 11 (4.9%)
* **Polars/DuckDB** – mentioned but minimal
* **No processing frameworks used** – 33 (14.8%)
\[DATA\]
Pandas remains the go-to tool for many professionals, primarily due to its intuitive interface and versatility for smaller to medium-sized data tasks. Meanwhile, Apache Spark’s adoption for scalability in handling big data workloads confirms its integral role in more demanding environments. The smaller footprints of newer or niche frameworks (like Flink, Dask, or Beam) suggest they are still establishing themselves as reliable alternatives in a competitive ecosystem.
Observability and Monitoring
----------------------------
Ensuring data quality and pipeline reliability is critical, yet formal observability solutions are still underutilized. A majority (76.6%) of respondents do not use dedicated observability tools.
Among those that do, Great Expectations (10.3%) and Monte Carlo (6%) are the most noted, with only minor representation from Soda.io and Databand (each 2.7%).
\[DATA\]
Do you use any data observability or monitoring tools for your pipelines? (Select all that apply)
* **Great Expectations** – 19 (10.3%)
* **Monte Carlo** – 11 (6%)
* **Soda.io** – 5 (2.7%)
* **Databand** – 5 (2.7%)
* **No observability tools** – 141 (76.6%)
* Others (custom scripts, Grafana) – <3% each
\[DATA\]
The lack of observability tool adoption suggests that many organizations are either early in maturing their data infrastructure or rely on ad hoc methods. This gap represents a significant area for improvement, as enhanced observability could lead to more proactive issue resolution and optimized system performance.
Cloud Platforms
---------------
Cloud adoption remains robust in data engineering, with the choice of platform often reflecting broader enterprise strategies and legacy considerations.
* **AWS:** Leads with 46.7%, underscoring its comprehensive service offerings and established market presence.
* **Azure and Google Cloud (GCP):** Hold significant shares at 35.2% and 33.8% respectively, illustrating competitive alternatives within the cloud sphere.
* **On-Premise:** Still in use by 21.9% of respondents, indicating that despite the cloud’s advantages, traditional infrastructures continue to serve essential roles in some organizations.
\[DATA\]
Which cloud platforms do you use for data engineering workloads? (Select all that apply)
* **AWS** – 98 (46.7%)
* **Azure** – 74 (35.2%)
* **Google Cloud (GCP)** – 71 (33.8%)
* **On-Premise** – 46 (21.9%)
* Others (Yandex, Huawei) – very limited
\[DATA\]
AWS leads in adoption, reflecting its long-standing prominence and breadth of services in the cloud ecosystem. Azure and GCP also maintain substantial market shares, driven by their unique strengths and integration with enterprise products. A notable segment still uses on-premise solutions, indicating that while cloud adoption is high, there remains a dependency on traditional infrastructures in certain contexts or regulatory environments.
Data Governance
---------------
Data governance remains a weak link for many teams. Automated tooling is lagging, with many organizations relying on manual methods.
* **Manual Cataloging:** Used by 20.1% of respondents.
* **Formal Tools:** Apache Atlas (6.3%), Collibra (5.3%), and Alation (4.2%) see limited use.
* **No Governance Tooling:** A substantial majority (64.6%) are not using any specialized governance tools, pointing to an area ripe for improvement as data ecosystems become more complex.
\[DATA\]
Which data governance tools or practices do you use? (Select all that apply)
* **Manual Cataloging** – 38 (20.1%)
* **Apache Atlas** – 12 (6.3%)
* **Collibra** – 10 (5.3%)
* **Alation** – 8 (4.2%)
* **Not using any governance tooling** – 122 (64.6%)
* Others (Atlan, OpenMetadata, DataHub) – very limited
\[DATA\]
The survey reveals a significant reliance on manual cataloging or a complete lack of governance tooling, highlighting an area where many organizations are lagging behind. Without automated governance frameworks, data quality, security, and compliance may be compromised, particularly as data ecosystems become complex. These gaps call for more robust and accessible solutions that can integrate seamlessly with existing workflows.
Real-Time Data Processing
-------------------------
There is a split in the approach to real-time processing, some teams invest in dedicated frameworks, while many continue with batch processing.
* **Dedicated Frameworks:** 26.7% actively use specialized tools for real-time data processing.
* **Minimal Real-Time Use:** An additional 28.1% engage in real-time processing on a limited basis.
* **No Real-Time Processing:** 47% of respondents have not adopted any real-time strategies, suggesting that for many, batch processing remains adequate.
\[DATA\]
Do you work with real-time data processing?
* **Yes, using dedicated frameworks** – 58 (26.7%)
* **Yes, minimally** – 61 (28.1%)
* **No** – 102 (47%)
\[DATA\]
Approximately half of the respondents do not engage in real-time processing, suggesting that for many organizations, batch processing remains sufficient for their current needs. However, the considerable share using dedicated or minimal real-time frameworks indicates that a segment of the industry is investing in faster, near-real-time analytics, potentially a precursor to more widespread adoption as data velocity increases.
Data Quality
------------
Data quality practices are mixed. Organizations are shifting toward automation, yet manual oversight still predominates.
* **Manual Checks:** Rely on human oversight (48.8%).
* **Automated Tests:** Used by 39.2%, showing a trend toward reducing manual intervention.
* **Validation Tools:** Such as Great Expectations, employed by 22.6%.
* **No Quality Practices:** Alarmingly, 26.7% do not implement any data quality measures, highlighting potential vulnerabilities.
\[DATA\]
How do you ensure data quality in your workflows? (Select all that apply)
* **Manual Checks** – 106 (48.8%)
* **Automated Tests** – 85 (39.2%)
* **Validation Tools (e.g., Great Expectations)** – 49 (22.6%)
* **No practices** – 58 (26.7%)
\[DATA\]
The prevalence of manual checks suggests that many teams rely on human oversight for data quality, a practice that is both time-consuming and prone to error. Adopting automated tests and validation tools shows a positive shift toward streamlining these processes. However, the notable share of organizations with no data quality practices underscores a critical vulnerability that could lead to operational inefficiencies or misinformed decisions.
Key Challenges
--------------
The survey identifies common hurdles that data engineering teams confront daily. The most significant challenge echoes through many facets of the data lifecycle.
* **Data Quality:** The top challenge at 68.8%, consistently affecting multiple stages of operations.
* **Data Integration:** Cited by 59.6%, reflecting difficulties in consolidating diverse data sources.
* **Scaling Pipelines:** A concern for 53.8% of respondents, emphasizing issues around growing data volumes.
* **Security and Compliance:** Reported by 40.4%, underscoring the increasing importance of regulatory and security measures.
\[DATA\]
What are the primary challenges you face in data engineering? (Select all that apply)
* **Data Quality** – 143 (68.8%)
* **Integration of Data Sources** – 124 (59.6%)
* **Scaling Pipelines** – 112 (53.8%)
* **Security and Compliance** – 84 (40.4%)
\[DATA\]
The most cited challenge, data quality, resonates throughout nearly every stage of the data lifecycle. Integration issues and scaling challenges further illustrate that while many organizations have embraced digital transformation, they still contend with legacy systems and infrastructural constraints. Security and compliance, while secondary in this survey, remain critical given the increasing regulatory scrutiny and the high stakes of data breaches. These challenges underscore the need for more robust, integrated solutions to address technical and organizational hurdles.
Team Sizes
----------
The survey reveals that many data operations are driven by small teams or solo practitioners.
* **Small Teams (1–5 members):** The largest group, with 117 teams.
* **Solo Practitioners:** 55 professionals operate independently.
* **Larger Teams:** There are fewer teams with 6+ members, which may explain the conservative approach toward adopting more complex, automated systems.
\[DATA\]
How many people are in your data engineering team(s)?
* **1–5 members** – 117
* **6–10** – 21
* **11–20** – 19
* **21–50** – 8
* **51+** – 12
* **Solo (0)** – 55
\[DATA\]
The data indicates that most teams are small, with many professionals operating in solo or very small groups. This size limitation likely contributes to a cautious approach toward adopting more complex infrastructures—such as automated observability stacks, advanced orchestration systems, or comprehensive lakehouse architectures. The lean nature of most teams stresses the importance of tools that are not only powerful but also easy to implement and maintain.
Conclusion
----------
The survey offers a clear perspective on where the industry stands today and highlights critical areas for potential improvement as organizations scale their data operations.
### Takeaway 1: Adoption of Mature vs. Emerging Technologies
Established tools like relational databases, Apache Airflow, and Pandas are widely used and form the backbone of many operations.
Emerging trends include the movement toward lakehouses and automated data quality and governance practices, areas where adoption is still in progress.
### Takeaway 2: Resource Constraints Drive Tool Choices
The prevalence of small teams and solo practitioners explains the reliance on tools that are both effective and straightforward to implement.
In contrast, the limited use of observability and automated governance indicates potential growth areas as organizations scale.
### Takeaway 3: Fragmentation and Opportunity
While cloud-based solutions are almost universal, the fragmentation in tooling for orchestration, governance, and real-time processing suggests ample opportunities for more unified and automated frameworks.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp
How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp
=========================================================================================
### Learn what you can build after ML Zoomcamp based on a real project from one of the graduates
11 Aug 2025 by [Alexander Daniel Rios](https://datatalks.club/people/alexanderdanielrios.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
In one of our previous articles, we covered the [waste classifier](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
built by [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
graduate Serena Haidar. This time, another graduate, Alexander Daniel Rios, walks through his final project: an end-to-end tool that segments and classifies blood cells from microscope images to assist in detecting signs of acute lymphoblastic leukemia (ALL).
Below, Alexander explains his approach, from problem framing and data preparation to modeling choices, evaluation, and deployment, and what it takes to make the system usable outside a notebook.
> [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
> is a free, four-month online course on core machine learning and taking models to production. A key requirement is an end-to-end capstone/final project that turns course concepts into a working system.
ML Zoomcamp Contribution
------------------------
Before the course, my projects were scattered: messy notebooks, ad-hoc experimentation, and model switching without proper validation. ML Zoomcamp gave me the structure I was missing. The recorded lessons, assignments, and projects pushed me to build a professional workflow.
What I gained:
* Methodical data preparation and reproducible pipelines
* Proper validation for regression and classification
* Sensible metric selection tied to use cases
* Practical confidence with neural nets and transfer learning
* Lightweight deployment paths (Flask, AWS Lambda)
My Final Projects
-----------------
I applied everything he had learned from ML Zoomcamp into several projects:
* Capstone 1 – [**Predicting cancer in blood cells**](https://github.com/aletbm/Blood_Cell_Cancer_Prediction)
: the focus of this article.
* Capstone 2 – [**Classifying disaster tweets**](https://github.com/aletbm/NLP_with_Disaster_Tweets)
: Another project using spaCy, TensorFlow, and BERT-based models for NLP. I [covered it separately](https://www.notion.so/Natural-Language-Processing-using-spaCy-TensorFlow-and-BERT-model-architecture-1895067176b380d09484d4b0338b0c5e?pvs=21)
.
* Kaggle Competition – [**ML Zoomcamp 2024 Retail Forecasting**](https://www.kaggle.com/code/aletbm/top4-retail-demand-forecast-mlzc-compet-24)
: A challenge to predict product demand for the next month using 25 months of historical retail data across multiple stores. My final model achieved 4th place globally, among dozens of participants.
In the rest of this piece, I walk through the blood-cell cancer project end-to-end.
Predicting Cancer in Blood Cells
--------------------------------
The goal of my project was to develop an **ML system** capable of **automatically classifying and segmenting blood cells** from microscopic images to detect the signs of acute lymphoblastic leukemia (ALL), a type of cancer that primarily affects the blood and bone marrow, particularly in children.
Diagnosis of such patients still relies heavily on microscopic analysis by trained specialists. That process is complex, subjective, and time-consuming. It also assumes consistent access to experienced pathologists and well-equipped laboratories, resources that are not uniformly available. In practice, this makes it difficult to achieve a timely and consistent diagnosis in many regions.
Within this context, an automated approach like the one developed here is intended as a support tool. It can help standardize the routine parts of analysis, reduce turnaround time, and provide clear visual evidence for review, contributing to faster triage and more efficient patient care.
### My Motivation
I’ve always been drawn to the application of data science in areas related to human health. Building a system that supports fast, efficient, and accurate diagnosis and can be used anywhere is my core motivation.
The system focuses on turning raw images into clear, structured outputs that a clinician can review: where the cells are, which ones look suspicious, and a concise summary of the findings.
### Key Goals of this Project
The goal of this project is to develop an **end-to-end** system that delivers accurate predictions through an intuitive and accessible interface. Specifically:
* Automate cell classification with high accuracy
* Reduce sample analysis time
* Visually identify affected regions
* Make the technology accessible through a simple interface
* Deploy the system in a scalable way across different environments
### Tools and Technologies Used
Here’s a workflow behind this project with the tools used in each step:
* Develop and evaluate models in Python with TensorFlow/Keras
* Use OpenCV/Matplotlib and scikit-learn for preprocessing, analysis, and metrics
* Package the model and services with Docker
* Orchestrate locally with Kubernetes/Kind
* Serve the trained model via TensorFlow Serving
* Expose predictions through a REST API built with Flask/Waitress
* Present results in a simple Streamlit web app.
My resulting project combines deep learning models, computer vision techniques, and an accessible user interface, providing an end-to-end path from microscope image to clear, reviewable output.
### Dataset Selection
I used the [**Blood Cell Cancer ALL 4-Class**](https://www.kaggle.com/datasets/mohammadamireshraghi/blood-cell-cancer-all-4class)
dataset. The images were prepared at the Bone Marrow Laboratory of Taleqani Hospital in Tehran, Iran.
Screenshot of the dataset page on Kaggle
The dataset includes **3,242** **peripheral** **blood** **smear** **(PBS)** **images** from 89 patients suspected of having acute lymphoblastic leukemia (ALL). It is divided into two main classes:
* **Benign:** normal hematopoietic cells
* **Malignant:** ALL-related cells, categorized into three subtypes of malignant lymphoblasts:
* Early Pre-B
* Pre-B
* Pro-B
All images were captured using a Zeiss camera mounted on a microscope at 100x magnification and saved as JPEG files. A specialist performed the final classification of cell types and subtypes using flow cytometry. Additionally, **segmentation** **masks** were included to facilitate training of models capable of visually identifying tumor regions.
Example images from the dataset
### Preprocessing
Before training any artificial intelligence model, it is essential to perform proper **data** **preprocessing**. This step helps standardize and optimize the model’s input, enhancing its learning ability and overall performance.
A diagram of the preprocessing steps I applied before training my models
In this project, I used several preprocessing techniques:
* **Image normalization and resizing:** All images were resized to a consistent shape and format, ensuring uniformity in the number of channels and making them easier to process with neural networks.
* **Class encoding for multiclass classification:** The labels associated with each image were encoded using suitable techniques so that classification algorithms could correctly interpret them in a multiclass setting.
* **Mask generation for semantic segmentation:** Since the original dataset did not include segmentation masks, a custom script was developed to generate binary masks. This process relied on converting images to the LAB color space, specifically focusing on the _a_ channel, which distinguishes red and green hues. This helped in identifying cell nuclei more effectively.
* **Efficient serialization with TFRecord:** To enable fast and structured data loading during training, the dataset was stored using the **TFRecord** format, widely used in TensorFlow environments for its performance advantages.
* **Data augmentation:** Various augmentation techniques were applied, including:
* Rotations
* Flips
* Brightness variations
* Contrast variations
* Saturation level changes
> These augmentations increased the diversity of the training set and helped improve the model’s generalization ability.
### Model development
For this project, I developed and trained two different models, each designed for a specific task. Additionally, I explored a unified multitask learning approach.
#### Classification
Convolutional neural networks (CNNs) were trained to identify the cell type in each image. Three main approaches were compared: a baseline model, a model with data augmentation, and a model based on transfer learning using the pre-trained VGG16 architecture.
##### Baseline Model
The first was a baseline model built from scratch. A custom CNN architecture was designed with convolutional, pooling, and fully connected layers, and was trained from scratch. This model served as a baseline for comparison.
##### Improved Model with Data Augmentation
The baseline model was enhanced by incorporating **data augmentation** techniques.
##### Model Based on Transfer Learning
The third model is based on **transfer learning** using the pre-trained **VGG16** architecture. I leveraged **VGG16** deep feature extraction capabilities. The convolutional layers were kept as a feature extractor, and new dense layers were added for classification specific to this task.
Model results
The models were evaluated using metrics like accuracy, F1 score, and AUC. Results showed **accuracy** **above** **98%** and an **AUC** **of** **0.9997**, demonstrating excellent performance in the classification task.
#### Segmentation
U-Net network architecture used for the semantic segmentation task from the [original paper](https://arxiv.org/abs/1505.04597)
by Ronneberger et al.
In addition to classifying cells, I developed a **semantic** **segmentation** model to identify specific regions of lymphoblasts in images. I modified the U-Net architecture from the [original paper](https://arxiv.org/abs/1505.04597)
by Ronneberger et al. for this task. An encoder-decoder structure, similar to U-Net but without skip connections, was used and trained with paired images and binary masks. Techniques such as class balancing and data augmentation were applied to enhance the model’s robustness.
Model results
The results were very good, with a **mean** **IoU** **of** **0.93** and **pixel-wise** **accuracy** **of** **98.5%**.
#### Combined architecture
In the final stage, both tasks were integrated into a **multitask** **architecture** capable of performing classification and segmentation simultaneously. This approach resulted in a more efficient and coherent system by leveraging the intersection between the two tasks.
Model results for the classification task
* **Classification metrics**: The combined model achieved an **AUC of 0.9993**. A **confusion matrix** was also generated to analyze classification errors among cell subtypes.
Confusion matrix
* **Segmentation metrics**: The multitask model achieved a **mean IoU of 0.935**.
Model results for the segmentation task
Finally, some examples of the model’s predictions can be visualized, showing both class predictions and segmentation overlays.
Model predictions showing both class predictions and segmentation overlays
System deployment
-----------------
The solution was designed to be easily deployable, both locally and in more complex production environments. A web service was implemented using **Flask** as the main framework and **Waitress** as the WSGI server, allowing the model to be exposed via a simple local prediction API. The trained model was packaged in a **Docker** container, ensuring portability, version control, and easy distribution.
The image depicts the local deployment architecture using **Kubernetes** **with** **Kind**. Client requests are received through ports 80 or 443 and managed by an **Ingress** **Controller**, which directs them to two pods within the cluster: one hosting the API gateway on port 9696 and another running the inference model with **TensorFlow** **Serving** on port 8500. This modular setup simulates a full production environment on a local machine.
To enable efficient and scalable inference, **TensorFlow Serving** was used as a specialized server for machine learning models. The entire system was orchestrated using **Kubernetes** with **Kind**, and components were configured using declarative YAML files to manage services, deployments, load balancers, and ingress controllers.
Streamlit app
Lastly, an [interactive web interface](https://bloodcellcancerprediction.streamlit.app/)
was developed using **Streamlit**, allowing any user to upload a microscopic image and instantly receive both the predicted class and a visual segmentation of the affected regions. This makes the model accessible even to users without technical knowledge, making it applicable in real-world clinical and lab environments.
Conclusion
----------
This project was made possible by the solid theoretical and practical grounding I gained in [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
, led by Alexey Grigorev and the DataTalksClub community. The course provided a clear, structured path, from core machine-learning fundamentals to pipeline design and model deployment.
Crucially, it provided me with hands-on experience in preparing and validating datasets, selecting suitable models, and rigorously evaluating their performance. Those skills directly enabled the end-to-end system presented here.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How Do Professionals Use LLM Tools and Frameworks? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How Do Professionals Use LLM Tools and Frameworks?
How Do Professionals Use LLM Tools and Frameworks?
==================================================
### Results of our DataTalks.Club Survey
11 Apr 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
We [surveyed](https://docs.google.com/forms/d/e/1FAIpQLScdx1FAIp2GDGgiMf7xu-I1PfhsQBJDvFstGmWmWbpP4S69Zg/viewform)
over 200 professionals working with large language models (LLMs) to capture insights on adopting managed services, self-hosting options, customization practices, infrastructure choices, and integration strategies.
In this article, we present key findings on managed LLM services, customization, fine-tuning practices, infrastructure choices, and current use cases among technical professionals.
Let’s explore the data.
Managed LLM Services
--------------------
Respondents primarily rely on managed LLM services, with OpenAI clearly in the lead:
* **OpenAI** - Used by 73.4% of respondents
* **Anthropic** - Adopted by 24.5%
* **AWS Bedrock** - Selected by 10.8%
* **Groq** - Used by 12.4%
* **No managed service** - 20.7% indicated they do not use any managed LLM services
* **Other Providers** - Platforms such as Google, Azure, and IBM Watsonx have only limited adoption, indicating a high level of concentration around a few key players
OpenAI dominates the managed LLM services space.
The dominance of OpenAI suggests that its offerings are considered reliable and well-suited to a broad set of applications. The concentrated market share may also indicate the trust and maturity these services have built over time.
Self-Hosting Open-Source LLMs
-----------------------------
Regarding self-hosting, the majority (74.1%) do not run open-source LLMs on their own infrastructure.
Majority do not self-host open-source LLMs.
Among those who do:
* **vLLM:** Used by 9.4%.
* **Custom solutions:** About 8.5% have developed their own inference setups.
* Other tools like TGI, Ollama, and cortex.cpp have smaller adoption rates.
Managed services clearly dominate due to their ease of use and reduced maintenance overhead. However, the minority pursuing self-hosted solutions typically do so for greater control over the models, potentially lower costs, or to customize the system for specific needs.
This suggests that while the majority prioritize convenience, a dedicated subset is willing to invest in infrastructure to tailor performance more closely to their requirements.
Customization and Fine-Tuning
-----------------------------
Fifty percent of respondents customize their LLMs, while the other half use them as provided.
The equal split in customization practices suggests that many professionals see value in tailoring models to specific applications.
Fine-tuning is uncommon – most don't do it.
Regarding fine-tuning:
* **Fine-tuning managed services** - 11.9% engage in fine-tuning
* **Fine-tuning self-hosted models** - 15.7% do so
* **No fine-tuning** - 72.5% do not fine-tune their LLMs
However, the relatively low rates of fine-tuning indicate that significant model modification remains niche, either due to the technical complexity involved or because the out-of-the-box performance is sufficient for many tasks. Over time, we might see an increase in fine-tuning as organizations look to optimize performance for specialized use cases.
Infrastructure for LLMs
-----------------------
Over half (54.7%) of respondents find GPU provisioning not applicable, likely because they do not run extensive training or fine-tuning workloads in-house.
Most rely on cloud GPUs, especially AWS and Azure.
For those who use GPUs:
* **Cloud-based GPUs:** Providers like AWS (39%), GCP (16.2%), Azure (22.9%), and other cloud services are used.
* **On-premise/Private Data Centers:** 12% reported using dedicated hardware.
The preference for cloud-based GPU solutions underscores a trend toward scalable, on-demand computing. This avoids the substantial capital expenditures associated with owning and maintaining physical hardware while meeting performance needs. The relatively small number of organizations using on-premise solutions may indicate either cost barriers or the complexity of managing dedicated hardware.
Integration and Observability Tools
-----------------------------------
While dedicated vector databases such as Chroma and Pinecone have gained traction, the majority of respondents either rely on more traditional systems or have yet to integrate these specialized tools. This may reflect a transitional phase where organizations are still exploring the best methods for handling vectorized data in LLM-powered applications.
Elasticsearch and Chroma lead among vector DBs used.
**Usage Statistics:**
* **Do Not Use:** 58.9%
* **Elasticsearch:** 21%
* **Chroma:** 15.6%
* **Pinecone:** 11.6%
* **Pgvector:** 8%
* **Quadrant:** 6.3%
* **Weaviate:** 2.7%
* **Milvus:** 1.3%
Integration Frameworks
----------------------
More than half of respondents (58%) do not use dedicated frameworks to integrate LLM applications.
LangChain is the most common LLM integration tool.
Among those who do:
* **LangChain:** The most popular integration library (34.1%).
* **LlamaIndex:** Used by 16.8%.
* Smaller communities use AutoGen, Haystack, or alternatives.
The data indicates that while frameworks like LangChain are becoming popular, many organizations still rely on ad hoc or custom-built solutions to integrate LLM capabilities into their workflows. The evolution of standardized integration tools could simplify deployment and maintenance, facilitating broader adoption.
#### Observability Tools
73.5% of respondents do not track performance with dedicated observability solutions.
Most do not currently use observability tools for LLMs.
Among those who do, use the following tools:
* **Weights & Biases:** 12.1%
* **LangSmith:** 9.8%
The low usage of observability tools suggests that performance monitoring for LLMs is still in its early stages. As these models become more critical to operational processes, improved monitoring and analytics will likely be essential to ensure reliability and optimize system performance.
### Primary Use Cases for LLMs
Respondents employ LLMs across a range of applications. The most common use cases include:
* Question-answering on internal knowledge bases: 58.1%
* Code generation: 62.1%
* Document summarization/extraction: 55.1%
* Customer support automation (e.g., chatbots): 25.6%
* Data annotation: 17.2%
* Agentic interactions (e.g., API connections): 12.8%
* Autonomous agents for task completion: 14.3%
* Content moderation/quality control: 9.9%
* Content generation (articles, blogs, social media): 31.5%
Code generation and Q&A are top LLM use cases.
The most common applications, such as code generation, question-answering, and document summarization, highlight the focus on internal process efficiency and productivity. Although customer-facing applications like chatbots and content generation have lower adoption rates, these areas present significant growth opportunities as organizations refine their LLM implementations.
Production Deployment and Organizational Focus
----------------------------------------------
Most respondents currently have no LLM systems in production.
When asked about production systems:
* 54.3% have no LLM-based systems live.
* 27.4% report one live system.
* 16.1% have between 2-5 live systems, with very few exceeding this range.
Most organizations do not yet have a dedicated GenAI team.
We also asked whether organizations have dedicated GenAI/LLM teams.
* 75.7% reported they do not have a dedicated team
* Only 24.3% have a special team
These results indicate that while experimentation with LLMs is widespread, a majority of organizations have not yet transitioned to full-scale production deployments. The limited presence of dedicated teams suggests that LLM initiatives are still integrated into broader technology projects rather than being standalone strategic units. This points to an opportunity for organizations to develop specialized expertise as LLM applications become more central to their operations.
Challenges and Future Opportunities
-----------------------------------
* **Integration and interoperability:** The varied landscape of managed and self-hosted solutions makes seamless integration a challenge. There is an opportunity for vendors to offer more unified and user-friendly platforms that can easily connect with existing systems.
* **Customization and quality assurance:** While many organizations engage in some form of model customization, extensive fine-tuning is still rare. As LLMs play a larger role in mission-critical applications, ensuring consistent performance and quality through more advanced fine-tuning techniques will become increasingly important.
* **Production readiness:** The high proportion of organizations with no live production systems reflects the current experimental phase of LLM adoption. As organizations gain confidence in these systems, the focus is expected to shift toward scalable, production-ready deployments and robust monitoring.
Conclusion
----------
The survey reveals a clear dominance of managed LLM services and highlights a growing interest in self-hosted, open-source options.
Key conclusions include:
1. **Dominance of managed services:** With over 70% reliance on platforms like OpenAI, managed services are currently the go-to solution for many organizations, likely due to their ease of integration and proven reliability.
2. **Exploration beyond convenience:** Although managed services prevail, a significant minority is experimenting with self-hosted solutions to obtain greater control and potentially reduce costs.
3. **Customization without extensive fine-tuning:** While half of the respondents customize their models, extensive fine-tuning remains relatively niche. This suggests that for many, the balance between convenience and the need for optimal configurations is still being evaluated.
4. **Cloud-centric infrastructure:** The preference for cloud-based GPUs over dedicated hardware underlines the trend toward scalable, cost-effective computing solutions.
5. **Emerging integration and monitoring tools:** With considerable room for improvement in integration frameworks and observability tools, future developments in these areas could simplify the transition from experimental setups to robust production systems.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rahul Jain – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rahul Jain
Rahul has over 12 years of experience in data and engineering, and he’s been a manager for the last 3 years. Now he works as a data engineering manager at Siemens.
[](https://linkedin.com/in/16rahuljain)
### Events
* Becoming a Data Engineering Manager ([watch on youtube](https://www.youtube.com/watch?v=FljnbUQ796w)
)
* Modern Data Warehouse ([watch on youtube](https://www.youtube.com/watch?v=x2yNK3LlWUc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aashish Nair – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aashish Nair
Aashish Nair is a Data Engineer at dltHub and the creator of the famous dlt deployment course, where he teaches best practices for running dlt pipelines in production.
[](https://linkedin.com/in/aashish-nair)
### Events
* [From APIs to Warehouses: AI-Assisted Data Ingestion with dlt](https://luma.com/hzis1yzp)
on 17 Feb 2026
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How to Build a Waste Classifier: A Case Study from ML Zoomcamp – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How to Build a Waste Classifier: A Case Study from ML Zoomcamp
How to Build a Waste Classifier: A Case Study from ML Zoomcamp
==============================================================
### Learn what you can build after ML Zoomcamp based on the final end-to-end ML engineering project from one of the graduates.
05 Aug 2025 by [Serena Haidar](https://datatalks.club/people/serenahaidar.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
GitHub of Serena’s final project from ML Zoomcamp
How does [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
turn course content into practical ML engineering? In this case study, **Serena Haidar** walks through her final project from ML Zoomcamp: training an Xception-based image classifier on ~15,000 waste images, reaching **93.3%** test accuracy, and serving predictions via a Flask API packaged in Docker.
If you’re evaluating this free course, this article shows how a graduate moves from a notebook to a small production app—what gets built, why each step matters, and how to approach the work methodically.
> At DataTalks.Club, [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
> is a free, four-month online course on core machine learning and taking models to production. To complete the course and earn a certificate, students deliver an end-to-end final project that turns course concepts into a working system.
Interview with Serena about Her Project Idea
--------------------------------------------
### Q: How did you select the problem for your final project?
The idea behind [my project](https://github.com/Serena-github-c/Deep-Learning--Waste-Classification/tree/main)
was to accurately classify waste, ensuring that different types of waste are sent to the appropriate locations.
For example, plastics, metals, and glass should go to recycling centers. When they end up in regular trash bins, they can remain for decades, contaminating soil and harming wildlife. Organic waste like food scraps, on the other hand, should be composted. If it ends up in landfills instead, it breaks down without oxygen and releases methane, a greenhouse gas many times more potent than carbon dioxide.
These two sorting mistakes can speed up environmental damage and worsen climate change.
### Q: How did you technically approach it?
The initial task was to differentiate between biodegradable and non-biodegradable waste, then classify it further into 8 subcategories: e-waste, food waste, leaf waste, metal waste, paper waste, plastic bags, plastic bottles, and wood waste.
For classification into subcategories, I chose to use deep learning, specifically a Convolutional Neural Network (CNN) based on the Xception pre-trained model. To consider the project successful, the benchmarks included: an accuracy score of over 90% and a small model size suitable for lightweight deployment.
### Q: Who could potentially use a similar approach to waste classification in real-life settings?
This project helps companies begin automating sustainable waste management. It’s the very first link in a chain of actions that lead to improved recycling.
Here’s how the process works:
* An embedded camera (for example, an ESP32‑CAM) monitors each piece of waste as it arrives.
* The device runs a simple image classification model to determine whether the item is plastic, metal, glass, organic, etc.
* It sends that label to the sorting equipment downstream.
Correct labels lead to fewer mistakes, less manual checking, and quicker routing of materials to recycling or composting.
Technical Overview
------------------
In this section, Serena will guide you through the key components of her project.
### Data Selection
Waste Segregation Image Dataset on Kaggle. Source: [https://www.kaggle.com/datasets/aashidutt3/waste-segregation-image-dataset/data](https://www.kaggle.com/datasets/aashidutt3/waste-segregation-image-dataset/data)
I used a public [dataset](https://www.kaggle.com/datasets/aashidutt3/waste-segregation-image-dataset/data)
from Kaggle, comprising 15,200 images, split into 3 subsets: train, test, and validation.
Each subset contains images for 8 classes:
* Train: 13,999 images
* Validation: 1,201 images
* Test: 1,201 images
Images were resized to 150x150 initially to reduce training time and later to 299x299 for final training.
### Model Selection
Instead of training a model from scratch, I used a pre-trained Keras model from the official website. After comparing models based on performance and size, I chose the [Xception model](https://keras.io/api/applications/xception/)
. It’s a CNN architecture trained on over a million ImageNet [images](https://www.image-net.org/)
with 1000 output classes.
A snapshot of the ImageNet data. Screenshot from the original paper: "ImageNet: A large-scale hierarchical image database
#### Xception Model Architecture
The Xception model consists of a series of convolutional layers arranged into a deep CNN architecture. It starts with two standard convolution layers, followed by 36 depthwise separable convolution layers organized into 14 modules.
The diagram above shows the Xception model architecture, displaying the three stages: entry, middle, and exit, and the layers with the number of neurons in each of them. Source: [https://viso.ai/deep-learning/xception-model/](https://viso.ai/deep-learning/xception-model/)
Each module contains a stack of operations like depthwise convolutions (which filter each input channel separately) and pointwise convolutions (which combine the outputs), allowing the model to be both efficient and powerful.
This diagram depicts the difference between the pointwise and depthwise convolution used in the Xception model. Source: [https://viso.ai/deep-learning/xception-model/](https://viso.ai/deep-learning/xception-model/)
These layers are designed to extract increasingly abstract features from input images, starting with basic edges and colors in early layers and progressing to complex textures and shapes in deeper layers. The architecture concludes with a Global Average Pooling layer and a fully connected Dense layer of 1000 neurons with a softmax activation, used to classify images into 1000 categories from the ImageNet dataset.
You can visualize the block diagram of the Xception model by running this code:
> model = Xception(weights=’imagenet’, input\_shape=(299, 299, 3))
>
> visualkeras.layered\_view(model, legend=True)
>
> 
This diagram illustrates that the model comprises an input layer, a functional layer, GlobalAveragePooling2D layers, and a dense layer.
#### My Modifications of the Original Model
For the waste classification task, I removed the top layer and added a new custom head to predict 8 waste classes. The “top layer” in a pre-trained model is the final Dense (fully connected) layer that maps extracted features into the original classes. In the case of Xception, the top layer maps to the original 1000 classes from the ImageNet dataset.
Since my dataset only contains 8 categories of waste, I replaced this layer with a custom classification head to output probabilities for only these 8 classes. This technique is known as **“transfer learning**,” where we retain the convolutional layers of the Xception model, remove the top (dense) layer, and add a new one to learn our specific classes and make predictions.
Transfer learning helps reduce training time and improves performance by leveraging features learned from a large dataset, such as ImageNet.
Specifically, I used:
* A dense layer for the 8 subcategories.
* Another dense layer for biodegradability.
After adding a new top layer, we can visualize the model summary using model.summary().
A summary of the resulting model
### Hyperparameter Tuning
The first model didn’t perform well enough. The validation accuracy was oscillating between 0.83 and 0.88. Hyperparameter tuning was necessary.
I determined the optimal learning rate by training the model on four different learning rates: \[0.0001, 0.001, 0.01, 0.1\]. After plotting, I chose **lr=0.001** as the best value, as it achieved the highest validation accuracy.
Then, I added Dropout for regularization, a technique that randomly deactivates a small number of neurons during training, changing which neurons are active between each epoch. It helps prevent overfitting and encourages the network to learn more robust features.
Finally, I used softmax activation with 8 units as the output layer of my model. This setup allows the model to output a probability for each of the 8 waste subcategories. Softmax is ideal for the output layer of a multi-class classification network because it assigns a probability score to each class that sums up to 1.
### Training Strategy and Optimization
In addition to tuning the learning rate and adding dropout, I implemented a few more strategies to improve performance and stability during training:
* **Checkpointing:** It’s a technique that saves the best-performing model from a list of training models based on a specific parameter. In my project, I used **ModelCheckpoint** callback from **keras.callbacks** to automatically save only the best version of the model (the one with the highest validation accuracy). This ensured that training didn’t overwrite a good model with a worse one in later epochs.
* **Inner Dense Layer Tuning**: I experimented with different values for the **inner\_size** (number of units) parameter in the Dense layer before the output. After testing 3 values: \[10, 100, 500\], I found that 100 worked best. This layer acts as the bridge between the extracted features and the final classification layer.
### Data augmentation
To enrich the dataset, instead of taking pictures of new images, we can apply a technique called **data augmentation**.
This adds images to the dataset from the existing images, by applying augmentations to the images, such as:
* flip (horizontally or vertically)
* rotate (by an angle, clockwise or counter-clockwise)
* shift (by height or width)
* shear (extend from one corner)
* zoom (in/out, horizontally/vertically)
* brightness or contrast shifts
This technique helps with overfitting by increasing data diversity.
After data augmentation, I trained a larger model with 299x299 images, which is closer to the production environment where users will upload photos and receive classification results for their waste.
### Final results
After tuning many hyperparameters and training the model on larger images, I tested it on the test data subset to evaluate its performance. Since the best model is saved using callbacks, we restart the kernel, load the best saved model, and then test it.
**model = keras.models.load\_model(‘xception\_299\_04\_0.934.keras’)**
**model.evaluate(test\_ds)**
The final accuracy is 93.3%, which is above our threshold.
Screenshot showing the model accuracy
Then, we can test it on a random image of a metal can that it hasn’t seen before from the val dataset, and get the final prediction, which features the predicted category, biodegradability, and confidence scores.
A ' make\_prediction' function that utilizes the final model and outputs predicted category, biodegradability, and confidence scores.
### Serving and Deployment
To deploy my model and create a user-friendly interface that allows users to upload an image and receive a prediction, I took the following steps.
#### Step 1: Built a Flask API for inference
I created a Flask web server that receives image uploads, passes them to the model, and returns the prediction. This allows the model to serve predictions through HTTP requests.
#### Step 2: Dockerized the app for consistent deployment
To ensure the app runs identically across different environments, I created a Docker container containing the Flask app, the trained model, and all necessary dependencies.
##### **Docker usage:**
> docker build -t waste-segregation-app .
This command builds the Docker image from the **Dockerfile** and names it **waste-segregation-app**
> docker run -p 9696:9696 waste-segregation-app
This runs the container and maps port **9696** on your machine to the container’s port **9696**, making the API accessible
#### Step 3: Inference
As a result, a user can call the model and retrieve results using either a command line or a web interface.
##### Option 1: Using the command line
curl -X POST -F “file=@your\_image.jpg” http://localhost:9696/predict
This sends a POST request with an image to the /predict endpoint, and the model returns the predicted waste category
##### Option 2: Using the web interface
I created a simple HTML web interface where a user can upload an image, click a button, and receive a classification result. This interface is connected to the same backend. At localhost:9696
> 
> Screenshot of the web interface and the result after uploading an image from the Internet.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Prepare for (Better) Structured Data Extraction – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Prepare for (Better) Structured Data Extraction
Prepare for (Better) Structured Data Extraction
===============================================
### Structured Data Extraction with OpenAI Function Calling
18 Nov 2023 by [Amber Roberts](https://datatalks.club/people/amberroberts.html)
_Authored in collaboration with Aman Khan_
Extracting structured information from unstructured data (i.e. audio, images, text) to make it easier to analyze and use is a common and often-critical task where machine learning is used in a larger software system.
This blog covers some of the differences between traditional and large language model (LLM) structured data extraction, then walks through an example of how to implement a structured data extraction application leveraging OpenAI function calling and [Phoenix](https://github.com/Arize-ai/phoenix)
, an open source AI observability tool.
### Extracting Data from Unstructured Sources
The costs and intricacies involved in deciphering and analyzing unstructured data can be immense. While research is still ongoing on exactly where LLMs are more cost efficient for data extraction tasks compared to traditional NLP methods, it is clear that [many teams](https://arize.com/blog/llm-survey/)
are beginning to implement LLM apps because they excel at intuitively grasping the intricacies of language, particularly on extracting structured data from unstructured text.
#### Traditional Unstructured Techniques
The exact methods and tools used for extracting structured information from unstructured data today depend on the use case in question, as well as exactly what data you are working with. For example, the most common strategy for extracting keywords from text data is Regular Expressions (Regex). It is rather baffling how fast we all went from using Regex to LLMs! In addition to Regex for text data extraction, natural language processing (NLP) techniques like Named Entity Recognition (NER) and Latent Dirichlet Allocation (LDA) have also been used to extract entities and determine the main topics via Topic Modeling respectively.
While there are many different methods for extracting text from images (object detection, image segmentation, optical character recognition) and audio (speech-to-text, Mel spectrograms, to name a few), this post only covers structured data extraction from written language to demonstrate the role of LLMs.
#### LLMs for Structured Data Extraction
While traditional text extraction methods are still used frequently, these approaches are starting to lose favor to LLMs due to their efficient and scalable approach to distill relevant data from unstructured text. While research is ongoing on whether or not LLMs are more [cost efficient for data extraction tasks](https://indatalabs.com/blog/large-language-model-benefits)
(such as scraping text from documents and pulling keywords from queries), it is clear that LLMs are built with comprehensive extraction capabilities. LLMs excel at intuitively grasping the intricacies of language, particularly on extracting structured data from unstructured text.
Structured data extraction using an LLM. Here, an LLM receives unstructured input together with a schema and outputs a structured representation of the information.
The example above shows how LLMs can efficiently convert unstructured text data into structured information. These extracted attributes can then be used to construct a structured query to find options that might be relevant to the user or enrich company databases for future model training.
### Implementing a Structured Extraction Application with Arize Phoenix OSS
The [OpenAI API](https://arize.com/blog-course/mastering-openai-api-tips-and-tricks/)
is great for developers looking to build LLM applications with few lines of code. However, it was still difficult for developers to work with unstructured data (mostly strings) without either regular expressions (RegEx) or prompt engineering to extract the information from the text string. OpenAI recently released new OpenAI’s function calling capabilities for GPT-3.5 and GPT-4 models to take user-defined functions as input and generate structure output. [With this](https://www.datacamp.com/tutorial/open-ai-function-calling-tutorial)
, you don’t need to write RegEx or perform prompt engineering.
Arize Phoenix comes in as a way to instruct your OpenAI client to record trace data in OpenInference tracing format, inspect the traces and spans of your application to visualize your trace data, and export your trace data to run an evaluation on the quality of your structured extractions. Now, let’s see how to use OpenAI’s function calling feature to perform structured data extraction: the task of transforming unstructured input (e.g., user requests in natural language) into structured format (e.g., tabular format).
Structured extraction is a place where it’s simplest to work directly with the OpenAI function calling API. OpenAI functions for structured data extraction recommends providing the following JSON schema object in the form of parameters\_schema (the desired fields for structured data output).
parameters_schema = {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": 'The desired destination location. Use city, state, and country format when possible. If no destination is provided, return "unstated".',
},
"budget_level": {
"type": "string",
"enum": ["low", "medium", "high", "not_stated"],
"description": 'The desired budget level. If no budget level is provided, return "not_stated".',
},
"purpose": {
"type": "string",
"enum": ["business", "pleasure", "other", "non_stated"],
"description": 'The purpose of the trip. If no purpose is provided, return "not_stated".',
},
},
"required": ["location", "budget_level", "purpose"],
}
function_schema = {
"name": "record_travel_request_attributes",
"description": "Records the attributes of a travel request",
"parameters": parameters_schema,
}
system_message = (
"You are an assistant that parses and records the attributes of a user's travel request."
)
The ChatCompletion call to Open AI would look like:
response = openai.ChatCompletion.create(
model=model,
messages=[\
{"role": "system", "content": system_message},\
{"role": "user", "content": travel_request},\
],
functions=[function_schema],
# By default, the LLM will choose whether or not to call a function given the conversation context.
# The line below forces the LLM to call the function so that the output conforms to the schema.
function_call={"name": function_schema["name"]},
)
One powerful feature of the OpenAI chat completions API is function calling, wherein a user describes the signature and arguments of one or more functions to the OpenAI API via a JSON schema and natural language descriptions, and the LLM decides when to call each function and provides argument values depending on the context of the conversation. In addition to its primary purpose of integrating function inputs and outputs into a sequence of chat messages, function calling is also useful for structured data extraction, since you can specify a “function” that describes the desired format of your structured output. Structured data extraction is useful for a variety of purposes, including ETL or as input to another machine learning model such as a recommender system.
While it’s possible to produce structured output without using function calling via careful prompting, function calling is more reliable at producing output that conforms to a particular format. For more details on OpenAI’s function calling API, see the [OpenAI documentation](https://platform.openai.com/docs/introduction)
.
### Tracing Structured Data Extraction with LLMs in Phoenix
Now let’s go through a colab notebook: _Tracing and Evaluating a Structured Data Extraction Application with OpenAI Function Calling_ ([colab](https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/tracing/openai_tracing_tutorial.ipynb)
| [github](https://github.com/Arize-ai/phoenix/blob/main/tutorials/tracing/openai_tracing_tutorial.ipynb)
).
You can use Phoenix spans and traces to inspect the invocation parameters of the function to verify:
1. The inputs to the model in form of the the user message
2. Your request to OpenAI
3. The corresponding generated outputs from the model match what’s expected from the schema and are correct
Viewing a batch of traces within Phoenix
The full tutorial covers how to:
* Use OpenAI’s [function calling feature](https://openai.com/blog/function-calling-and-other-api-updates)
to perform structured data extraction: the task of transforming unstructured input (e.g., user requests in natural language) into structured format (e.g., tabular format),
* Instrument your OpenAI client to record trace data in [OpenInference tracing](https://github.com/Arize-ai/open-inference-spec/blob/main/trace/spec/traces.md)
format,
* Inspect the traces and spans of your application to visualize your trace data,
* Export your trace data to run an evaluation on the quality of your structured extractions.
Topics include:
* OpenAI
* Structured data extraction
* Function calling
* Note: This notebook requires an OpenAI API key.
### Additional Evaluations for Structured Extraction with LLMs
When working with unstructured data, it’s important to have a clear understanding of the desired outcome, as the methods and tools you choose will depend on the specific task at hand. It’s also crucial to continuously evaluate and refine the extraction processes to ensure accuracy and relevancy.
Evals help you continuously understand your system’s performance after deployment. Evaluating LLM applications needs to take place across three environments: pre-production when you’re doing the benchmarking, pre-production when you’re testing your application and production when it’s deployed. Life is messy. Data drifts, users drift, models drift, all in unpredictable ways. Just because your system worked well once doesn’t mean it will do so on Tuesday at 7 p.m.
_LLM evals need to work in several different environments: pre-production (benchmarking, testing) and production to understand a system’s performance._
Remember that verifying correctness of extraction at scale or in a batch pipeline can be challenging and expensive. It is a best practice not to do [LLM evals](https://arize.com/blog-course/llm-evaluation-the-definitive-guide/)
with one-off code but rather a library that has built-in prompt templates. This increases reproducibility and allows for more flexible evaluation where you can swap out different pieces.
_Workflow for evaluating an LLM retrieval span using Phoenix_
Evaluating data extraction tasks performed by LLMs is challenging with these models being non-determinism and often not having “ground truths” for their language tasks; However, with careful monitoring and LLM observability using LLMs for structured data extraction can scale faster and perform better than any previously used methods.
Ready to dive deeper? Get certified in key areas like [Traces and Spans](https://courses.arize.com/p/llm-observability-traces-spans?dl_cd=eyJrdiI6IktWX2VjOGE0OTI2YTE5MGVhNzBiZTMyMWUzMWFlZDM2YWEwIiwiaXNQcmV2aWV3IjpmYWxzZX0%3D)
or [Agents, Tools, and Chains](https://courses.arize.com/p/agents-tools-and-chains?dl_cd=eyJrdiI6IktWX2VjOGE0OTI2YTE5MGVhNzBiZTMyMWUzMWFlZDM2YWEwIiwiaXNQcmV2aWV3IjpmYWxzZX0%3D)
or ask questions on the [Arize community](https://arize.com/community/)
.
_This post is sponsored by [Arize](https://arize.com/)
. We thank the [Arize](https://arize.com/)
team for their ongoing support of the community._
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aaron Wishnick – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aaron Wishnick
Aaron Wishnick works as a Senior Software Engineer at Amazon, where he has been for 7 years. During that time he has worked on Amazon’s payment systems, financial intelligence systems, as well as working for AWS on Athena and AWS Proton. When not at work, Aaron and his fiance, Alyssa, are on a quest to determine just how much dog fur is too much, with their husky and malamute, Mina and Wally.
[](https://linkedin.com/in/https://www.linkedin.com/in/aaron-wishnick-93691a28)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aaisha Muhammad – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aaisha Muhammad
Aaisha is a self-taught bioinformatician, machine learning engineer and data scientist from Johannesburg, South Africa. She is a Datatalks.Club ML zoomcamp graduate and has studied python and basic web development, and she maintains a blog on bioinformatics and ML topics. Beyond that, she is interested in a variety of topics such as photography and digital art.
[](https://twitter.com/ZealousMushroom)
[](https://linkedin.com/in/aaisha-muhammad)
[](https://github.com/AaishaMuhammad)
[](http://www.aaishamuhammad.co.za/)
### Events
* Mastering Self-Learning in Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=Kc3Puh3UCRQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# 8 Newsletters for Data Science, AI, and ML Enthusiasts – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
8 Newsletters for Data Science, AI, and ML Enthusiasts
8 Newsletters for Data Science, AI, and ML Enthusiasts
======================================================
### Follow insights and trends with these publications
11 Dec 2023 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
In the landscape of modern technology, there is a surge of interest in data science. Before 2023, one could count on one hand the number of newsletters that tackled this niche. Now, they cascade into inboxes, each promising the key to unlocking the mysteries of AI, ML, and data science.
To help you, we decided to curate a list of eight newsletters that excel in discussing data science, AI, and ML. We made sure this collection is useful for everyone, whether you’re a seasoned expert or just beginning your journey in these fields.
1\. DataTalks.Club
------------------
DataTalks.Club is a vibrant community of data enthusiasts. We host regular online events such as podcasts, webinars, and workshops centered around data-centric topics. DataTalks.Club also provides free online courses on machine learning engineering, data engineering, and MLOps.
Subscribing to DataTalks.Club newsletter ensures that you’ll get weekly updates filled with the latest community developments: access to free courses, announcements of live online events, collaborative learning projects, and plenty of other engaging community activities.
Link: [https://datatalks.club/](https://datatalks.club/)
2\. Ahead of AI by Sebastian Raschka
------------------------------------
“Ahead of AI” shares practical coverage of deep learning techniques, research papers, and the latest concepts like foundation models.
Readers receive:
* Monthly insights into AI/ML advancements.
* Bi-monthly overviews of noteworthy research papers.
* Intermittent deep dives into topics like transformer architecture and LLM nuances.
Sebastian Raschka, the creator of the newsletter, previously researched as a PhD. Now, he educates enthusiasts through his books and this newsletter. Additionally, at Lightning AI, he melds AI research with open-source software development and the crafting of educational resources. and now educates people on the topics of AI/ML through his books and a newsletter. He also combines AI research with open-source development and creating learning materials at Lightning AI.
Link: [https://magazine.sebastianraschka.com/](https://magazine.sebastianraschka.com/)
3\. Data Elixir
---------------
“Data Elixir” is your curated mix of top data science finds from around the web. This newsletter expertly blends a range of content, keeping readers updated on the latest in data science.
Features include:
* Content Highlights: Insightful blog posts, engaging interviews, and comprehensive tutorials.
* Learning Opportunities: Upcoming online events and free courses.
* Technical Resources: Updates on open-source tools, package enhancements, and other coding assets.
Link: [https://dataelixir.com/](https://dataelixir.com/)
4\. TheSequence
---------------
TheSequence newsletter shares concise insights on ML research, tools, and trends.
It covers:
* Tuesdays: Core ML insights, summaries of pivotal research, and the latest in AI tech.
* Thursdays: Deep dives into standout research or essential technology frameworks.
* A Sunday snapshot of AI’s latest in research and innovation, all in 5 minutes.
* Bi-weekly interviews with ML’s leading minds
5\. Turing Post
---------------
Turing Post aims to be an AI knowledge hub for all, from novices to seasoned professionals.
They offer:
* Weekly Digests: summaries of over 150 AI/ML-centric newsletters.
* In-depth Series: the history of LLMs and offering practical insights from the FM/LLM series
* Company Spotlights: In-depth looks at AI front-runners like OpenAI, Anthropic, Inflection, Hugging Face, and Cohere.
* Global AI Outlook: the nuances of AI adoption and practices across different countries.
Link: [https://www.turingpost.com/](https://www.turingpost.com/)
6\. Interconnects by Nathan Lambert
-----------------------------------
“Interconnects” explores the complex world of AI research and its intersection with society. The newsletter aims to simplify and communicate intricate AI topics.
Subscribers enjoy:
* Weekly deep dives, lasting 5-15 minutes, into pressing AI concerns and their societal implications
* Occasionally, a bonus post or exclusive content reserved for paid subscribers
Nathan Lambert, the voice behind “Interconnects”, is a seasoned machine learning researcher. Having previously been part of HuggingFace’s research team and bootstrapped an RLHF team, Nathan brings a wealth of experience.
Link: [https://www.interconnects.ai/](https://www.interconnects.ai/)
7\. Palindrome by Tivadar Danka
-------------------------------
Palindrome shares stories about how mathematics makes our world work. Not textbook chapters, or dull technical exercises.
Tivadar, the passionate mathematician behind the newsletter, believes there’s a better way to teach and appreciate math. His journey spans from being a curious student to becoming a computational biology researcher and a machine learning specialist. Today, Tivadar dedicates his time to education, sharing mathematical wisdom in digestible stories and working on a comprehensive guide to the math behind machine learning.
Link: [https://thepalindrome.org/](https://thepalindrome.org/)
8\. Data science weekly
-----------------------
“Data Science Weekly” brings a weekly dose of the most relevant articles, news, and tutorials—handpicked through daily research across multiple platforms.
The newsletter curators, Hannah and Sebastian, blend expertise from Business Strategy and Financial Modeling to Data Visualization and Economics. They’ve also created resources like the guide “Get A Data Science Job Course” and the book “Data Scientists at Work”.
Link: [https://www.datascienceweekly.org/](https://www.datascienceweekly.org/)
Summary
-------
We’ve chosen these eight newsletters because they’re insightful, reliable, and, frankly, just interesting. They’re great reads for anyone looking to stay up-to-date on data science, AI, and ML, no matter your expertise level. Give them a try. Who knows, they might just spark your next big idea or project. Happy reading!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won! – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won!
A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won!
=================================================================================
### A deep dive into the Kaggle Competition on Kitchenware Classification, focusing on the winning approach.
21 Feb 2023 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
An overview of the Kitchenware Classification competition on Kaggle
The Kaggle competition with the theme: Kitchenware Classification has just ended. Below is a summary of how it unfolded, some statistics, and the approach followed by the winners.
What we’ll cover
* Overview of the competition
* How the dataset was built
* The starter notebooks
* The solution of the winner
* Making the solutions production-ready
Overview of the Kitchenware Classification Competition
------------------------------------------------------
A Kaggle competition is an online competition where teams compete to develop the most accurate model based on certain data. Teams usually comprise data scientists, statisticians, and machine learning engineers who work together to develop a model that can accurately predict and classify data. The team with the best model wins the competition.
The Kaggle competition on Kitchenware Classification started on December 5th, 2022, and ended on February 1st, 2023. Participants were tasked with correctly identifying the type of kitchenware in a given image. There were six categories: cups, glasses, plates, spoons, forks, and knives.
### How the dataset was built
The dataset contains 9337 images. The following figure shows some sample images:
Some sample images were extracted from the dataset provided by the competition.
The dataset was generated using [Toloka](https://toloka.ai/)
, a scalable AI and Machine Learning development platform. Toloka covers the entire Machine Learning life cycle, starting from data collection and ending with model monitoring.
Toloka’s offerings include a crowdsourcing platform that enables companies to collect and label data using human insight. The annotators, called Tolokers,will complete simple tasks for money, such as labeling images or transcribing audio.
The Kitchenware dataset defined the following rules for image collection in Toloka:
* The picture should be yours. It’s best to use your phone and take pictures of your own kitchenware.
* It’s strictly not allowed to download pictures from the internet. It cannot be images from websites like Ikea or similar.
* There should be only one item on the picture, not multiple
* The item should be placed in the center of the picture
* The item should be visible
* If you have a set with many similar items (e.g., 6 forks, 6 spoons, etc.), don’t take pictures of all six. Just take a picture of one spoon, fork, and so on.
* Don’t upload duplicates of your previous pictures. Each picture you submit should be unique
* Don’t upload multiple pictures of the same item. One is enough
* Screenshots are not allowed. It has to be your picture
Checkthe related [documentation](https://docs.google.com/document/d/e/2PACX-1vRK9pEAbmrNm0kD-u09no2l8KP6gwZdBF87fB_RoF7duVDNi8NrVq4XqUYISOdZ6n1H33iEVoW9wBLZ/pub)
and [workshop](https://www.youtube.com/watch?v=POGiLFWxQWQ)
for more details on how the dataset was collected.
### The Starter Notebooks
The competition provided the competitors with two basic notebooks. The [first notebook](https://github.com/DataTalksClub/kitchenware-competition-starter/blob/main/keras-starter.ipynb)
trains an xception model, and the [second](https://www.kaggle.com/code/harpdeci/supergradients-starter-notebook)
, developed by Harpreet Sahota, uses SuperGradients. Competitors were asked to start from one of these two notebooks and then improve it to achieve a higher score.
Both notebooks are available in Saturn Cloud, an enterprise-level cloud computing platform enabling users to manage workloads, process data, and run applications in the cloud. Check [this repo](https://github.com/DataTalksClub/kitchenware-competition-starter)
for instructions on running the notebooks in Saturn Cloud.
### Statistics
The following table resumes some statistics on the Kitchenware Competition:
| **Statistic** | **Value** |
| --- | --- |
| Number of participants | 115 teams |
| Average number of people per team | Most people did the competition individually |
| Lowest score | 0.05445 |
| Highest score | 0.99145 |
### The Prize
The winner of the Kitchenware Competition won the NVIDIA GeForce RTX 3080 Ti shown below:
The NVIDIA GeForce RTX 3080 Ti given as a prize to the winner.
The competition also gave other prizes, including $3000 in AWS credits.
The Winner
----------
The competition’s winner is Olufemi Victor Tolulope with the [Kitchenware Classification 3rd Place Notebook](https://www.kaggle.com/code/victorolufemi/kitchenware-classification-3rd-place-notebook)
. Olufemi Victor improved the [solution](https://www.kaggle.com/code/miwojc/starter-notebook-with-fastai)
proposed by MIWOJC, who used the [fastai](https://docs.fast.ai/)
Python library to train the model.
### The MIWOJC’s Solution
MIWOJC executed the following steps to solve the kitchenware classification problem:
1. Import load data as ImageDataLoaders
2. Create the learner model. Use the vision\_learner() function provided by fastai. This line of code creates a vision learner object using fastai. MIWOJC used a convnext\_nano model, which is an efficient and accurate model.
3. Use the lr\_find() method to find the best learning rate
4. Use the fine\_tune() method to fine-tune the model. The method takes two arguments: the number of training epochs and the learning rate.
### The Winner’s Solution
Olufemi Victor Tolulope started from the MIWOJC’s solution and improved it by modifying the following elements:
1. Augment the dataset size using the [albumentations](https://albumentations.ai/)
library. This library provides you with various augmentations, including geometrical transformations, color adjustments, and various other augmentations.
2. Use the convnext\_xlarge\_384\_in22ft1k model instead of convnext\_nano. While the first model is an advanced convolutional neural network designed for larger image datasets, the second one is designed for small datasets. The convnext\_xlarge\_384\_in22ft1k model is, therefore more powerful but requires more resources and computing power, while the convnext\_nano model is lightweight and quick to train.
3. Set the training floating point precision to 16 through the to\_fp16() method. This setting provides a more accurate result by allowing the model to make calculations with higher precision. This is especially important when dealing with large datasets or complex models with many layers, as it can help to reduce numerical errors and improve the overall accuracy of the results.
4. Use test time augmentation (TTA) to improve the model accuracy through the tta() method.
The winner also provided a solution ready for production by saving the export() method.
Making it Production-Ready
--------------------------
Building a well-performing model is only the first step of a project life cycle. When the model is ready, you should move it to production, that is, making the model usable for predictions in a real environment. To make the competitors aware of this challenge, the Kitchenware Competition did not conclude with building a model. There was a second part of the competition, named [make it production ready](https://www.kaggle.com/competitions/kitchenware-classification/overview/make-it-production-ready)
, which invited the competitors to build a GitHub repository for the project and, optionally, a blog post.
### Evaluation
A jury composed of two experts decided on the winner in the following categories:
* The most production-ready ML solution (the overall winner of the contest).
* The best GitHub repo with a solution.
* The smallest model size (measured by the size of the models used in the final solution).
#### The Winners
The proposed solutions included some modern practices and frameworks for production: Docker containerization, serverless inference, optimized inference engines, package managers, and additional UI like Streamlit and Telegram.
The winning solution was proposed by **Martin Uribe**, who also got the nomination for “The best GitHub repo with a solution”. Martin’s solution: [https://github.com/clamytoe/kitchenware\_classifier](https://github.com/clamytoe/kitchenware_classifier)
.
**Bhaskar Sarma** won the prize for the smallest model size. Bhaskar’s solution: [https://github.com/bhasarma/kitchenware-classification-project](https://github.com/bhasarma/kitchenware-classification-project)
.
Acknowledgments
---------------
We thank our sponsors, [Toloka](https://toloka.ai/)
, [SaturnCloud](https://saturncloud.io/)
, and [NVIDIA](https://www.nvidia.com/)
. We couldn’t have done it without their support, and we’re grateful for their help in making education more accessible and fun for our community.
We also want to thank [Rustem Feyzkhanov](https://www.linkedin.com/in/ryfeus/)
for donating $2000 in AWS Credits.
And, most importantly, we thank our amazing community of participants. This competition would not have been the same without you. Your passion and dedication are truly inspiring.
Thank you all for making this competition a success!
Summary
-------
The Kaggle Competition on Kitchenware Classification was a great learning experience for many data scientists and computer vision enthusiasts. It provided an opportunity to apply the latest state-of-the-art algorithms, learn from others’ approaches and gain valuable insights into kitchenware classification problems.
Congratulations are due to all participants who worked hard throughout the competition and congratulations in particular to the winner of this amazing event!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Naming Variables in Machine Learning – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Naming Variables in Machine Learning
Naming Variables in Machine Learning
====================================
### Why df is a bad name for a dataframe
02 Oct 2022 by [Igor Demidov](https://datatalks.club/people/igordemidov.html)
For a long time there has been a significant discrepancy in coding style between software engineers and data scientists. The first have been following established practice of good code writing, and the last… have just been writing some code. It worked well for us, data scientists, for quite some time. But then our projects became more complex, the code grew larger. Eventually, we faced the same complications in understanding our own code, as software engineers faced a long time ago.
Did you think that strict code writing conventions are for software engineers, not data scientists? Just like I did. It’s time to use our colleagues’ experience and adopt those good code practices one by one.
Starting from the very beginning. Can you tell what each variable in your project contains at any moment of time? Do you know its type? And most importantly, are you sure the variable contains what you expect it to contain?
If your answer is “ehm”, then read on - let’s reinvent the wheel.
Even in Python names are important (image by the author)
Clean Code
----------
If you read Robert Martin’s “Clean Code” book, most of the points described here will be familiar to you. If you didn’t… just read it right after you finish this article and press the clap button. The book really provides a life changing experience. It describes a huge amount of techniques to make your code structurally better. The main idea that I took out of the book is that your code should strive to be reader-friendly as well as flexible for modification.
You may argue that it is the case for big “real” software applications, on which many engineers are working. If you are a machine learning specialist, most probably you work on your code alone. So why bother about readability? You know every corner of the code, right? Of course you are. Now imagine you get back to your code several months after. To fix something or to take one of its parts to a recent project. And this is where you face the “s” variable in the middle of a huge function named “func”. And now you need a huge cup of coffee, as you know you will need to spend time trying to remember what’s going on here.
“Okay, I’m convinced”, - I hope you say. If so, follow me.
Naming Tips
-----------
### Intention-revealing names
This one is simple. Variable and function names should reveal intentions.
if e > 10:
break
If you look at this code abstractedly, can you guess what’s happening here? Probably not.
With a tiny tweak like this:
if epoch > 10:
break
it makes perfect sense: it’s definitely a part of a neural network training procedure, specifically, a training loop termination condition.
You say _epoch_ is a standard name for data science? You are perfectly right! And this is the example of a meaningful name. You don’t need neither a comment nor additional guessing to understand what this part of the code is about.
What about bad names? I’d probably incur the wrath of the data science community, but “data” and “df” are poor names.
Let’s look at an example. Common data scientific code could look like this:
def process_data(df):
df = df.fillna()
df['column_name'] = df['another_column'] * 5
df = df.groupby('major_column').sum()
df = pandas.concat([df.iloc[0:100], df.iloc[200:300])\
return df\
\
\
Can you tell right away, what this code does? Probably you’d need some time to dig into it. And it could be much more complicated.\
\
To solve this, it could be a good idea to detach operations of one purpose into a separated function with a meaningful name. There you’d know what happens inside, what comes in and out. Then organise everything into a data pipeline, which would eliminate the need to reassign all results into one variable (this we will discuss in one of the following articles).\
\
### Comments to explain variables\
\
Another hint is a comment. If you need it to explain what the variable means, then there probably might be a better name for it.\
\
Compare two code examples, which do exactly the same. The first one is brute and straightforward:\
\
p = numpy.percentile(df.groupby('user')['sales'].mean(), 0.95)\
x = df.groupby('user')['date'].min().max()\
df = df[(df['sales'] >= p) & (df['date'] > x]\
u = df['user'].unique()\
\
\
The second one provides more readable solution:\
\
user_mean_sale = sales.groupby('user')['sales'].mean()\
mean_sales_percentile = numpy.percentile(user_mean_sale, 0.95)\
\
user_first_sale_date = sales.groupby('user')['date'].min()\
latest_first_sale_date = user_first_sale_date.max()\
\
recent_high_value_sales = sales[\
(sales['sales'] >= mean_sales_percentile) & \
(sales['date'] > latest_first_sale_date)]\
unique_hv_users = recent_high_value_sales['user'].unique()\
\
\
You would probably need to add comments into the first code block to explain what each variable means in order to save you thinking efforts. Does the second part need a comment to describe intentions? Probably not. I can agree it looks more bulky, but look me in the eye and say you don’t understand what the second code example does. I won’t believe you. Now when you return to your code in several months or years, it doesn’t matter, you will be right on board with the code.\
\
### Meaningful distinctions\
\
\
\
The distinction should be really meaningful (image by [Jørgen Håland](https://unsplash.com/@jhaland)\
\
Sometimes you need to create several objects with similar content. It is important to make names, which express the difference between the objects. The most obvious example here might be variables _df_ and _data._ What’s in _data_ and what’s in _df_? How do they differ? Impossible to understand without a detective investigation about origins of both objects.\
\
Similarly, naming objects like _df1_ and _df2_ does not help to understand the code.\
\
Try to check your naming by answering the question if this specific word in the name helps to understand. For example, if you name a variable the\_user\_data, do all the words work for the purpose? The first word in the name (_the)_ is definitely not important. If you omit it, the name’s meaning won’t change at all. The second one (_user)_ – definitely is, extract it and you lose a vast share of information. _The third on (data)_ – probably, as adding data type to naming isn’t considered a good practice, but still it can be useful to distinguish such important objects in machine learning projects as dataframes.\
\
### Searchable names\
\
This one seems quite self-explanatory. Imagine you want to rename a variable in your class or a notebook. Let’s say you want to change it from _train_ to _test_. If you did the naming right, all you’ll need to do is to find and replace all the names. If not – you will have a hard time understanding if the current _train_ is _the train_ you want to change or if it’s meant for something else.\
\
If you name your variable _sum_. How many false search results will you filter in order to find your variable.\
\
Another use case is when you decide to get rid of some function and you want to make sure that variables generated by this function do not affect any other piece of code.\
\
Finally, when you try to understand why your predicted validation data frame missing columns, it’s quite useful to run a search on a validation set variable to look through all the places where it’s involved.\
\
### Class and method names\
\
Perfect code can be read like a book. And this tip is one of the most elegant ways to get closer to it. Here is the idea: you name your classes in nouns, because a class is some service, providing you with specific processing; and functions are named in verbs, because a function executes one specific task. A function’s arguments are nouns as well, as they are mostly some data structures or primitive data types. As a result you can get a really smooth notation, like this:\
\
train_data = DataLoader().load_data(path)\
\
\
or\
\
encoded_train = Encoder().encode_country(train_data)\
\
\
You don’t really need any comments about these functions, do you? Proper naming tells the story by itself. Actually, you regularly meet such examples when using some properly designed libraries.\
\
RandomForestClassifier().train(train_data, train_target)\
\
\
pandas.DataFrame().corrwith(another_df)\
\
\
Looks pretty and quite easy to read.\
\
Conclusion\
----------\
\
Well written code will not make your model work faster or better. In some cases it can make your code shorter, but pretty often it will on the contrary enlarge the program. It may seem that for machine learning projects it’s redundant and only requires additional time and space. Of course it’s neither mandatory nor demanded as a must. Moreover, if you adopt this practice, you’d probably have numerous disputes with colleagues about whether this is a good or a bad thing. I can’t promise it will work for you, but it surely worked for me, and I hope I can put your mind to another helpful habit in writing code. As it will surely enhance your code readability and facilitate its support for the next reader, even if the next reader is the future you.\
\
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)\
. \
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.\
\
Email \
\
Join\
\
* * *\
\
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)\
. We use cookies.
---
# DataOps: Similarities and Differences with Data Engineering and Data Science – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
DataOps: Similarities and Differences with Data Engineering and Data Science
DataOps: Similarities and Differences with Data Engineering and Data Science
============================================================================
### A comparison among the DataOps, Data Engineering, and Data Science roles.
23 Dec 2022 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
Image by [💖MORE ON 👉 https://melovess.com 💖](https://pixabay.com/users/buffik-17824401/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5475661)
from [Pixabay](https://pixabay.com//?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=5475661)
Data Operations (DataOps), Data Science, and Data Engineering are three closely related fields in data management and analytics. While Data Science involves the development of statistical and machine learning models to extract insights and make predictions from data, DataOps focus on the processes and tools used to manage, integrate, and deploy these models in production environments. Data Engineering is the practice of designing, building, maintaining, and managing data pipelines and systems that enable organizations to store, process, and analyze large and complex datasets.
What we’ll cover
* What is DataOps?
* DataOps and Data Engineering
* Data Ops and Data Science
What is DataOps?
----------------
DataOps is a term used to describe developing and managing data-driven applications. It encompasses the entire data lifecycle, from its collection and storage to its processing and analysis.
DataOps is a relatively new field, and as such, there is no one agreed-upon definition of it. However, most experts agree that DataOps involves the following key components, as shown in the following figure:
The DataOps main components
* **Continuous data integration**: This refers to the automated process of moving data from its source to its destination. This can be done using various tools, including ETL (extract, transform, load) tools and data pipelines.
* **Continuous data quality assurance**: This component ensures that the integrated data is high quality and free of errors. Data quality assurance can be achieved through manual and automated methods.
* **Continuous monitoring**: This refers to continually monitoring all aspects of the DataOps process, from data collection to storage to processing and analysis. Monitoring helps identify issues early on so they can be addressed in a timely manner.
DataOps engineers help people produce meaningful results faster, in a more pleasant, less scary way. DataOps is not only about infrastructure; it’s also about helping people write better SQL queries and learn how to use the infrastructure.
### How is DataOps related to infrastructure?
DataOps is related to infrastructure in several ways. First, DataOps relies on automation to manage and provision infrastructure resources. This allows for faster deployments and updates and reduces the need for manual intervention.
Additionally, DataOps uses monitoring and logging tools to track the performance of infrastructure resources. This helps identify issues early and prevent them from becoming critical problems.
Finally, DataOps includes disaster recovery plans that ensure the continuity of operations in the event of an infrastructure failure.
### Suitable skills for DataOps
DataOps is a data management approach that emphasizes collaboration, automation, and monitoring to help organizations improve their data pipelines’ speed, quality, and security. To succeed in a DataOps role, you must have strong technical skills and work well with others.
You will serve as a middleman between the platform, security, SRE, and users, meaning data analysts, engineers, and scientists. And as a DataOps role, you work across different teams and business units. You are also observing some Slack channels.
DataOps and Data Engineering
----------------------------
There is a lot of overlap between DataOps and Data Engineering. Both disciplines deal with data management, including data acquisition, storage, transformation, and analysis. Both also strongly emphasize automation and using tools and technologies to streamline processes and improve efficiency.
However, there are some critical differences between the two disciplines. DataOps primarily focuses on operations, ensuring that data pipelines run smoothly and efficiently. On the other hand, Data Engineering focuses on designing and implementing those data pipelines.
DataOps practitioners need to have a strong understanding of both the technical aspects of data management and the business goals that need to be achieved. Data Engineers need to be experts in designing and building scalable data architectures that can support the needs of a rapidly growing business.
DataOps VS. Data Engineering
DataOps and Data Science
------------------------
In the past, Data Science and DataOps were often treated as separate disciplines, with data scientists developing models and data engineers handling the deployment and maintenance of those models. However, as the demand for data-driven decision-making has increased, the importance of DataOps has grown, and the lines between these two fields have become increasingly blurred.
One reason for this is that the success of a data science project often depends on the quality and availability of data, as well as the speed and reliability of the model deployment process. As a result, data scientists are increasingly being asked to take on DataOps responsibilities, such as data integration, model deployment, and monitoring.
This shift has led to the emergence of a new role known as the “data scientist-engineer,” or “data scientist 2.0,” skilled in both data science and DataOps. These professionals bridge the gap between the development of statistical models and their practical implementation, ensuring that data science projects are delivered on time and with high quality.
Another important factor driving the convergence of Data Science and DataOps is the increasing use of cloud-based platforms and tools. These platforms, such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure, provide data scientists with access to a wide range of tools and services for data storage, processing, and analytics as automatic scaling and deployment capabilities.
This has made it easier for data scientists to deploy and maintain models in production environments. It has also led to the developing of new tools and frameworks designed explicitly for DataOps, such as Apache Airflow and Databricks.
In summary, the distinction between Data Science and DataOps is becoming increasingly blurred as data scientists are increasingly being asked to take on DataOps responsibilities, and cloud-based platforms and tools make it easier for data scientists to deploy and maintain models in production environments. The emergence of the data scientist-engineer role, with skills in data science and DataOps, reflects this trend.
Summary
-------
Congratulations! You have just learned what DataOps is and its differences and similarities with Data Engineering and Data Science.
* DataOps involves developing and managing data-driven applications
* DataOps focuses on operations, ensuring that data pipelines run smoothly and efficiently. Data Engineering focuses on designing and implementing those data pipelines.
* Data Science and DataOps are two separate fields. However, data scientists are increasingly being asked to take on DataOps responsibilities.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Important SQL Fact That Everyone Should Know – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Important SQL Fact That Everyone Should Know
Important SQL Fact That Everyone Should Know
============================================
### How this has several impacts
21 Oct 2022 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
SQL is the primary programming language used in databases (picture [fromearvine95](https://pixabay.com/users/earvine95-2534314/)
in [Pixabay](https://pixabay.com/)
)
A database-management system (DBMS) is a group of linked data and a collection of software tools for accessing that data. The database, a collection of data, often contains information important to an organization. A DBMS’s main objective is to offer a simple and effective method for storing and retrieving database data.
SQL (also called SeQueL or Structured Query Language) has been designed to make it easier for programmers to interact with databases. SQL provides a variety of language constructs for database queries, such as the SELECT, FROM, and WHERE clauses but also mechanisms to rename both attributes and relations, and to order query results by sorting on specific attributes.
According to the book “[Database System Concepts](https://www.mheducation.com/highered/product/database-system-concepts-silberschatz-korth/M9780078022159.html)
” from McGraw-Hill Education (7th Edition) the history around SQL is as follows:
_“IBM developed the original version of SQL, originally called Sequel, as part of the System R project in the early 1970s. The Sequel language has evolved since then, and its name has changed to SQL (Structured Query Language)._
_In 1986, the American National Standards Institute (ANSI) and the International Organization for Standardization (ISO) published an SQL standard, called SQL-86. ANSI published an extended standard for SQL, SQL-89, in 1989. The next version of the standard was SQL-92 standard, followed by SQL:1999, SQL:2003, SQL:2006, SQL:2008, SQL:2011, and most recently SQL:2016.”_
One important fact about SQL is the way the SQL syntax is read by the SQL processor (also called SQL engine).
In this article, we are going to talk about how the real order of the SQL syntax affects:
* The final result of the query;
* The performance of the query;
* Readability of the query using an alias;
### How does the processor read the SQL statement?
SQL is considered by some authors as a programming language of the fourth level because it is very similar to a speaking language (in this case, similar to English), i.e. If you know how to write in English then you will quickly learn the basics of SQL. However, the real order of the statement is a little different.
Below I present to you the genuine order of the SQL statement.
| **Order** | **Clause** | **Description** |
| --- | --- | --- |
| 1st | FROM | Gets all the needed tables for the query - it includes all the JOIN clauses (Mandatory) |
| 2nd | WHERE | Filters the rows. |
| 3th | GROUP BY | Aggregates the data according to common information. |
| 4th | HAVING | Filters the groups created before. |
| **5th** | **SELECT** | **Select the attributes(columns) that you need. (Mandatory)** |
| 6th | ORDER BY | Orders the result based on one attribute. |
| 7th | LIMIT | Limit the number of rows represented. |
As you saw above the first clause to be read is the **FROM** with all the respective joins clauses (in case of existence), after that, we will filter the tables with the **WHERE** clause, then we create groups with **GROUP BY** followed by the group filtering using the **HAVING**.
Only after all these clauses, are we going to run the **SELECT** clause.
In case of need the clause **ORDER BY** is going to be read and then the **LIMIT** clause.
In a very general point of view, the SQL processor goes “from big to small” (by [Michael Shoemaker](https://medium.com/@dataslinger)
) to get the wanted result, and then it will “think” in the way of presenting the result by ordering and limiting.
The real order of the SQL statement has implications on the query if you don’t know this important SQL fact.
I created three tables running [PostgreSQL](https://www.postgresql.org/)
locally using [Docker](https://www.docker.com/)
containers, then I inserted some rows in these tables (clients, invoice, and client\_invoice) that were used for the purpose of this article.
[Learn how to run PostgreSQL locally on Docker](https://medium.com/p/c388483712e9)
(picture created using the official PostgreSQL and Docker logos)
### The impact of the SQL statement order
#### 2.1. The final result of the query
The obvious consequence of this order is the query’s final result. You may be filtering a query in the FROM clauses (the first clause to run) when you have to filter with the **WHERE** clause.
One classic example is when you want to know the data present in one table and not another table. A simple way of analyzing this information is by doing a **LEFT JOIN** between the tables and then checking if there is no corresponding to the second table using a IS NULL filter (see query below). For this query I got as a final result 42 k rows, meaning that there are 42 k clients that are not in the client\_invoice table.
SELECT COUNT(c.nickname) AS number_nickname
FROM clients c
LEFT JOIN client_invoice ci ON c.id=ci.user_id
WHERE ci.id IS NULL
However if you run the query like this below you will get a warning saying that _““ci.id IS NULL” is always false”_ and the total result we get is 172k rows. This query is obviously incorrect because we are trying to filter during the **JOIN** clause.
SELECT COUNT(c.nickname) AS number_nickname
FROM clients c
LEFT JOIN client_invoice ci ON c.id=ci.user_id
AND ci.id IS NULL
Another important piece of information regarding the SQL statement’s real order is the awareness of the difference between the **WHERE** clause and the **HAVING** clause.
Since the SQL processor is running the **WHERE** clause in the second it will always filter the rows based on that clause, while the **HAVING** clause always needs the **GROUP BY** clause because the filter will be made according to the groups.
Knowing the difference between these two clauses is very important if you want to get a correct final result.
#### 2.2. The performance of the query
Another effect of the order is the performance of the query. Since the **FROM** clause is the first to run if we have a join between two very large tables the performance can decrease. Sometimes it can be a good idea to filter one table before joining with another.
For this article, I made two simple queries and analyzed the time of each one:
* For the first query, I did a simple join using the three tables and then filtering at the end.
SELECT nickname,
SUM(value) AS total_to_pay
FROM client_invoice ci
INNER JOIN clients cl ON cl.id=ci.user_id
INNER JOIN invoice i ON i.id=ci.invoice_id
WHERE nickname NOT ILIKE 'The Singer%'
GROUP BY nickname
HAVING sum(value) > 200000
ORDER BY total_to_pay
* In the second query I filtered one table in a CTE before joining with the others.
WITH CTE_not_singer AS
(SELECT id,
nickname
FROM clients
WHERE nickname NOT ILIKE 'The Singer%')
SELECT nickname,
SUM(value) AS total_to_pay
FROM client_invoice ci
INNER JOIN CTE_not_singer cl ON cl.id=ci.user_id
INNER JOIN invoice i ON i.id=ci.invoice_id
GROUP BY nickname
HAVING sum(value) > 200000
ORDER BY total_to_pay
I made several iterations and got the final time results as you may see in the next table.
| **Iterations** | **Query 1** | **Query 2** |
| --- | --- | --- |
| 1 | 09 sec 017 ms | 09 sec 921 ms |
| 2 | 09 sec 470 ms | 08 sec 252 ms |
| 3 | 08 sec 485 ms | 07 sec 660 ms |
| 4 | 07 sec 622 ms | 07 sec 275 ms |
| 5 | 07 sec 711 ms | 07 sec 591 ms |
| 6 | 09 sec 185 ms | 07 sec 477 ms |
| Average | 08 sec 582 ms | 08 sec 029 ms |
As you can see from the table above the second query had slightly better performance, not much but in a larger scale, you can imagine the possible effects (my tables have around 100 k rows so we can’t mention them as “big data” tables).
Of course, today’s SQL processors feature one or more SQL planners, which will “improve” query performance (using indexes, choosing the most important attributes, etc.). However, if we are utilizing vast data or the database is continually updating (for example, daily UPDATES), these improvements will take some time. As a result, staying on top of our queries’ performance is always a wise habit.
#### 2.3. Readability of the query using an alias
One golden rule while working with data is “If the solution is “nasty” then is wrong” (see all the Golden Rules [here](https://medium.com/@lgsoliveira/do-you-know-the-golden-rules-while-working-with-data-110385bc9b25)
). This means that your code should be as “beautiful” as possible, for example, a query should have good readability.
Knowing the real order of the query you can apply several alias and therefore your SQL statement is easier to understand (for other developers or for “future you”).
The most common use of the alias is to give one to a table and then you can use it everywhere. Perhaps you didn’t know before why was that possible but now you know… the **FROM** clause is the first one to be read.
Since the clause **ORDER BY** only runs after the **SELECT** clause you also can use the alias like the example below.
SELECT c.nickname,
count(DISTINCT ci.invoice_id) total_invoice_id
FROM clients c
INNER JOIN client_invoice ci ON c.id=ci.user_id
GROUP BY 1
ORDER BY total_invoice_id DESC
But be aware that the **WHERE** clause is the second to be read and the **SELECT** clause is only the fifth. So if you try to run the query below you will get the error - “ \[42703\] ERROR: column “number\_invoice\_id” does not exist”
SELECT c.nickname,
ci.invoice_id number_invoice_id
FROM clients c
INNER JOIN client_invoice ci ON c.id=ci.user_id
WHERE number_invoice_id > 1000
Use alias to get beautiful queries but always remember the real order.
### Summary
SQL is a simple programming language to master the fundamentals of, but it does include several “secrets” and information that you must be aware of if you want to become a SQL expert.
One important fact that you should know is that the SQL statement order is not totally as it looks. The first clause to be read is the **FROM**, followed by the **WHERE** clause, then the **GROUP BY** and the HAVING clauses, followed by the **SELECT** clause, and at the end the **ORDER BY** and the **LIMIT** clause.
This information is significant since it can influence the query’s final result, performance, and internal code quality.
I would like some feedback from your side 🙂
* Did you know this was the real order of the statement?
* Do you have any important information you know and could share with me?
* Did you know the **SELECT** was almost the last to be… selected? 😁
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Interview with Ken Wu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Interview with Ken Wu
Interview with Ken Wu
=====================
### Ken’s story about his interest in machine learning, Machine Learning Zoomcamp and work on NLP during his internship at delphai
22 Aug 2022 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
In this interview, Ken Wu, a graduate of ML Zoomcamp, tells us about:
* His background
* His interest in ML and his school essay about ML
* Machine Learning Zoomcamp projects
* Working on Chinese NER at delphai
**Please tell us a bit about yourself. What’s your background?**
I’m an incoming student at the University of Waterloo and Wilfrid Laurier University studying computer science and business administration. I completed Machine Learning Zoomcamp when I was a senior in high school ([American School of Milan](https://www.asmilan.org/)
). Since middle school, I have been programming a bit but only really started focusing on it after my sophomore year in high school. You can find more about me on my [website](https://kenwu.is-a.dev/)
, [GitHub](https://github.com/KenWuqianghao)
, and [LinkedIn](https://www.linkedin.com/in/qianghao-wu-798246204/)
.
**When and how did it occur to you that you’re interested in ML?**
I had to write an extended essay on a subject to complete the [International Baccalaureate](https://www.ibo.org/)
program at my school, and I chose computer science since it was my intended major in university. I talked to a past IB student, [Matvey](https://www.linkedin.com/in/matvey-ryabov/)
(who also happened to be a UWaterloo student), and he guided me toward ML and computer vision, which were the basis of the essay I wrote. Here is a [link to my final essay](https://drive.google.com/file/d/14p_EKxl0ZL26waGNz-h1P06uJoZr9JT6/view?usp=sharing)
and [GitHub repo](https://github.com/KenWuqianghao/Covid-CNN)
, if anyone is interested.
[Ken’s essay](https://drive.google.com/file/d/14p_EKxl0ZL26waGNz-h1P06uJoZr9JT6/view)
**What did you do to enter the field and what were your main challenges?**
Just for the sake of completing my essay assignment, I started researching about the field and found it interesting. There weren’t many challenges, as I’m still a beginner in the field, but I would say finding adequate resources was initially somewhat challenging. Machine Learning feels like an overwhelming topic to get into.
**What attracted you to Machine Learning Zoomcamp? Why did you decide to take it?**
I saw a post on Reddit advertising a free 4-month introductory course to ML, and I thought it sounded like a great opportunity to expand my current knowledge and lead to a future career. I decided to enroll simply because it sounded like an outstanding deal (free, professional support, certification, internship opportunity). It also sounded super interesting.
**What do you do now? Can you tell us more about the internship that you got and the work you needed to do?**
I’m currently an intern at [delphai](https://www.delphai.com/)
, an AI-driven B2B company search engine. I’m working on Chinese NER. So far it has been very fulfilling, and I’ve learned a lot along the way. The colleagues were so helpful and welcoming. During the internship, I have learned about NLP from scratch, while deploying a working model for our company. I will be ending my internship this August and will continue to study computer science at university. However, I will be very happy to have the opportunity to come back to delphai next summer for another internship.
[Named entity recognition example](https://explosion.ai/demos/displacy-ent)
: detecting company names and dates
**Do you have any suggestions for future students of ML Zoomcamp?**
Hang in there. Sometimes you might feel lazy and want to give up. Just think about how much you will learn and how interesting this entire world of ML is. You won’t ever regret the time you spent learning for yourself.
**What kind of other materials would you recommend to people who read this interview?**
I don’t really know, I’m a beginner myself who learned mostly from ML Zoomcamp. Personally, I’m opposed to reading books as a form of learning. I would recommend thinking of cool projects that you would like to create and use Google and StackOverflow from there.
**What was the most helpful thing for you in the course?**
Definitely the capstone projects, where you actually get to do something cool you learned in the previous modules. Practicing with hands-on coding will always be the best method of learning.
[](https://github.com/KenWuqianghao/ML-Zoomcamp-Capstone-Project)
[Ken’s capstone project for classifying Doom vs Animal Crossing images](https://github.com/KenWuqianghao/ML-Zoomcamp-Capstone-Project)
**Anything else you’d like to mention?**
Feel free to connect with me and contact me if you have any cool projects and opportunities.
Machine Learning Zoomcamp is open for registrations. Sign up [here](http://mlzoomcamp.com/)
.
**About delphai**: delphai collects unstructured public company data all over the internet and then uses ML and AI to transform this unstructured data into a structured format to make it searchable and usable for our customers.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Abouzar Abbaspour – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Abouzar Abbaspour
Abouzar Abbaspour is a machine learning and data engineer with experience across industries as diverse as telecom, e-commerce, theme parks, and automotive.
He began his career as a software engineer in Iran, co-founded a startup for telecom network monitoring, and later worked on forecasting models and recommendation systems at the Dutch theme park Efteling. At bol.com, he helped design and deploy ML models at scale, including a recommendation engine for six million users. Today, at Tesla, he focuses on predictive maintenance, integrating LLM agents into existing applications, and building scalable data pipelines. Abouzar holds an EngD in Data Science from Eindhoven University of Technology.
[](https://linkedin.com/in/abouzar-abbaspour)
[](https://www.abouzar-abbaspour.com/)
### Events
* From Theme Parks to Tesla: Building Data Products That Work ([watch on youtube](https://www.youtube.com/watch?v=gXvVMvhfrIY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Adam Sroka – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Adam Sroka
Dr. Adam Sroka, Head of Machine Learning Engineering at Origami Energy, is an experienced data and AI leader helping organizations unlock value from data by delivering enterprise-scale solutions and building high-performing data and analytics teams from the ground up. Adam shares his thoughts and ideas through public speaking, tech community events, on his blog, and in his podcast.
[](https://twitter.com/adzsroka)
[](https://linkedin.com/in/aesroka)
[](https://adamsroka.co.uk/)
### Events
* Defining Success: Metrics and KPIs ([watch on youtube](https://www.youtube.com/watch?v=H4P2RfKvXGs)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Defining-Success-Metrics-and-KPIs---Adam-Sroka-e17gfp0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aditya Seshaditya – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aditya Seshaditya
Aditya Seshaditya is a Data Scientist with background in Quantum Computing, Digital Twins and AI.
[](https://linkedin.com/in/a-seshaditya-7180822a5)
[](https://github.com/adytiaa/physAI_CFDsim)
[](https://quasi.digital/)
### Events
* Combining Quantum and AI for Accelerating CFD Simulations - Part 1 ([watch on youtube](https://www.youtube.com/watch?v=UHRN7cX_ieE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Interview with Valerii Chetvertakov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Interview with Valerii Chetvertakov
Interview with Valerii Chetvertakov
===================================
### Valerii’s story about his work in property valuation, Machine Learning Zoomcamp, and his internship at delphai
29 Sep 2022 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
In this interview, Valerii Chetvertakov, a graduate of ML Zoomcamp, tells us about:
* His work in property valuation and interest in machine learning
* The main challenges of entering the fields of ML and data
* Machine Learning Zoomcamp
* Valerii’s internship project: extracting business relationships between companies
* Suggestions for the future students
**Please tell us a bit about yourself. What’s your background?**
I have a higher technical education, specialising in underground mining of mineral deposits. Later, I switched to the field of property valuation. I have been a certified appraiser since 2006.
**When and how did it occur to you that you’re interested in ML?**
Working as an appraiser on various equipment, goods and real estate valuation projects, I came across the probabilistic nature of market values and prices.
I had to apply various statistical and regression approaches in property valuation to deal with a lack of sufficient data, variety of property features, and uncertainty. I enjoyed learning such techniques and wanted to explore this outlet professionally. My property valuation reports looked like a mathematical reference — a lot of formulas, calculations, tables, graphs.
When doing property valuation, we need to take into account a lot of features (Image generated with DALL-E)
At some point, the capabilities of Microsoft Excel became insufficient for me to analyse the data and build statistical and regression models. Then I learned about Python and Machine Learning.
In the summer of 2018, I decided to learn Python, and in September I enrolled in an evening course on Machine Learning basics, organised by a local IT school.
**What did you do to enter the field and what were your main challenges?**
First, I consumed a lot of introductory information about machine learning and data science — where, why and how Machine Learning is applied, what are the current achievements, and future challenges. I was amazed at how widely Machine Learning was introduced into our lives, and what great prospects it holds for business, technology, and humanity. We need to find that sense of belonging to something great and outstanding in order to overcome obstacles when pursuing a goal. I got that feeling from the start.
Next, I found a mentor, Artem Gruzdev from Moscow, who helped me to sort through the field of classical Machine Learning. Self-study has its drawbacks, as at some point, you do not have enough knowledge, experience, or wisdom to move forward. You get stuck. Therefore, I think that having a mentor is a big advantage.
Having a mentor can help move forward (Image generated with DALL-E)
Additionally, I attended many online courses, read/watched many tutorials, and studied some books. After a year of such self-education and with the help of my mentor, I planned a dedicated path and began to choose courses more wisely.
Still, the main challenge for me was the practical application of acquired knowledge. I tried taking part in Kaggle competitions, but honestly speaking, this is a second job and requires a lot of time and resources. The benefits of Kaggle is that you get good practical experience, even on well studied tutorial datasets. One of these datasets I later used for my midterm project in **Machine Learning Zoomcamp**.
To be more competitive on the data science job market, I began to dive into Deep Learning and NLP. So from the spring of 2020, I began to take courses on Neural Nets and NLP during which I explored and met trials with neural networks theory, backpropagation, Pytorch, architectures of RNNs, CNNs and transformers.
**What attracted you to Machine Learning Zoomcamp? Why did you decide to take it?**
While studying Computer Vision with Deep Learning, I wanted to refresh my ML knowledge and improve skills in practical deployment. The syllabus of **ML Zoomcamp,** and the community around it — DataTalks.Club — looked great for me, so I enrolled. My expectations were absolutely exceeded. I also had my first experience in peer-to-peer review of other students’ projects and in going public in social networks about what you’ve learned.
Valerii sharing his progress on social media
**What do you do now? Can you tell us more about the internship that you got and the work you needed to do?**
Thanks to **ML Zoomcamp**, I got an internship at an AI start-up, **delphai**, that is focused on extracting, analysing, and structuring firmographic data.
My internship project was to extract information from texts. I worked on detection and extraction of business relations between companies. The task is, given a sentence and a pair of tagged entities, to predict the relation between those entities and the direction of the relation.
For example, there is a sentence in the [news article](https://www.evconnect.com/news/ev-connect-acquired-by-schneider-electric)
“_EL SEGUNDO, Calif., June 21, 2022 —_ _[EV Connect, Inc.](https://www.evconnect.com/?utm_source=release&utm_medium=email&utm_campaign=se)
, a premier electric vehicle (EV) charging solution provider, announced that it has been acquired by_ _[Schneider Electric](https://www.se.com/ww/en/)
, the leader in energy management and automation.”_, the relation between two companies is about _acquisition_, and the direction is _[Schneider Electric](https://www.se.com/ww/en/)
| [EV Connect, Inc.](https://www.evconnect.com/?utm_source=release&utm_medium=email&utm_campaign=se)
| Acquirer - Acquired_.
A screenshot from delphai showing the discovered relationships
Relationship extraction is one of the hottest NLP tasks now. I was able to experience the essential steps of the ML cycle – from dataset creation and annotation to model deployment and serving. It was an invaluable practical experience for me to work with databases, Azure storage, annotation tools, Github and so much more during the project.
As a result of the project the service for partnership relation extraction was deployed and included into “news-relation-extraction” service. Initially, I used Spacy pipelines with transformer models for my project, but recently, we switched to a more sophisticated and promising approach.
**Do you have any suggestions for future students of ML Zoomcamp?**
If you are a total beginner and starting self-education, here are my suggestions:
* Try to get a mentor or supervisor;
* Plan your path;
* Follow professionals on social networks like Twitter or Telegram;
* Group up during courses with other students and communicate;
* If you are non-native English speaker like me, find time for translating English articles or tutorials on topics your are studying to your language;
* Although there are a lot of free courses and materials, don’t hesitate to pay for tuition if you are sure it is worth it and you need it. You will also feel more responsible for outcome;
* A lot of coding practice is a must;
* Keep the motivation! If you don’t feel like studying or coding, follow the advice of Daniel Bourke, a self-taught Machine Learning Engineer and enthusiast, _“Ask yourself: What’s the alternative not to study?”_
**What kind of other materials would you recommend?**
My first introductory book was _“An Introduction to Statistical Learning with Applications in R”_ by Trevor Hastie Et al. As for Python and Pandas I’d recommend books by Ted Petrou and Matt Harrison.
ML practitioner - Abhishek Thakur, his YouTube channel and book. In general, YouTube is a great supplementary source of information and tutorials, use it wisely. Don’t forget to subscribe and give a like!
As for NLP, a good starting point is the bestseller _“Speech and Language Processing”_ by Daniel Jurafsky and James H. Martin, there was a third edition in 2020.
As for coding practice, the best material is a package / library documentation!
**What was the most helpful thing for you in the ML Zoomcamp course?**
* Office hours - the feedback from lecturer on home tasks is essential;
* Slack channel;
* Alexey’s drawings during lectures.
An illustration for ROC curves from Machine Learning Zoomcamp
**Anything else you’d like to mention?**
I take this opportunity to thank everyone involved in the **ML Zoomcamp** creation and who kept it going. Congratulations on great continuations with **DE Zoomcamp** and **MLOps Zoomcamp!**
Machine Learning Zoomcamp is open for registrations. Sign up [here](http://mlzoomcamp.com/)
.
**About delphai**: delphai collects unstructured public company data all over the internet and then uses ML and AI to transform this unstructured data into a structured format to make it searchable and usable for our customers.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Engineers Aren't Plumbers – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Engineers Aren't Plumbers
Data Engineers Aren't Plumbers
==============================
### But almost identical to a less known profession
02 Sep 2022 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
Water pipes (Photo by [LoggaWiggler](https://pixabay.com/users/loggawiggler-15/)
on [Pixabay](https://pixabay.com/)
)
Every time we open an article with a title similar to “**What is a data engineer?**” or “**The difference between data engineer and data scientist**” we get a cliche answer: _Data engineers are like plumbers._
No! No! No! That is wrong. A data engineer can work with pipelines like a plumber but the role is very different.
In this article, I will show you that a data engineer is similar to another profession/job: **hydraulic and water resources engineer.**
I will explain why in three simple arguments:
* Working titles;
* Task goals;
* Tools needed to develop work.
1\. The titles are the same
---------------------------
Well, this is obvious, right? 🙂 They are both engineers.
Mathematics (Photo by [ArtsyBee](https://pixabay.com/users/artsybee-462611/)
on [Pixabay](https://pixabay.com/)
)
According to the [Oxford English Dictionary](https://www.dictionary.com/browse/engineer)
, an engineer is _“someone who designs, builds or maintains engines, machines, or structures”_.
Each role may focus on different resources/products/processes, “water” for hydraulic and water resources engineers, and “data” for data engineers but both handle engineering on it.
They are more “thinking” roles than “manual” roles (like a plumber) since they have to reflect and calculate what are the best solutions for their processes and not act by simple guidelines.
2\. Both have similar working goals
-----------------------------------
This section is similar to the previous but we are focusing on the goal of each engineer.
For example, a mechanical engineer has the working scope to build, maintain and improve mechanical machines that will perform some tasks.
I’m considering a data engineer has the objective of building, maintaining, and improving data pipelines (ETL or ELT processes), data storage structures (data warehouses or data lakes), and providing solid data to the stakeholders.
In a detailed way, the professional has the working scope of guarantee the a) extraction of data from various sources (both internal like relational databases and external sources), b) transformation of data using solid programming skills or software, c) good organization of the data in the correct storage structures and d) quality/organization of all the end-to-end processes and data using orchestrator tools, monitoring tools or other control tools.
A data engineer needs to think of the process as a whole considering downstream and upstream mechanisms.
Water Treatment plant (Photo by [Marcin Jozwiak](https://www.pexels.com/@marcin-jozwiak-199600/)
on [Pexels](https://www.pexels.com/)
)
The specialization of [Hydraulic and Water Resources Engineering](https://www.mcgill.ca/civil/undergrad/areas/water)
by the McGill University of Canada describes these two disciplines as follows:
_“**Water resources engineering** is the quantitative study of the hydrologic cycle — the distribution and circulation of water linking the earth’s atmosphere, land and oceans. (…) Applications include the management of the urban water supply, the design of urban storm-sewer systems, and flood forecasting.”_ and _“**Hydraulic engineering** consists of the application of fluid mechanics to water flowing in an isolated environment (pipe, pump) or in an open channel (river, lake, ocean). Applications include the design of hydraulic structures, such as sewage conduits, dams and breakwaters, the management of waterways, such as erosion protection and flood protection, and environmental management, such as prediction of the mixing and transport of pollutants in surface water.”._
Therefore I’m considering hydraulic and water resources engineers need to guarantee (besides other tasks)
a) the extraction of water from various sources,
b) the correct water cleaning in water treatment facilities (see image above),
c) good organization of the water in the correct storage structures, and
d) quality/organization of all the end-to-end processes with several control tools.
In the table below it is possible to see how identical both roles are in terms of working processes, tasks, or goals (with some examples).
| **Processes/Task/Scope** | **Data Engineer** | **Hydraulic and Water Resources Engineer** |
| --- | --- | --- |
| Extraction of raw product from sources | Relational databases, External API, or CRM data. | Surface water, groundwater, or wastewater. |
| Development and maintain transformation processes | Data transformation by cleaning, deduplication, or data type correction. | Water cleaning by removing organic compounds, or non-organic compounds. |
| Development and maintain storage structure | Data warehouse, data Lakes. | Water towers, water dams. |
| Development of the full process construction | Data orchestration tools. | Computer tools to draw all systems, and wastewater treatment plants. |
| Controlling/Monitoring processes and product | Software tools for data lineage or process control | Sensors all over the process |
| Stakeholders | Data analysts, Data Scientists. | Cities, industrial. |
Therefore you can see that even having different targets both engineers do similar tasks.
3\. They use identical tools
----------------------------
In that cliché of “data engineer equals plumber” it is often written that both have tools. However, the plumber tools are different from the data engineer tools. But both data engineers and hydraulic and water resources engineers use similar tools.
Considering the processes present in the table above I will present you some examples for each role.
For data engineers:
1. SQL for analysis of the data sources;
2. Python, Scala or other programming languages for development.
3. Airflow, Luigi or other for the development of the full process construction
4. Grafana and data testing tools to control and monitor.
For hydraulic and water resources engineer:
1. Tools for geo analysis or GIS tools for analysis of the sources area;
2. Excel or similar tool for calculus;
3. CAD software tools for the development of the full process construction;
4. Sensors for quality and quantity water control.
So all the tools for both engineers are complex tools (mostly software) with the purpose of proceeding to the estimation of the best solution. They are not manual tools like hammers.
Conclusion
----------
In summary it was presented in three simple subjects that data engineers are less identical to plumbers and more to hydraulic and water resources engineers.
Hydraulic and water resources engineers and data engineers resemble because
* Both are engineers, a “mind role” and not a “manual role” like a plumber;
* They have a similar working scope of extracting/studying raw product, transforming it, storing it and deliver to the stakeholder;
* These positions always have to understand all the process end-to-end by being aware of downstream and upstream operations;
* The tools that both positions use are complex tools aiming calculation and analysis.
And the cliché is down!
What do you think, do you agree with me?
Do you think I am going to be attacked by Mario Bros? 🧑🔧
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Regularization in Regression – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Regularization in Regression
Regularization in Regression
============================
### Remedy for numerical instability
22 Sep 2022 by [Ksenia Legostay](https://datatalks.club/people/ksenialegostay.html)
There are a lot of methods of how you can improve your ML model accuracy. They include feature engineering, missing value imputation, improvements in data quality, etc.
One of the effective approaches is regularization. It is a popular concept that helps to control coefficients under the numerical instability in computation taste such as model training.
In this article, we will take a closer look at why you might want to regularize your model. As an example, we will apply a basic regularization technique to a simple linear regression model and learn how it influences the model.
### Linear Regression
Linear regression is a supervised machine learning model, which can be expressed in a matrix form as follows:
\\\[g(X) \\approx y\\\]
$X$ is a matrix where the features of observations are rows of the matrix and $y$ is a vector with the values we want to predict. Function $g(X)$ can be represented as a matrix-vector multiplication of feature matrix $X$ and some weight vector $w$:
\\\[Xw = y\\\]
After some transformations described in [Training Linear Regression: Normal Equation](https://www.youtube.com/watch?v=hx6nak-Y11g&list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR&index=18)
lecture of [Machine Learning Zoomcamp](https://github.com/alexeygrigorev/mlbookcamp-code#machine-learning-zoomcamp)
, weight vector $w$ can be represented as:
\\\[w = (X^T X)^{-1} X^T y\\\]
where
* $X^T$ is the [transpose](https://en.wikipedia.org/wiki/Transpose)
of $X$,
* $X^T X$ is a [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix)
,
* $(X^T X)^{-1}$ is the [inverse](https://en.wikipedia.org/wiki/Invertible_matrix)
of Gram matrix.
A matrix inversion should be considered with caution. If a matrix contains a column that is a linear combination of its other columns the matrix is singular, which means the inverse matrix does not exist.
### Why do we need regularization in Linear Regression
Linear dependent columns in a matrix is not a typical case in real-world problems, even though due to noise in the data, characteristics of your machine, OS, or NumPy version there might be some similar vectors in the above sense. When it happens, the weight vector $w$ can result in very large values (both positive and negative) and the overall model predictions won’t be useful.
To overcome this numerical instability problem we can refer to regularization. Regularization in linear regression guarantees the existence of inverse matrix $(X^T X)^{-1}$
One of the regularization techniques is adding a factor to the diagonal of matrix $X^T X$ like this:
\\\[w = (X^T X + \\alpha I)^{-1} X^T y\\\]
where
* $I$ is an [Identity matrix](https://en.wikipedia.org/wiki/Identity_matrix)
and
* $\\alpha$ is a (typically small) factor.
This modification of the linear regression is commonly called [Ridge Regression](https://en.wikipedia.org/wiki/Ridge_regression)
.
### How regularization can fix your Regression
Let’s demonstrate the effect of regularization through an example and see that the more regularization we add (factor $\\alpha$), the smaller the weights $w$ become.
We will build a Linear Regression model for predicting car prices based on a dataset from Kaggle - [Car Prices Dataset](https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-02-car-price/data.csv)
.
_The full code is in the notebook [here](https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/course-zoomcamp/02-regression/notebook.ipynb)
._
For the sake of simplicity we won’t use any specific ML packages, instead we train a simple linear regression model in a vector form:
# define feature matrix X of size 6x3 with nearly same second and third column
X = np.array([[4, 4, 4],\
[3, 5, 5],\
[5, 1, 1],\
[5, 4, 4],\
[7, 5, 5],\
[4, 5, 5.00000001]])
# define vector y of size 1x6
y= np.array([1, 2, 3, 1, 2, 3])
# calculate Gram matrix for X
XTX = X.T.dot(X)
XTX
array([[140. , 111. , 111.00000004],\
[111. , 108. , 108.00000005],\
[111.00000004, 108.00000005, 108.0000001 ]])
# take inverse matrix of Gram matrix
XTX_inv = np.linalg.inv(XTX)
XTX_inv
array([[ 3.86409478e-02, -1.26839821e+05, 1.26839770e+05],\
[-1.26839767e+05, 2.88638033e+14, -2.88638033e+14],\
[ 1.26839727e+05, -2.88638033e+14, 2.88638033e+14]])
# calculate a weights vector w:
w = XTX_inv.dot(X.T).dot(y)
W
array([-1.93908875e-01, -3.61854375e+06, 3.61854643e+06])
As you can see the second and the third values of the weights vector $w$ are huge. It comes from the fact that initial feature matrix $X$ contains almost the same columns: 2 and 3.
Let’s introduce a regularisation term and see how the vector $w$ changes:
# add regularization factor 0.01 to the main diagonal of Gram matrix
XTX = XTX + 0.01 * np.eye(3)
# take inverse matrix of Gram matrix
XTX_inv = np.linalg.inv(XTX)
XTX_inv
array([[ 3.85624712e-02, -1.98159300e-02, -1.98158861e-02],\
[-1.98159300e-02, 5.00124975e+01, -4.99875026e+01],\
[-1.98158861e-02, -4.99875026e+01, 5.00124974e+01]])
# calculate a weights vector w:
w = XTX_inv.dot(X.T).dot(y)
W
array([0.33643484, 0.04007035, 0.04007161])
The weights in vector $w$ now are reasonable and suitable for prediction.
The example of applying regularization in Linear Regression for car price prediction can be found in this [notebook](https://github.com/Ksyula/ML_Engineering/blob/master/02-regression/Regularization%20in%20Linear%20Regression.ipynb)
.
### Summary
The main purpose of regularization techniques is to control the weights vector $w$ and not let it grow too large in magnitude. A regularized regression considered in this article is called Ridge Regression and you can typically find it in various ML packages (e.g. [scikit-learn](https://scikit-learn.org/)
).
Regularization is capable of finding a solution when there are correlated columns, reduce overfitting and improve your model performance in many cases.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Agita Jaunzeme – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Agita Jaunzeme
Agita has designed a career spanning DevOps/DataOps engineering, management, community building, education, and facilitation. She has worked on projects across corporate, startup, open source, and non-governmental sectors. Following her passion, she founded an NGO focusing on the inclusion of expats and locals in Porto. Embodying the values of innovation, automation, and continuous learning, Agita provides practical insights on promotions, career pivots, and aligning work with passion and purpose.
[](https://linkedin.com/in/agita)
### Events
* Career choices, transitions and promotions in and out of tech ([watch on youtube](https://www.youtube.com/watch?v=QKWu5-6_6TE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Admond Lee Kin Lim – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Admond Lee Kin Lim
Admond is a data scientist, writer, and speaker who wants to empower people with clarity and insights through data and technology. He is currently a Data Science Instructor at Hackwagon Academy with a mission to make data science accessible to everyone through education.
Prior to joining Hackwagon Academy, he was a Data Scientist at Micron focusing on Smart Manufacturing and AI while working as an independent data scientist and consultant, helping startups with various data science and machine learning projects.
Being a passionate data science communicator (writer & speaker) who shares about data science, his data science work and experience have been featured by various publications, including KDnuggets, Medium, Tech in Asia, and AI Time Journal and Data Science UA Conference 2020.
[](https://twitter.com/admond1994)
[](https://linkedin.com/in/admond1994)
### Events
* DataTalks.Club Conference: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=ltFkvoiA57M)
)
* Personal Branding ([watch on youtube](https://www.youtube.com/watch?v=tQRQnz_aHYQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Personal-Branding---Admond-Lee-Kin-Lim-ern77e)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Guidelines to Get a Data Engineer Job Against the Odds – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Guidelines to Get a Data Engineer Job Against the Odds
Guidelines to Get a Data Engineer Job Against the Odds
======================================================
### It worked for me
04 Jan 2023 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
Getting a data engineer job is not accessible if you don’t have an IT background (Image from Geralt in [Pixabay](https://pixabay.com/)
)
Data engineers are responsible for managing big data sets and building applications to move, transform and analyze them. They work closely with business analysts, data analysts, data scientists, and project managers to build solutions that help businesses make sense of their data. If you want to understand deeper what the data engineer functions there are several articles and videos explaining it but I advise you to read this [article](https://www.theseattledataguy.com/what-skills-do-data-engineers-need/)
with a video from [Seattle Data Guy](https://www.theseattledataguy.com/)
and this [article](https://www.coursera.org/articles/what-does-a-data-engineer-do-and-how-do-i-become-one)
from Coursera. But be aware of not mixing concepts by falling into the common cliché of saying that data engineers are similar to plumbers because both work with pipelines (see [my article](https://medium.com/@lgsoliveira/data-engineers-arent-plumbers-c3dbb585a901)
about this).
For someone without an IT background (software engineering, computer science technical courses, or similar), getting a data engineer job may not be an easy task and straightforward_._ Even for someone with an engineering master’s degree or another technical background, it can be difficult to be accepted by companies that need data engineers.
In this article, I am going to present the following:
* Why is it so hard to get a data engineer job without an IT background?
* Five guidelines to get a data engineer job
1\. Why is it so hard to get a data engineer job without an IT background?
--------------------------------------------------------------------------
There is a shortage of data engineers but several companies need this professional leading to a mismatch between the number of data engineers needed and the available candidates. Even with the job market unbalanced, it is weird to think why a professional without an IT background can’t easily get a data engineer job.
Why is this happening? For several reasons:
* A data engineer is not typically an entry-level position. Many employers look for data engineers with two or more years of experience in analytics, IT management, computer programming, or a role-related field. And getting into an IT job without a computer science background is difficult;
Meme on “experience vs job” (extracted from [me.me](https://me.me/i/i-cant-hire-you-without-experience-i-need-a-job-e8aad172769c4d14bc42c4dc14b76228)
* Because data engineering is the intersection of software engineering and data science, some employers prefer candidates with at least a bachelor’s degree in a related field such as computer science or data science;
* There are not many specific specializations or pos-degrees in data engineering. Only now colleges and universities are attracting more interest in the study of data engineering by offering specializations and bachelor’s degrees in the field. Here are some exceptions online, [Data Engineering Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
and [Learn Data Engineering](https://www.linkedin.com/company/learn-data-engineering/)
.
* Data engineer jobs are relatively new, and the industry is rapidly changing, with new technologies being developed all the time. This is excellent for someone already familiar with the area, but it will be quite perplexing for a newcomer. (“The programming language A is better than the B”, “The cloud provider X is the best” 😨)
The number of tools for data engineering (Picture from the blog of the lake)
* Many companies give preference to hard skills rather than soft skills. So even if a professional is very dedicated and willing to learn it will be set aside for someone with some knowledge in IT;
* Prejudice and bias. 😐 It is weird to think about it but there is still some bias among IT professionals against non-IT background professionals. It is strange to happen this in 2022 but it still happens (It occurred to me more than once…);
I will explain in the next section how you can try to overcome all these problems and get a data engineer job against the odds.
2\. Some guidelines to get a data engineer job
----------------------------------------------
### 2.1. Education, education, education!!!
Hard skills in IT it is important to get a data engineer job so you need to learn some basics (picture from [Jeko](https://unsplash.com/@jexo)
in [Unsplash](https://unsplash.com/)
)
As previously said, one barrier to obtaining a position as a data engineer for someone without IT experience is the lack of IT hard skills/knowledge. So the solution is “simple,” but laborious: you must complete a large number of courses, boot camps, or other sorts of education that any data engineer requires. A comprehensive and well-designed course/specialization or Bootcamp will provide you with the essential tools you need to become a data engineer. This is a great option if you don’t want to go to college (and honestly, it’s expensive).
“But which topics should I learn to work as a data engineer?” you may ask. There are numerous articles, websites, and publications that detail the courses of study required to become a data engineer. One great article is this [Data Engineering Roadmap For 2021](https://medium.com/coriers/data-engineering-roadmap-for-2021-eac7898f0641)
from [Ben Rogojan](https://medium.com/@SeattleDataGuy)
(AKA Seattle Data Guy) or if you want something more visual you also have this [Modern Data Engineer Roadmap](https://github.com/datastacktv/data-engineer-roadmap)
by [data stack. tv](https://datastack.tv/)
.
In my humble point of view, you should first learn:
* **Coding**: This is a very important skill in data engineering. My favorite programming language is Python but Scala and Java are also widely used in data engineering. Choose one and learn the basics, the more important modules, and something of programming object-oriented;
* **SQL and database fundamentals**: SQL is the most important database language and it is essential in data engineering. First, you should learn the basics of DML and DDL, then understand how a relational database is organized, and in an advanced way you should learn data warehousing;
* **Tools provided in clouds**: To be a good data engineer, you should be able to work with tools available from cloud providers (AWS, Azure, Google Cloud Platform). Nowadays it is important to know how to work with cloud storage (eg. AWS S3, Google Storage), with databases on the cloud (eg. Redshift, BigQuery), with tools for data orchestration (eg. Azure Data factory) or with tools regarding Spark (eg. Databricks). I advise you to know, at least, which tools the main three cloud providers have to offer;
* **Some other software knowledge:** You should also know some bash commands, how to work with version control, and some knowledge regarding the basics of IT.
To learn all these contents you can access the multiple free MOOC platforms available. If you don’t know which one to choose from see my favorites [here](https://link.medium.com/vD9y7B3thvb)
.
### 2.2. Show yourself with data projects
A good portfolio is mandatory if you have 0 experience in IT (picture from [Joshua Aragon](https://unsplash.com/@goshua13)
in [Unsplash)](https://unsplash.com/)
Like in the meme presented before you know it is harder to get a job without experience but you will get experience with a job. So, how do we break this negative cycle? By doing some personal projects and creating a portfolio.
Data engineering job applicants are frequently asked about their projects. If you’ve never worked as a data engineer, you can explain a project you worked on for a course or submitted to GitHub after doing some IT challenges like [HackerRank](https://www.hackerrank.com/challenges/challenges/problem)
or [LeetCode](https://leetcode.com/problemset/all/)
.
You can also volunteer for some associations that do IT jobs pro-bono. For example, I did a web scraper for the [Data Science For Social Good Portugal](https://www.dssg.pt/)
.
Data Science for Social Good Portugal is a community of volunteers in data science and data engineering (official logo)
Also, a good idea is to “show yourself” with certificates and certifications. In this [article](https://link.medium.com/xe1ufXKvhvb)
, I explained which are the best.
### 2.3. Be the best friend with networking
Networking is very important nowadays (picture from [Chris Montgomery](https://unsplash.com/@cwmonty)
in [Unsplash](https://unsplash.com/)
)
We have a popular saying in Portugal “Who has friends will not die in prison!” that says that if you have a strong network you will have great advances in life (unfortunately some people take that saying to the extreme😒 ).
**How to build and maintain a solid network?**
* First of all, you need to follow Linkedin the best professionals in data science and data engineering. If you don’t know who they are check my [article here](https://link.medium.com/tD6kT84Mgvb)
.;
* Then you should use several platforms to show yourself. For example, after an achievement writes on Twitter: “Hello everyone, today I finished my 100 HackerRank challenge”;
* You must connect to recruiters or professionals in companies you like. After the first connection it is important to keep in touch with them;
* Last but not least, In platforms with professional purposes like Linkedin or Landing. jobs you must always (!!!!) keep your CV updated.
### 2.4. Learn from failure
Failure is important to learn (picture from [Brett Jordan](https://unsplash.com/@brett_jordan)
on Unsplash)
This is a very important guideline because if you think like this you will always win, it is a win-win situation. Just remember that learning from failure means that sometimes you win and sometimes you learn something that will lead to winning in job seeking.
**How can this be applied in job searching?**
When looking for a new job and you fail you have to take the time to reflect on the “WHY” you were not successful in a particular job search and to use that information to make changes and improvements in your approach. This might involve analyzing the feedback you received from employers, assessing your job search strategy, and identifying areas where you can improve your skills or qualifications.
In my case, I used this principle, especially in the technical interviews. When, during the technical interview, I was asked one theme of data engineering that I didn’t know how to answer I went to study that theme for the next technical interview.
### 2.5 Be resilient, persistent, patient, and never give up
I finished my Master’s in Engineering in 2007, and only in 2017, I got my first good and stable engineering job. During those 10 years, I’ve been unemployed, in a foreign country, working in a non-technical area, etc… You can see the history of my professional experience in my article [Every Time I am Cleaning the Dishes I Remember Norway …](https://link.medium.com/lPNKFECKEvb)
"A river cuts through a rock not because of its power but because of its persistence." by James N Watkins (picture from [Àlex Rodriguez](https://unsplash.com/@alexabad)
in Unsplash)
If you read my article you may be thinking I am an optimistic and positive person but… “Au Contraire My Friend”, I am a very pessimistic person and always with negative thoughts but I am **not a quitter**!! 😁I was able to achieve my current position because I was very persistent, I had to be patient and resilient.
Resilience and persistence are important when looking for a new job because the process can be challenging and unpredictable. It will be common for you to face rejection or to have to apply to many jobs before finding the right fit. Being resilient can help you stay motivated and keep trying, even when faced with setbacks.
Overall, being resilient and persistent can be a valuable asset in the job search process and in your career.
Summary
-------
Getting a data engineer job may not be an easy task and straightforward, especially for someone without an IT background.
In this article, I presented to you the main reasons why getting a data engineer position is hard and how you can surpass some difficulties by following these five pieces of advice:
1. Education, education, education!!!
2. Show yourself with data projects
3. Be the best friend with networking
4. Learn from failure
5. Be resilient, persistent, patient, and never give up
And you, are you trying to get a data engineer position?
Are you following these pieces of advice?
Good luck 😉
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Do You Know the Golden Rules While Working With Data? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Do You Know the Golden Rules While Working With Data?
Do You Know the Golden Rules While Working With Data?
=====================================================
### With these, you will be a great professional
29 Sep 2022 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
Following the rules will help you to be a great data professional (Photo by [LoggaWiggler](https://pixabay.com/users/loggawiggler-15/)
on [Pixabay](https://pixabay.com/)
)
Data is an asset that can be captured, stored, analyzed, and retrieved whenever necessary. However, to have data is not enough to be so useful for businesses, rather, it is how we use data that makes all the difference.
Working with data isn’t as simple as “just” receiving information from a source and storing it in a designated location. Rather, it requires skillful manipulation to extract only what you need and no more, and the best possible way while being very efficient.
While working as a data engineer I always try to follow some rules. Some rules were given to me by my own Data Master: [“Data Mr. Myagi”](https://www.linkedin.com/in/guillermo-franco-garcia/)
(“[Wash in, Wash out, Daniel-San](https://www.imdb.com/title/tt0087538/)
” 🥋 ), and some I compose on my own.
The following five golden rules (see “Golden Rule” definition in [Cambridge Dictionary definition](https://dictionary.cambridge.org/dictionary/english/golden-rule)
) will help you work efficiently with data:
* Automate repetitive tasks;
* Always work with data as a “defensive driver”;
* If the solution is “nasty” then is wrong;
* Do an extra effort to develop correctly from the start;
* Data sources aren’t always right.
### 1\. Automate repetitive tasks
One of the most important principles when working with data is to automate processes whenever possible. This will allow you to work efficiently, make fewer mistakes, and potentially save money.
Automate repetitive tasks to be a “Data Jedi” (Photo by [Dan Pupek](https://www.flickr.com/photos/danandkir/)
from [Flickr](https://www.flickr.com/)
)
Automation can include a number of things, including integrating data from one system with another or using software to replace certain manual tasks. You cannot automate everything, but you should be able to identify areas where automation would be beneficial.
Likewise, you can automate processes in your personal life. A simple example is to set recurrent monthly payments (eg. water bill) by the direct debit transaction (where the bank extracts the amount for that bill on the same day every month).
### 2\. Always work with data as a “defensive driver”
Once I heard the best definition for being a “defensive driver”:
* To drive defensively is not about following the road rules but assuming that the other drivers will NOT follow these rules.
And this is what I want you to do, to work with data assuming that the data and/or the users that work with data will get wrong one day.
If you are going to work with data, you need to think about the future and what might happen in that future. Having data is good, but only if you can use it to make informed decisions not only today but also in the future. Therefore, you need to know what you may expect in future years.
To be more clear I will use examples:
1) If we are doing a SQL query based on dates this below is wrong because it will fail in January when the result will be 0 and, of course, there is no month 0.
SELECT MONTH(current_date) - 1 AS previous_month
FROM table
2) If you use hard-coded in your programming code sooner or later it will fail because the user can change the way he writes the information. In this example, the user can write the city in French, “Lisbonne”, and it will fail.
if City_Name == 'Lisboa' OR City_Name == 'Lisbon':
Country = 'Portugal'
### 3\. If the solution is “nasty” then is wrong
For your ETL pipeline, Machine Learning model, or structure for visualization, you can have more than one solution but I can assure you that one will be the wrong solution: The “Nasty” Solution (by definition, the [bad or very unpleasant](https://dictionary.cambridge.org/dictionary/english/nasty#:~:text=bad%20or%20very%20unpleasant%3A)
solution).
When you are developing your code or process you should:
* Keep it as simple as possible (Do you know the KISS principle? - see definition [Wikipedia](https://en.wikipedia.org/wiki/KISS_principle)
);
* Avoid redundancies. For example, enter a filter and then remove it later;
* Guarantee each function or sub-process only performs one task;
* Not create multiple nested codes or queries. More than three nested queries means something is wrong;
* Document your code.
This golden rule takes into consideration internal quality code and the table below shows some properties the code must have to have good internal quality.
| **Code Properties** | **Definition** |
| --- | --- |
| Concision | Code does not suffer from duplication |
| Cohesion | Each \[module\|class\|routine\] does one thing and does it well |
| Low coupling | Minimal interdependencies and interrelation between objects. |
| Simplicity | The quality or condition of being easy to understand or do. |
| Generality | The problem domain bounds are known and stated |
| Clarity | The code enjoys a good auto-documentation level |
Internal quality code proprieties and its definitions (adaptation of [Good Code](https://wiki.c2.com/?GoodCode)
information)
With this advice, I assure you that it will have a “beautiful” and readable code, and will allow future needed changes because internal quality code will affect external quality code.
And the golden rule is simple:
* If the code is “ugly” then it is wrong. 🙂
### 4\. Do an extra effort to develop correctly from the start
This is a rule that can be seen as almost impossible to follow but it is very important because it will save you lots of time.
When developing your process you can follow two approaches:
1) Write all the code/process to just give results and then, in the end, get the exact result proposed, correct some errors and write documentation, or
2) Develop everything from the start.
I am a big fan of number 2) due several reasons:
* If you correct the errors and write documentation while you are developing you will have a “fresh idea” of each sub-process and it will be easier for you;
* If you take more time on each sub-process you will reflect more on it allowing you to “think on the future” and be a data “defensive driver”. (see golden rule number 2.)
* Developing everything from the start will help you to reach a cleaner code (see golden rule number 3.);
Of course, this is a difficult golden rule to follow because it means to do an extra effort on each sub-process but I assure you it will compensate.
### 5\. Data sources aren’t always right
While you should trust data, you shouldn’t trust it too much. For example, if you are working with survey data and have found some interesting statistics, you should be careful about drawing too many conclusions from it. You need to understand the source of the data and, if possible, try to replicate the results so that you know they are accurate.
There are three main reasons why you should be careful about trusting the data:
* Data can be wrong: Data can be wrong for a number of reasons. It may have been collected incorrectly, the sample size may have been too small, or the data may have been entered incorrectly;
* Data can be biased: Even if the data is collected and entered correctly, it can still be biased. For example, if you are using online surveys, people who are visiting that site may be different than the rest of your customer base;
* Data can be outdated: Data can also be outdated. For example, if you are looking at sales data from last year, this data may not be applicable to this year.
Of course, it is impossible to always be “on top” of the data source quality but if you are doing a process, an analysis, a model, or visualization you have to have a critical vision of the data.
### **Summary**
Data is an important asset for businesses, but it is how you work with the data that makes all the difference.
These five golden rules mentioned in the article will help you work efficiently to be a better data professional. Just remember that:
1. If you have repetitive tasks then they should be automated;
2. While working with data you have to be a “chess player”, always thinking about the next moves;
3. Confusing solutions are always wrong and will affect your process;
4. Make additional efforts to develop properly from the beginning;
5. Don’t always trust the data source quality.
Do you think these golden rules are possible to follow and will improve your work?
Do you have personal data guiding principles?
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# What is DataOps exactly? – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
What is DataOps exactly?
What is DataOps exactly?
========================
### An overview of DataOps and what makes it different from the other DevOps practices.
11 Jun 2022 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
DataOps aligns teams and automation across the data lifecycle ([photo](https://unsplash.com/@fabioha?utm_source=medium&utm_medium=referral)
via [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)
).
It’s hard to overstate the importance of data in modern enterprises. As a new practice, DataOps is aimed at helping organizations overcome obstacles in their data analytics processes. **But what exactly is this emerging practice, and how can it help businesses better leverage their data?** In this article, we’ll explore how important DataOps has become by looking at its various aspects and examining the ways it complements other DevOps and MLOps practices.
The article is organized as follows:
* What is DataOps
* The seven steps of DataOps
* DataOps vs MLOps
* DataOps tools
1 What is DataOps
-----------------
DataOps is the result of applying DevOps principles to the data lifecycle.
The basic idea of DataOps is, **“if you build a system around that — that automates a lot of the monitoring, deployment, and collaboration—your productivity goes way up, your customers are much happier, and you end up doing better work.”**
DataOps focuses on three processes:
* **Error Reduction**, which improves your customer’s trust in your data. In practice, you should monitor all the software, aiming at checking the stuff that you’re doing.
* **Cycle Time of Deployment,** which involves how fast you can get new models, new datasets, and new visualizations from your mind into production. This aspect involves both velocity and risk.
* **Increasing Team Productivity**, with a reduction in the number of meetings and collaboration.
All the previously defined processes are measurable, and you should measure them. For example, you should measure metrics that answer the following questions:
* How much work is your team doing?
* How often are things broken?
* How fast are you getting new things into production?
Those are really important metrics. It’s not so much about data science or data engineering. You always optimize the whole and not just the part. The idea is to show your work to people, get feedback from them, and then iterate on this feedback.
2 The seven steps of DataOps
----------------------------
DataOps involves the following seven steps:
DataOps steps: add data tests, version control, branch/merge, prod tests, multiple environments, containerize and reuse, parameterize processes.
* **Add data to logic tests,** to move from DevOps to DataOps.
* **Take your code and put it in version control.** Don’t have it on your hard disk somewhere or file share. You could use frameworks such as Github or GitLab.
* **Branch and Merge**. When you’re changing something in development, run automated tests against that to judge regression or impact analysis. If you change something on the back end, you’re able to tell if the front end is broken in a very simple way.
* **Write automated tests that run in production**. You should test whether your code behaves as expected or not.
* **Use multiple environments**. Many developers working on the same project at the production level may lead to conflicts. For this reason, it is important that every team member can work on a local copy of the project.
* **Reuse and Containerize**. You should reuse your code and make it work independently on your local machine. For this reason, you should wrap your software in containers, such as Docker.
* **Process Parameterization**, to make your software pipeline flexible to changes.
You can find more information about the seven steps of DataOps at [this link](https://em360tech.com/sites/default/files/2020-07/Data-Kitchen_WP_7Steps_710816A_LR.pdf)
.
To run your tests, you could use tools such as Great Expectations to run your tests in ways that are simple. However, you can also write tests on your own, in a lot of ways that are pretty simple. For example, you can do row count checks, or you can write SQL queries to do the tests.
So the whole idea is that those tests:
* should be done automatically,
* are in version control
* run during production,
* are run during development.
In practice, about 10% of your work should be developing automated tests.
3 DataOps vs MLOps
------------------
What are the differences between DataOps and MLOps? Are they the same thing or different?
There are two answers to that. **From the point of view of an engineer, the answer is no.** It’s the same idea. It’s just DevOps applied to data, so you call it DataOps or MLOps.
**From a more general point of view, the answer is yes.** You can use the term DataOps to encompass the data, the model, the visualization, and the governance. The objective of DataOps is to optimize the whole of that, not just a single part.
More formally, DataOps is the concept of building an organization’s data infrastructure in a way that will allow you to not only perform better as an organization but also be more agile. It’s not just about having good data; it’s about having trustworthy and reliable data.
DataOps can lead to the following benefits:
* Increased quality of data
* Increased speed of data
* Increased efficiency of data
* Increased accuracy of data
* Improved consistency (i.e., fewer errors) across teams or departments that are working with the same dataset(s).
The following figure shows the specific focus of Data, Machine Learning, Development, and Operations:
DataOps vs MLOps: data, models, development, and operations responsibilities and overlaps.
4 DataOps tools
---------------
DataOps tools automate and simplify all parts of the data life cycle. They increase the agility of data management for organizations and accelerate data analytics for users.
There are four types of DataOps tools:
* **All-in-One Tools**, which focus on data management, such as data ingestion, transformation, analysis, and visualization.
* **DataOps Orchestration Tools**, which permit you to manage complex data pipelines in a centralized manner.
* **Component Tools**, which focus only on a single component of the whole pipeline.
* **Case-Specific Tools**, which focus on specific domains.
Some popular tools for DataOps include [Great Expectations](https://greatexpectations.io/)
, [Dataform assertions](https://dataform.co/blog/data-assertions)
, [Monte Carlo](https://www.montecarlodata.com/)
, and [dbt tests](https://docs.getdbt.com/docs/building-a-dbt-project/tests)
.
Summary
-------
Congratulations! You have just learned the basic concepts behind DataOps!
The DataOps approach can be a powerful tool for any company, and it’s worth taking the time to understand the framework and its benefits. The most important thing to remember is that this practice is about collaboration.
It’s about building a culture where everyone, from engineers to data scientists, works together with their stakeholders in order to produce data-driven results faster and more efficiently.
The content of this article has been inspired by the podcast episode [Storytime for DataOps](https://datatalks.club/podcast/s08e05-storytime-for-dataops.html#the-essence-of-dataops)
with Christopher Bergh at DataTalks.Club.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Agnes van Belle – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Agnes van Belle
Agnes van Belle works as a Data Scientist at Berlin-based HeyJobs, a scale-up focused on intelligently targeting talent for jobs and vice versa.
Formerly she was employed at OLX Group (online classifieds), where she build a large part of their internal A/B testing framework. In addition, she was a regular consumer of said framework, when developing and testing solutions regarding search and recommendation.
Further back she worked as Search R&D team lead at Textkernel, an Amsterdam-based company using parsing and ranking for matching CVs and vacancies.
Agnes studied Artificial Intelligence at the University of Amsterdam, and has spoken or be a panel member at Berlin Buzzwords, ECIR, ECML-PKDD, and Haystack Europe. Her main interests are around representation learning, NLP, information retrieval and statistics.
[](https://linkedin.com/in/agnesvanbelle)
[](https://github.com/agnesvanbelle)
### Events
* Setting up an A/B Testing Framework ([watch on youtube](https://www.youtube.com/watch?v=u3afDiIrKo4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aishwarya Jadhav – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aishwarya Jadhav
With over 4 years of industry experience as a Machine Learning Engineer, I have demonstrated success driving impactful projects spanning Multimodal LLMs, Generative AI, and Computer Vision domains. My passion lies in translating applied research into cutting-edge AI solutions, leveraging a diverse skill set across ML and Engineering to build and deploy high-quality products.
I hold a Master’s degree from Carnegie Mellon University, where my focus was on research in Multimodal Deep Learning and Text Information Extraction.
Check out our research on assistive technologies for the visually impaired: https://aiguidedog.wordpress.com/
[](https://linkedin.com/in/aishwaryajadhav8)
### Events
* Lessons from Applied AI: Tesla, Waymo, and Beyond ([watch on youtube](https://www.youtube.com/watch?v=vK_SxyqIfwk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aditya Gautam – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aditya Gautam
Aditya Gautam is an AI researcher and engineer whose work spans industrial innovation, academic research, and AI policy. He has held roles at Google and Meta, working on recommendation systems, integrity, and large-scale generative AI deployment. His research covers topics including reel multimodal understanding, multi-agent systems, and LLM evaluation, and he has published in top-tier conferences such as ICWSM while serving as a peer reviewer for venues like NeurIPS, ICML, and AAAI.
Aditya is also an active voice in the AI community: he speaks at industry events such as the Databricks Data + AI Summit and Analytics Vidhya, and shares insights on the economics and practical adoption of LLMs and AI agents. He holds a Master’s degree from Carnegie Mellon University.
[](https://linkedin.com/in/aditya-gautam-68233a30)
### Events
* [The Future of AI Agents](https://luma.com/0s0dcpdl)
on 10 Feb 2026
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Agostino Calamia – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Agostino Calamia
Agostino is a Senior Data Scientist at Shopify who has a background in ecommerce and fintech. While now working for a pioneer in ecommerce, he previously worked for N26 while the company went from a small start-up to one of Germany’s biggest banks. During his time in both companies, he gained a lot of knowledge on how to standardize processes for ML development in fast-growing companies and organizations that contain more than 300 data scientists. His interest in delivering high quality while repeatedly building ML models fast, led to the fact that he created his own framework on how to develop models.
[](https://linkedin.com/in/agostino-calamia)
[](https://www.agostinocalamia.com/)
### Events
* From Data to Deployment ([watch on youtube](https://www.youtube.com/watch?v=DaW7I5ag9CI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Agnieszka Mikołajczyk – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Agnieszka Mikołajczyk
Agnieszka Mikołajczyk - a deep machine learning enthusiast preparing her PhD thesis at the Gdansk University of Technology. On a daily basis, she conducts her research as part of her grant “Detecting and overcoming bias in data with explainable artificial intelligence” and as part of her R&D work at Voicelab.ai. In her free time she organizes and actively contributes to the scientific community, among others by participating in numerous open source projects (e.g. detectwaste.ml, [Omdena](https://omdena.com/)
- AI for Good).
[](https://linkedin.com/in/agnieszkamikolajczyk)
[](https://github.com/AgaMiko)
[](https://amikolajczyk.netlify.app/)
### Events
* Data Science for Social Good ([watch on youtube](https://www.youtube.com/watch?v=arwHfVX8_cc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alena Astrakhantseva – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alena Astrakhantseva
Alena is a Python Developer, Data Scientist, and Developer Relations professional at dltHub, the company behind the dlt (data load tool). I build and optimize data pipelines (ELT/ETL), engage with our fantastic community, and create educational content to make data processing more accessible.
[](https://linkedin.com/in/alenaastra)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How to run PostgreSQL and PgAdmin with Docker – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How to run PostgreSQL and PgAdmin with Docker
How to run PostgreSQL and PgAdmin with Docker
=============================================
### And an alternative with Docker Compose
07 Mar 2023 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
PgAdmin GUI welcome page (picture by author)
As you saw in one of the previous articles I set [PostgreSQL to run in a Docker container](https://medium.com/dev-genius/learn-how-to-run-postgresql-locally-c388483712e9)
. However, to manage and query databases in the [PostgreSQL](https://www.postgresql.org/)
server I had to use other software like DBeaver or DataGrip.
What if I set a database management tool running in a Docker container too? Then we don’t need to install anything less but Docker 😁
This article is made under the [Data Engineering Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
by [Data Talks Club](https://datatalks.club/)
and will show up some code related to these videos:
* [Connecting Postgres and pgAdmin](https://www.youtube.com/watch?v=hCAIVe9N0ow&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb)
;
* [Running Postgres and pgAdmin with Docker-Compose.](https://www.youtube.com/watch?v=hKI6PkPhpa0&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb)
In this article I will present:
* A brief introduction to Docker, PostgreSQL, and pgAdmin;
* What do you have to do to run PostgreSQL and pgAdmin in two Docker containers;
* What do you have to do to run PostgreSQL and pgAdmin with Docker compose.
1\. A brief introduction to Docker, PostgreSQL, and pgAdmin
-----------------------------------------------------------
### 1.1 Docker
[Docker](https://www.docker.com/)
uses OS-level virtualization to allow fantastic customization of software in containers that can be easily shared with your colleagues or between development environments, “and be sure that everyone you share with gets the same container that works in the same way” (from the Official Docker website).
Containers are isolated from one another and bundle their own software, libraries, and configuration files.
Because all of the containers share the services of a single operating system kernel, they use fewer resources than virtual machines.” (Source [Wikipedia](https://en.wikipedia.org/wiki/Docker_(software)%7B:target=%22_blank%22%7D)
)
### 1.2 PostgreSQL
Official PostgreSQL logo
According to the [official website](https://www.postgresql.org/)
“PostgreSQL is a powerful, open source object-relational database system with over 35 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.”.
“It was originally named POSTGRES but in 1996, the project was renamed to PostgreSQL to reflect its support for SQL.
It features transactions with Atomicity, Consistency, Isolation, Durability (ACID) properties, automatically updatable views, materialized views, triggers, foreign keys, and stored procedures.
It is designed to handle a range of workloads, from single machines to data warehouses or Web services with many concurrent users.” (Source [Wikipedia](https://en.wikipedia.org/wiki/PostgreSQL)
)
### 1.3 PgAdmin
[PgAdmin](https://www.pgadmin.org/)
is a free and open-source graphical user interface (GUI) tool for managing and developing databases. It is specifically designed for PostgreSQL, a powerful and popular open-source database management system.
PgAdmin provides users with an intuitive interface for creating and maintaining database objects, as well as a powerful SQL editor for writing and executing SQL queries and scripts. It is commonly used by database administrators and developers to manage and develop PostgreSQL databases.
Some features of pgAdmin for PostgreSQL include:
* Graphical user interface for managing and interacting with databases;
* Support for multiple PostgreSQL versions and operating systems;
* Connection management, allowing users to connect to and switch between databases easily;
* Query tool for executing SQL statements;
* Object browser for managing database objects such as tables, views, and functions;
* Table data editor for modifying table data directly;
* Import/export capabilities for transferring data between databases;
* Job scheduling and management for automating tasks;
* Debugging tools for troubleshooting queries and functions.
2\. Running PostgreSQL and pgAdmin in two Docker containers
-----------------------------------------------------------
In my previous article, I set the PostgreSQL server to run with two commands.
First, we needed to set a volume to persist the data of the server:
docker volume create --name postgres_volume_local -d local
Then we needed to run a Docker command:
docker run -it \\
--rm --name postgresql \\
-e POSTGRES_USER="root" \\
-e POSTGRES_PASSWORD="root" \\
-e POSTGRES_DB="ny_taxi" \\
-v postgres_volume_local:/var/lib/postgresql/data \\
-p 5432:5432 \\
postgres:13
After running this command a PostgreSQL server starts to run with all the needed variables. For more details about each line of the full command please check the previous article.
However, now we are going to run two Docker containers and, therefore, we need to set a connection between both (remember that containers are isolated?). And the previous command to run PostgreSQL is incomplete.
To create a network in Docker you have to run the following command:
docker network create pg-network
Now we are going to adapt the previous command to run **PostgreSQL in a Docker container that can be connected to another container by adding the network and setting a name to the container**:
docker run -it --rm \\
-e POSTGRES_USER="root" \\
-e POSTGRES_PASSWORD="root" \\
-e POSTGRES_DB="ny_taxi" \\
-v postgres_volume_local:/var/lib/postgresql/data \\
-p 5432:5432 \\
--network=pg-network \\
--name pg-database \\
postgres:13
So now we have the server PostgreSQL in a Docker container with the name “pg-database” and a network that allows connection with other Docker containers.
Now we need to run pgAdmin in a Docker container by setting a full command calling the image “dpage/pgadmin4” with the following information:
* Environment variables: User/Email and Password
* Port: 8080
* Network: the same network that you set for PostgreSQL
* Name for the container
docker run -it --rm \\
-e PGADMIN_DEFAULT_EMAIL="admin@admin.com" \\
-e PGADMIN_DEFAULT_PASSWORD="root" \\
-p 8080:80 \\
--network=pg-network \\
--name pgadmin \\
dpage/pgadmin4
With the command above we started a docker container for running pgAdmin.
To see it running you have to go to your browser and call the localhost:8080 and you will be asked for the user and password similar to the picture below.
After writing the user as “admin@admin.com” and the password “root” you will enter the main page of pgAdmin like presented at the beginning of the article.
To connect to PostgreSQL you have to click on “Add a New Server” which will pop up a new window.
In the “General” part you can call your server whatever you want. I called mine “postgresql”. 😂
Then in “Connection” tab you have to fulfill the following information:
* Host Name: **pg-database**
* Port: **5432**
* Maintenance database: **ny\_taxi** (this is the name of the database you used to create the PostgreSQL server in a docker container)
* Username: **root** (this is the user you used to create the PostgreSQL server in a docker container)
* Password: **root** (this is the password you used to create the PostgreSQL server in a docker container)
AND… VOILÁ 🎉🎉
Now you have a database administrator tool running in Docker connected to PostgreSQL server also running on a Docker container.
3\. How to run PostgreSQL and pgAdmin with docker-compose
---------------------------------------------------------
However, we can run both containers already connected with docker-compose.
**“Docker Compose** is a tool for defining and running multi-container Docker applications. It uses YAML files to configure the application’s services and performs the creation and start-up process of all the containers with a single command (…) The **docker-compose.yml** file is used to define an application’s services and includes various configuration options.” (text extracted from [Wikipedia](https://en.wikipedia.org/wiki/Docker_(software)%7B:target=%22_blank%22%7D)
_)._
Then we need to write in our docker-compose.yml file:
**The information for the PostgreSQL server:**
services:
pgdatabase:
image: postgres:13
environment:
- POSTGRES_USER=root
- POSTGRES_PASSWORD=root
- POSTGRES_DB=ny_taxi
volumes:
- postgres_volume_local:/var/lib/postgresql/data:rw"
ports:
- "5432:5432"
**The information for the pgAdmin part:**
pgadmin:
image: dpage/pgadmin4
environment:
- PGADMIN_DEFAULT_EMAIL=admin@admin.com
- PGADMIN_DEFAULT_PASSWORD=root
ports:
- "8080:80"
**And a code informing that our volume is external:**
volumes:
postgres_volume_local:
external: true
**The full docker-compose Yaml file is as follows:**
services:
pgdatabase:
image: postgres:13
environment:
- POSTGRES_USER=user_name
- POSTGRES_PASSWORD=a_cool_but_difficult_password
- POSTGRES_DB=my_database
volumes:
- postgres_volume_local:/var/lib/postgresql/data:rw"
ports:
- "5432:5432"
pgadmin:
image: dpage/pgadmin4
environment:
- PGADMIN_DEFAULT_EMAIL=admin@admin.com
- PGADMIN_DEFAULT_PASSWORD=root
ports:
- "8080:80"
volumes:
postgres_volume_local:
external: true
Now we can call the command to start docker-compose
docker compose up
Now when starting a server in pgAdmin remember that our server is called “pgdatabase” like the image below.
Summary
-------
This article, it was presented a tutorial on how to run PostgreSQL and pgAdmin in docker containers.
Using PostgreSQL and pgAdmin inside docker containers can provide a simple and convenient way to set up and manage your database environment. By utilizing containerization technology, you can easily deploy your database environment on different machines with minimal configuration.
With this tutorial, you can practice your SQL skills without any extra installation.
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How I Landed a Job As a Product Analyst – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How I Landed a Job As a Product Analyst
How I Landed a Job As a Product Analyst
=======================================
### An account of my job search in the field of analytics during Covid-19
01 Feb 2021 by [Nishant Mohan](https://datatalks.club/people/nishantmohan.html)
Background
----------
I did my postgraduate studies in computer science with specialization in data science. After I completed the course in 2020, it was time to look for jobs. Though I had relevant professional experience in analytics, searching for jobs was new for me because I landed my earlier job fairly quickly during campus placements after graduate studies, and therefore I had never really searched for jobs nor given many interviews. Adding to my woes, the covid-19 had largely weakened the job market.
After hundreds of unsuccessful applications, I finally secured a great job. In this article, I talk about what I think helped in retrospect. If you are reading this, I suppose you are looking for jobs too. Hope I can be of help!
Preparation Before Applications Began
-------------------------------------
There are four things I gave attention to-
1. **CV-** They say that you should change your CV according to the job description. I did too. I took help of articles that suggest how to build a great CV. With every application, my CV improved. I even received appreciation for my CV from two or three interviewers.
2. **GitHub-** There have been interviews where I actually showed my GitHub repos via screen sharing. It’s important you have a great Readme for your projects. Not everyone would look at your GitHub, but if they do, they should be able to understand what, why, and how of your projects through your Readme.
3. **Portfolio-** I designed my personal website, which had an overview of my professional career, my blogs, and my projects. Not because I felt it was needed, but because I saw someone else’s, and I was thrilled to have my own. Took only a day or two build but helped enormously when I reached out to someone on LinkedIn. Always think from the perspective of the other person. If someone was reaching out to you, would you rather download their CV, or click on a link?
4. **Blog-** Getting a great job had been on my mind since half my degree was over. One day, I decided to write about one of my projects on medium. I discovered that I liked writing. I felt like I was interacting with the readers face to face. It felt good to be explaining about a project that I was proud of. It felt good to see my name on a blogging website. But at the same time, I knew this might help me build a profile that would be different than my peers.
> Proud fact #1- I got contacted by a startup because its founder read one of my articles and it related to their business problem.
> Proud fact #2- A very accomplished data scientist reached out to me on LinkedIn and appreciated my article that cited his paper and explained his algorithm.
The Interview Process
---------------------
Out of the hundreds of applications that I filled for data analysts, product analysts, and data scientists, I got replies from only a couple of them. Though I had good prior analytics experience, the fight was tough because the job market was cold. However, there were still some takers. All the interview processes were more or less like this- a recruiter call, a technical take-home assessment, hiring manager interview and finally a series of interviews with the team which included product managers, analysts, directors. Let’s talk about each one of them in detail.
### The Recruiter Call
All the interviews begin with an introduction. I prepared a detailed introduction, which specifically addressed these points-
* Where I am currently in my career phase- 1-2 lines stating that I have recently completed master’s studies, and have these many years of relevant work experience
* An overview of what I did in the previous role- My responsibilities, nature of work and end objectives
* One liner each about some specific projects that highlighted my nature of work- Forming and analyzing customer journey funnels, a/b testing and customer segmentation for marketing campaigns, statistical hypothesis testing for deeper analyses, predicting costs for incoming customers and so on.
* About postgrad- How and why it was necessary to pursue further studies, what I learned during it, about master’s thesis, if relevant.
* What’s next? Now that everything is done, what am I looking for? How does this role and company fit into my future plans? Do the dots connect?
In my experience, if you can explain to the recruiter how you fit the job description, you will get to the next round. Not a big deal. Be prepared for the obvious. For example, I often got asked during analytics interviews, “your academics have been in data science, why do you want an analytics job?”. Have your arrow ready.
The recruiters generally discuss your rough salary expectations at this stage itself. So, do your research beforehand. Do remember to ask them about the stages involved in the process.
### The Technical Assessment
Companies may ask you to perform an analysis on their own data, or on a public dataset. Often, a set of questions is given, which need to be answered using the given data. There are no shortcuts here. Generally, you get one week to complete an assessment. Bear in mind the business objective of the task. Answer the questions and provide your solutions from a strategic perspective. Think big.
If I differentiate between the assessments that I cleared and the ones I didn’t, I can only tell one difference- I did not pour my heart into the ones I didn’t clear. Be mindful, companies do not expect you to provide a shining, production ready solution for the problem. Rather, they can sense the effort. For the ones that I cleared, I did not just submit my codes, but I also made cool PowerPoint presentations explaining my methodologies, along with what more could have been done.
### The Hiring Manager Round
The hiring manager interviews are part technical and part experience based. There may be three components in this interview-
* **Assessment Review-** Revise your assessment submission. Think why you took the approach you did. Think how else you could have done it. Think what assumptions you made about the data. Think what else would you do if you had more time. Think!
* **Behavioral Questions-** In all my interviews at this stage, I was asked situation-based and behavioral questions. Some examples are “Which is the most interesting project you worked on?”, or “Who were your stakeholders? How did you collaborate with them?”, or “Tell me about a time when you had to stand up to your manager”. Expect that the interviewer will go dive deep in whichever project you talk about. Therefore, it’s important to recollect some interesting stories from your experience before the interview. Use [STAR methodology](https://en.wikipedia.org/wiki/Situation,_task,_action,_result)
to answer, quantify your results, if possible. Phrase your stories such that the listener gets hooked. Show your excitement while narrating your story; if you are not excited, who else would be? Imagine if you were hearing this story from someone else, would you have liked it?
* **Questions?-** Hiring managers also tell about the role and teams in detail. It’s important to ask good questions. It shows that you are a curious soul. Ask questions that may help you later in deciding if you get more than one offer simultaneously. Ask what the career path is for this role. Ask who are the stakeholders for this role. Ask how much flexibility and freedom is given to do day-to-day tasks. Ask!
### The Final Interview(s)
There could be one or more back-to-back interviews with your prospective team. Their focus is to gauge your product mindset, proficiency with data, and team fit.
* **Product Mindset-** Some questions to gauge your understanding of how a product works. How would you, as an analyst, define the KPIs? Which KPIs did you measure in your last role? In one of the interviews, they suggested me an A/B test with regard to their product and asked my views on it- does the test make sense? Will it be successful? What are its assumptions? Which one is the control group? How would I measure its success? Which KPIs? How to decide on a KPI? Which statistical test? What is the importance of measuring product performance?
* **Data Proficiency-** What are the technical challenges you’ve solved with data? Have some real examples to share. Formulate your experiences in the form of a story. This may entail some live coding as well. Initially, live coding used to give me the heebie-jeebies. But if you practice well-enough (try codility and hacker-rank), you should be good. Remember, it’s okay to use google even in live-coding sessions. Maybe ask your interviewer before you do.
* **Team Fit-** Consider how you partner with business or product leaders. What’s important to share with business partners? How do you approach this? Have you worked in collaboration with other teams? How did you communicate? Did you ever have a situation when you disagreed with your team? Can you explain technical stuff to non-technical people? Here’s my favorite one- Explain A/B testing as if telling it to your mom!
Summary
-------
While there is no set pattern for the number or order of interviews in the field of analytics, I have tried to make out a pattern based on my little experience of job hunting. It’s always important to know about the company you’re interviewing with. For all the companies I had hots for, I made sure that I know about the product in depth. I read online as much as possible. These days many companies have their own blogs, which give insight into their tech stack and core values. I found them useful. During the interviews, I would intelligently slip in such information. For example, I read on a company’s tech blog that they use a particular tool, say X, for conducting A/B tests. When asking questions towards the end of an interview, I asked if there is an active collaboration with the tech team of X. How does the integration of X with this company’s product work? Such questions do get noticed, as was evident to me in feedback later. They hint at your passion for the product and the company.
> I believe that Spotify would rather hire a musician than a chef, if they are otherwise similar. Don’t you think so!?
Hope this helps!
* Connect with me on [LinkedIn](https://www.linkedin.com/in/mohannishant/)
!
* Check out some of my cool projects on [GitHub](https://github.com/mohannishant6)
!
Originally published [here](https://link.medium.com/ttGt2sfswdb)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aleksander Molak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aleksander Molak
My name is Aleksander Molak. Friends call me Alex. My mission is to translate complex concepts into understandable bite-size pieces and share them with you. I am an independent machine learning researcher, author, consultant, educator and an entrepreneur.
I am specialized in causality, natural language processing (NLP) and AI startegy.
[](https://twitter.com/AleksanderMolak)
[](https://linkedin.com/in/aleksandermolak)
[](https://alxndr.io/)
### Events
* Democratizing Causality ([watch on youtube](https://www.youtube.com/watch?v=0I2FHH95Ofs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Starting a Career in Data Science at 45 – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Starting a Career in Data Science at 45
Starting a Career in Data Science at 45
=======================================
### From microbiologist to data scientist
20 May 2022 by [Tatyjana Ankudo](https://datatalks.club/people/tatyjanaankudo.html)
A citizen of Belarus Tatiana Ankudo left the country 20 years ago. A graduate of the Biological Faculty of the Belarusian State University, she worked as a microbiologist in laboratories in Minsk, Hungary, Sweden and Germany, received a PhD in biology, Hungarian citizenship, got married, gave birth to a daughter. And a few years ago she changed her life and became a data scientist. Tatiana has been working for the international company dunnhumby for 7 months, analyzing data from a large grocery store chain and growing as a data analyst. We talked about what it is like to go into the IT field when you are over 40 and start building your career over again.
Tatiana’s mid-career pivot: applying scientific rigor to data science.
“The first two months I cried and said - its not for me”
--------------------------------------------------------
**Why did you decide to get a new specialty, where you have to achieve everything from scratch?**
I couldn’t continue growing as a microbiologist. Reagents, equipment - all this costs a lot of money, and Hungary is not the richest country. It seemed logical to choose a new path: at the last job I generated a lot of data, and I had to to process it. Suddenly it turned out to be more interesting than experimenting.
**It’s one thing to try, another is to change your profession. Was it scary for you?**
I always tried to live so that fears do not influence my choice. The thought of leaving was frightening - I wanted to be a biologist since I was little. I took half of a year to make a decision about my profession change.
**Trying to enter the IT sphere, young people in Belarus, where you are from, take three-month courses and try to get a job. It’s enough?**
To go to DS - no, you need knowledge in several areas - programming, domain area, mathematics. They study programming only for at least six months.
**How much did you study, and what was your path?**
The first step was to enroll in Java courses, which were held offline. For 5-6 hours a day, 5 times a week, we studied with teachers. It cost about 500 thousand forints a month - this is about 1.5 thousand dollars. Then, for about the same amount, I bought a two-year online DS course.
I always liked computers and programming, and the courses made it clear thatI could do it on a professional level. The first two months I cried and said - “it’s not for me”, but then it started working out for me .
Early challenges in programming give way to confidence with practice.
**Why did you cry?**
Programming requires a certain mindset. When you learn your first language, there are many rules and conventions to follow. Immersion in them is incomprehensible. It’s very hard to remember it all. You understand what the teacher is saying. But then a teacher gives you a task and you have no idea how to do it. You have to think, remember and think quickly.
**What were the main difficulties in mastering DS?**
To understand the tasks that are set for you. Requests come from real customers who form the question, adapting to themselves. They say one thing, but mean another. And you must understand them. Sometimes you can ask what exactly is needed, but people often get annoyed with clarifications.
**Isn’t it too much to enter the specialty - 2 years?**
The courses allow you to have a job and study in parallel. 6 hours a week is enough to work through the materials at a very good pace.
The knowledge acquisition process in DS should be viewed as circular. You go through linear algebra, statistics, programming, and after a while you return to them, deepening your knowledge with each cycle.
My employer is paying for Udemy courses and giving 2 days a month for self-education, training is prescribed in my career plan. DS is also about communication with a large group of people, meetings, conferences, competitions and constant growth.
In the course I took, there was a career module. After a year of study, you can start looking for a job. I went through this course in 5 months and started mailing out resumes. I wrote to about 60 companies, giving preference to those who worked in biology. They were interested in talking to me, but after the test tasks, they rejected my application
Storytelling and clear business narratives help interviews stand out.
“My resume had work experience”
-------------------------------
**How many interviews did you go through?**
About 10. This is a good response rate - I had work experience, which was given by a six-month internship in an American company: during my studies I was offered to work with clients on real data for free.
**Did it take a lot of time to find a job?**
2 months and 4 interviews. First, I completed 2 test tasks on knowledge of programming and theory, then there was a meeting with the employer - all in an online format. I was tasked with preparing a report, presentation and storytelling for a fictional company with fictional problems.
It was strange to talk about a non-existent company, but at the beginning of the interview, the employer began to actively nod, showing that he liked my train of thought and I was going in the right direction, half an hour of slideshow turned into a pleasure.
The fourth interview was a kind of face control: an employee of the company found out if I fit their team as a person. I went over.
“People aged 40+ have advantages over young people”
---------------------------------------------------
**What from your previous experience helped you change your career?**
Life experience. When, in addition to studying at the university, you have 3 years of graduate school, 2 years of postdoc and 10 years of work experience, you develop self-confidence and a professional approach to life.. After getting a job, I found out that they selected me among 200 other applicants. The employer took into account my work experience, scientific degree and soft skill I gained through the long career path.
Life experience and soft skills complement technical growth in data science.
**A number of Belarusian companies are looking for young people - it is easier for them to enter the specialty.**
I was looking for a job in Hungary, where ageism issues fade into the background. On the contrary, people aged 40+ have advantages over younger ones. The age problem is typical for Russia, when the age in the resume becomes an obstacle. But this is not a typical problem in Europe.
**Without what knowledge it is impossible to start in DS?**
Programming. Even though you can do a lot of data processing in Excel, not knowing how to program severely limits your options. Storytelling is also important: you need to communicate the results of your work in an accessible language. If a great programmer cannot explain what they did, in a simple language, the value of their analysis is low.
**What level is expected of an applicant for a DS job?**
Sufficient knowledge of programming and Python, and some knowledge of mathematics - at the school level. You also need to be able to communicate with others - it is quite difficult for introverts in analytics. There is a fundamental part of DS, though, when there is a requirement for communication.
**What skill was the most difficult to master?**
Everyone is afraid of mathematics. I had this barrier as well: when I saw a formula, I didn’t know what to do with it. My fellow students helped with it - a guy from Russia and a girl from Australia. I noticed that the material in the 2-year courses was not good enough, and I invited them to study additional material together on our own. I came up with a lesson plan and suggested a format: listen to the topic, make a presentation about once a week and answer questions.
We were a team, pulling each other, and no one wanted this to end. When someone was not ready, the next week they had to give 2 lectures. We completed the course in six months and were very proud of it.
**Have you met people who tried to switch to DS, but failed?**
Yes. Some students did not like the quality of the material, they took the money back, continued their studies independently and got lost in the wilds of free courses.- They didn’t have enough self-organization. Others switched to full-time courses, and this added another 2 years to their studies. Others just disappeared..
“To be specialists in three areas”
----------------------------------
**In what areas is the analyst most in demand?**
Wherever there is data. The most developed analytics is in the financial sector, work is underway in the product segment. In medicine and biology, everything is just beginning.
Promising path: combining biology expertise with machine learning for drug discovery.
**What direction do you think is the most promising?**
Pharmacology and the search for new drugs. For example, a task arises: find a cure for a cold. Scientists need to carry out a thousand experiments and a lot of time. The analyst simulates part of the results on a computer and offers 50 experiments, this is already a solvable problem.
**Should such an analyst understand pharmacology?**
He must be a specialist in three areas. When they explain what DS is, they draw three overlapping circles - knowledge of the subject area, mathematics and programming.
**Are you planning to use your knowledge in microbiology?**
Certainly. The result can be successful work in a pharmaceutical company: I understand the business problem, read scientific literature, communicate with scientists in their language.
**Do you need to control the work of the analyst?**
Yes, because the success of the company and financial investments depend on his conclusions. Therefore, we analysts double-check each other’s results. To be able to admit a mistake, to be glad that it was found on time, and to go forward is an important quality in DS.
Magic or analytics
------------------
**DS specialists receive the highest salaries in the IT field. Did you come here for money?**
Rather, for opportunities. The financial sphere attracts, but the tasks are also interesting. People always wanted to control their lives, went to fortune tellers, laid out cards. DS tools allow you to predict the course of events on your own - and no magic.
**What advice do you have for a newbie who has decided to work with data analytics?**
Don’t give up and go forward. I often hear questions: I’m already 30, isn’t it too late to change my profession? I am 45, and I made it in time. In the first half of my life I made a career as a microbiologist, in the second I will achieve success in DS.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Key Lessons from ML Zoomcamp: Serena Haidar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Key Lessons from ML Zoomcamp: Serena Haidar
Key Lessons from ML Zoomcamp: Serena Haidar
===========================================
### Interview with ML Zoomcamp grad Serena Haidar: practical lessons, getting unstuck, and the impact of the final project
05 Aug 2025 by [Serena Haidar](https://datatalks.club/people/serenahaidar.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
In our previous article, we profiled [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
graduate Serena Haidar’s [end-to-end waste-classifier capstone](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
. In this follow-up, Serena shares the lessons that mattered most—study cadence, how to get unstuck, and how the program shaped her approach to ML engineering.
> DataTalks.Club’s [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
> is a free, four-month online course on machine-learning fundamentals and deploying models to production. To earn a certificate, students must deliver an end-to-end final project and participate in peer review.
Key Lessons Learned
-------------------
### Could you tell us a bit about your background and the moment you decided to join ML Zoomcamp?
**Serena**: Sure! I study computer and communications engineering (CCE) at college. While much of my coursework focuses on electronics, communications systems, and network protocols, I have always been moer drawn to math and programming. When I discovered the field of machine learning, I instantly knew it was the area I wanted to pursue.
When I discovered the course in September 2024, I didn’t have any hands-on experience with machine learning. The course only required basic knowledge of Python and a willingness to learn. I checked out the [course’s repository](https://github.com/DataTalksClub/machine-learning-zoomcamp#syllabus)
, and I liked how organized and detailed everything was. The topics ranged from the basics of data pre-processing to building models and finally deploying them, so I decided to join right away.
### What are the key practical lessons for ML that you’ve learned from this course?
**Serena:** I’d say I have 3 main takeaways from this course:
**1\. Leverage a cloud provider:** Training deep‑learning models demands significant compute and often parallel workflows. By using a cloud platform, I can spin up GPU or TPU instances on demand, run experiments in parallel, and scale resources up or down as needed.
2\. **Start from a pre‑trained backbone:** Rather than building and training a model from scratch, I used a pre‑trained network. This way, I only needed to fine‑tune the final layers for specific waste classes. It drastically cuts training time, reduces data requirements, and lets me focus on optimizing performance rather than reinventing basic feature extractors.
**3\. Begin with reduced image dimensions:** To iterate quickly, first train on smaller input sizes (for example, 128×128 or 224×224 pixels). This speeds up each epoch, allowing me to identify promising architectures and hyperparameter settings. Once I’ve narrowed down the best candidates, I scale up to full resolution for final tuning and production‑ready performance.
Advice For the Next Cohort
--------------------------
### What study pace do you recommend for the next cohort?
**Serena:** Aim to complete one module per week. This steady rhythm gives you enough time to absorb the material, apply each lesson in hands‑on exercises, submit the homework, and benefit from the practical tips shared at every stage.
### What learning outcomes can you expect by following that pace?
**Serena:** You’ll develop a solid understanding of every step in exploratory data analysis (EDA) and model training. More importantly, you’ll immediately put those concepts into practice through weekly homework and two to three real‑world projects.
### What should you do if you ever feel stuck?
**Serena:** First, check the [course’s public repository](https://github.com/DataTalksClub/machine-learning-zoomcamp)
for comprehensive notes. If you need more clarification, rewatch the [YouTube lectures](https://youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR&si=VHtM9Eia-PWQakVx)
. Every topic is explained clearly in the ML Zoomcamp material. In my experience, external resources were rarely necessary because the curriculum covers everything you need.
ML Zoomcamp public repository: [https://github.com/DataTalksClub/machine-learning-zoomcamp](https://github.com/DataTalksClub/machine-learning-zoomcamp)
Course Impact
-------------
### What foundational skills did this course give you for new machine learning projects?
**Serena:** The course provided me with the essential information I need to approach any new machine learning project. I can now build and deploy an end‑to‑end ML solution: from choosing the dataset all the way through Dockerization.
From the very first module, the course laid out the machine‑learning lifecycle you need to follow on every project. Each subsequent video offered valuable insights and concrete, actionable steps. So if you dedicate the time and keep your goal in mind, you’ll know exactly what to do next.
### What was your personal goal in taking this course? Have you achieved them?
**Serena:** My goal was to deepen my understanding of ML to land a job or a research position after I graduate from university. The course’s clear road map, from theory to deployment, helped me move confidently toward that objective. I learned a great deal and am continuing my learning journey with a solid foundation.
### What aspects of the Zoomcamp’s design stood out to you?
**Serena:** This zoomcamp is very well designed. It motivated me to join the [LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
in its next cohort. One of the best aspects is that the material is free and publicly accessible; you receive high-quality education in a structured format, with detailed short videos, and access to notebooks and code files. Additionally, you can easily [ask questions on Slack](https://datatalks.club/slack.html)
, add your notes to the public repository, and benefit from others’ notes, which makes the courses much more interactive.
ML Zoomcamp course channel on Slack
### How did peer interaction contribute to your learning?
**Serena:** Reviewing other people’s projects was one of the most interesting parts. Seeing different approaches taught me new techniques, and getting actionable feedback on my own work helped me improve quickly.
### Would you recommend this course to others?
**Serena:** Yes! If you’re willing to put in the effort, this bootcamp offers a structured, hands-on path through ML, with community support and real-world project reviews to keep you on track.
What makes this course stand out is its focus on practice over theory, the projects where you apply everything you’ve learned, and most importantly, the deployment phase, which is rarely covered in other courses but is crucial in real-world scenarios. This course teaches you to see the bigger picture and implement ML solutions that truly benefit users and businesses.
Thank you, Serena!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences
Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences
===========================================================================
### Use softening language, relatable examples, and clear summaries to avoid communication walls and explain data science effectively
17 Dec 2020 by [David Gates](https://datatalks.club/people/davidgates.html)
Effective communication is a vital skill that all data professionals must master. Data science has a lot of potential for businesses. One major obstacle, however, is that decision-makers often lack the relevant background knowledge to truly understand the field. This creates a major problem for data professionals: how can you communicate technical concepts with a non-technical audience?
This problem isn’t unique to data professionals. Imagine you take your car to a mechanic after hearing a mysterious rattle or seeing your engine light turn on. After taking a look at your car, the mechanic calls you and explains the problem in a way that only other mechanics would understand. They discuss parts you’ve never heard of. You’re completely unsure what this part does, what happened to cause this issue, and what will happen if you don’t fix it. All you know is that it’s going to cost you quite a bit of money.
Image by [Ulrike Mai](https://pixabay.com/users/counselling-440107/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=707699)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=707699)
The mechanic has failed to communicate effectively with you.
For those of you who have a trusted mechanic, it’s likely that their communication skills match their ability to fix cars.
When you enter a shop you know that they’ll be able to explain any potential issues in a clear way. They can give you details on what likely caused the problem, what the solution is, and what will happen if it is and is not addressed in a timely manner.
This situation is similar to what data professionals encounter. Having the ability to discuss your work with anyone is vital. It’s a differentiating skill. It makes you approachable and an indispensable part of an organization.
Language choice
---------------
So how does our language choice influence our audience?
When you begin a technical explanation and go too in-depth too soon your audience is likely to build a “communication wall.” They’ll assume they’re not able to understand this topic.
So for example, if I use the dictionary definition of regression:
“linear regression is a linear approach to modeling the relationship between a scalar response and one or more explanatory variables.”
This type of language quickly leads to a communication wall.
So it’s vital to use language to tear down communication walls. We can do so using a 3 step process.
1. Softening language- First, use language to prime your audience for a technical explanation. You can use phrases like “If you ever…” or “If you know how…” This language is vital because it shows your audience that you understand their situation. You know that this topic may be challenging, but are already demonstrating that you have the communication skills needed to explain it in a clear way
2. Relatable example- After priming the audience, choose a relatable example that highlights the concepts you’re going to introduce. By choosing something the audience is familiar with, they’re going to be more receptive to your talk.
3. Summarize- Go over the details one more time.
Three-step method: soften, give a relatable example, and summarize clearly
Putting It Together
-------------------
So how can we put this process together in a concise and logical way? Let’s go back and think about how we can explain regression to a non-technical audience: The IBM Skills Developer course “What is data science” has an excellent explanation of regression. They use softening language, a clear example, and a summary. Here’s how they explain it:
“Let me explain regression in the simplest possible terms. If you have ever taken a cab ride or taxi ride you understand regression.
> Here’s how it works
>
> The moment you sit in a cab you see there is a fixed amount there. It says $2.50. Rather the cab moves or you get out this is what you owe to the driver, the moment you step into a cab. That’s a constant. You have to pay that amount if you’ve stepped into a cab. Then as it starts moving, for every meter or hundred meters the fare increases by a certain amount. There’s a relationship between the distance and the amount you would pay above and beyond that constant.”
Image by [LEEROY Agency](https://pixabay.com/users/life-of-pix-364018/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=498437)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=498437)
> And if you’re not moving, if you’re stuck in traffic then every additional minute you have to pay more. So as the minutes increase your fare increases, as the distance increases your fare increases, and while all this is happening you’ve already paid the base fare which is the constant.
>
> This is what regression is. Regression tells you what the base fare is and what is the relationship between time and the fare you’ve paid and the distance you’ve traveled and the fare you’ve paid.
>
> Because in the absence of knowing those relationships and just knowing how much people traveled for and how much they’ve paid… Regression allows you to compute that constant that you didn’t know was $2.50 and it will compute the relationship between fare and distance and fare and time. That is regression”
_Source: Coursera IBM Data Science Certificate: What is data science?_
Overall, the language choices in this example were perfect.
The softening language was used well: “If you’ve ever taken a cab ride or taxi ride you understand regression. Here’s how it works”
This immediately ensures your audience is open and receptive.
The example is relatable. Everyone has taken a taxi and understands the pricing structure. Finally, it ends with an excellent summary: “This is what regression is…”
Introducing a complex topic to a non-technical audience in this way ensures they have a clear understanding. When used correctly, this approach will ensure that your audience leaves with the ability to explain this topic to someone else. This is the true sign of success.
Putting It To Use
-----------------
Now that you have the framework, you can put it to use. A great way to get started is to think: how you could explain classification in a similar way.
Image by [Free-Photos](https://pixabay.com/photos/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1245776)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1245776)
Are you able to use softening language? Find a relatable example? Summarize? These are the steps needed to explain any complex topic. Once you develop these skills you’ll have the ability to confidently discuss your work with anyone.
You can learn more about this topic from my talk about essential communication skills for data professionals:
You can find me on [LinkedIn](https://www.linkedin.com/in/david-gates-a84750b/)
and on my website [AccentsWelcome.com](https://www.accentswelcome.com/)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices
MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices
=====================================================================
### Fast, practical overview of experiment tracking, pipelines, deployment, and monitoring
02 May 2022 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
MLOps applies DevOps practices to machine learning across design, training, and operations.
It’s a common misconception that MLOps is solely about the tools we use for deploying models and preparing the infrastructure for it. Partly it is, but it’s not the whole story — there’s much more. In this post, I’ll break down a machine learning project into several stages and explain how MLOps helps at each of them.
MLOps is a new topic and there’s no consensus on what it is or what it is not. In this post, I’ll share my personal take on it. You don’t have to agree with it, but I hope it’ll still be useful.
So let’s start!
MLOps
-----
MLOps is a set of practices of putting machine learning in production. Let’s see what they are.
To do it, we’ll start with a helicopter view of a typical process for ML projects. In the simplest form, it consists of 3 stages:
* Design
* Train
* Operate
Machine learning lifecycle split into three stages: design, training, and operations.
It starts with the design stage: we understand what the problem is and decide if ML is the right solution for that.
If we think we need ML, we train the model. This is the train state.
When it’s ready, we need to apply the model to new data regularly. This is the operate stage.
Train
-----
In the train stage, we experiment with different models and try to find the best set of parameters and features for the model.
Iterative model training and hyperparameter tuning to improve performance.
Most data scientists do it in Jupyter notebooks.
The typical process for experimenting is “change a parameter → re-execute the cell → see if the results improve”. After a few iterators, it makes the notebook a total mess: you can’t easily track the changes, the parameters and the results. Also, it becomes non-trivial to reproduce these results later.
Two things help us deal with these problems:
* Experiment tracking
* Training pipelines
### Experiment tracking
In experiment tracking, we establish the process of saving the results of each experiment. Each time we experiment with a new set of parameters, or with a new feature, we log the parameters and the results with an experiment tracking tool. Later we can use it for seeing which settings led to the best model and how different parameters compare with each other.
Experiment tracking logs runs, parameters, and metrics for reproducible comparisons.
There are many tools for tracking experiments. The most popular one is [MLFlow](https://mlflow.org/)
.
### Training pipelines
Experiment tracking doesn’t solve all our training problems. We still have messy Jupyter notebooks with cells that need to be executed in a specific order to get the final solution.
To solve it, we decompose the notebook into a set of building blocks executed one after another. We call a sequence of such blocks as a “machine learning pipeline”.
End-to-end ML pipeline orchestrates data prep, training, evaluation, and retraining.
Once we express the training process as a pipeline, we can execute it and retrain the model on new data with just one click. Or without any clicks at all.
In practice, you do it by converting the notebook into a Python script and then breaking it into several functions. For more advanced pipelines, you can use general-purpose workflow orchestrations tools like [Airflow](https://airflow.apache.org/)
. There are also specialized ML-focused tools like [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/introduction/)
.
To summarize, in the training stage, MLOps helps with reproducibility and automating model training and evaluation.
Automate evaluation with experiment tracking and automate training with pipelines.
Operate
-------
The output from an ML pipeline is a model — often a pickle file. Now we need to take the model and start applying it to new data.
This process is called model deployment.
### Deployment
Depending on a use case, we can deploy our model in two ways: batch and online.
In the batch mode, we don’t need to immediately react to all new data. Instead, we process the data regularly in batches. For example, we can execute it every hour, every day or once per week.
Batch inference runs models on scheduled data windows (e.g., previous day’s data).
We run training pipelines in a batch mode as well, that’s why batch deployment typically becomes an extension of ML pipelines. Often we just use simple Python scripts that can be executed as Kubernetes jobs or AWS Batch jobs. Or, quite often, we might use Spark for that.
The batch mode is the most common way of using ML models. It’s simple and sufficient for a lot of cases.
In contrast to batch, in the online mode we need to react to new data as soon as it appears. It’s usually more complex than batch because the model services need to be always up and running, ready to process new data all the time.
Two deployment modes for ML: batch processing and online real-time serving.
We can further break down the online case into two ways of deploying ML models: web services and streaming.
In the first one we deploy the model as a web service. The users of our model send HTTP requests with features (typically in JSON format) and get back the predictions.
Online serving exposes a REST API for low-latency predictions on demand.
For deploying models as web services we use libraries like Flask or FastAPI and run them on Kubernetes or a similar system. There are also more specialized ML-oriented solutions like [KServe](https://github.com/kserve/kserve)
.
In the streaming case, our model becomes a consumer of an event stream.
Every time there’s a new event, the service reacts to it and applies the model. The predictions are saved to another stream. Our users now can subscribe to the output stream and make decisions based on these predictions.
Streaming inference consumes event streams and publishes prediction events to downstream topics.
Typically we use message brokers like Kafka for streaming and implement the model service as a Kafka consumer. Instead of Kafka, we can use other streaming platforms like Kinesis.
### Model monitoring
Our job doesn’t end when the model is deployed. When it goes live, we need to keep an eye on it and make sure it remains functional.
First of all we monitor the traditional “DevOps” metrics:
* CPU unitization
* memory
* network usage
* the number of requests per second
* and others.
If some metric exceeds a threshold, we send an alert to the support team, who quickly reacts to these problems and fix them.
But that’s not enough. We also need to make sure that the predictions of our model are still good and the model doesn’t become stale. That’s why we need model monitoring.
Monitoring detects drift and SLA breaches and triggers alerts ([slack icon source](https://commons.wikimedia.org/wiki/File:Slack_icon_2019.svg)
).
If we detect that the performance drops, we can trigger the training pipeline and retrain it on the fresh data.
To summarize, in the operate stage, MLOps helps us deploy the model and monitor its performance.
Operate stage: deploy the model, monitor performance and data drift, and retrain as needed.
People, processes and best practices
------------------------------------
So far we discussed the practices and tools for training and operating. That’s already good but not sufficient.
First of all, we need to make sure we understand the problem we want to solve and make sure ML is the right solution. We need to be methodical about it: start with the goal, come up with a baseline, then gradually improve it. We don’t always need a full-blown training pipeline in Kubeflow with the model deployed in KServe and the state-of-the-art monitoring system. Instead, we often need to start simple and first show the value in the project — and then iterate.
Processes help us with that. There are tools and frameworks like [CRISP-DM](https://mlbookcamp.com/article/crisp-dm)
, [ML Canvas](https://www.ownml.co/machine-learning-canvas)
and [MLOps Canvas](https://ml-ops.org/content/mlops-stack-canvas)
that help us work together and solve the problems.
CRISP-DM framework emphasizes iterative ML: define goals, prepare data, model, evaluate, and deploy ([image source](https://commons.wikimedia.org/wiki/File:CRISP-DM_Process_Diagram.png)
).
MLOps is “DevOps for Data Science”. This means that all the DevOps practices still apply:
* Testing our services with unit and integration tests
* Automating everything
* CI/CD
* Using templates to get started quickly (e.g. cookie-cutter) and makefiles
* Writing comprehensive documentation
Both processes and best practices help us in all the three states of the project.
There’s more
------------
In this article, we only scratched the surface. There are more concepts that I didn’t cover here:
* Feature stores
* Model registries
* Experimentation platforms
And possibly much more. You can read more about them and see which problems they solve and which stage they belong to (hint: some can belong to multiple).
Summary
-------
We discussed what MLOps is and looked at the helicopter view of the process. We broke down the process into 3 stages: design, train and operate.
For each of these stages we saw how MLOps helps:
* Processes help us work together and make sure we bring value
* Experiment tracking helps us stay sane when trying different model parameters
* Training pipelines make it easier to reproduce results and retrain the model with just a few commands
* Model deployment takes care of using the model in the best way — either in a batch mode, as a web service or as a part of a streaming thing
* Model monitoring alerts us when the model goes stale and we need to retrain it
* Best engineering practices keep our code clean and reliable
MLOps Zoomcamp
--------------
Join the free MLOps Zoomcamp to practice production ML tools end to end.
Are you interested in learning MLOps? At DataTalks.Club we launch a free online course. We will cover all the topics discussed here and teach you how to apply all these concepts in practice.
More information here: [https://github.com/DataTalksClub/mlops-zoomcamp](https://github.com/DataTalksClub/mlops-zoomcamp)
.
See you on the course!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aleksander Kruszelnicki – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aleksander Kruszelnicki
An almost archaeologist. Ex Delivery Hero product manager turned co-founder in the data field with a few battle scars from failed start-ups ideas. Currently co-running leukos, a boutique data analytics agency in Berlin.
[](https://twitter.com/alkrusz)
[](https://linkedin.com/in/alkrusz)
### Events
* Starting a Consultancy in the Data Space ([watch on youtube](https://www.youtube.com/watch?v=rh_pE35m3vE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alexander Guschin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alexander Guschin
Alexander Guschin is a Machine Learning Engineer with 10+ years of experience, a Kaggle Grandmaster ranked 5th globally, and a teacher to 100K+ students. He leads DS and SE teams and contributes to open-source ML tools.
[](https://linkedin.com/in/1aguschin)
[](https://github.com/aguschin)
[](https://www.aguschin.com/)
### Events
* Competitive Machine Learning and Teaching ([watch on youtube](https://www.youtube.com/watch?v=NfAJAr7FvyY)
)
* MLOps Zoomcamp Competition - Bot or Not? ([watch on youtube](https://www.youtube.com/live/ZxUVBG4z5uE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# DataTalks.Club Community Demographics – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
DataTalks.Club Community Demographics
DataTalks.Club Community Demographics
=====================================
### Results of our DataTalks.Club Survey
16 May 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Previously, we published 4 articles on the results of a big survey about the usage of AI tools, data engineering tools, MLOps tools and LLMs-related tools by our community members:
* [How Do Data Professionals Use MLOps Tools and Frameworks?](https://datatalks.club/blog/how-do-data-professionals-use-ml-and-mlops-tools-and-practices.html)
* [How Do Professionals Use Data Engineering Tools and Practices?](https://datatalks.club/blog/how-do-data-professionals-use-data-engineering-tools-and-practices.html)
* [How Do Professionals Use LLM Tools and Frameworks?](https://datatalks.club/blog/how-do-professionals-use-llm-tools-and-frameworks.html)
* [How Do Professionals Use AI Tools for Personal Productivity?](https://datatalks.club/blog/ai-tools-for-personal-productivity.html)
Check them out if you still haven’t!
What we also asked our community members was about their background to better understand who makes up DataTalks.Club audience.
In this article, we share insights about where our members are from, their experience levels, what they do, where they work, and what industries they’re in, completing the picture of our diverse global data community.
Geographic Distribution
-----------------------
Our community spans the globe, with members from more than 65 countries across all continents. Here’s the regional breakdown of our global community:
* North America: 37.2%
* Europe: 25.1%
* Asia-Pacific: 24.5%
* Africa: 6.8%
* South America: 3.8%
* Middle East: 1.8%
* Other: 0.8%
This distribution shows our strong global presence across all major regions. North America leads with over one-third of our community (37.2%), primarily driven by significant participation from the United States (31.84%) and Canada (4.48%). Europe (25.1%) and Asia-Pacific (24.5%) follow closely behind, together representing nearly half of our members. We also have substantial representation from Africa (6.8%), South America (3.8%), and the Middle East (1.8%), demonstrating the truly global reach in our community.
Regional distribution of DataTalks.Club community survey respondents.
Career Level / Seniority
------------------------
Looking at experience levels, our community has a mix of seasoned professionals and newcomers:
* Senior-level practitioners: 40.6%
* Entry-level professionals: 35.6%
* Team leads and managers: 10.1%
* Middle-level professionals: 3.0%
* Directors: 2.4%
* Students and interns: 3.0%
* Freelancers and entrepreneurs: 2.0%
* Executives: 1.3%
* Others: 2.0%
This balance shows that DataTalks.Club is both a place where experienced professionals share their knowledge and where newcomers can learn and grow. While leadership roles make up a smaller portion, they bring valuable strategic perspectives to our discussions.
Distribution of career levels among survey respondents.
Most respondents occupy senior or entry-level roles, with a smaller fraction in leadership/executive positions. The presence of students, interns, freelancers, and entrepreneurs adds diversity to our community’s perspective.
Job Role
--------
The roles in our community reflect the diverse landscape of data professions:
* Data Engineering and Infrastructure: 28.5% (Data Engineers, Database Specialists, DevOps/Platform Engineers)
* Data Science and ML: 26.8% (Data Scientists, Machine Learning Engineers)
* Software Development: 13.1% (Developers and Software Engineers)
* Analytics: 16.8% (Data/Product Analysts, Business Analysts)
* Management and Consulting: 7.0% (Project Managers, Product Managers, Consultants)
* Research and Education: 3.4% (Researchers and Teachers)
* Students: 1.7%
* Others: 2.7%
This variety shows how interconnected the world of data has become, from building data pipelines to creating ML models and developing data products. It’s why our courses and events often appeal to professionals across different specializations.
Distribution of job roles among survey respondents.
Data engineering and infrastructure roles lead in representation, closely followed by data science and ML positions. The significant presence of analytics and software development roles shows the diverse technical expertise in our community.
Organization Size
-----------------
Our community members work in organizations of all sizes:
* Large enterprises (1,000+ employees): 29.9%
* Mid-sized companies (201-1,000 employees): 17.8%
* Small-medium companies (51-200 employees): 12.4%
* Small businesses (11-50 employees): 12.4%
* Micro businesses (1-10 employees): 8.1%
* Freelancers and independent professionals: 14.8%
* Academic/Research institutions: 2.3%
* Others: 2.3%
From the structured approaches of large enterprises to the agility of startups and the flexibility of independent consultants, this diversity brings together different perspectives.
Distribution of organization sizes among survey respondents.
Nearly one-third work in large enterprises (1,000+), while freelancers make up the third-largest group at about 15%. The remaining respondents are distributed across organizations of various sizes, from small startups to mid-sized companies, showing the diverse nature of data work across different organizational contexts.
Industry / Sector
-----------------
The technology sector leads in representation, but our community spans many industries:
* Technology/Software: 40.6%
* Finance/Banking: 9.4%
* Education/Research: 9.1%
* Healthcare: 8.1%
* Retail/E-commerce: 7.4%
* Manufacturing: 5.4%
* Telecommunications: 4.7%
* Government/Public Sector: 4.4%
* Travel/Tourism/Hospitality: 1.4%
* Consulting: 1.0%
* Other sectors (including Energy, Real Estate, Media): 8.5%
This spread shows that data skills are valuable across many sectors, from technology giants to traditional industries embracing data-driven approaches.
Distribution of industries among survey respondents.
Technology and software companies dominate the survey sample, but there is healthy representation from regulated sectors (finance, healthcare) and academia, illustrating the broad applicability of data skills across different domains.
Key Takeaways
-------------
1. **Truly global**: Engagement spans six continents and dozens of languages.
2. **Experience spectrum**: Senior experts and entry-level professionals represent the majority of our community dividing it almost equally, with leadership roles forming a focused minority.
3. **Role diversity**: Data engineering, analytics, ML, and software development all well represented, plus niche specialties.
4. **Organizational breadth**: Active participants range from one-person consultancies to multinational enterprises.
5. **Cross-sector relevance**: Dominant tech presence balanced by finance, healthcare, research, and public-sector voices.
Conclusion
----------
This demographic profile confirms that DataTalks.Club serves a richly varied community, professionals at every stage, in every role, and across every type of organization, united by a shared commitment to practical, fact-based discussion of data and AI.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# What Open Source Can Do For Your Data Career – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
What Open Source Can Do For Your Data Career
What Open Source Can Do For Your Data Career
============================================
### From baby steps on GitHub to Developer Advocate Engineer hero
15 Jul 2022 by [Mehdi OUAZZA](https://datatalks.club/people/mehdiouazza.html)
Image by Kushagra Kevat on [Unsplash](https://unsplash.com/photos/KZs5Bt5VDng)
In today’s data world, we use many open source tools. From data engineering to data science and deep learning. This is a unique opportunity to grow your career in multiple ways. Merve Noyan, Developer Advocate Engineer, shared her journey up to Hugging Face, leader in making Machine Learning more accessible. She also advised on how to get started and where NLP is heading. Here are my takeaways from the excellent podcast at [DataTalks.Club](https://datatalks.club/)
.
The article is organized as follows:
* What’s a Developer Advocate Engineer
* How to get started with Open Source
* How platforms like Hugging Face Space can help you to showcase your project
* A word on the future of NLP
Developer Advocate Engineer got you covered
-------------------------------------------
First off, Merve Noyan mentioned that there are some differences between Developer Advocate definitions. Some are more oriented to community growth and will limit their contribution to primarily educational content.
Hugging Face added the Engineer part to explicitly mention that the role is deeply technical and you will work on product features that support the team horizontally.
But how do we get there?
Baby steps in Open Source
-------------------------
Contributing to open source can be scary. With an unknown codebase, unknown way of working, a lot of automation during the PR, where do you get started?
Well, Merve Noyan emphasizes that you don’t need to code to do your first steps!
For instance, you can:
* Update documentation
* Helping on StackOverflow
* Submitting an issue with reproducible steps to this one.
* Promote it, write a blog post (or even a video!).
Next level
----------
If you want to get your hands dirty, it’s worth looking at the first good issue label on GitHub and starting the discussion before implanting anything.
It can be frustrating to have your PR rejected because it’s not in line with the design decision. Merve Noyan highlighted that maintainers will always be happy to discuss with you, as they respect your time and commitment on the project.
Multiple events will promote open-source contributions.
Here are a few of them:
* Contribution sprint: Many opensource projects have dedicated contribution sprints where maintainers will focus their time onboarding and helping new contributors.
* Hacktoberfest
* Google Summer of Code
From contributing to Open Source to landing your dream job
----------------------------------------------------------
There’s a great secret about doing work in public: it’s public. Anyone can look it up. It could also speed up technical interviews as you may have already proven your abilities through some PR’s.
Merve Noyan contributed to different Open Source projects already before joining Hugging face. She also gave a few workshops later on NLP/TensorFlow.
After these contributions to the open-source community, Hugging face project included, they seamlessly reached out to offer a job opportunity.
From contributing in public to showcasing a portfolio that lands interviews
How to get started in NLP
-------------------------
### Your first project
Sentiment analysis is an excellent first entry into the NLP world as data representation is pretty simple: sentences and labels. There are a lot of use cases where we need to know people’s perceptions about a product, service, or brand. On the opposite, summarisation or paraphrasing are more challenging NLP tasks.
Merve also recommends upskilling yourself in Transfer Learning in general. There are a lot of pre-trained models available today like BERT or GPT. You usually get better results when you fine-tune them on downstream tasks than training your own from scratch.
### How Hugging face can help promote your project
Nobody wants to git clone and read your README to set up your project. The last mile would be to deploy your project so that any non-technical user can easily access it.
Fortunately, there are a couple of platforms that help you to do so. Kaggle provides a notebook runtime to show off your projects.
Hugging Face goes a step further and has “Spaces.” They will host your model demos, made by Streamlit or Gradio, and it will be open for everyone. You can see it as a personal portfolio for models.
About the future of NLP
-----------------------
NLP solves tasks which are shaped according to your data. For instance: Question-answering, and speech tagging.
The next big thing is solving multiple tasks with one big model without fine-tuning : zero-shot learning. It’s a trend not only in the NLP domain but also in visions like DALL-E. There are a lot of multi-models or generative models.
Two exciting papers that Merve recommends on these are [T0 by Hugging face](https://bigscience.huggingface.co/blog/t0)
and [Flamingo by Deepmind](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model)
.
Summary
-------
In this article, we covered through the story of Merve how impactful it can be to contribute to Open source for your career. We discussed the different action points you can take to get started and all the events that can help you bootstrap your Open Source journey. On top of that, we mentioned how to get started with NLP and its future.
There has never been a better opportunity to contribute to Open Source.
There are tons of projects
Many platforms to lower the technical barrier to deploying and showcasing your work.
And everything that you will do will be public, which is gold for future reference.
So don’t hesitate, and make the leap!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Adrian Brudaru – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Adrian Brudaru
I studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side.
I ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, I had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge I wanted.
As going back to startups was not a desirable option either, I decided to postpone decision by taking freelance work. I since never looked back, and now, 5 years later, I am co-founding a company in the data space so I can try new things. This company is also looking to release a bunch of open source tooling (the kind we use ourselves) to help democratise data engineering.
[](https://linkedin.com/in/data-team)
[](https://adrian.brudaru.com/)
### Events
* Freelancing and Consulting with Data Engineering ([watch on youtube](https://www.youtube.com/watch?v=9DTTrN-khCk)
)
* The Entrepreneurship Journey: From Freelancing to Starting a Company ([watch on youtube](https://www.youtube.com/watch?v=vOpEQiCsaLw)
)
* Data Ingestion From APIs to Warehouses ([watch on youtube](https://datatalks.club/people/adrianbrudaru.html)
)
* Trends in Data Engineering ([watch on youtube](https://www.youtube.com/watch?v=AlCFKbFIEM8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alex Chung – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alex Chung
Alex Chung is a former Senior Product Manager at Amazon SageMaker, Facebook Data Operations, and Lyft Account Security. His first job out of college was to build MLOps infrastructure in a quantitative hedge fund. He now runs industry working groups on MLOps and builds open-source tools.
[](https://linkedin.com/in/alex-chung-gsd)
### Events
* Orchestrating Enterprise ML Workload Jobs Across Clouds ([watch on youtube](https://www.youtube.com/watch?v=ocDP-94YFjI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alex Ioannides – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alex Ioannides
Alex is the co-founder of Bodywork Machine Learning, the creators of Bodywork - an open-source deployment framework for machine learning projects developed in Python.
Prior to this, Alex has spent 8 years with technology companies large and small, developing solutions to their data engineering and machine learning problems: he has been Chief Data Officer for LiveMore Capital, a Machine Learning Engineer within Oracle’s Adaptive Intelligent Apps team, and the Head of Data Science for Perfect Channel.
[](https://twitter.com/ioannides_alex)
[](https://linkedin.com/in/alexioannides)
[](https://github.com/AlexIoannides)
[](https://alexioannides.com/)
### Events
* Bodywork: GitOps for Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=m4cn7HJUxng)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals
Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals
==============================================================================
### A practical framework and example to turn analysis into action
25 Dec 2020 by [David Gates](https://datatalks.club/people/davidgates.html)
The Challenges of Storytelling
------------------------------
Whether you’re working in data analytics or data science, the ability to turn your data into a story is an essential communication skill. Your insights are only valuable if you can communicate them effectively. Many professionals clearly see what the data is telling them. They present their findings by repeating these insights as they see them. They are then surprised when others fail to see the value in their data. The issue is not the quality of the insights, it is the lack of a story. We all process information a bit differently. Facts connect with some, and not others. Stories, however, are universal.
Image by [Free-Photos](https://pixabay.com/photos/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1245690)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=1245690)
_Data professionals do not need to be natural storytellers. Instead, they simply must understand the elements of a good story._
Great stories don’t simply communicate events and facts. They frame them in a way that engages the audience from start to finish. Great stories have 3 essential parts: characters, conflicts, and a conclusion.
I have a good friend who is terrible at telling stories. No matter how interesting the topic, his stories always seem to fall flat. He tends to focus on facts and is generally unable to create conflict or tension with relatable characters. Simply put, his stories are boring. When listening to him, I am rarely concerned about the conclusion. His stories end with a casual laugh and then someone quickly changes the topic.
My sister, however, is a fantastic storyteller. She is able to make the most mundane events engaging. Her stories seem to connect with _everyone_. People laugh. Listeners pay attention. Others retell her stories. She combines characters, conflict, and a conclusion seamlessly. A good business story must do the same.
Characters + Conflict + Conclusion
----------------------------------
First, every story needs characters. Data affects somebody. Identify these people. It could be stakeholders such as customers or employees. It could be the general public. It could even be your audience members.
Next, identify your conflict. It should relate directly to what the characters are experiencing. What problems are they facing? Why are these problems so important? How are these issues affecting their well-being? Maybe the characters are your audience members. In that case, _what problem are they causing_?
Three essentials of a compelling narrative: characters, conflict, and a conclusion
Finally, a good story must have a conclusion. A conflict needs resolution. This is where your insights into the situation become essential. As a data expert, you understand the problems and know how to solve them. You can restore balance by eliminating the conflict. This motivates your audience to act and provides a satisfying ending to the story.
Creating a narrative
--------------------
Let’s take a look at a simple example using demand forecasting at a shoe store. We could directly state the problem:
_We do not have enough sizes or styles at the right times. Demand forecasting models can mitigate this problem._
This approach, however, lacks engagement.
So how do we engage our audience? We’ll want to use a hook during our introduction that introduces our conflict and characters. In our shoe store example, our conflict is the lack of stock. The characters could be customers, employees, or the organization as a whole. Here’s a potential introduction:
_I don’t buy shoes very often but they’re something I’m willing to spend money on. Last year, my beloved pair of Clarks desert boots reached the end of their life. It was unfortunate timing as they’re my go-to shoe and I had big weekend plans._
_So I ran to the nearest “Shoes R Us.” I arrived and was immediately greeted by a helpful assistant named Paul. I told Paul what I was looking for: “brown Clark desert boots size 10.” He went to the back and returned empty-handed. I received the dreaded “Sorry we’re out of stock but we have….” I told Paul sorry but I’m not interested._
Image by [David Mark](https://pixabay.com/users/12019-12019/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=81310)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=81310)
_I knew I didn’t want any of the alternative options, so I left the store and was able to find what I wanted at a competitor down the street._
_Shoes R Us lost out on a sale of a shoe that costs over $120._
_As the customer, I lost out on time and had to reevaluate my loyalty to the company._
_Paul lost out on commission._
_This situation can lead to decreased customer loyalty, employee turnover, and lost revenue._
_Now, this problem isn’t unique. Given the vast stocks that we must have at every “Shoes R Us” location, it will certainly happen from time to time._
_So how can we solve this issue?_
_Luckily, we can create machine learning models that will help us predict the demand for each style and size at all of our retail locations across the country._
_Here’s how it works……._
After hooking your audience during an introduction, you’d then go on to explain the issue more in-depth. This is a more functional part of the story where you can introduce your methodology, data, and model. You can expand on the issue, highlight your knowledge, and offer ideas on what can be done to solve the problem.
Finally, it’s important to end your story on a strong note. During your conclusion, you can relate it back to what was discussed during your introduction. One of my favorite techniques is to highlight how things may be different for various stakeholders. So using our previous example I could mention how customers like myself would have increased loyalty, how employees like Paul will make more money (and likely be happier in the process), and the company will see an increase in sales.
Your story has first highlighted the problem, you then showed how your knowledge can resolve the issue, and finished by describing how the situation will change. This can be incredibly motivating. It’s likely your audience wants this problem solved. The information you’ve presented is the bridge between the current problematic situation (which you highlighted during the introduction) and an ideal situation (which you introduced during your conclusion). You are giving them the power to eliminate this problem.
Excellent storytellers are made, not born
-----------------------------------------
The most successful data professionals combine world-class analytical abilities with expert communication skills. These individuals are well-known for their talent to turn insights into action. How do they do it? Through stories. Storytelling is a do-it-all tool. It helps you simplify complex ideas, engage an audience, and motivate action.
Image by [Roché Oosthuizen](https://pixabay.com/users/rocheartist-7638257/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=4610564)
from [Pixabay](https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=4610564)
So do your communication skills match your analytical skills?
Developing storytelling abilities in a foreign language, such as English, is challenging, but it’s a skill that can be developed by anyone. It will allow you to rapidly advance in the field. It opens doors to international opportunities, can lead to promotions, and makes you an invaluable member of your organization.
You can learn more about this topic from my talk about essential communication skills for data professionals:
You can find me on [LinkedIn](https://www.linkedin.com/in/david-gates-a84750b/)
and on my website [AccentsWelcome.com](https://www.accentswelcome.com/)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# The Hiring Process for Data Professionals – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
The Hiring Process for Data Professionals
The Hiring Process for Data Professionals
=========================================
### Advice from a Recruiter
01 Sep 2022 by [Pavel Chernetsov](https://datatalks.club/people/pavelchernetsov.html)
Photo by [Christina Morillo](https://www.pexels.com/@divinetechygirl?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/woman-sitting-while-operating-macbook-pro-1181676/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
The hiring process can be a daunting task for data professionals, whether you’ve just left university and are seeking a Junior position or even if you are a Senior that is looking for a change. After talking with Alicja Notowska, a recruiter with nearly a decade of hiring experience under her belt, we’ve compiled some key points and advice for data professionals that want to know the ins and outs of the hiring process.
The article is split into 7 sections, based loosely on the chronological order of the process and the logical sequence of the ideas, as follows:
* Courses, projects, education
* Cover letters
* LinkedIn profile and CV content, format, and other tips
* Interview
* Salaries
* Recruiters
* Switching professions
Courses, projects, education
----------------------------
Photo by [Pixabay](https://www.pexels.com/@pixabay?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/adult-blur-books-close-up-261909/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
Naturally, before you can look for a job, you must consider the skills you already have and where you’ve acquired them. This may come from a traditional university education, online courses, and bootcamps, and from simply learning on the job, which is usually what we mean when we say “experience”. So let’s start with a traditional education – how important is having a Master’s degree? Perhaps you only have a Bachelor’s degree level of education or maybe you don’t have any degree at all. How important is this factor to a recruiter?
Although Alicja believes that there isn’t that much difference between a Bachelor’s and a Master’s, there is a bit more distinction when it comes to having a Ph.D. For instance, it would be a must for a team that is very research-heavy, where maybe they aren’t working on a specific product. The role may require doing a lot of research and writing papers, so a Ph.D. would be a must in such cases. However, this is certainly not the majority and then a Bachelor’s or Master’s would be sufficient. In some cases, the degree may not be listed as a requirement at all, but an employer will want the candidate to have a solid education in terms of understanding the maths behind some of the tools they would be using and the algorithms. But this doesn’t mean that it absolutely has to be a Master’s degree.
So what about somebody having two Bachelor’s degrees? For example, one in IT and one in account management. Would this make a candidate stand out among others? As impressive as having two Bachelor’s is, Alicja doesn’t think that this would be something that tips the scales one way or the other. It’s important that there is a technical degree, whether it’s computer science-related or related to machine learning. But if there is also another Bachelor’s degree in a non-technical field at the same time, it is more of a quirk than an advantage. As always, this depends on the role, to be honest. If someone has account management experience, this naturally leads to assumptions like the person may be good at presenting and have soft skills like managing stakeholders. However, an assumption is not a concrete thing and thus doesn’t offer a significant advantage in and of itself.
With traditional education out of the way, there are plenty of other things a person can do to further their education. So do recruiters consider portfolio projects such as courses from Coursera when reviewing a potential candidate? Similar to the double Bachelor’s, Alicja believes that experience plays a greater role than courses. Experience means what specific models or tools you used and how you implemented them. A lot of people tend to do the Andrew Ng Coursera course – if this adds value and helps you, even if it’s just to pass the technical interviews and refresh your knowledge, then that’s great. But it is more of a “nice to have” rather than a must, unless, of course, it is listed as a requirement in the job description.
Generally speaking, when it comes to courses, pet projects, and other things that enriched your skills, it’s something you accomplished and something you did from beginning to end. That also says something about you – you committing to doing that and putting in those hours. Even if it may not be the thing that a recruiter or hiring manager will base their entire decision on, this type of work also adds value and makes a good addition to a CV.
Cover letters
-------------
As anyone who has spent a bit of time researching on the internet while looking for jobs, the question of the cover letter will inevitably come up. Is having a cover letter even important or should the focus be more on the CV?
If the cover letter is obligatory during the application process, and sometimes it will be – then, of course, it’s important. If it is mentioned as a must in the job description, that means that the recruiter or hiring manager, or both, will read it. With that in mind, a lot of recruiters will not read the cover letter because they simply do not have the time to do so, but that doesn’t mean that someone involved in the hiring process will not either. If the job is for an entry-level position, the importance of the cover letter raises substantially.
If you’re a graduate and there is a junior-level data scientist position available, then a cover letter would typically be something that the recruitment team would look at. This is primarily because you probably don’t have that much experience to put in your CV, so the team wants to understand your motivations a bit more than your actual experience. However, if it’s a more seasoned or a more senior-experience level position, it is advisable to omit the cover letter unless it is explicitly mentioned as mandatory.
LinkedIn profile and CV content, format, and other tips
-------------------------------------------------------
Photo by [cottonbro](https://www.pexels.com/@cottonbro?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/person-holding-black-ipad-with-black-case-5989925/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
Many people, especially professionals, treat their LinkedIn profile quite similar to their CV. This makes sense considering that recruiters and hiring managers tend to look at both of these sources when reviewing a potential candidate. So what information do recruiters look for when scrolling through the potential candidate’s profile or CV and how are these resources different from one another?
CVs and LinkedIn are very similar because they are used to convey a person’s experience and education. A CV is only slightly different in the sense that on LinkedIn, there is a specific format that is set for all users, whereas a CV is more or less open-ended. Typically, the first things that recruiters and hiring managers see are the experience, the role, and the responsibilities, after the name and the picture, of course. This differs from a CV, where people tend to put their education first. However, regardless of the resource, experience is usually the first thing recruiters look at, and the education after that. Of course, this depends on whether you are a recent graduate who doesn’t have much experience, in which case it is better to highlight the education as a first point. With that in mind, the experience typically beats most of that. Even if it’s just a six-month internship, it’s already better to put that on top of the CV, because that’s what a recruiter would look for anyway.
On CVs, people often tend to put a title and a list of responsibilities, or just a list of tools. However, you must keep in mind that the person looking at this list can’t be sure what you did with the things you listed or your depth of knowledge on them. Another thing to avoid is an overuse of buzzwords, as this can make things very confusing for the person looking at your profile or CV, especially if you actually have no experience with the things you mentioned. To combat this, it would be a good idea to showcase the things you and your team actually worked on, what your role in the project(s) was, and what you accomplished. It’s all about finding a balance when it comes to the use of keywords and showcasing your actual skills.
Your responsibilities and what _you_ did at your job should be very clear, as opposed to listing what the team did. Of course, this doesn’t mean “don’t give credit to the team,” but you should not own up to things that you didn’t do on your own. Instead, highlight what your part in that team effort was. This will help to drive interviews and be useful for the interviewers further down the line because they will be able to ask you more informed questions and avoid asking you totally irrelevant things, which could inevitably confuse you as well.
Other points may initially seem small from the outside but can be critical when a hiring manager or recruiter has to pick your CV over that of 100 other candidates. One such detail is to put the month and the year in the job history, and not just a year to year because that could be up to a 12-month difference sometimes. There are, of course, quite practical things like making sure there are no typos, but that’s quite obvious.
Some less obvious nuances are things that do not reflect your knowledge, skills, or experience, but instead, serve to add bias to the recruitment process. Things such as a photo and your birthday are better left off a resume if they are not a specific requirement of a potential employer. We are all biased, and it’s unconscious bias. Not intentionally, but when you see a photo first, people immediately start forming assumptions without even knowing it. Leaving these things off a resume also helps you as a candidate to reduce that bias, and hopefully be more successful.
Interview
---------
Photo by [fauxels](https://www.pexels.com/@fauxels?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/man-and-woman-near-table-3184465/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
Alright, so you’ve got your CV, LinkedIn profile, and cover letter down pat. You’ve done a great job on all three counts and have been invited to an interview. So what does the hiring process look like and what are the typical steps at the interview stage? A lot of the time, of course, the overall process will depend on the company. But typically, especially in tech and data science, the steps are as follows:
**Recruiter interview**: This is often the first step. These interviews tend to be quite short, but this depends on many factors, such as the size of the company, the system they have in place, and on the recruiters themselves. This interview involves primarily soft-skill questions like what you’ve been doing over the course of your career, what your responsibilities and accomplishments were in your current or previous job, your education, etc. There may also be a
**Technical screening:** This is usually an hour-long interview with a data science interviewer, which involves many more technical questions.
**The final round**: typically involves physical on-site interviews, hence the name “on-sites”. Due to the pandemic, these have mostly become virtual but may go back to being physical as restrictions are lifted. The final run could be anything from two or three, all the way to five or seven interviews. Again, this depends on the company and how big they are.
For the majority of interviews, you are likely to be speaking to a person who is not technical, especially for the recruiter interview. Therefore, it’s important to keep in mind that it’s not just about you proving that you have the technical ability and skills, but that you are also able to explain those complex concepts to people who have no idea about machine learning.
Be prepared for behavioral questions as well, such as “Can give me an example of a situation when you had to work with a difficult stakeholder? Why was it difficult?” A lot of people tend to answer in very general or hypothetical scenarios, so try to answer by giving an actual example from the past and walk the interviewer through your actions and what you did in that situation specifically. What was the outcome and what was your role in it? The purpose of these behavioral questions is to see if you fit in with the culture of the company and the values that the company has.
Aside from behavior, there are also likely to be practical questions. This will typically be about the notice period – if you’re currently employed, how soon you could start if you decided to join us, and, obviously, salary. What are your salary expectations and how active you are interviewing currently?
Naturally, this process may sometimes not be as organized or as structured as what is listed here. After the interviews, there is another stage where the decision is made. If the decision is positive (the company wants to hire you) the company then makes an offer and talks about the conditions. This most likely comes in the form of a contract and then the candidate is onboarded.
Salaries
--------
Photo by Monstera from [Pexels](https://www.pexels.com/photo/concept-of-waiting-for-cash-credited-to-bank-card-5849580/)
When it comes to salary expectation questions, you may have heard the advice – “Never say the number first.” So is this good advice and should you play this game of ping-pong with the recruiter or hiring manager when it comes down to negotiating a salary? Essentially this can be broken down into the following points:
* Your current salary
* Your expectations
* The company’s salary structure / Negotiations
### Your current salary
If the recruiter asks about your current salary, you should know that it’s not necessary for you to respond. In most European countries, at least, your future employer won’t be able to check your actual salary because it’s personal data, which is protected by privacy laws. Thus, you can ignore that question or just say, “Look, that’s not something that I’m willing to share.” Sometimes recruiters are told by their bosses that this is a question they have to ask and thus, they will ask you, but just know that you are not obligated to answer.
### Your expectations
If you ask the recruiter to give a number first, you may be met with a response such as “It would be super helpful to know the expectations.” This is an honest question since the recruiters want to know if your expectations are completely off from what they can offer you. If this is the case, then you probably shouldn’t be wasting each other’s time. However, you can also counteract this by saying “This is my initial estimate of what I want, but this may change because I’m interviewing with other companies. I will also do my research.” Recruiters typically will expect that you are applying to more than one company.
In the data science recruitment market, it would be strange to expect that people will not have other offers. If you’re just a graduate, it may be harder to find a job, but people change their minds and expectations change and that’s fine. As long as you’re open about this from the beginning and say, “This may change,” and keep the recruiter posted – they will not hold it against you. Just let us know so that we can counteract and try to still maybe do something about it.
When setting expectations, it is a good idea to do some research about the typical salary for the role that you’re applying for. The resource Glassdoor, among others, allows you to see the salaries of people from the same company you’re interviewing with. If you set expectations that are backed by research – checking salaries for the same role globally, for different companies, and setting a range – you become informed of the market and have a much higher chance of achieving success.
### The company’s salary structure / Negotiations
For the majority of companies, the way they approach salaries can be divided into one of two categories: either they have a set range for roles (typically this approach is used by bigger companies) or it all depends on what you negotiate with the person that’s hiring you.
In either case, it’s good to ask “How does your salary structure work? How does the leveling and seniority tier work in this company?” before you answer the question of salary expectations. If it is the type of company where it all depends on how well you interview and what you ask for, then don’t tell them your expectations. However, in a lot of companies, especially the bigger ones, there is a level system in place. That means that to every level, whether it’s Junior, Mid-level, Senior, or above – there is a band attached. In this case, no matter what you’re going to say, even if the number is very low, you will be told what that specific band is. A lot of companies try to avoid giving a salary that is out of that range if you truly belong to that tier, even if you’ve given a lower number, as it leads to a weird atmosphere between the members. If the company has established ranges and leveling, then it’s okay for you to share your expectations. You don’t have to actually say a specific number as a range will usually suffice.
If the company does not have a salary level structure in place, your salary will likely depend on how good you are at negotiating, which is typically the case in smaller companies. This is where your salary research will come in handy as you will be less likely to have overly high or low expectations. Generally, if you set a price that’s too low, you will likely be stuck with it, given that you accept the position. However, the tendency for data professions, according to Alicja, is that they don’t have a huge gap between what the company can offer and their expectations.
Interacting with recruiters
---------------------------
Photo by [RODNAE Productions](https://www.pexels.com/@rodnae-prod?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/marketing-people-woman-art-5922537/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
If you want to impress a recruiter, it is best to try to answer their questions as much as possible. Try to think about the fact that you’re speaking to someone who doesn’t understand much about your profession in the technical sense. Although a lot of recruiters do try to understand machine learning and can sometimes tell the difference between certain aspects, just try to explain as much as possible in words that are maybe not as technical.
You may often be met with a question that is more open-ended, in which case, try to avoid giving a simple ‘yes’ or ‘no’ because the interviewer’s goal is to understand your motivation. They want to know why you were doing this particular thing or what it was that you even did. Do your best to come prepared for the interview. This doesn’t necessarily mean you have to know everything about the company where you’re interviewing, but be prepared and have some specific examples.
The behavioral questions you are typically asked do not vary that much and are usually about similar values. Companies want to know if someone is a team player, so the questions will likely be around that. If you were more in a senior role and leadership, then you may get asked something like “How did you lead the team? How big was it? Can you give me an example of how you would grow someone?” So try to be ahead of that and try to think about it before attending the interview. The interviewer will probably ask you questions that need specific examples, so try to come up with some specific examples as opposed to hypothetical scenarios.
Coming prepared for the interview with your own questions is also sure to impress a recruiter. If you’re motivated and you’re actually interested in joining the company that recruiters are representing, then ask questions about the company or the culture. If the person being interviewed is not asking those questions, the recruiter may question whether they are actually interested in the company or just attending the interview as a backup plan.
Commitment is a sticking point for recruiters. If things change, just explain what happened, whether it be salary expectations or something else. The name of the game is respecting people’s time. Albeit recruiters can also be quite difficult about this or not very good at keeping you posted and you never hear from them after you were rejected. But if you’re interviewing with many companies and some company makes you an offer, let the recruiter know as soon as possible. Be open with the recruiter about your intentions.
Switching professions
---------------------
There are many situations in life when it’s time for a change. The change can vary greatly, from simply switching companies to choosing a completely different field of interest. So how can career changers achieve success when searching for a data profession?
A lot of companies want candidates to have a specific degree, even if it’s just a Bachelor’s – but they do want one from a university and in something related to computer science. If you come from a different background and are changing careers from something not tech-related, it can be quite challenging. Although it is not uncommon for people to take some Coursera courses and then try to swap, this can prove very difficult because experience is what matters the most for recruiters.
Therefore, the best advice is to try to gather experience, even if it’s an unpaid internship or apprenticeship. Of course, it’s not easy to just do an internship that is unpaid for six months because you have bills to pay, but this is where the challenge comes in. Try to network as much as possible with people who are data scientists in the companies you’re interested in and connect with recruiters on LinkedIn. This can be in the form of meetups or any kind of community you can find.
Then there are also headhunters from recruitment agencies who may typically have worked with many different clients. They work with many different companies and have a wider net, so they may have some opportunities somewhere to offer you. The more recruiters you connect with, the better.
Photo by [Henri Mathieu-Saint-Laurent](https://www.pexels.com/@henri-mathieu?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
from [Pexels](https://www.pexels.com/photo/a-man-in-plaid-long-sleeves-sitting-near-his-colleagues-while-giving-an-opinion-8348623/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)
All in all, while being a data professional can be a rewarding career and the job hunt rigorous, the main thing is that you are happy with what you do. Armed with the knowledge you have gathered here, you can proceed further and establish a good foundation that will prepare you for any interview that you come across. Be sure to know what you want, learn how to express these needs, and do not be afraid to voice your concerns. I am certain that you can succeed – you’ve already taken the first step!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data
Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data
========================================================================
### Build a 5D RFM+ framework, engineer metrics, and segment responders/non-responders with k-means to power targeted in‑game marketing
29 Nov 2020 by [Nishant Mohan](https://datatalks.club/people/nishantmohan.html)
Background
----------
There’s a specific part of job-hunting that I look forward to. It’s called take-home assessments. These assignments are a great way to learn about what you would do in the company on a typical day. As a data scientist, it gives me immense joy to take a sneak peak into what kind of data I’d be working with. It helps me judge a company’s attitude towards its data science initiatives too!
Towards the end of my master’s course, I started applying for jobs. One of the most interesting projects I did was from a gaming company.
Introduction
------------
They asked me to perform customer segmentation for their in-game marketing campaign.
I was given a user level dataset and the attributes showed user’s purchase date for the base game, expansion packs and downloadable content of the game. That was it!
When I first saw the data, I thought, really? What can I do with merely these attributes!?
Turned out, more than I could initially think! Not only I figured a way of doing an RFM Analysis, I managed to take it up a notch. I call it Customer Segmentation 2.0!
Oh and, in case you didn’t know, RFM stands for Recency, Frequency and Monetary value. An RFM analysis is a generally accepted method for customer segmentation. For the purpose of this article, I would not dive into details of RFM analysis, as there are already many such resources available. My focus is to explain what I did for this particular project.
So let’s start, shall we!?
The data
--------
Let’s take a quick look at the available features.
Snapshot of available features used for segmentation: base game, expansions, and DLC install dates
So the last 8 features are the names of either an expansion pack of the game or a downloadable content. The dataset has 500k rows. That’s good because it means we can make more segments, right!?
The Methodology
---------------
I begin by studying the distributions or unique value counts of each of the feature. This helps me get familiar with the data. There are a lot of blanks in the data, considering not many players install an expansion pack or downloadable content. I replace such values with -1.
> I translate all other dates to number of days passed since the game was launched. This makes the data numeric.
In other words, I convert all the install dates to numeric by counting number of days passed at the date of installing since the game was launched. That is, the days between install date and game launch date. This puts all the dates in perspective.
I tag the users as responders or non-responders based on whether they buy any add-on or not. For responders, I intend to use k-means clustering for segmentation.
Now I can begin defining my key metrics for segmenting the responders:
Recency metric: days since last activity, highlighting more recent engagement in 2019
### Recency
This is the number of days passed since the user was seen active on the gaming platform.
The chart shows that more users have been active in 2019, as compared to the users in 2017.
Frequency metric: number of active days since install, skewed toward fewer days for most players
### Frequency
Since the day a player installed the game, how many days did he play the game?
The chart is concentrated towards left, meaning that most players are active for lesser days. However, it should be noted that new players have less number of days where they could be active, as compared to older players.
Monetary value metric: spend estimated by mapping store prices to user add-on purchases
### Monetary Value
Since this information is not available in the data, I went to the game store website and mapped the prices to the add-ons. This way, I now have the amount spent by each player. Neat, eh!?
Most players spend less than a hundred bucks. This is expected because the base game costs 55 bucks. And the downloadable content is generally cheap!
Responses metric: count of prior add-on purchases per player; most buyers purchase only one
### Responses
How many add-ons did the player buy previously? This will not be correlated with the monetary value, because the prices vary across add-ons!
It can be seen that most people who bought any add-on, only bought one.
Purchase frequency metric: intervals between purchases with peaks around expansion release periods
### Purchase Frequency
Maybe the player buys everything together, or maybe he spreads it out?
While most players buy everything soon after they buy the game, we see other highs near 400 days and 800 days. Incidental? No! These bumps can be attributed to launch dates of the two expansion packs roughly every year.
Clustering/Segmenting The Responders
------------------------------------
Using the 5 key metrics, I apply k-means clustering to segment the users.
Elbow plot suggests k=5 as a balanced choice for k-means clustering complexity and cohesion
Looking at the chart, I select 5 as the optimum number of clusters/segments. This gives me a balance between homogeneity within clusters and complexity of the analysis.
Segmenting The Non-Responders
-----------------------------
Since these are the users who have not interacted much, we only have two measures to judge them: Recency and Frequency.
Non-responder segmentation using a recency threshold to separate recently active from lapsed users
As can be seen in the above chart, I segment such users by a threshold of 1000 days. That is, those who have been active in last 200 days are in Cluster 6, others are in Cluster 5 (Cluster 0–4 being the responders).
Analysis and Strategy
---------------------
Following table gives means of all the features across the user segments.
Summary statistics by segment for recency, frequency, responses, monetary value, and purchase cadence
Look at the first row. On average, players in Cluster 0 were active for nearly 15 days, bought 1.5 add-ons, were active 477 days from the beginning (long back), spent 65 bucks, and purchased an add-on every 33 days. Since these were active long back, they have probably forgotten about the game. So, in-game marketing may not work on them! On the other hand, email marketing might!
Now look at the second row. On average, players in Cluster 1 were active for a whopping 92 days, bought nearly 3 add-ons, were active fairly recently, have spent much more than others have, but purchase relatively rarely. These could be the players who have recently bought an add-on. These are the customers who seem to be loyal. We could target them with more exciting features!
Following figure gives similar summary of each cluster/segment.
Actionable strategy guidance for each segment to tailor in-game and email marketing
Conclusion
----------
In this article, I presented my methodology of attacking a customer segmentation problem with limited data. I utilized all that was available, and instead of a more popular RFM analysis, I performed a 5-d segmentation.
The analysis resulted in 7 customer segments. These segments consist of users with similar behaviour. Looking at their behaviour across metrics helps in targeting them with custom advertisements.
Hope that helps!
Find a detailed explanation in my YouTube video:
* Here’s [my GitHub repo](https://github.com/mohannishant6/Customer-Segmentation/tree/master/2K)
with the data and code.
* Connect with me on [LinkedIn](https://www.linkedin.com/in/mohannishant/)
!
* Check out some of my cool projects on [GitHub](https://github.com/mohannishant6)
!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Akela Drissner – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Akela Drissner
Akela is the Head of Solutions Engineering at dltHub, a company building open source tooling for data ingestion. She has a background in machine learning with a focus on NLP, having previously worked in conversational AI at Rasa.
[](https://twitter.com/oakela)
[](https://linkedin.com/in/oakela)
### Events
* Open source data ingestion for RAGs with dlt ([watch on youtube](https://datatalks.club/people/akeladrissner.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alicja Notowska – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alicja Notowska
Alicja is a seasoned talent acquisition specialist a.k.a. a self-professed ‘professional stalker’. She has 10+ years of tech recruitment experience from big companies such as Google and Zalando as well as, most recently, an embedded talent agency, WeAreKeen.
[](https://linkedin.com/in/alutka)
### Events
* Recruiting Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=WSMDXsjKYx4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Recruiting-Data-Professionals---Alicja-Notowska-e1dj2qn)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alex Kim – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alex Kim
Most of Alex’s work experience involved solving data science problems in various domains: physics, aerospace, telemetry/log analytics, image, and video processing.
In the last couple of years, he became increasingly interested in the engineering side of ML projects: processes and tools needed to go from an idea to a production solution. Currently, he works as an MLOps Solutions Engineer at Iterative.ai, helping customers extract the most value from the Iterative ecosystem of tools.
[](https://twitter.com/alex000kim)
[](https://linkedin.com/in/alex000kim)
[](https://github.com/alex000kim)
### Events
* Best MLOps Practices for Building End-to-End ML Projects ([watch on youtube](https://www.youtube.com/watch?v=2vDwAX9Bf8c)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alexander Hendorf – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alexander Hendorf
Alexander Hendorf is responsible for data and artificial intelligence at the boutique consultancy KÖNIGSWEG GmbH. Through his commitment as a speaker and chair of various international conferences as PyConDE & PyData Berlin, he is a proven expert in the field of data intelligence. He’s been appointed Python Software Foundation and EuroPython fellow for this various contributions. He has many years of experience in the practical application, introduction and communication of data and AI-driven strategies and decision-making processes.
[](https://twitter.com/hendorf)
[](https://linkedin.com/in/hendorf)
[](https://github.com/alanderex)
### Events
* Lessons Learned About Data & AI at Enterprises ([watch on youtube](https://www.youtube.com/watch?v=Vms29u9xC3k)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alex Litvinov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alex Litvinov
Alex is an experienced Software Engineer with a background in bioinformatics, dedicated to the Healthcare and Life Sciences sector throughout his career. He is also deeply interested in everything related to ML/AI, with a recent focus on GenAI-related topics.
[](https://linkedin.com/in/alex-l-45464424)
### Events
* How to Build an LLM-powered QA bot ([watch on youtube](https://www.youtube.com/watch?v=QhFLeZV-PVk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How to Build a Data Science Team from Scratch: Complete Hiring Guide – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How to Build a Data Science Team from Scratch: Complete Hiring Guide
How to Build a Data Science Team from Scratch: Complete Hiring Guide
====================================================================
### Complete guide to hiring data scientists, ML engineers, and data engineers. Learn who to hire first, specialists vs generalists, and how to pick the right projects.
10 Aug 2025 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
Photo by [Hannah Busing](https://unsplash.com/@hannahbusing?utm_source=medium&utm_medium=referral)
on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)
.
Working as a data scientist is one of the most in-demand jobs today. Companies are looking for data scientists to help them make better decisions, drive growth, and improve operational efficiency. But building a data science team from scratch can be a challenge. In this article, we’ll explore some tips on how to build an effective data science team.
The article is organized as follows:
* How to start building a data science team
* Specialists vs generalists
* Who to hire first
* What is a strong product team
* Motivating team to write articles and contribute to open source
* Picking the next projects to work on
How to start building a data science team
-----------------------------------------
There are two ways to start building a data team, as shown in the following figure:
Two main approaches to starting your data science team
* **Defining a project and then starting recruiting** — In this case, you should start with the following questions: What problems are you trying to solve? What kind of data do you need to build your project? Answering these questions will help you determine the skillsets and expertise you need on your team. Once you have an idea of the team you need, it’s time to start recruiting.
* **Starting immediately hiring and then defining a project** — In this case, you start the recruiting campaign, and then define the project.
Indeed, there is no solution better than the other. In fact, some companies firstly define the project and then start hiring, and other companies that do exactly the opposite. A good trade-off could be a combination of both, in the sense that **you could start from a vision**, which defines what the end goal should be, but the final product is still unclear.
Although you could have in your mind your final goal, you may not know which features will lead to the final product. For this reason, **you could start hiring different roles** that would take you to that point to get a better understanding of your vision.
The best way to find talented data scientists is through referrals from people you trust. Attend meetups and conferences related to data science and big data, and post job listings on relevant online communities and forums.
Once you have a few candidates, it’s important to evaluate their skillset carefully. In addition to technical ability, you want to make sure they have strong communication skills and are able to work well in a team environment. The best way to assess these skills is through coding challenges and interviews that test both their technical abilities and soft skills.
Building a data team can be a challenge, but it’s definitely doable with careful planning and execution. By keeping these tips in mind, you’ll be well on your way to creating a strong team.
Specialists vs generalists
--------------------------
When you start hiring, you may search for two types of profiles, as shown in the following figure:
Specialists vs Generalists: choosing the right profile for your team
* **generalists** — professionals who have a general knowledge of all the Data Science workflow, from project setup to project deployment, but they do not have specific knowledge of the single steps of the Data Science workflow.
* **specialists** — professionals who have specific knowledge of a single step of the Data Science workflow, but do not know the general Data Science workflow.
If you decide to hire specialists, you’ll need to make sure that each team member is able to work together and share their knowledge. This can be a challenge, but if you have the right mix of people it can lead to a very strong team. The advantage of this approach is that each team member will be an expert in their own area and they’ll be able to bring a lot of knowledge to the table.
If you decide to go with a team of generalists, the advantage of this approach is that everyone will have a good understanding of the whole data science process.
**In a startup, you should hire first generalists — people who can do pretty much everything.** In fact, when you start, you have no lines of code. There’s nothing, which means you need to do back end, front end, DevOps, and whatever.
Instead, if your company is already stable, probably you need more specialists, who focus on specific problems, such as tuning hyperparameters of a Machine Learning model.
As your company grows, you may begin hiring more and more specialists and fewer generalists.
Who to hire first
-----------------
When building a data science team, it can be difficult to decide who to hire first. The answer, of course, depends on your specific needs.
But in general, we recommend hiring first the following three profiles, as shown in the following figure:
The three key roles to prioritize when building your data science team
* **Machine Learning engineers** or Software engineers with Machine Learning skills.
* **Data engineers**, that build and maintain the systems that collect and store all that data.
* **Product managers**, that prioritize the work to do.
At the beginning, you need people who can work on the prototype, who will write a lot of coding, and who will work on the product. It doesn’t make sense for you to hire a UX/UI designer when you have no work for them.
Of course, every team is different, and you may find that you need to hire a different combination of roles depending on your unique circumstances. So you really need to understand, at which stage you are. And what kind of roles do you need now to solve this problem.
What is a strong product team
-----------------------------
To build a strong product team, it is important to first understand what a product team is. **A product team is responsible for the development and management of a company’s products.** This includes everything from the initial idea and conception of the product, to its design, production, and eventual distribution.
A strong product team is a team that is building a product that the customer wants. Strong means:
* Very customer-centric
* Delivering features very fast
* Testing these things out with customers very fast.
Key characteristics of a strong product team
In other words, a strong product team will have a clear understanding of the market they are targeting and will work together to ensure that the final product meets the needs of their target audience. In order to build a strong product team, it is important to have a clear vision for the product and to assemble a team of individuals with the skills and experience necessary to bring that vision to fruition.
Motivating team to write articles and contribute to open source
---------------------------------------------------------------
Data science is a collaborative field, and one of the best ways to learn and improve your skills is to share your knowledge with others. Writing blog posts and articles is a great way to do this, and it can also be a good motivator for your team.
You should encourage your team members to write articles about their work and provide them with any resources they need to get started. You can also offer incentives for writing articles, such as bonus points or rewards.
Contributing to open-source projects is another great way for your team to learn and improve their skills. You could even encourage your team members to find projects they’re interested in and contribute their time and expertise. Again, you can offer incentives for open source contributions, such as bonus points or rewards.
Hiring a data scientist
-----------------------
When it comes to building a data science team, one of the most important things to consider is hiring a data scientist. While there are many different ways to go about this, there are a few key things to keep in mind.
First, it’s important to find someone with the right skillset. You can look at the **candidate curriculum**. A data scientist should be able to not only analyze data, but also understand how to use that data to solve real-world problems. Additionally, they should be well-versed in statistics and modeling, and be able to communicate their findings effectively.
Once you’ve found a few candidates with the right skillset, it’s important to **interview** them thoroughly. In addition to questions about their experience and qualifications, be sure to ask them about their approach to data science and how they would go about solving a specific problem. This will help you get a better sense of their thought process and whether or not they would be a good fit for your team.
The interview should also include questions on personal experiences, to check whether that person is interesting or not. If someone is just plain boring, this is very difficult for the team. That person also needs a hobby, such as going to the cinema, doing some sports, or whatever. But that person needs to do something. You work more time with that person than you spend with your girlfriend or your wife. You spend more time with them than with the person that you love. So that’s why you need to understand that person really well.
You could also include a second interview, which could be a **homework assignment**. You could send out some homework, which is not very difficult. Then the candidate sends you the code, whether it is Jupyter Notebook or whatever. Then you should check it. From this simple task, you could already see how much people are working.
Three-step process for hiring data scientists
Picking the next projects to work on
------------------------------------
There are a lot of data science projects out there, and it can be tough to decide which ones to tackle next. Here are a few things to keep in mind when choosing projects:
* Consider the impact of the project. Will it make a big difference for your company or customers?
* Think about the difficulty of the project. Is it something you and your team are confident you can handle?
* Take into account the resources you have available. Do you have the time and budget to dedicate to this project?
* Lastly, don’t forget to have some fun! Data science is supposed to be enjoyable, so pick projects that seem interesting and exciting to you and your team.
Summary
-------
Congratulations! You have just learned some tips on how to build a data science team!
Data science is a rapidly growing field, and it can be difficult to build an effective data science team. However, by following the advice in this article, you can create a team that is well-equipped to tackle your data science projects.
When building a data science team, it is important to consider the skills and experience of each team member. The team should have a mix of technical and non-technical skills, as well as a variety of domain expertise. In addition, the team should be able to work together collaboratively to solve complex problems.
Once you have assembled your team, you need to provide them with the resources they need to be successful. This includes access to data, computing power, and software tools. In addition, you need to create an environment that encourages creativity and collaboration.
By following these tips, you can build a data science team that is poised for success!
Happy Data Science!
The content of this article has been inspired by the podcast episode [Building a Data Science Team](https://datatalks.club/podcast/s01e03-building-ds-team.html)
with Dat Tran at DataTalks.Club.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# LLM Zoomcamp: Free LLM Engineering Course and Certification – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
LLM Zoomcamp: Free LLM Engineering Course and Certification
LLM Zoomcamp: Free LLM Engineering Course and Certification
===========================================================
### Master LLM Engineering: Build Production-Ready AI Applications from Scratch
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Teams turn to large language models (LLMs) because they want applications that can answer questions or search information more intelligently. But once they start building, they discover how unstable these systems can be: answers change from one run to the next, search quality depends heavily on how data is indexed, and a small prompt adjustment can break a feature that worked yesterday. Without a clear way to retrieve the right context, measure output quality, or monitor how the system behaves in the real world, it’s difficult to trust the results.
LLM Zoomcamp course cover
LLM Zoomcamp is designed to close this gap. This free course covers the fundamentals of LLMs and RAG, how vector and hybrid search work in practice, how to evaluate retrieval and model responses, and how to monitor user behavior once a system is live. The focus is on practical, grounded methods that make LLM-powered applications more predictable, stable, and easier to maintain.
Table of Contents
-----------------
* [Why Learning to Build LLM Applications is Important](https://datatalks.club/blog/llm-zoomcamp.html#why-learning-to-build-llm-applications-is-important)
* [Who the Course Is For](https://datatalks.club/blog/llm-zoomcamp.html#who-the-course-is-for)
* [Course Curriculum](https://datatalks.club/blog/llm-zoomcamp.html#course-curriculum)
* [How LLM Zoomcamp Works](https://datatalks.club/blog/llm-zoomcamp.html#how-llm-zoomcamp-works)
* [What is DataTalks.Club Community?](https://datatalks.club/blog/llm-zoomcamp.html#what-is-datatalksclub-community)
* [How to Join LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html#how-to-join-llm-zoomcamp)
* [Testimonials](https://datatalks.club/blog/llm-zoomcamp.html#testimonials)
* [Frequently Asked Questions](https://datatalks.club/blog/llm-zoomcamp.html#frequently-asked-questions)
[Join the 2026 cohort →](https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q)
Why Learning to Build LLM Applications Is Important
---------------------------------------------------
Large language models are rapidly expanding across industries, yet transforming experimental prototypes into reliable, production-ready systems remains a significant challenge.
According to Red Hat’s [research](https://www.redhat.com/en/blog/enterprise-ai-survey-ambition-value-gap-and-importance-open-source)
, AI has become a core strategic priority for 72% of organizations in Europe, the Middle East, and Africa, with plans to increase AI investments by an average of 32% by 2026. However, only 7% of organizations currently report “driving customer value” at scale from their AI investments.
This gap between ambitious AI strategies and real outcomes is why expertise in building, evaluating, and maintaining LLM-powered applications is increasingly sought after in today’s market.
Who the Course Is For
---------------------
LLM Zoomcamp is designed for anyone who wants to build practical, reliable LLM-powered applications. If you can write Python, use the command line, and work with Docker, you have everything you need to follow the course.
It’s a good fit for:
* **Software engineers** who want to add LLMs, RAG, and modern search capabilities to real products.
* **Data engineers** interested in how vector search, hybrid search, and retrieval pipelines fit into production systems.
* **ML practitioners** who want a structured way to evaluate and monitor LLM-based applications.
* **Developers new to LLMs** who already know Python and want a clear, practical introduction to building end-to-end AI applications.
* **Technical product managers or tech leads** who need a working understanding of how LLM systems behave in real usage.
* **Anyone maintaining an existing LLM feature** and struggling with drift, inconsistent answers, or unreliable retrieval.
You don’t need prior experience with AI or ML. The course focuses on the engineering side of modern LLM applications and guides you through the concepts step by step.
Course Curriculum
-----------------
The course follows a practical, production-focused approach to building LLM applications. Each module adds a new layer—from RAG foundations to vector search, evaluation, monitoring, and a complete end-to-end project.
| Module | Topic | Focus | Tools You'll Use |
| --- | --- | --- | --- |
| 1 | Introduction to LLMs and RAG | Build your first RAG pipeline with LLM fundamentals and text search | OpenAI API, Elasticsearch |
| 2 | Vector Search | Create embeddings, index documents, and retrieve with semantic search | Qdrant, dlt |
| 2A | Agents (Bonus) | Add agentic behavior and function calling to RAG pipelines | OpenAI Function Calling |
| 3 | Evaluation | Measure retrieval quality and answer accuracy with offline and online evaluation | LLM-as-a-Judge, evaluation frameworks |
| 4 | Monitoring | Track user feedback, chat logs, and system performance in production | Grafana, monitoring dashboards |
| 5 | Best Practices | Optimize retrieval with hybrid search, reranking, and orchestration patterns | LangChain, hybrid search tools |
| 6 | End-to-End Project | Build a complete RAG application combining all components | All tools from previous modules |
### What You’ll Build: Course Project
For the final project, you’ll create a complete end-to-end RAG application. The goal is to show that you can move from raw data to a working, searchable AI system that users can interact with, and that you can evaluate and monitor.
Example project: Fitness Assistant - a conversational AI that helps users choose exercises and find alternatives, making fitness more accessible for beginners who find gyms intimidating or lack access to personal trainers
You’ll build:
1. A searchable knowledge base: Choose a dataset and ingest it, clean it, and store it in a format suitable for retrieval.
2. A retrieval pipeline that works in practice: Implement the core RAG flow—retrieve relevant context from your knowledge base, assemble prompts, send them to an LLM, and return grounded answers.
3. An evaluation process: Measure how well your system retrieves and answers information. This can include search metrics, LLM-as-a-judge, or your own evaluation framework.
4. A user-facing interface: Build a simple UI or API so others can try your application. This can be Streamlit, FastAPI, a small web page, or anything that makes interaction straightforward.
5. Monitoring and feedback loops: Once your app works, you’ll add a basic monitoring layer—tracking queries, feedback, or performance so you can see how it behaves over time.
By the end, you’ll have a working RAG system that demonstrates your ability to design, build, evaluate, and deploy an LLM-powered application.
How LLM Zoomcamp Works
----------------------
### GitHub Repository: Your Source of Truth
All lessons, homework, and cohort updates live in the [LLM Zoomcamp GitHub repository](https://github.com/DataTalksClub/llm-zoomcamp)
. The structure mirrors our other Zoomcamps, so you can quickly find weekly folders, homework forms, and project guidelines.
[LLM Zoomcamp GitHub repository](https://github.com/DataTalksClub/llm-zoomcamp)
showing course materials and structure
### Video Lectures
Lectures are pre-recorded and available in this [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIB4fSqLy_0AfTjVLpgjV3R)
, so you can follow the live cadence or binge-watch at your own pace.
[LLM Zoomcamp YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIB4fSqLy_0AfTjVLpgjV3R)
with pre-recorded lectures
### Homework Assignments
Examples of homework assignments from the 2024 cohort of LLM Zoomcamp covering RAG, vector search, and LLM engineering
We release homework assignments for each week of the course. Your scores are added to an anonymous leaderboard, creating friendly competition among course members and motivating you to do your best.
Course leaderboard displaying student progress and homework scores anonymously
### Learning in Public
A unique feature is our “learning in public” approach, inspired by [Shawn @swyx Wang](https://www.youtube.com/watch?v=tkBCPqWKCL8&list=PL7NIGf5_PlM-Dk3lgPsZFT94Ng7PpRQEh&index=5&t=195s)
’s [article](https://www.swyx.io/learn-in-public)
. We believe everyone has something valuable to contribute, regardless of their expertise level.
Extract from [Shawn @swyx Wang's article](https://www.swyx.io/learn-in-public)
explaining the benefits of learning in public
Throughout the course, we actively encourage and incentivize learning in public. By sharing your progress, insights, and projects online, you earn additional points for your homework and projects.
Sharing your work online also helps you get noticed by social media algorithms, reaching a broader audience and creating opportunities to connect with individuals and organizations you may not have encountered otherwise.
### How to Get a Certificate
LLM Zoomcamp certificate for completing the course
To receive a certificate, you’ll need to complete the [final project](https://datatalks.club/blog/llm-zoomcamp.html#what-youll-build-course-project)
and peer review 3 other students’ projects:
1. **Complete the final project**: Build a real-world LLM application (RAG project) that demonstrates your mastery of all course concepts
2. **Peer review**: Evaluate and provide feedback on 3 fellow students’ projects during the peer review process
3. **Submit on time**: Meet the project submission deadlines to qualify for certification
What is DataTalks.Club Community?
---------------------------------
DataTalks.Club has a supportive community of like-minded individuals in [our Slack](https://datatalks.club/slack.html)
. It is the perfect place to enhance your skills, deepen your knowledge, and connect with peers who share your passion. These connections can lead to lasting friendships, potential collaborations in future projects, and exciting career prospects.
Active discussions and support in the LLM Zoomcamp [Slack community](https://datatalks.club/slack.html)
channel
How to Join LLM Zoomcamp
------------------------
You can join LLM Zoomcamp either by **following a live cohort** or **learning at your own pace**.
All materials are freely available in the [LLM Zoomcamp GitHub repository](https://github.com/DataTalksClub/llm-zoomcamp)
. Each module has its own folder, and cohort-specific homework and deadlines are in the `cohorts` directory. Lectures are pre-recorded and available in this [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIB4fSqLy_0AfTjVLpgjV3R)
.
### Option 1: Self-Paced Learning
You can start anytime and move at your own speed. You get full access to materials and community support on Slack.
You can complete homework assignments on the [course platform](https://courses.datatalks.club/)
and build a project for your portfolio, even outside a live cohort.
> Under self-paced learning, homework isn’t scored, your project isn’t peer-reviewed, and you can’t earn a certificate.
### Option 2: Live Cohort
LLM Zoomcamp runs once per year and typically starts in June.
When you join a live cohort, you get:
* Updated homework
* Automatic homework scoring and a leaderboard
* Project peer review
* Eligibility for a certificate after meeting all requirements
> Even if you join after the official start date, you can still follow along. Note that some homework forms may already be closed. All active deadlines are listed on the [course platform](https://courses.datatalks.club/)
> .
[Join the 2026 cohort →](https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q)
Testimonials
------------
> This course gave me hands-on experience in building LLM-powered applications, including prompt engineering, retrieval-augmented generation (RAG), pipeline orchestration, and vector search optimization.
>
> — [Alexander Daniel Rios](https://www.linkedin.com/in/alexander-daniel-rios)
> , LLM Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/alexander-daniel-rios_llmzoomcamp-ai-llm-activity-7391098999820406784-ByF1)
> )
> Not gonna lie – this course took longer than planned. By the end, I was running on fumes, forcing myself to push through the final modules. But I made it.
>
> What I loved about this course: hands-on experience building real AI systems (not just theory!), deep dives into RAG, vector databases, evaluation, and monitoring, the wealth of knowledge beyond the workshops – production-ready practices that matter in enterprise environments.
>
> — [Vasiliy Chernykh](https://www.linkedin.com/in/v4siliy)
> , LLM Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/v4siliy_llm-machinelearning-ai-activity-7394774137300402177-dFY9)
> )
Frequently Asked Questions
--------------------------
What is the LLM Zoomcamp?
The LLM Zoomcamp is a free, community-driven program by [DataTalks.Club](https://datatalks.club/)
that teaches practical applications with large language models through hands-on project work.
This 10-week course covers a comprehensive [curriculum](https://datatalks.club/blog/llm-zoomcamp.html#course-curriculum)
with all materials open and available anytime on [GitHub](https://github.com/DataTalksClub/llm-zoomcamp)
. You’ll work with an industry-standard stack including OpenAI API, Hugging Face, Elasticsearch, Ollama, Streamlit, Grafana, LangChain, and vector databases, and earn a [certificate](https://datatalks.club/blog/llm-zoomcamp.html#can-i-get-a-certificate)
.
What does zoomcamp mean?
“Zoomcamp” is a term that originated from [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, the founder of DataTalks.Club. It started with his book “ML Bookcamp.” When Alexey decided to create a video course based on the book, he called it “[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
” - a free, cohort-based course in video format. The name “zoomcamp” is a play on “bookcamp,” referring to the video format of the course. The Zoomcamp series has since expanded to include other free courses like the [Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
, [MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
, and [LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
, all following the same community-driven, open-source philosophy.
Is it really free?
Yes, the LLM Zoomcamp is completely free. There are no hidden costs, no tuition fees, and no paid tiers. All course materials, videos, homework assignments, and access to the [community](https://datatalks.club/blog/llm-zoomcamp.html#what-is-the-datatalksclub-llm-community)
are provided at no cost. Unlike traditional bootcamps that charge $10,000-$20,000+, this course is entirely community-driven and open source.
How does the LLM Zoomcamp compare to traditional LLM bootcamps?
The LLM Zoomcamp differs from traditional LLM bootcamps in several key ways:
1. **Cost**: Completely free vs. $10,000-$20,000+ for bootcamps
2. **Community**: Community-driven and open source with all materials available forever on GitHub vs. content locked behind paywalls
3. **Flexibility**: Can continue at your own pace after the cohort ends vs. rigid schedules and limited access periods
How does the LLM Zoomcamp certificate work?
To earn a [certificate](https://datatalks.club/blog/llm-zoomcamp.html#can-i-get-a-certificate)
, you need to complete one [capstone project](https://datatalks.club/blog/llm-zoomcamp.html#course-projects-for-your-portfolio)
that demonstrates your mastery of building an end-to-end RAG application. After submitting your project, you must also review at least 3 other students’ projects by the deadline and provide constructive feedback.
When does the next cohort of the LLM Zoomcamp start?
The next cohort of the LLM Zoomcamp typically starts in the fall each year. Register here: [https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q](https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q)
before the course starts.
Who runs the LLM Zoomcamp?
The LLM Zoomcamp is run by [DataTalks.Club](https://datatalks.club/)
, a global online community of data professionals and learners. While the initial idea and most of the content were created by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, members of the DataTalks.Club community contribute as instructors and maintainers.
DataTalks.Club is often referred to as “the DataTalks Club”, “data talks club”, or “datatalks club”.
What background knowledge do I need to take the LLM Zoomcamp? Course prerequisites.
To get the most out of this course, you should be comfortable with programming and Python, command line, and Docker. No previous exposure to AI or ML is required. The course is designed to teach you everything you need to know about LLMs and RAG from the ground up.
How much time should I expect to spend?
Expect to spend 5-15 hours per week, depending on your background. This includes watching videos, completing homework, and working on [the capstone project](https://datatalks.club/blog/llm-zoomcamp.html#course-projects-for-your-portfolio)
. More time may be needed during the final project weeks.
Can I take the course in self-paced mode?
Yes! All course materials, videos, and recordings remain available after the cohort ends, and you can learn at your own pace. You’ll have access to the [Slack community](https://datatalks.club/slack.html)
for support. However, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a [certificate](https://datatalks.club/blog/llm-zoomcamp.html#can-i-get-a-certificate)
. To receive a certificate, you need to join an active cohort.
Can I get a certificate in self-paced mode?
No, certificates are only available when completing the course with a live cohort. Self-paced mode does not include homework submissions, project evaluations, or certificates. This is because the certification process requires you to peer-review three capstone projects, and these peer reviews only happen during the active course period. Additionally, the submission form closes after the peer-review list is compiled. Self-paced learners can access all course materials and the Slack community, but must join a live cohort to earn a certificate.
Can I still join if I just discovered the course?
Yes, but if you want to receive a certificate, you need to submit your project while we’re still accepting submissions.
Do I need a confirmation email after registration?
You don’t need a confirmation email. You are automatically accepted into the course. You can even start learning and submitting homework (while the form is open) without registering, as submissions are not checked against any registered list. Registration is primarily used to gauge interest before the start date.
How do I join the office hours or live sessions?
There are no office hours—all lectures are pre-recorded and available in the [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, so you can watch them whenever it suits you.
All course materials are in the [GitHub repository](https://github.com/DataTalksClub/llm-zoomcamp)
. Each module has its own folder (for example, 01-intro or 03-classification), while cohort-specific homework and deadlines are located in cohorts/2025.
Occasionally, additional workshops or updated implementation videos are released—there will be additional announcements if this happens.
Can I still get a certificate if I missed the first homework?
Yes! The only requirement for receiving a certificate is completing the capstone project. Homework assignments are not mandatory, though they are recommended for reinforcing concepts and understanding the material better. Your homework points will count toward your position on the course leaderboard, but they don’t affect certificate eligibility.
What if I get stuck?
You have multiple support channels available. Join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
where you can ask questions and get help from instructors and fellow students. We also have an [FAQ repository](https://github.com/DataTalksClub/faq)
with answers to common questions and a @ZoomcampQABot in Slack for quick help.
Where is the GitHub repository?
The GitHub repository is [https://github.com/DataTalksClub/llm-zoomcamp](https://github.com/DataTalksClub/llm-zoomcamp)
.
Where can I find the course videos?
Course videos are available in the [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
. For easier navigation, refer to the [GitHub repository](https://github.com/DataTalksClub/llm-zoomcamp)
. We also maintain [year-specific playlists](https://www.youtube.com/@DataTalksClub/playlists)
for updates.
How many projects do I need to complete for the certificate?
You need to complete one capstone project to earn a certificate. The capstone project is a comprehensive end-to-end RAG application that demonstrates your mastery of all course concepts including retrieval, vector databases, evaluation, monitoring, and LLM orchestration. After submitting your project, you’ll also need to review at least 3 other students’ projects. [Learn more about the capstone project](https://datatalks.club/blog/llm-zoomcamp.html#course-projects-for-your-portfolio)
and [certificate requirements](https://datatalks.club/blog/llm-zoomcamp.html#can-i-get-a-certificate)
.
What are LLMs and why should I learn them?
Large Language Models (LLMs) are AI systems trained on vast amounts of text data that can understand and generate human-like text. They power applications like ChatGPT, code assistants, and intelligent search systems. Learning LLMs is essential for anyone looking to build AI-powered applications, implement RAG (Retrieval-Augmented Generation) systems, or work with modern NLP technologies. The demand for LLM skills is rapidly growing across industries.
What is RAG and why is it important?
RAG (Retrieval-Augmented Generation) is a technique that combines information retrieval with LLM generation to create AI systems that can answer questions about specific knowledge bases. RAG is crucial because it allows LLMs to access up-to-date information and domain-specific knowledge that wasn’t in their training data. This course teaches you to build production-ready RAG systems from scratch.
What tools and technologies will I learn?
The course covers essential LLM tools and platforms including OpenAI API for LLM integration, Hugging Face for open-source models, Elasticsearch for vector search and text search, Ollama for running LLMs locally, Streamlit for building UIs, Grafana for monitoring, LangChain for LLM orchestration, and vector databases for efficient retrieval. You’ll also learn evaluation techniques, monitoring best practices, and data ingestion pipelines.
What is the DataTalks.Club LLM community?
The DataTalks.Club LLM community is a supportive network of 80,000+ data professionals and learners. As part of the LLM Zoomcamp, you’ll have access to a dedicated course channel in [Slack](https://datatalks.club/slack.html)
where you can ask questions, get help from instructors and peers, share your progress, and connect with like-minded individuals. The community provides technical support, peer learning opportunities, and networking that can lead to collaborations and career opportunities. This active community is one of the key differentiators of the LLM Zoomcamp experience.
What LLM training does this course provide?
This comprehensive training covers the complete large language model application lifecycle. You’ll receive hands-on training in building RAG systems, working with open-source and commercial LLMs, implementing vector databases and search, evaluating and monitoring LLM applications, orchestrating LLM pipelines, and deploying production-ready LLM systems. The course includes 7 core technical modules, weekly homework assignments, and a capstone project. This practical training prepares you for real-world AI applications and is taught by expert instructors from the DataTalks.Club community.
Is this a free LLM course with certificate?
Yes! This is a completely free LLM course, with a certificate available when you complete the course with a live cohort. There are no hidden costs or tuition fees. To earn your certificate, you’ll need to complete the technical modules, build one capstone RAG project, participate in peer reviews, and follow LLM best practices. This free course provides the same quality training as paid bootcamps but at no cost. Certificates, homework submissions, and project evaluations are only available when participating in a live cohort, not in self-paced mode.
Can I get a certificate?
Yes, certificates are available when completing the course with a live cohort. Requirements include completing the technical modules, building one capstone RAG project, participating in peer reviews (reviewing at least 3 other students’ projects), and following LLM best practices. Certificates, homework submissions, and project evaluations are not available in self-paced mode.
[Join the 2026 cohort →](https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q)
Related Posts
-------------
[### Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps\
\
Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics\
\
Read more](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
[### Data Engineering Zoomcamp: Free Data Engineering Course and Certification\
\
Become a Data Engineer: Master Modern Data Engineering with Hands-On Training\
\
Read more](https://datatalks.club/blog/data-engineering-zoomcamp.html)
[### MLOps Zoomcamp: Free MLOps Course and Certification\
\
Learn to deploy, monitor, and maintain ML models in production with MLflow, Docker, AWS, and monitoring tools\
\
Read more](https://datatalks.club/blog/mlops-zoomcamp.html)
[### AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers\
\
Learn how to integrate AI into real developer workflows, from AI coding assistants to agents, CI/CD, DevOps, and no-code automation.\
\
Read more](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
[### ML Zoomcamp: Free Machine Learning Engineering Course and Certification\
\
Master Machine Learning Engineering with Python in 4 Months\
\
Read more](https://datatalks.club/blog/machine-learning-zoomcamp.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# How to Setup a Lightweight Local Version for Airflow – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
How to Setup a Lightweight Local Version for Airflow
How to Setup a Lightweight Local Version for Airflow
====================================================
### With Docker and Docker Compose
17 Sep 2022 by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
Forget about the “Low Memory” issues when running Airflow (logos are taken from [Apache Airflow](https://airflow.apache.org/)
and [Docker](https://www.docker.com/)
)
This article was created under the scope of the first edition of the [Data Engineer Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
by [DataTalksClub](https://datatalks.club/)
. I will present a technical variation I made to the initially proposed development to run **Apache Airflow** locally (see [_What means “to run one software locally”_](https://medium.com/@lgsoliveira/what-means-to-run-one-software-locally-a8b556d6f34c)
) with **Docker** and **Docker Compose**. This adaptation was later incorporated into the DE Zoomcamp as seen in this [video](https://www.youtube.com/watch?v=A1p5LQ0zzaQ&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=23)
.
What we’ll cover in this article:
* What are Apache Airflow and Docker
* How to setup the lightweight Docker version to run Airflow
* Reasons why this setup is lighter
This is not a tutorial about Airflow or Docker but an explanation on how to set up a less demanding version of Docker environment to run Airflow locally.
The “full” proposed version to run Airflow inside a Docker container is highly resource-intensive, and hence pushes a lot of one computer/laptop (the cooling fan of my laptop was always ON). For more information about the full version, I advise you to see the Data Engineering Zoomcamp mentioned above and this [article](https://www.linkedin.com/pulse/airflow-o-que-%C3%A9-o-faz-hands-on-leandro-bueno/)
(in Portuguese) by Leandro Bueno.
1\. Introduction to Airflow and Docker
--------------------------------------
### 1.1 Apache Airflow
**Apache Airflow** is one of the most known tools in the data engineering world therefore I will not take long to explain it.
This software is an open-source data orchestrator tool allowing to build full end-to-end pipelines by connecting several processes in **Directed Acyclic Graphs** (**DAG**s). It connects and organizes tasks that manage data, and is not a data streaming tool as mentioned on the official [Airflow website](https://airflow.apache.org/docs/apache-airflow/stable/#)
: “_Airflow is not a data streaming solution. Tasks do not move data from one to the other (though tasks can exchange metadata!)”._
The official logo for Apache Airflow
[Airbnb](http://airbnb.com/)
developed Airflow in 2014, it was made available as a free tool in 2015, and it was donated to the [Apache Foundation](https://www.apache.org/)
in the following year.
In addition to being open-source, Airflow has the following main advantages:
* It is more maintainable, versionable, testable, and collaborative because it was all developed in code, according to the official Airflow website;
* Python, a well-known programming language, is used throughout;
* It has several built-in operators, but you can write your own custom operator if they don’t fulfill your requirements;
* Even being all develop with code it has a fantastic web interface allowing the correct understatement of the flow;
* It is highly scalable.
### 1.2. Docker and Docker Compose
One year ago I started working on **Docker**, a fantastic set of Platform As A Software (PAAS) products, and I’m now a huge fan.
This tool uses OS-level virtualization that allows fantastic customization of software in containers that can be easily shared with your colleagues or between development environments, “_and be sure that everyone you share with gets the same container that works in the same way”_ (by the [Official Docker website](https://docs.docker.com/get-started/overview/#:~:text=Docker%20provides%20the%20ability%20to,simultaneously%20on%20a%20given%20host.)
).
The way Docker works is very simple because it uses a client-server architecture. “_The Docker client talks to the **Docker daemon**, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface. Another Docker client is Docker Compose, which lets you work with applications consisting of a set of containers.”_ (text extracted from the Official Docker Website).
Docker architecture (from the [Official Docker website](https://docs.docker.com/get-started/overview/#:~:text=Docker%20provides%20the%20ability%20to,simultaneously%20on%20a%20given%20host.)
)
The most common way of using Docker is by setting several instructions in a text file called **Dockerfile**. This file first “calls” an image from the public Docker repository (e.g., Python Image, Airflow Image, etc) setting the Base Image, and then it will run several user-defined commands to customize your new image. Then after running the “docker build” command a new image is created and the entire context (recursively) is sent to the daemon.
_**“Docker Compose** is a tool for defining and running multi-container Docker applications. It uses YAML files to configure the application’s services and performs the creation and start-up process of all the containers with a single command (…) The **docker-compose.yml** file is used to define an application’s services and includes various configuration options.”_ (text extracted from [Wikipedia](https://en.wikipedia.org/wiki/Docker_(software)%7B:target=%22_blank%22%7D)
_)._
2\. Setup and run Airflow locally with a less demanding Docker-compose version (with bonus information)
-------------------------------------------------------------------------------------------------------
The setup I am presenting in this section was built and set to run successfully under the scope of my final Capstone of the Zoomcamp mentioned above. You can see my full capstone [here](https://github.com/guoliveira/data-engineer-zoomcamp-project)
.
I divided this part of the article into two:
1. The main design to enable Airflow to run in Docker containers and
2. The adaptation I created to allow a less demanding configuration.
### 2.1. Structure to allow Airflow to run in Docker
In order to run Airflow locally (in a Docker container), I used an extended image, containing some additional dependencies.
Therefore I firstly created a Dockerfile pointing to the Airflow version I needed, such as apache/airflow:2.2.3, as the base image.
FROM apache/airflow:2.2.3
Then I customized this Dockerfile by adding some custom packages to be installed. The one I was going to need the most was _gcloud_ to connect with the GCS bucket/Data Lake and integrate “requirements.txt” to install libraries via pip install.
And here is the bonus I promised: I decided to run Spark (using Pyspark) in my DAG so I had to configure Spark in the Dockerfile. This was possible by adding bash commands to set Java env and inserting commands to download all the necessary files to run Spark. Besides, it was necessary to insert Pyspark as a requirement.
ENV JAVA_HOME=/home/jdk-11.0.2
ENV PATH="${JAVA_HOME}/bin/:${PATH}"
RUN DOWNLOAD_URL="https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz" \
&& TMP_DIR="$(mktemp -d)" \
&& curl -fL "${DOWNLOAD_URL}" --output "${TMP_DIR}/openjdk-11.0.2_linux-x64_bin.tar.gz" \
&& mkdir -p "${JAVA_HOME}" \
&& tar xzf "${TMP_DIR}/openjdk-11.0.2_linux-x64_bin.tar.gz" -C "${JAVA_HOME}" --strip-components=1 \
&& rm -rf "${TMP_DIR}" \
&& java --version
Then I ran curl to import the official docker-compose setup file (docker-compose.yml) from the latest Airflow version into my laptop.
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
Then I adapted the Yaml file for running docker-compose:
* In x-airflow-common, a) I removed the image tag, to replace by my Dockerfile, b) mounted my google\_credentials in volumes section as read-only and c) set environment variables `GOOGLE_APPLICATION_CREDENTIALS` and `AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT`;
* And I changed `AIRFLOW__CORE__LOAD_EXAMPLES` to `false`;
With all the steps mentioned before I was ready to run the full version to run Airflow locally with Docker.
### 2.2. Modification to allow a lightweight version
To achieve my objective of having a lightweight version I had to remove several parts of the docker-compose.yml file. All these processes were removed for a solid reason as I will explain in the next section.
1. I firstly removed the _redis_ part:
redis:
image: redis:latest
expose:
- 6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
1. It was to remove the _airflow-worke_r service. A controversial approach since we are talking about workers but in the next section you will understand this:
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
environment:
<<: *airflow-common-env
# Required to handle warm shutdown of the celery workers properly
# See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
DUMB_INIT_SETSID: "0"
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
1. Then I removed the section for _airflow-triggerer_:
airflow-triggerer:
<<: *airflow-common
command: triggerer
healthcheck:
test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
1. And finally, the _flower_ section was removed:
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
depends_on:
<<: *airflow-common-depends-on
airflow-init:
condition: service_completed_successfully
With those parts removed some dependencies need to be corrected:
user: "${AIRFLOW_UID:-50000}:0"
depends_on:
&airflow-common-depends-on
redis:
condition: service_healthy
postgres:
condition: service_healthy
And ultimately I set the Executor from CoreExecutor to LocalExecutor. In the next section, I’ll explain why we changed the Executor and why this is the **most important part**.
`AIRFLOW__CORE__EXECUTOR`: CeleryExecutor LocalExecutor
The original and official version of the docker-compose.yml is [this](https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml)
and my final version is presented [here](https://github.com/guoliveira/data-engineer-zoomcamp-project/blob/main/Airflow/docker-compose.yaml)
.
3\. Why this setup is lighter
-----------------------------
The main reason this setup is lighter than the full version is due to the choice of setting the core executor as **LocalExecutor** (single-node). Then, in a relationship of correlation, some features that were dependent on **CeleryExecutor** (multi-node) could be removed.
In Airflow a DAG will be executed and completed due to three main components, a) the **Metadata** **Database**, b) the **Scheduler** and c) the **Executor**(from the [Astronomer.io](https://www.astronomer.io/guides/airflow-executors-explained/)
website)
The function of the Executor is to work together with the Scheduler to understand what resources will actually complete those tasks as they’re queued.
A CeleryExecutor (related to the [Celery](https://docs.celeryq.dev/en/stable/)
distributed system) is specially made for horizontal scaling because CeleryExecutor works with a “pool” of independent workers (in a reliable distributed system) across which it can delegate tasks, via messages (according to the [Astronomer.io](https://www.astronomer.io/guides/airflow-executors-explained/)
website). However, this Executor is highly resource-intensive. The LocalExecutor exemplifies single-node architecture (hence its resource light) but it still allows parallelism.
Therefore it is recommended to use LocalExecutor for local tests (run locally) and CeleryExecutor for production.
Since I set the Executor to LocalExecutor I could remove the parts _airflow-worke_r and _flower_ because they only work for Celery architecture.
_Redis_ is a simple caching server (see [Redis.io](https://redis.io/)
) and it is necessary to setup as a Celery backend (see [CeleryExecutor](https://airflow.apache.org/docs/apache-airflow/1.10.13/executor/celery.html#:~:text=For%20this%20to%20work%2C%20you%20need%20to%20setup%20a%20Celery%20backend%20(RabbitMQ%2C%20Redis%2C%20%E2%80%A6){:target=%22_blank%22})
) for the CeleryExecutor. Since we will not use this Executor it is safe to remove this service-
I decided to remove the _airflow-trigger_ because it is a new airflow service specially made for asyncio event loop, that I was not going to use.
Conclusion
----------
Now I can run Airflow locally without damaging my laptop too much or worrying about the integrity of my cooling face (it doesn’t feel like I’m operating an airplane 😉) because we are running it with a single-node executor (but still having parallelism) and having fewer processes running.
I would like to have your feedback on the information in this article.
Do you think I was clear or unclear on the technical point of view?
Did I write anything technically wrong?
Did you like this article? Follow me for more articles on [Medium](https://medium.com/@lgsoliveira)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anahita Pakiman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anahita Pakiman
Anahita is a Data scientist-engineer with a strong background in the knowledge graph, semantic web, digitalization, and mechanical engineering. Currently the Senior Knowledge Graph-Data Scientist Consultant at brox IT-Solutions
[](https://linkedin.com/in/anahita-pakiman-65988342)
### Events
* Knowledge Graphs and LLMs Across Academia and Industry ([watch on youtube](https://www.youtube.com/watch?v=YncdlUscUOo)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alvaro Navas Peire – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alvaro Navas Peire
Born in Barcelona, Spain. Grew up there until my teenage years, in which I moved to Andorra and lived there until university time. Studied “Informatics engineering” (a sort of mix between computer science and engineering) back in Barcelona and had a wild time in university by joining lots of clubs and not studying at all. After graduating, I joined a small company which sold Chinese-designed Android smartphones under their own brand; I tested a couple of models and underwent the required homologation process that cell phone carriers used to require in order to have them sell your phones and managed to get a model approved before the company changed investors and decided to focus their efforts in Latin America, so back in 2016
I moved to Mexico to keep working with them under their new brand name. I got tired of the cell phone industry and decided to quit my job, return to Barcelona and get back to studying and fell upon Machine Learning and Data Science. After being unemployed for a couple of years in which I managed to finish a postgraduate course and a few more informal courses (including DataTalks’ ML zoomcamp and DE zoomcamp), I managed to get hired by a consultancy company and I’m currently managing an internal project which makes use of NLP models in order to classify user posts on social networks.
[](https://twitter.com/ziritrion)
[](https://linkedin.com/in/alvaronavas)
[](https://github.com/ziritrion)
### Events
* From Testing Phones to Managing NLP Projects ([watch on youtube](https://www.youtube.com/watch?v=-xumbiXOlA8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andreas Kretz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andreas Kretz
Data Engineer and Plumber of Data Science. He writes and talks about platform architecture, tools and techniques that are used to build modern data science platforms
[](https://twitter.com/andreaskayy)
[](https://linkedin.com/in/andreas-kretz)
[](https://github.com/andkret)
[](https://www.teamdatascience.com/)
### Events
* DataTalks.Club Summer Marathon: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
* Build Your Own Data Pipeline ([watch on youtube](https://www.youtube.com/watch?v=IrZPAG6OBqo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Build-Your-Own-Data-Pipeline---Andreas-Kretz-e139se1)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andreas Syrén – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andreas Syrén
Andreas Syrén is a Principal Applied Scientist at Zalando. He found his love for mathematics while studying Mechatronics Engineering, pivoting into a master’s in Engineering Mathematics & Computational Science at Chalmers University of Technology - before heading into industry. “I am passionate about the Applied part of Applied Science. Massive impact is out there whenever we can successfully apply scientific methodology with real-world outcomes. But for that to happen we need much more than only theory - from an engineering mindset; to people, product, and sales.” In his free time, you can find him with his friends playing TTRPGs, doing some kind of boardsport, or reading up on some non-applied area of mathematics.
[](https://linkedin.com/in/andreassyren)
### Events
* Inventory Optimization in E-commerce ([watch on youtube](https://www.youtube.com/watch?v=RlXlJK8OXhY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anastasia Karavdina – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anastasia Karavdina
Anastasia is a particle physicist turned data scientist, with experience in large-scale experiments like those at the Large Hadron Collider. She also worked at Blue Yonder, scaling AI-driven solutions for global supply chain giants, and at Kaufland e-commerce, focusing on NLP and search. Anastasia is a mentor for Ml/AI, dedicated to helping her mentees achieve their goals. She is passionate about growing the next generation of data science elite in Germany: from Data Analysts up to ML Engineers.
[](https://linkedin.com/in/dr-anastasia-karavdina)
[](https://www.karavdina.com/)
### Events
* Large Hadron Collider and Mentorship ([watch on youtube](https://www.youtube.com/watch?v=kV0ZDy2UtJA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrada Olteanu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrada Olteanu
I am a Data Scientist at Endava, a Kaggle Notebooks Master and one of the 16 data Science Ambassadors for Z by HP & NVIDIA. I also have completed an undergrad in Stats while working as a full time Data Analyst at Avon Cosmetics and a Masters in DS and Analytics in the UK.
[](https://twitter.com/andradaolteanuu)
[](https://linkedin.com/in/andrada-olteanu-3806a2132)
### Events
* Shifting Career from Analytics to Data Science ([watch on youtube](https://www.youtube.com/watch?v=ixmTewD5Waw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Shifting-Career-from-Analytics-to-Data-Science---Andrada-Olteanu-ev19ma)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# MLOps Zoomcamp: Free MLOps Course and Certification – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
MLOps Zoomcamp: Free MLOps Course and Certification
MLOps Zoomcamp: Free MLOps Course and Certification
===================================================
### Learn to deploy, monitor, and maintain ML models in production with MLflow, Docker, AWS, and monitoring tools
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Most companies struggle with machine learning operations. Models get trained, but turning them into reliable, monitored, production systems often falls on ML engineers, data scientists, or whoever “owns” the model—usually without clear processes or tooling. As a result, teams reinvent workflows, rely on ad-hoc scripts, and operate ML systems with limited visibility and no reproducibility.
MLOps Zoomcamp is a free MLOps course designed to close that gap. It teaches you how to put ML models into production using real tools: MLflow, orchestration frameworks, deployment patterns, monitoring, and AWS examples, without drowning you in theory. By the end, you’ll know how to build, deploy, and maintain a complete ML pipeline the way modern teams expect.
MLOps Zoomcamp course curriculum showing 6 modules from infrastructure to production deployment
If you’re an ML engineer, data scientist, or software developer working with ML models and want to elevate your MLOps skills, MLOps Zoomcamp provides a practical, end-to-end foundation you can apply immediately.
Table of Contents
-----------------
* [What is MLOps Zoomcamp?](https://datatalks.club/blog/mlops-zoomcamp.html#what-is-mlops-zoomcamp)
* [Why is MLOps Important?](https://datatalks.club/blog/mlops-zoomcamp.html#why-is-mlops-important)
* [Who is This Course For?](https://datatalks.club/blog/mlops-zoomcamp.html#who-is-this-course-for)
* [Course Curriculum](https://datatalks.club/blog/mlops-zoomcamp.html#course-curriculum)
* [How MLOps Zoomcamp Works](https://datatalks.club/blog/mlops-zoomcamp.html#how-mlops-zoomcamp-works)
* [What is DataTalks.Club Community?](https://datatalks.club/blog/mlops-zoomcamp.html#what-is-datatalksclub-community)
* [How to Join MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html#how-to-join-mlops-zoomcamp)
* [Frequently Asked Questions](https://datatalks.club/blog/mlops-zoomcamp.html#frequently-asked-questions)
[Join the waitlist →](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
What is MLOps Zoomcamp?
-----------------------
MLOps Zoomcamp is a free MLOps course that takes you from experiment tracking to production deployment in 6 modules plus a portfolio project.
You’ll learn infrastructure setup with Docker and AWS, experiment tracking with MLflow, pipeline orchestration with Mage, model deployment (batch, real-time, and streaming), monitoring with Prometheus and Evidently AI, and testing/CI/CD best practices.
The course culminates in a real-world project where you build, deploy, and monitor a complete ML pipeline that you can showcase to employers.
Why is MLOps Important?
-----------------------
MLOps bridges the gap between model development and production deployment by turning machine learning experiments into reliable, continuously running services. Unlike traditional software, ML systems introduce challenges such as data drift, model degradation, reproducibility issues, and the need for ongoing retraining. These characteristics make MLOps a standalone discipline with its own workflows, tools, and operational requirements.
Yet, MLOps is still a relatively new field and many organizations struggle with it. In [practitioner surveys](https://mlops.community/3-takeaways-from-our-survey-of-top-ml-teams/)
, **84.3%** of data scientists and ML engineers report that detecting and diagnosing production model issues is a recurring problem; **26.2%** say it takes one week or more to resolve an issue. Without proper MLOps practices, organizations may spend millions on ML models that don’t deliver or can’t be sustained.
As machine learning becomes a strategic capability, MLOps is the engine that makes it reliable, maintainable, and scalable. This recognition is driving rapid market growth:
* The global MLOps market [was valued](https://www.grandviewresearch.com/industry-analysis/mlops-market-report)
at approximately **US $2.19 billion in 2024** and is projected to reach around **US $16.6 billion by 2030**, growing at a compound annual growth rate (CAGR) of ~40.5%.
* [Another estimate](https://www.gminsights.com/industry-analysis/mlops-market)
puts the global MLOps market at **US $1.7 billion in 2024**, with a forecast CAGR of 37.4% through 2034.
Who is This Course For?
-----------------------
This course is for practitioners who work with machine learning models and want to learn how to operationalize them—from experimentation to deployment, monitoring, and ongoing maintenance. It’s especially relevant for data scientists, ML engineers, and software engineers who are expected to manage the full lifecycle of ML systems.
If you’re comfortable with the command line and Python, and you have prior exposure to machine learning and Docker (either from work or from courses like the [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
), you have the right background to follow the MLOps Zoomcamp successfully.
Course Curriculum
-----------------
Course overview: a complete journey through modern MLOps tools and technologies
The curriculum follows a logical progression from experimentation to production deployment and monitoring, culminating in an end-to-end project. Here’s what you’ll learn each week:
| Module | Topic | Focus | Tools You'll Use |
| --- | --- | --- | --- |
| 1 | Infrastructure & Prerequisites | Build your dev environment with Docker, AWS, and containerized deployment basics | Docker, AWS, Terraform, cloud shells |
| 2 | Experiment Tracking & Model Management | Track experiments, manage model versions, and compare runs | MLflow Tracking, MLflow Model Registry |
| 3 | Orchestration & ML Pipelines | Create reproducible pipelines and manage dependencies end-to-end | Mage, Airflow, Prefect |
| 4 | Model Deployment | Ship models via batch jobs, web APIs, and streaming services | Flask, Docker, AWS Lambda, AWS Kinesis |
| 5 | Model Monitoring | Detect drift and monitor health with production-grade dashboards | Prometheus, Grafana, Evidently AI |
| 6 | Testing & CI/CD | Add testing, CI/CD, and cloud infrastructure fundamentals | Pytest, GitHub Actions, LocalStack |
### What You’ll Build: Course Project
The final part of the course is dedicated to a **hands-on MLOps project** where you apply everything you’ve learned to build a real, production-ready ML system. This project becomes a strong addition to your portfolio and demonstrates that you understand not only how to train models but also how to **operationalize** them.
You’ll:
* **Choose a dataset** that interests you
* **Train a model** and **track experiments** using MLflow or Weights & Biases
* **Build an automated training pipeline** using tools like Mage, Airflow, or Prefect
* **Deploy your model** as a batch job, web service, or streaming system
* **Set up monitoring** with Evidently AI, Prometheus, or Grafana
* **Implement CI/CD workflows** using GitHub Actions or GitLab CI/CD
* **Document your architecture** and make the project reproducible
You have freedom to choose your stack: cloud providers (AWS, GCP, Azure), orchestration tools, experiment trackers, and monitoring systems, while still following the core MLOps lifecycle required by the course.
Completing the project and reviewing peers’ work is also what qualifies you for the **MLOps Zoomcamp certificate**.
[Join the waitlist →](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
How MLOps Zoomcamp Works
------------------------
### GitHub Repository: Your Source of Truth
All lessons, homework, and cohort updates live in the [MLOps Zoomcamp GitHub repository](https://github.com/DataTalksClub/mlops-zoomcamp)
.
[MLOps Zoomcamp GitHub repository](https://github.com/DataTalksClub/mlops-zoomcamp)
showing course materials and structure
### Video Lectures
Lectures are pre-recorded and available on the [official YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, so you can follow the live cadence or binge-watch at your own pace.
### Homework Assignments
We release homework assignments for each week of the course. Your scores are added to an anonymous leaderboard, creating friendly competition among course members and motivating you to do your best.
Course leaderboard displaying student progress and homework scores anonymously
### Learning in Public
A unique feature is our “learning in public” approach, inspired by [Shawn @swyx Wang](https://www.youtube.com/watch?v=tkBCPqWKCL8&list=PL7NIGf5_PlM-Dk3lgPsZFT94Ng7PpRQEh&index=5&t=195s)
’s [article](https://www.swyx.io/learn-in-public)
. We believe that everyone has something valuable to contribute, regardless of their expertise level.
Extract from Shawn @swyx Wang's article explaining the benefits of learning in public
Throughout the course, we actively encourage and incentivize learning in public. By sharing your progress, insights, and projects online, you earn additional points for your homework and projects.
Course leaderboard highlighting bonus points earned through learning in public activities
This not only demonstrates your knowledge but also builds a portfolio of valuable content. Sharing your work online also helps you get noticed by social media algorithms, reaching a broader audience and creating opportunities to connect with individuals and organizations you may not have encountered otherwise.
### How to Get a Certificate
To receive a certificate, you’ll need to complete the [final project](https://datatalks.club/blog/mlops-zoomcamp.html#what-youll-build-course-project)
and peer review 3 other students’ projects:
1. **Complete the final project**: Build a real-world MLOps project that demonstrates your mastery of all course concepts
2. **Peer review**: Evaluate and provide feedback on 3 fellow students’ projects during the peer review process
3. **Submit on time**: Meet the project submission deadlines to qualify for certification
What is DataTalks.Club Community?
---------------------------------
Active discussions and support in the MLOps Zoomcamp [Slack community](https://datatalks.club/slack.html)
channel
[DataTalks.Club](https://datatalks.club/)
is a global community of 80,000+ data professionals who connect on [Slack](https://datatalks.club/slack.html)
to share knowledge, ask career questions, and collaborate across analytics, ML, and data engineering. When you join MLOps Zoomcamp, the dedicated Slack channel becomes your daily workspace for troubleshooting, accountability, and celebrating wins with peers following the same modules.
How to Join MLOps Zoomcamp
--------------------------
You can join MLOps Zoomcamp either by **following a live cohort** or **learning at your own pace**.
All materials are freely available in the [MLOps Zoomcamp GitHub repository](https://github.com/DataTalksClub/mlops-zoomcamp)
. Each module has its own folder, and cohort-specific homework and deadlines are in the `cohorts` directory. Lectures are pre-recorded and available in this [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
.
### Option 1: Self-Paced Learning
Start anytime. You get full access to materials and community support on Slack.
Complete homework assignments: homework and solutions are available on the [course platform](https://courses.datatalks.club/)
. Build a project for your portfolio.
> Under self-paced learning, homework isn’t scored, your project isn’t peer-reviewed, and you can’t earn a certificate.
### Option 2: Live Cohort
Runs once per year (typically starts in the spring).
Includes:
* Updated homework
* Automatic homework scoring and a leaderboard
* Project peer review
* Eligibility for a certificate after meeting all requirements
Even if you join after the official start date, you can still follow along — but note that some homework forms may already be closed. All active deadlines are listed on the [course platform](https://courses.datatalks.club/)
.
> To earn a [certificate](https://datatalks.club/blog/mlops-zoomcamp.html#can-i-get-a-certificate)
> , you’ll need enough time to complete one [final project](https://datatalks.club/blog/mlops-zoomcamp.html#project-phase)
> and the required peer reviews. Details are in the Projects and Certificate sections.
[Join the waitlist →](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
Frequently Asked Questions
--------------------------
What is the MLOps Zoomcamp?
The MLOps Zoomcamp is a free, community-driven program by [DataTalks.Club](https://datatalks.club/)
that teaches core MLOps skills through hands-on project work.
This 3-month course covers a comprehensive [curriculum](https://datatalks.club/blog/mlops-zoomcamp.html#course-curriculum)
with all materials open and available anytime on [GitHub](https://github.com/DataTalksClub/mlops-zoomcamp)
. You’ll work with an industry-standard stack including MLflow, Docker, AWS, Prometheus, Grafana, Mage, and GitHub Actions and earn a [certificate](https://datatalks.club/blog/mlops-zoomcamp.html#can-i-get-a-certificate)
.
What does zoomcamp mean?
“Zoomcamp” is a term that originated from [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, the founder of DataTalks.Club. It started with his book “ML Bookcamp.” When Alexey decided to create a video course based on the book, he called it “[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
” - a free, cohort-based course in video format. The name “zoomcamp” is a play on “bookcamp,” referring to the video format of the course. The Zoomcamp series has since expanded to include other free courses like the [Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
, [MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
, and [LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
, all following the same community-driven, open-source philosophy.
Is it really free?
Yes, the MLOps Zoomcamp is completely free. There are no hidden costs, no tuition fees, and no paid tiers. All course materials, videos, homework assignments, and access to the [community](https://datatalks.club/blog/mlops-zoomcamp.html#what-is-the-datatalksclub-mlops-community)
are provided at no cost. Unlike traditional bootcamps that charge $10,000-$20,000+, this course is entirely community-driven and open source.
How does the MLOps Zoomcamp compare to traditional MLOps bootcamps?
The MLOps Zoomcamp differs from traditional MLOps bootcamps in several key ways:
1. **Cost**: Completely free vs. $10,000-$20,000+ for bootcamps
2. **Community**: Community-driven and open source with all materials available forever on GitHub vs. content locked behind paywalls
3. **Flexibility**: Can continue at your own pace after the cohort ends vs. rigid schedules and limited access periods
How does the MLOps Zoomcamp certificate work?
To earn a [certificate](https://datatalks.club/blog/mlops-zoomcamp.html#can-i-get-a-certificate)
, you need to complete one [end-to-end MLOps project](https://datatalks.club/blog/mlops-zoomcamp.html#project-phase)
that demonstrates your mastery of all course concepts. After submitting your project, you must also review at least 3 other students’ projects by the deadline and provide constructive feedback.
When does the next cohort of the MLOps Zoomcamp start?
The next cohort of the MLOps Zoomcamp typically starts in the spring each year. Register here: [https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
before the course starts.
Who runs the MLOps Zoomcamp?
The MLOps Zoomcamp is run by [DataTalks.Club](https://datatalks.club/)
, a global online community of data professionals and learners. While the initial idea and most of the content were created by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, members of the DataTalks.Club community contribute as instructors and maintainers. Expert instructors include Cristian Martinez, Emeli Dral, and others.
DataTalks.Club is often referred to as “the DataTalks Club”, “data talks club”, or “datatalks club”.
What background knowledge do I need to take the MLOps Zoomcamp? Course prerequisites.
To get the most out of this MLOps course, you should have prior programming experience (1+ year), basic understanding of machine learning concepts, familiarity with Python, basic command line knowledge, and previous exposure to Docker (recommended). Prior exposure to machine learning from work or other courses (e.g., from [ML Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
) is helpful.
How much time should I expect to spend?
Expect to spend 5-15 hours per week, depending on your background. This includes watching videos, completing homework, and working on [the final project](https://datatalks.club/blog/mlops-zoomcamp.html#project-phase)
. More time may be needed during the final project weeks.
Can I take the course in self-paced mode?
Yes! All course materials, videos, and recordings remain available after the cohort ends, and you can learn at your own pace. You’ll have access to the [Slack community](https://datatalks.club/slack.html)
for support. However, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a [certificate](https://datatalks.club/blog/mlops-zoomcamp.html#can-i-get-a-certificate)
. To receive a certificate, you need to join an active cohort.
What if I get stuck?
You have multiple support channels available. Join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
where you can ask questions and get help from instructors and fellow students. We also have an [FAQ repository](https://github.com/DataTalksClub/faq)
with answers to common questions and a @ZoomcampQABot in Slack for quick help.
Where is the official MLOps Zoomcamp GitHub repository?
The GitHub repository is [https://github.com/DataTalksClub/mlops-zoomcamp](https://github.com/DataTalksClub/mlops-zoomcamp)
.
Where can I find the course videos?
Course videos are available in the [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
. For easier navigation, refer to the [GitHub repository](https://github.com/DataTalksClub/mlops-zoomcamp)
. We also maintain [year-specific playlists](https://www.youtube.com/@DataTalksClub/playlists)
for updates.
How do I join the office hours or live sessions?
There are no office hours—all lectures are pre-recorded and available in the [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, so you can watch them whenever it suits you.
All course materials are in the [GitHub repository](https://github.com/DataTalksClub/mlops-zoomcamp)
. Each module has its own folder (for example, 01-intro or 03-classification), while cohort-specific homework and deadlines are located in cohorts/2025.
Occasionally, additional workshops or updated implementation videos are released—there will be additional announcements if this happens.
How many projects do I need to complete for the certificate?
You need to complete one end-to-end MLOps project to earn a certificate. The project is a comprehensive MLOps solution that demonstrates your mastery of all course concepts including experiment tracking, pipeline orchestration, model deployment, and monitoring. After submitting your project, you’ll also need to review at least 3 other students’ projects. [Learn more about the final project](https://datatalks.club/blog/mlops-zoomcamp.html#project-phase)
and [certificate requirements](https://datatalks.club/blog/mlops-zoomcamp.html#can-i-get-a-certificate)
.
What is MLOps and why should I learn it?
MLOps (Machine Learning Operations) is a set of practices that combines Machine Learning, DevOps, and Data Engineering. It enables automating ML pipelines, deploying models to production, monitoring model performance, and ensuring ML system reliability. Learning MLOps is essential for anyone working with machine learning in production environments.
What tools and technologies will I learn?
The course covers essential MLOps tools and platforms including MLflow for experiment tracking, Docker for containerization, AWS services (including Kinesis), Prometheus and Grafana for monitoring, Mage for ML pipeline orchestration, and GitHub Actions for CI/CD.
How does MLOps experiment tracking work in this course?
MLOps experiment tracking is a core component of the course. You’ll learn to use MLflow for comprehensive experiment tracking, including tracking model parameters, metrics, and artifacts. The course covers implementing model registry and versioning, organizing ML experiments effectively, and comparing experiment results. This hands-on training ensures you can manage and reproduce ML experiments in production environments.
Does this course provide MLflow certification?
While this course does not provide an official MLflow certification from the MLflow creators, you’ll receive comprehensive MLflow training as part of the MLOps Zoomcamp. The course includes in-depth coverage of MLflow for experiment tracking, model registry, and versioning. Upon completing the course with a live cohort, you’ll earn a DataTalks.Club MLOps Zoomcamp certificate that demonstrates your proficiency with MLflow and other MLOps tools. This practical training is more valuable than a standalone certification as it’s integrated into a complete production ML workflow.
What is the DataTalks.Club MLOps community?
The DataTalks.Club MLOps community is a supportive network of 80,000+ data professionals and learners. As part of the MLOps Zoomcamp, you’ll have access to a dedicated course channel in [Slack](https://datatalks.club/slack.html)
where you can ask questions, get help from instructors and peers, share your progress, and connect with like-minded individuals. The community provides technical support, peer learning opportunities, and networking that can lead to collaborations and career opportunities. This active community is one of the key differentiators of the MLOps Zoomcamp experience.
What MLOps training does this course provide?
This comprehensive training covers the complete machine learning operations lifecycle. You’ll receive hands-on training in experiment tracking with MLflow, containerization with Docker, ML pipeline orchestration with Mage, model deployment (batch, real-time, and streaming), monitoring with Prometheus and Grafana, and CI/CD with GitHub Actions. The course includes 6 core technical modules, weekly homework assignments, and a final end-to-end project. This practical training prepares you for real-world production ML systems and is taught by expert instructors including Cristian Martinez, Alexey Grigorev, and Emeli Dral.
Is this a free MLOps course with certificate?
Yes! This is a completely free MLOps course, with a certificate available when you complete the course with a live cohort. There are no hidden costs or tuition fees. To earn your certificate, you’ll need to complete the technical modules, build one end-to-end MLOps project, participate in peer reviews, and follow MLOps best practices. This free course provides the same quality training as paid bootcamps but at no cost. Certificates, homework submissions, and project evaluations are only available when participating in a live cohort, not in self-paced mode.
Can I get a certificate?
Yes, certificates are available when completing the course with a live cohort. Requirements include completing the technical modules, building one final end-to-end MLOps project, participating in peer reviews, and following MLOps best practices. Note that certificates, homework submissions, and project evaluations are not available in self-paced mode.
[Join the waitlist →](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
Related Posts
-------------
[### Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps\
\
Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics\
\
Read more](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
[### ML Zoomcamp: Free Machine Learning Engineering Course and Certification\
\
Master Machine Learning Engineering with Python in 4 Months\
\
Read more](https://datatalks.club/blog/machine-learning-zoomcamp.html)
[### Data Engineering Zoomcamp: Free Data Engineering Course and Certification\
\
Become a Data Engineer: Master Modern Data Engineering with Hands-On Training\
\
Read more](https://datatalks.club/blog/data-engineering-zoomcamp.html)
[### LLM Zoomcamp: Free LLM Engineering Course and Certification\
\
Master LLM Engineering: Build Production-Ready AI Applications from Scratch\
\
Read more](https://datatalks.club/blog/llm-zoomcamp.html)
[### AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers\
\
Learn how to integrate AI into real developer workflows, from AI coding assistants to agents, CI/CD, DevOps, and no-code automation.\
\
Read more](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alexander Daniel Rios – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alexander Daniel Rios
Alexander Daniel Rios is about to graduate with a degree in Electronic Engineering from Universidad Tecnológica Nacional (UTN – FRBA), a public engineering university in Buenos Aires. It was through ML Zoomcamp program that he first discovered my passion for programming and data science, which led me to explore the world of Python, machine learning, and data-driven problem solving. He has been a Python developer for over 7 years, and for the past 3 years, he has been focused on applying data science techniques to real-world challenges. He is now transitioning into a role as a Data Scientist or Machine Learning Engineer.
[](https://linkedin.com/in/alexander-daniel-rios)
[](https://github.com/aletbm)
[](https://aletbm.github.io/Portfolio/)
### Articles
* [How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
* [Building Discipline in Machine Learning with ML Zoomcamp](https://datatalks.club/blog/building-discipline-in-machine-learning-with-ml-zoomcamp.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alexia Audevart – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alexia Audevart
Alexia Audevart, also a Google Developer Expert in machine learning, is the founder of datactik. She is a data scientist and helps her clients solve business problems by making their applications smarter. Her first book is a collaboration on artificial intelligence and neuroscience.
[](https://twitter.com/aaudevart)
[](https://linkedin.com/in/alexia-audevart)
[](https://github.com/aaudevart)
[](https://www.datactik.com/)
### Books
* [Machine Learning Using TensorFlow Cookbook](https://datatalks.club/books/20210503-machine-learning-using-tensorflow-cookbook.html)
(the book of the week from 03 May 2021 to 07 May 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alex Petrov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alex Petrov
Alex is an Infrastructure Engineer, Apache Cassandra Committer, working on building data infrastructure and processing pipelines. He’s interested in CS Theory, algorithms, Distributed Systems, understanding how things work and sharing it with others through blog posts, articles and conference talks.
[](https://twitter.com/ifesdjeen)
[](https://github.com/ifesdjeen)
[](http://coffeenco.de/)
### Books
* [Database Internals](https://datatalks.club/books/20210315-database-internals.html)
(the book of the week from 15 Mar 2021 to 19 Mar 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrey Cheptsov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrey Cheptsov
Andrey Cheptsov is the founder and CEO of dstack, an open-source alternative to Kubernetes and Slurm, built to simplify the orchestration of AI infrastructure. Before dstack, Andrey worked at JetBrains for over a decade helping different teams make the best developer tools.
[](https://twitter.com/andrey_cheptsov)
[](https://linkedin.com/in/andrey-cheptsov)
[](https://github.com/dstackai)
[](https://dstack.ai/)
### Events
* Trends in AI Infrastructure ([watch on youtube](https://www.youtube.com/watch?v=1aMuynlLM3o)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrei Tserakhau – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrei Tserakhau
Andrei Tserakhau is an Engineer and Technical Leader at DoubleCloud with over 10 years of experience in IT. For the last 4 years, he has been working on distributed systems. In particular, his focus was on data delivery systems. He gradually merged disparate data delivery systems at Yandex into a single cross-system data delivery service - Yandex Data Transfer. He is also a big fan of moving data from A to B
[](https://linkedin.com/in/andrei-tserakhau)
### Events
* Terraform: Reshaping the Data Engineering Experience ([watch on youtube](https://www.youtube.com/watch?v=YFr_5NTjv0Q)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andre Schumacher – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andre Schumacher
André has been working as a data scientist at Simplaex on machine-learning problems related to adtech for about two years. Prior to that his focus was on recommender systems and search ranking in the online tourism and retail industry. He has industry experience in a wide range of fields and also worked as a data-science consultant for several years. He holds a doctoral degree in computer science from Aalto University and a master’s degree from TU Darmstadt. He is interested in classical machine-learning problems, linear and convex optimization in operations research, and Bayesian modeling and statistics.
[](https://linkedin.com/in/andreschumacher)
[](https://github.com/AndreSchumacher)
### Events
* Hypothesis Testing: Bayesian or Frequentist? Two ways of looking at the same coin ([watch on youtube](https://www.youtube.com/watch?v=AkWmyE3kQ9k)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andreea Munteanu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andreea Munteanu
Andreea Munteanu is a Product Manager at Canonical, for Machine Learning Operations. With a background in Data Science in various industries, she used AI techniques to enable enterprises to benefit from their initiatives and make data-driven decisions. She collaborated on retail, telco, and shared economy projects, focusing on customer behavior patterns, cost efficiency and scalability.
[](https://linkedin.com/in/andreea-mihaela-munteanu)
### Events
* Open Source MLOps on Public Cloud ([watch on youtube](https://www.youtube.com/watch?v=kfCTwyfwV_s)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Amber Roberts – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Amber Roberts
Amber Roberts is Machine Learning Engineer at ML observability company Arize AI. Previously, Amber was a product manager of AI at Splunk and the Head of Artificial Intelligence at Insight Data Science. A Carnegie Fellow (2016–2017), Amber has a MS in Astrophysics from the Universidad de Chile.
[](https://twitter.com/astronomeramber)
[](https://linkedin.com/in/amber-roberts42)
### Articles
* [Prepare for (Better) Structured Data Extraction](https://datatalks.club/blog/prepare-for-better-structured-data-extraction.html)
### Events
* Keeping Your Model In Production ([watch on youtube](https://www.youtube.com/watch?v=UG_z9ekrJt0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Angela Ramirez – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Angela Ramirez
Angela is a data engineer working at Sam’s Club with their fraud team. She has 4 years of experience as a data engineer. Currently specialising in machine learning for fraud prevention, Angela works on designing and maintaining the data for a machine learning system identifying fraudulent transactions.
In addition, Angela has a masters in Natural Language Processing where she focused on coreference resolution and dialogue systems. Her most recent research includes generating quality synthetic data from LLMs for dialogue systems.
[](https://twitter.com/angelamaria__r)
[](https://linkedin.com/in/aramirez1305)
[](https://github.com/aramir62)
### Events
* Data Engineering for Fraud Prevention ([watch on youtube](https://www.youtube.com/watch?v=ZXNKjrrKU_I)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anish Shah – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anish Shah
Anish loves turning ML ideas into ML products. Anish started his career working with multiple Data Science teams within SAP, working with traditional ML, deep learning, and recommendation systems before landing at Weights & Biases. With the art of programming and a little bit of magic, Anish crafts ML projects to help better serve our customers, turning “oh nos” to “a-ha”s!
[](https://twitter.com/ash0ts)
[](https://linkedin.com/in/anish-shah)
[](https://wandb.ai/a-sh0ts)
### Events
* Large Language Models Agents ([watch on youtube](https://www.youtube.com/watch?v=m5CZzhXPgd0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide
Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide
=======================================================================
### Step-by-step workflow to secure branches, automate linting, and run tests using GitHub Actions, pre-commit, black/isort/flake8/mypy, and pytest.
07 Dec 2020 by [Oleg Polivin](https://datatalks.club/people/olegpolivin.html)
Introduction
------------
In this article I describe how one can set up a continuous integration / continuous delivery (CI/CD) pipeline using GitHub. This text is inspired by two sources. First one is “[Nine simple steps for better-looking python code](https://ternaus.blog/tutorial/2020/04/09/Nine-simple-steps-for-better-looking-python-code.html)
” by [Vladimir Iglovikov](https://github.com/ternaus)
. He gives the keys to better coding practices with CI/CD systems as its core. It is a great article and I wrote mine following the steps outlined there. So, “A practical guide for better-looking python code” is an accompanying/practical guide to Vladimir’s text. I advice you to go through the “Nine steps” for an overview of various approaches, although “A practical guide” could be read independently.
Second source is more subtle, and it is Joel Grus’ [Ten Essays on Fizz Buzz](https://joelgrus.com/2020/06/06/ten-essays-on-fizz-buzz/)
where he describes ways to solve the Fizz Buzz coding exercise. While it provides the 10 solutions, it is more a discussion on Python, Maths, Testing and Coding. And if you don’t know Joel Grus, check out his presentation about his everlasting [love towards Jupyter Notebooks](https://youtu.be/7jiPeIFXb6U)
.
Here are some resources that are associated with this article:
1. [GitHub repository](https://github.com/olegpolivin/FizzBuzz-CI-CD)
where I performed my experiments. It is in a state after all the steps I describe below are implemented. However, you can see the development through git commits history.
2. Along with setting up a CI/CD pipeline, it is possible to publish code documentation using GitHub pages. That’s what I did, and you can find the “documented” version of this article [here](https://olegpolivin.github.io/FizzBuzz-CI-CD/index.html)
. There are no changes to text, it is the same in the article and in the docs. I used [Sphinx](https://www.sphinx-doc.org/en/master/)
to create documentation.
I. Do not push to the master branch
-----------------------------------
The idea is that we want the master branch to contain the main code for our project. Even if we work on our own, it might be a good idea to always push to a different branch, and then integrate the code to the main branch through a pull request (PR). That way we can introduce various checks on pull requests, and impose structure on them. Let’s see how we can do it in GitHub.
I create an empty repository to illustrate how one sets up a CI/CD pipeline step by step. So far it only contains a `README.md` file. It also has only one main branch, and nothing else. I clone it locally:
git clone https://github.com/olegpolivin/Fizz-Buzz-CI-CD.git
Starting point: an empty repo with a single README on main
### Rules for branches
As usual I can work on the code, and then push to the `main` branch. That’s what I want to prohibit. Go to the `Settings` menu for a given repo and choose `Branches`.
Navigate to Settings → Branches to configure protection rules
There are two ways to prevent pushing to the main branch, and you can choose it in the Add rule section. They are:
1) Require pull request reviews before merging. As it is written below:
> when enabled, all commits must be made to a non-protected branch and submitted via a pull request with the required number of approving reviews and no changes requested before it can be merged into a branch that matches this rule.
However, indeed, this will prevent you from pushing to `main` branch, but you cannot be a reviewer of your own pull request as of November 2020. Therefore, if you are working on a project alone, this won’t let you merge PR into your `main` branch.
2) Setting up a CI/CD pipeline.
Click on `Add rule`, and here is the rule that I’ve added:
Add a protection rule for main with required status checks and admin inclusion
In particular, I have added:
* Require status checks to pass before merging (+ Require branches to be up to date before merging). So far there are no checks, but we will add them later.
* Include administrators: Even if we are alone on the project we don’t want to allow ourselves to push to main.
Let’s see how one can set up a CI/CD pipeline, so as to prevent pushing to the `main`. See the next section.
II. Continuous Integration / Continuous Delivery
------------------------------------------------
In order to start with CI/CD using GitHub Actions one just needs to add a config file to the repository under .githib/workflows folder. You can find my configuration [here](https://github.com/olegpolivin/FizzBuzz-CI-CD/blob/main/.github/workflows/ci.yml)
.
### Basic example
The most basic configuration file might be the following one:
# This workflow will install Python dependencies, run tests and
# lint with a variety of Python versions
# For more information see: https://bit.ly/3mX0m9V
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip # This path is specific to Ubuntu
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
It is not doing anything important. It makes use of GitHub Actions and the only thing it does is printing a python version, in my case “3.7”. Creating a pull request will run the script above. Pull request will always pass all checks, because the script checks nothing. However, the whole procedure prevents you now from pushing directly to main. Later we will add code formatters and a linter to this script.
### Set up a rule for your branch
It is necessary just to add some modifications to the `Settings -> Branches -> Rules part`. See what’s new:
Required check “build (3.7)” appears after configuring the workflow
Notice that `build (3.7)` has appeared among status checks. This corresponds to the name of the job (`build`) and python version `3.7`. I made a small modification to the `README.md` file, and let’s see if I can push it now to the main branch. Here is the error I get:
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: error: Required status check “build (3.7)” is expected.
To https://github.com/olegpolivin/FizzBuzz-CI-CD.git
! [remote rejected] main -> main (protected branch hook declined)
error: failed to push some refs to ‘https://github.com/olegpolivin/FizzBuzz-CI-CD.git'
Nice! The commit is rejected because a required status check is needed. Therefore, let’s push to a new branch. Locally, let’s create a new branch
git checkout -b dev
git push origin dev
A new branch called `dev` is created on the remote repository. What’s left is to create a pull request, and merge it to the `main` branch.
Create a PR from your feature branch to main to trigger checks
It becomes possible to merge after all checks are run:
All required checks pass—your PR is ready to merge
We would like to introduce actions or tests to be performed, before the pull request is ready to be approved, so let’s provide code that will be actually checked. We will consider solving the `FizzBuzz` problem, see the next section.
III. FizzBuzz
-------------
OK, it is time to write some code!
### FizzBuzz Definition
Fizz Buzz problem is a task that sometimes people get during coding interviews. It goes like this (I take the definition from Joel’s book):
> Print the numbers from 1 to 100, except that if the number is divisible by 3, instead print “fizz”; if the number is divisible by 5, instead print “buzz”; and if the number is divisible by 15, instead print “fizzbuzz”
Imagine we come up with a great code:
import scipy
import pandas
import numpy
import matplotlib
from matplotlib import pyplot as plt
####### Here I start the solution to the fizz buzz problem #######
def fizz_buzz(n: int) -> str:
if n % 15 == 0: return 'fizzbuzz'
elif n % 5 == 0: return 'buzz'
elif n % 3 == 0: return 'fizz'
else: return str(n)
We were in a hurry, so we first imported everything that we usually import, made comments to visually show where the code starts, and printed `if - return` statements on the same line. Clearly, there is no newline at the end of file, who cares since the code is so great!
Actually, we do care because the code needs to be readable and beautiful, and we decide that it is a good idea to impose structure on every pull request. Also, code must pass linter checks and be formatted in a unified manner. It is possible to add all the necessary checks that we want to impose in the `ci.yml` file that we created in the previous section. Let’s say we add:
* _black_ formatter of the code
* _isort_ to sort the imports in alphabetical order
* _flake8_ and _pylint_ to inspect the code for conformity with good code practices
* _MyPy_ as a static type checker
### Extended GitHub Actions file
That’s how the `ci.yml` file looks like now:
# This workflow will install Python dependencies, run tests and
# lint with a variety of Python versions
# For more information see: https://bit.ly/3mX0m9V
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.7]
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip # This path is specific to Ubuntu
# Look to see if there is a cache hit for the corresponding requirements file
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-
# You can test your matrix by printing the current Python version
- name: Display Python version
run: python -c "import sys; print(sys.version)"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install black flake8 mypy pytest hypothesis isort pylint
- name: Run black
run:
black --check . --exclude docs/
- name: Run flake8
run: flake8 fizzbuzz.py
- name: Run pylint
run: pylint fizzbuzz.py
- name: Run Mypy
run: mypy fizzbuzz.py
- name: Run isort
run: isort --profile black fizzbuzz.py
Let’s now try to push the solution above to the repository.
Example of a failing CI run—fix issues locally and push updates
And we see that it fails on the first check. When it fails it does not proceed to the next steps, but it turns out that the code above for solving the `FizzBuzz` problem will fail on every check.
### A better version of `fizzbuzz.py`
The code below passes all of the checks that we have imposed on it.
"""Function to solve the fizzbuzz problem."""
def fizz_buzz(num: int) -> str:
"""This is my great and neat function to solve the famous
Fizz Buzz problem.
:param num: That's the number which we want the answer for
:return: fizz, buzz, fizzbuzz or the number itself
"""
if num % 15 == 0:
return "fizzbuzz"
if num % 5 == 0:
return "buzz"
if num % 3 == 0:
return "fizz"
return str(num)
Now when we push it to the `dev` branch, pull requests could be merged into the `main` branch since all checks are passed.
### Catch the problem before committing
It might be that you want to learn that there are problems with your code (that is, it does not pass a check that one imposed) before committing. Yes, you will run the tests locally, but what if there is an additional point of control that does not allow you to commit your changes unless all the checks are passed? It is called a “Pre-commit hook”, and more on how to set it up in the next section.
IV. Pre-commit hook
-------------------
A pre-commit hook is kind of a script that will be run when you do
git commit -m ""
Link to the original and complete description of the pre-commit hook [here](https://pre-commit.com/)
.
### Setting up the pre-commit hook
First install the pre-commit hook by running:
pip install pre-commit
It is necessary to create a `.pre_commit-config.yaml` file in the repository, where you would specify all the steps that should be done before the commit is performed. If an error is encountered, commit does not happen. Below is a simple `.pre_commit-config.yaml` configuration that
* Checks that code is formatted according to _black_.
* Sorts imports using _isort_.
* Uses _flake8_ and _pylint_ as linters.
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.2.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-added-large-files
- repo: https://github.com/pre-commit/mirrors-isort
rev: f0001b2 # Use the revision sha / tag you want to point at
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/psf/black
rev: 20.8b1
hooks:
- id: black
- repo: https://gitlab.com/pycqa/flake8
rev: 3.7.9
hooks:
- id: flake8
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: system
types: [python]
After the file is created in the repository, run `pre-commit install` to install pre-commit into your git hooks. Et voilà, now the checks will run each time before the commit.
### Testing the pre-commit hook
Here is a small test: let’s change the neat `fizzbuzz.py` code to get back to the one that does not pass the checks and see what happens. Here is a part of the result: we see where it fails. Note that the pre-commit hook modifies files for some commands (like black or isort).
Pre-commit prevents bad commits by running formatters and linters before commit
Coming back to the neat version of the `fizzbuzz.py`, the pre-commit hook test is passed. That’s how it looks like in my case:
All hooks pass—your changes are clean and consistent
Nice!
Finally, we want to not only check the formatting of our code, but also make sure that the code works correctly. We can add unit tests to the CI/CD pipeline!
V. Testing the code
-------------------
In this section I show how to integrate unit tests into the CI/CD pipeline, and again we will make use of the `ci.yml` file. Personally, I like the _pytest_ framework, and that’s what I will use in this section.
### Setting up the test
There is only one function to test (`fizz_buzz.py`), and it is quite simple. I will put the `test_fizzbuzz.py` function directly into the root folder. The structure of the current github project is as follows:
├── fizzbuzz.py
├── .github
│ └── workflows
│ └── ci.yml
├── .gitignore
├── .pre-commit-config.yaml
├── README.md
└── test_fizzbuzz.py
`test_fizzbuzz.py` contains:
"""Perform tests of the fizz_buzz function."""
import pytest
from fizzbuzz import fizz_buzz
inputs = [3, 5, 15, 4, 10, 115, 7]
outputs = ["fizz", "buzz", "fizzbuzz", "4", "buzz", "buzz", "7"]
@pytest.mark.parametrize("inp,out", zip(inputs, outputs))
def test_fizzbuzz(inp, out):
"""Takes inputs, gets the output of the fizz_buzz function.
Asserts whether equality holds.
"""
assert fizz_buzz(inp) == out
Append the code below to the `ci.yml` file:
- name: tests
run: pytest
### Passing the test
And here is the result:
Unit tests executed by pytest pass successfully in CI
But that was the case when everything is ok. We are happy.
### Failing the test
Now a new and innovative idea comes to our mind. Why overcomplicate the code, why do we start from “15”? Let’s “sort” the if-conditions in the code, the code will like so nice! So, we change the fizzbuzz code to the following one:
def fizz_buzz(num: int) -> str:
"""This is my great and neat function to solve the famous
Fizz Buzz problem.
:param num: That's the number which we want the answer for
:return: fizz, buzz, fizzbuzz or the number itself
"""
if num % 3 == 0:
return "fizz"
if num % 5 == 0:
return "buzz"
if num % 15 == 0:
return "fizzbuzz"
return str(num)
Great, let’s push and see that one test has failed:
Failing test demonstrates how CI guards against regressions before merge
That is, by introducing unit tests into the CI/CD pipeline we were able to catch the problem before merging pull request into the `main` branch.
Conclusions
-----------
That’s it for now. I hope you find this practice guide useful, and will apply it in your work. Try implementing just some of the steps or all of them: it feels great when you see it working in practice. I encourage you to read the “[Nine simple steps for better-looking python code](https://ternaus.blog/tutorial/2020/04/09/Nine-simple-steps-for-better-looking-python-code.html)
” for even more ideas on the subject.
I would be happy to get your comments and learn your ways to perform CI/CD and keep your code clean.
Acknowledgements
----------------
I would like to thank
* [Andrew Lukyanenko](https://github.com/Erlemar)
for helping me with details on CI/CD implementation.
* [Vladimir Iglovikov](https://github.com/ternaus)
for his article and his investment into openness and development of the whole Data Science community.
* I am grateful to the Open Data Science community ([ods.ai](https://ods.ai/)
) as a source of inspiration.
This article was originally publised on [medium](https://medium.com/@olegpolivin/a-practical-guide-for-better-looking-python-code-5a032508bb20?source=friends_link&sk=4214333a288a7aa39f0d6dabfbd9d65c)
.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrey Shtylenko – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrey Shtylenko
Andrey Shtylenko is a Director of Engineering at Honeywell where he leads the Advanced Technology Group within the Safety & Sensing business developing technologies and solutions based on signal processing, perception, computer vision, and AI/ML for healthcare, industrial, and logistics market verticals.
In the last 5 years at Honeywell Andrey was leading Honeywell’s AI practice, as well as building and growing Honeywell’s Advanced Robotics Group where he led the perception team.
Andrey had a scholarship and multiple degrees in Engineering Management and has been leading conventional technology and advanced research teams for the last decade. Before Honeywell, Andrey co-founded multiple technology businesses several of which were later acquired.Andrey lives with his wife and son in Dallas, TX.
[](https://linkedin.com/in/shtylenko)
### Events
* Building Data Science Practice ([watch on youtube](https://www.youtube.com/watch?v=XbDQv8FTA4U)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrew McMahon – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrew McMahon
Andy McMahon is a machine learning engineer and data scientist with experience of working in, and leading, successful analytics and software teams. His expertise centers on building production-grade ML systems that can deliver value at scale. He is currently ML Engineering Lead at NatWest Group and was previously Analytics Team Lead at Aggreko. He has an undergraduate degree in theoretical physics from the University of Glasgow, as well as master’s and Ph.D. degrees in condensed matter physics from Imperial College London. In 2019, Andy was named Data Scientist of the Year at the International Data Science Awards. He currently co-hosts the AI Right podcast, discussing hot topics in AI with other members of the Scottish tech scene.
[](https://twitter.com/electricweegie)
[](https://linkedin.com/in/andymcmahon629)
[](https://github.com/AndyMc629)
[](https://electricweegie.com/)
### Books
* [Machine Learning Engineering with Python](https://datatalks.club/books/20220117-machine-learning-engineering-with-python.html)
(the book of the week from 10 Jan 2022 to 21 Jan 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andy Petrella – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andy Petrella
Andy is an entrepreneur with a Mathematics and Distributed Data background focused on unleashing unexploited business potentials leveraging new technologies in machine learning, artificial intelligence, and cognitive systems.
In the data community, Andy is known as an early evangelist of Apache Spark (2011-), the Spark Notebook creator (2013-), a public speaker at various events (Spark Summit, Strata, Big Data Spain), and an O’Reilly author (Distributed Data Science, Data Lineage Essentials, Data Governance, and Machine Learning Model Monitoring).
Andy is the CEO of Kensu, bringing the Data Intelligence Management (DIM) Platform for data-driven companies to leverage AI sustainably, combining AI Observability with Data Usage Catalog.
[](https://twitter.com/noootsab)
[](https://linkedin.com/in/andypetrella)
[](https://github.com/andypetrella)
[](https://www.kensu.io/)
### Events
* Hands-on AI Observability ([watch on youtube](https://www.youtube.com/watch?v=ARfrqa-t-Xk)
)
### Books
* [Fundamentals of Data Observability](https://datatalks.club/books/20240429-fundamentals-of-data-observability.html)
(the book of the week from 29 Apr 2024 to 03 May 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Andrew Jones – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Andrew Jones
Andrew Jones is a Principal Engineer at GoCardless, one of Europe’s leading Fintechs. He has over 15 years’ experience in the industry, with the first half primarily as a software engineer before moving into the data infrastructure and data engineering space.
Joining GoCardless as its first data engineer, he led his team to build their data platform from scratch. After initially following a typical data architecture and getting frustrated with facing the same old challenges we’ve faced for years, he started thinking there must be a better way, which led to him coining and defining the ideas around “data contracts”.
Andrew is a regular speaker and writer, and passionate about helping organizations get the most value from data.
[](https://twitter.com/andrewrjones)
[](https://linkedin.com/in/andrewrhysjones)
[](https://andrew-jones.com/)
### Books
* [Driving Data Quality with Data Contracts](https://datatalks.club/books/20230807-driving-data-quality-with-data-contracts.html)
(the book of the week from 07 Aug 2023 to 11 Aug 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset
NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset
==================================================================
### Discover NER fundamentals, Transformer vs Reformer basics, and a hands‑on Trax implementation on a Kaggle dataset with training and evaluation.
17 Dec 2020 by [Saurav Maheshkar](https://datatalks.club/people/sauravmaheshkar.html)
Link to the [Kaggle Kernel](https://www.kaggle.com/sauravmaheshkar/trax-ner-using-reformer)
which is referenced in this post
Introduction to Named Entity Recognition
----------------------------------------
Named entity recognition(NER) is the task to identify mentions of rigid designators from text belonging to predefined semantic types such as person, location, organization etc. NER always servers as the foundation of many natural language applications such as question answering, text summarization, and machine translation.
Despite the various definitions of NE(Named Entity), researchers have reached common consensus on the types of NEs to recognize. We generally divide NEs into two categories:
* **Generic NEs:** Person and Location
* **Domain-specific NEs:** proteins, enzymes, and genes.
4 mainstream approaches used in NER are:
* **Rule-Based Approaches:** Don’t need annotated data as they rule on hand-crafted rules
* **Unsupervised Learning Approaches:** Rely on unsupervised algorithms without hand-labelled training examples
* **Feature-based Supervised Learning:** Rely on supervised algorithms with a lot of feature engineering involved
* **Deep Learning Approaches:** Automatically discover representations from raw input
Formal Definition
-----------------
A named entity (NE) is a word or a phrase that clearly identifies one item from a set of other items that have similar attributes. Examples being organizations, person, location names. NER is the process of locating and classifying named entities in text into predefined entity categories.
Formally, given a sequence of tokens $s = {w\_1,w\_2,…,w\_N}$, NER outputs a list of tuples $ {I\_s, I\_e, t}$, each of which is a named entity mentioned in $s$. Here, $I\_s \\in \[1, N\]$ and $I\_e \\in \[1, N\]$ are the start and end indexes of a NER; t is an entity type from a predefined category set.
### Evaluation
NER systems are usually evaluated by comparing their outputs against human annotations. The comparison can be quantified by either exact-match or relaxed match.
#### Exact-Match Evaluation
NER essentially involves two subtasks: boundary detection and type identification. In “exact-match evaluation”, a correctly recognized instance requires a system to correctly identify its boundary and type, simultaneously.
#### Relaxed-Match Evaluation
A correct type is credited if an entity is assigned its correct type regardless its boundaries as long as there is an overlap with ground truth boundaries; a correct boundary is credited regardless an entity type’s assignment.
### Approaches
#### Deep Learning Techniques for NER
There are three core strengths of applying deep learning techniques to NER.
1. NER benefits from the non-linear transformations, which generates non-linear mappings from input to output. DL models are able to learn complex and intricate features from data compared to linear models (log-linear HMM, linear chain CRF).
2. DL saves a significant amount of effort on designing NER features. The traditional models required a considerable amount of engineering skill and domain expertise.
3. Deep NER models can be trained on an end-to-end paradigm which enables us to build complex NER systems.
#### General Deep Learning Architecture for NER
* **Distributed representations for input** consider word- and character-level embeddings as well as the incorporation of additional features.
* **Context encoder** is to capture the context dependencies using CNN, RNN, or other networks.
* **Tag decoder** predicts tags for tokens in the input sentence.
### Distributed Representations for Input
Distributed representation represents words in low dimensional real-valued dense vectors where each dimension represents a latent feature. Automatically learned from the text, distributed representation captures semantic and syntactic properties of word
#### Word-Level Representation
Some studies employed word-level representation, which is typically pre-trained over large collections of text through unsupervised algorithms such as continuous bag-of-words (CBOW) and continuous skip-gram models. Recent studies have shown the importance of such pre-trained word embeddings. Using as the input, the pre-trained word embeddings can be either fixed or further fine-tuned during NER model training. Commonly used word embeddings include Google Word2Vec, Stanford GloVe, Facebook fastText and SENNA.
#### Character-Level Representation
Instead of only considering word-level representations as to the basic input, several studies incorporated character-based word representations learned from an end-to-end neural model. Character-level representation has been found useful for exploiting explicit sub-word-level information such as prefix and suffix. Another advantage of character-level representation is that it naturally handles out-of-vocabulary tokens. Recent studies (like CharNER) have shown that taking characters as the primary representation is superior to words as the basic input unit.
#### Hybrid Representation
Besides word-level and character-level representations, some studies also incorporate additional information (e.g., gazetteers, lexical similarity, linguistic dependency and visual features ) into the final representations of words, before feeding into context encoding layers. In other words, the DL-based representation is combined with a feature-based approach in a hybrid manner. Adding additional information may lead to improvements in NER performance, with the price of hurting generality of these systems
### Context Encoders
#### Convolutional Neural Networks
Some studies proposed a sentence approach network where a word is tagged with the consideration of the whole sentence. Each word in the input sequence is embedded in an N-dimensional vector after the stage of input representation. Then a convolutional layer is used to produce local features around each word, and the size of the output of the convolutional layers depends on the number of words in the sentence. The global feature vector is constructed by combining local feature vectors extracted by the convolutional layers. Finally, these fixed-size global features are fed into a tag decoder to compute a distribution score for all possible tags for the words in the network input.
#### Recurrent Neural Networks
Recurrent neural networks, together with its variants such as a gated recurrent unit (GRU) and long-short term memory (LSTM), have demonstrated remarkable achievements in modelling sequential data. In particular, bidirectional RNNs efficiently make use of past information (via forward states) and future information (via backward states) for a specific time frame. A token encoded by a bidirectional RNN will contain evidence from the whole input sentence.
#### Transformer
Neural sequence labelling models are typically based on complex convolutional or recurrent networks which consist of encoders and decoders. Transformer, proposed by Vaswani, dispenses with recurrence and convolutions entirely. A transformer utilizes stacked self-attention and pointwise, fully connected layers to build basic blocks for encoder and decoder.
### Tag Decoder
Tag decoder is the final stage in a NER model. It takes context-dependent representations as input and produces a sequence of tags corresponding to the input sequence.
#### MLP + Softmax
Tag decoder is the final stage in a NER model. It takes context-dependent representations as input and produces a sequence of tags corresponding to the input sequence.
#### Conditional Random Fields
A conditional random field (CRF) is a random field globally conditioned on the observation sequence. CRFs have been widely used in feature-based supervised learning approaches. Many deep learning-based NER models use a CRF layer as the tag decoder. CRF is the most common choice for tag decoder and the state-of-the-art performance on CoNLL03 and OntoNotes5.0 is achieved with a CRF tag decoder.
Introduction to Transformers
----------------------------
### Motivation
Traditional architectures like Recurrent Neural Network or Convolutional Neural Networks ( where we use encoders to encode sentences into representations and then decode these representations into our desired format ) for sequence transduction tasks ( language modelling and machine translation ) have shown promising results but they are affected by long sequence lengths. Although using conditional computations and certain factorisation tricks have resulted in increased computational efficiency but the constraints of sequence computation still remain. Thus, this new architecture relies entirely on attention mechanisms to draw global dependencies between input and output sequences.
Using self-attention we can:
* Reduce the Computational Complexity per layer
* Parallelize more computations
* Capture long-range dependencies effectively
### Past Works
#### Convolutional Models
* Neural GPU based on a type of convolutional gated recurrent unit, is highly parallel and easy to run. Neural GPU can be trained on short instances of an algorithmic task and successfully generalize to long instances.
* The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. It uses dilation in convolutional layers to increase its receptive field.
* As per the ConvS2S model, compared to recurrent models, computations over all elements can be fully parallelized during training to better exploit the GPU hardware and optimization is easier since the number of non-linearities is fixed and independent of the input length. The use of gated linear units eases gradient propagation and equips each decoder layer with a separate attention module.
#### Recurrent Attention Models
* In the paper, End-To-End Memory Networks a neural network with a recurrent attention model over a possibly large external memory was introduced.
### The Architecture
As mentioned earlier, the earlier neural sequence transduction models had an encoder-decoder structure. Where the encoder mapped an input sequence $ (x\_1, …, x\_n) $ into a sequence of representations $ z = (z\_1, …, z\_n) $ and then given $z$ , the decoder generates an output $ (y\_1, …, y\_n) $. As the model is auto-regressive in nature, it takes as input the previously generated symbols as an additional input.
#### Encoder
The encoder is shown on the left consists of a stack of 6 identical layers, which contain two sub-layers: a multi-head self-attention mechanism and a simple Feed-Forward Network. A residual connection is added around each sub-layer. Due to this attention mechanism, the encoder can “attend” to all positions in the previous layer
#### Decoder
The decoder block is shown on the right also consists of a stack of 6 identical layers, which contain three sub-layers: two layers from the encoder blocks and another layer which performs attention over the outputs of the encoder stack. We also have residual connections around sub-layers. Due to this attention mechanism, the decoder can “attend” to all positions in the previous layer
### Attention: Scaled Dot-Product
\\\[Attention(Q,K,V) = softmax(\\frac{QK^T}{\\sqrt{d\_k}})V\\\]
The inputs are:
* queries and keys of dimensions $ d\_k $
* values of dimensions $ d\_v $
#### Multi-Head Attention
Instead of performing single attention functions, the keys, values and queries were linearly projected $ h $ times and then scaled dot-product attention was applied in parallel. These were then concatenated and projected again. The queries come from the previous decoder block and the keys and values are the outputs of the encoder.
### Positional Encoding
As the model doesn’t use any sort of recurrence or convolutional layer, to enable transformers to learn about the positions, “positional encodings were added” to the embedding at the start and bottom of our encoder-decoder stacks.
Implementing NER using Trax
---------------------------
Install the latest version of the [Trax](https://github.com/google/trax)
Library.
!pip install -q -U trax
### Importing Packages
import trax # Our Main Library
from trax import layers as tl
import os # For os dependent functionalities
import numpy as np # For scientific computing
import pandas as pd # For basic data analysis
import random as rnd # For using random functions
### Pre-Processing
#### Loading the Dataset
Let’s load the `ner_dataset.csv` file into a dataframe and see what it looks like
data = pd.read_csv("/kaggle/input/entity-annotated-corpus/ner_dataset.csv",
encoding='ISO-8859-1')
data = data.fillna(method = 'ffill')
data.head()
#### Creating a Vocabulary File
We can see there’s a column for the words in each sentence. Thus, we can extract this column using the `.loc()` and store it into a `.txt` file using the `.savetext()` function from numpy.
## Extract the 'Word' column from the dataframe
words = data.loc[:, "Word"]
## Convert into a text file using the .savetxt() function
np.savetxt(r'words.txt', words.values, fmt="%s")
#### Creating a Dictionary for Vocabulary
Here, we create a Dictionary for our vocabulary by reading through all the sentences in the dataset.
vocab = {}
with open('words.txt') as f:
for i, l in enumerate(f.read().splitlines()):
vocab[l] = i
print("Number of words:", len(vocab))
vocab[''] = len(vocab)
### Extracting Sentences from the Dataset
For extracting sentences from the dataset and creating `(X,y)` pairs for training.
class Get_sentence(object):
def __init__(self,data):
self.n_sent=1
self.data = data
agg_func = lambda s:[(w,p,t) for w,p,t in zip(s["Word"].values.tolist(),\
s["POS"].values.tolist(),\
s["Tag"].values.tolist())]
self.grouped = self.data.groupby("Sentence #").apply(agg_func)
self.sentences = [s for s in self.grouped]
getter = Get_sentence(data)
sentence = getter.sentences
words = list(set(data["Word"].values))
words_tag = list(set(data["Tag"].values))
word_idx = {w : i+1 for i ,w in enumerate(words)}
tag_idx = {t : i for i ,t in enumerate(words_tag)}
X = [[word_idx[w[0]] for w in s] for s in sentence]
y = [[tag_idx[w[2]] for w in s] for s in sentence]
### Making a Batch Generator
Here, we create a batch generator for training.
def data_generator(batch_size, x, y,pad, shuffle=False, verbose=False):
num_lines = len(x)
lines_index = [*range(num_lines)]
if shuffle:
rnd.shuffle(lines_index)
index = 0
while True:
buffer_x = [0] * batch_size
buffer_y = [0] * batch_size
max_len = 0
for i in range(batch_size):
if index >= num_lines:
index = 0
if shuffle:
rnd.shuffle(lines_index)
buffer_x[i] = x[lines_index[index]]
buffer_y[i] = y[lines_index[index]]
lenx = len(x[lines_index[index]])
if lenx > max_len:
max_len = lenx
index += 1
X = np.full((batch_size, max_len), pad)
Y = np.full((batch_size, max_len), pad)
for i in range(batch_size):
x_i = buffer_x[i]
y_i = buffer_y[i]
for j in range(len(x_i)):
X[i, j] = x_i[j]
Y[i, j] = y_i[j]
if verbose: print("index=", index)
yield((X, Y))
### Splitting into Test and Train
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=1)
### Building the Model
We will perform the following steps:
* Use input tensors from our data generator
* Produce Semantic entries from an Embedding Layer
* Feed these into our Reformer Language model
* Run the Output through a Linear Layer
* Run these through a log softmax layer to get predicted classes
We use the:
* `tl.Serial():` Combinator that applies layers serially(by function composition). It’s commonly used to construct deep networks. It uses stack semantics to manage data for its sublayers
* `tl.Embedding():` Initializes a trainable embedding layer that maps discrete tokens/ids to vectors
* `trax.models.reformer.Reformer():` Creates a Reversible Transformer encoder-decoder model.
* `tl.Dense():` Creates a Dense(fully-connected, affine) layer
* `tl.LogSoftmax():` Creates a layer that applies log softmax along one tensor axis.
def NERmodel(tags, vocab_size=35181, d_model = 50):
model = tl.Serial(
# tl.Embedding(vocab_size, d_model),
trax.models.reformer.Reformer(vocab_size, d_model, ff_activation=tl.LogSoftmax),
tl.Dense(tags),
tl.LogSoftmax()
)
return model
model = NERmodel(tags = 17)
### Train the Model
from trax.supervised import training
rnd.seed(33)
batch_size = 64
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab[''], True),
id_to_mask=vocab[''])
eval_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_test, y_test,vocab[''], True),
id_to_mask=vocab[''])
def train_model(model, train_generator, eval_generator, train_steps=1,
output_dir='model'):
train_task = training.TrainTask(
train_generator,
loss_layer = tl.CrossEntropyLoss(),
optimizer = trax.optimizers.Adam(0.01),
n_steps_per_checkpoint=10
)
eval_task = training.EvalTask(
labeled_data = eval_generator,
metrics = [tl.CrossEntropyLoss(), tl.Accuracy()],
n_eval_batches = 10
)
training_loop = training.Loop(
model,
train_task,
eval_tasks = eval_task,
output_dir = output_dir)
training_loop.run(n_steps=train_steps)
return training_loop
### Training
train_steps = 100
training_loop = train_model(model, train_generator, eval_generator, train_steps)
Step 1: Ran 1 train steps in 815.40 secs
Step 1: train CrossEntropyLoss | 2.97494578
Step 1: eval CrossEntropyLoss | 5.96823492
Step 1: eval Accuracy | 0.85458949
Step 10: Ran 9 train steps in 6809.59 secs
Step 10: train CrossEntropyLoss | 5.27117538
Step 10: eval CrossEntropyLoss | 5.19212604
Step 10: eval Accuracy | 0.85005882
Step 20: Ran 10 train steps in 5372.06 secs
Step 20: train CrossEntropyLoss | 6.68565750
Step 20: eval CrossEntropyLoss | 4.00950582
Step 20: eval Accuracy | 0.81635543
Step 30: Ran 10 train steps in 1040.84 secs
Step 30: train CrossEntropyLoss | 3.92878985
Step 30: eval CrossEntropyLoss | 3.32506871
Step 30: eval Accuracy | 0.78096363
Step 40: Ran 10 train steps in 3624.02 secs
Step 40: train CrossEntropyLoss | 3.41684675
Step 40: eval CrossEntropyLoss | 3.47973170
Step 40: eval Accuracy | 0.84054841
Step 50: Ran 10 train steps in 195.43 secs
Step 50: train CrossEntropyLoss | 2.64065409
Step 50: eval CrossEntropyLoss | 2.21273057
Step 50: eval Accuracy | 0.84472065
Step 60: Ran 10 train steps in 1060.08 secs
Step 60: train CrossEntropyLoss | 2.35068488
Step 60: eval CrossEntropyLoss | 2.66343498
Step 60: eval Accuracy | 0.84561690
Step 70: Ran 10 train steps in 1041.36 secs
Step 70: train CrossEntropyLoss | 2.30295134
Step 70: eval CrossEntropyLoss | 1.31594980
Step 70: eval Accuracy | 0.84971260
Step 80: Ran 10 train steps in 1178.78 secs
Step 80: train CrossEntropyLoss | 1.15712142
Step 80: eval CrossEntropyLoss | 1.15898243
Step 80: eval Accuracy | 0.84357584
Step 90: Ran 10 train steps in 2033.67 secs
Step 90: train CrossEntropyLoss | 1.06345284
Step 90: eval CrossEntropyLoss | 0.93652567
Step 90: eval Accuracy | 0.84781972
Step 100: Ran 10 train steps in 2001.96 secs
Step 100: train CrossEntropyLoss | 1.04488492
Step 100: eval CrossEntropyLoss | 1.02899926
Step 100: eval Accuracy | 0.85163420
References
----------
* [Google AI Blog- Reformer: The Efficient Transformer](https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html)
* [Google AI Blog- Transformer: A Novel Neural Network Architecture for Language Understanding](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html)
* [Trax: Deep Learning with Clear Code and Speed](https://github.com/google/trax)
* [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/)
* [Attention Is All You Need](https://arxiv.org/abs/1706.03762)
* [Illustrating the Reformer](https://towardsdatascience.com/illustrating-the-reformer-393575ac6ba0)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anusha Akkina – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anusha Akkina
Anusha is the co-founder of Aurelytix, an AI-driven finance platform built to give CFOs and finance teams clarity and speed without adding complexity.
She is Alumni of CIMA, UK ,after almost a decade in corporate finance strategy and transformation, leading ERP implementations and major finance change programs at companies like Deloitte, Parexel, Survitec, Chubb, she grew frustrated with data silos & slow, fragmented tools that keep enterprises & SMEs reactive instead of strategic.
In 2025 she launched Aurelytix to change that: building Insight Pulse, an AI-powered decision layer that spots anomalies, explains variances, and surfaces opportunities while keeping a human firmly in the loop. She’s passionate about augmented finance using AI to supercharge decision-making while respecting compliance and trust, especially under GDPR in Europe.
[](https://linkedin.com/in/anusha-akkina-acma-cgma-56154547)
[](https://aurelytix.com/)
### Events
* From Black-Box Systems to Augmented Decision-Making ([watch on youtube](https://www.youtube.com/watch?v=YZNaLm-_zwA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# 20+ Best Free Machine Learning Courses: Learn from Stanford, MIT and Google Without Paying Tuition – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
20+ Best Free Machine Learning Courses: Learn from Stanford, MIT and Google Without Paying Tuition
20+ Best Free Machine Learning Courses
======================================
### Learn ML from top universities and platforms with these free courses, covering foundations, core algorithms, and practical projects.
21 Oct 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Traditional machine learning education costs thousands of dollars and takes years to complete. This curated collection of 20+ free machine learning courses from Stanford, MIT, Coursera, edX, and specialized ML platforms delivers the same foundational knowledge and practical skills without the financial barrier.
These carefully selected free ML courses cover everything from fundamental concepts and core algorithms to hands-on projects and real-world applications. You’ll master industry-standard tools like Python and TensorFlow, work with actual datasets, and build a portfolio of ML projects that showcase your abilities to potential employers.
What are Free Machine Learning Courses?
---------------------------------------
Free courses provide access to educational content without upfront payment, though they vary in what’s included. Understanding the different types can help you choose the right option for your learning goals.
**Fully Free Courses:** These rare gems offer complete access to all materials and often include certificates at no cost. They’re typically provided by organizations, open-source communities, or educational initiatives.
**Audit-Only Courses:** Most major platforms like Coursera and edX allow you to audit course materials, including videos, readings, and lecture notes, for free. However, graded assignments, peer feedback, and certificates require payment. Pro tip: On Coursera, look for the “Audit” option on individual course pages rather than Specialization landing pages, where it’s often hidden.
**University Open Courseware:** These offerings from institutions like MIT and Stanford provide comprehensive access to lectures, notes, problem sets, and sometimes even recorded class sessions, all completely free. The main limitation is that they typically don’t offer certificates or instructor interaction.
**Freemium Models:** Some platforms offer basic content for free with premium features (advanced modules, mentorship, career services), available for a fee.
20+ Best Free Machine Learning Courses Comparison Table
-------------------------------------------------------
| Course | Provider | Difficulty | Duration | Certificate | Platform | Access Type |
| --- | --- | --- | --- | --- | --- | --- |
| [ML Zoomcamp](https://datatalks.club/blog/free-machine-learning-courses.html#ml-zoomcamp) | DataTalks.Club | Beginner-Intermediate | 4 months | Free (cohort only) | DataTalks.Club | Fully Free |
| [Google ML Crash Course](https://datatalks.club/blog/free-machine-learning-courses.html#google-ml-crash-course) | Google | Beginner-Intermediate | Self-paced | No | Google Developers | Fully Free |
| [ML for Beginners](https://datatalks.club/blog/free-machine-learning-courses.html#microsoft-ml-for-beginners) | Microsoft | Beginner | 12 weeks | No | GitHub | Fully Free |
| [ML with Python](https://datatalks.club/blog/free-machine-learning-courses.html#ibm-ml-with-python) | IBM | Intermediate | 2 weeks | Audit free, paid certificate | Coursera | Audit-Only |
| [ML Specialization](https://datatalks.club/blog/free-machine-learning-courses.html#deeplearning-ai-ml-specialization) | DeepLearning.AI | Beginner | 2 months | Audit free, paid certificate | Coursera | Audit-Only |
| [Statistical Learning with Python](https://datatalks.club/blog/free-machine-learning-courses.html#stanford-statistical-learning) | Stanford | Introductory | 11 weeks | Audit free, paid certificate | edX | Audit-Only |
| [ML with Python: Linear to Deep Learning](https://datatalks.club/blog/free-machine-learning-courses.html#mit-ml-linear-to-deep) | MIT | Intermediate-Advanced | 15 weeks | Audit free, paid certificate | edX | Audit-Only |
| [Data Science: Machine Learning](https://datatalks.club/blog/free-machine-learning-courses.html#harvard-data-science-ml) | Harvard | Introductory | 8 weeks | Audit free, paid certificate | edX | Audit-Only |
| [CS229 Machine Learning](https://datatalks.club/blog/free-machine-learning-courses.html#stanford-cs229) | Stanford | Intermediate-Advanced | Summer 2025 | No | Stanford | University OCW |
| [CS189/289A](https://datatalks.club/blog/free-machine-learning-courses.html#berkeley-cs189) | UC Berkeley | Intermediate-Advanced | 16 weeks | No | UC Berkeley | University OCW |
| [Introduction to ML (6.036)](https://datatalks.club/blog/free-machine-learning-courses.html#mit-6036) | MIT | Intermediate | 13 weeks | No | MIT OCW | University OCW |
| [Learning from Data (CS156)](https://datatalks.club/blog/free-machine-learning-courses.html#caltech-cs156) | Caltech | Introductory-Intermediate | 18 lectures | No | Caltech | University OCW |
| [Machine Learning (10-601)](https://datatalks.club/blog/free-machine-learning-courses.html#cmu-ml) | Carnegie Mellon | Intermediate-Advanced | 16 weeks | No | CMU | University OCW |
| [Mathematics for ML Specialization](https://datatalks.club/blog/free-machine-learning-courses.html#mathematics-for-ml) | Imperial College | Foundational-Intermediate | 12-18 weeks | Audit free, paid certificate | Coursera | Audit-Only |
| [ML with Python](https://datatalks.club/blog/free-machine-learning-courses.html#freecodecamp-ml) | FreeCodeCamp | Intermediate | Self-paced | Free certification | freeCodeCamp | Fully Free |
| [Practical Deep Learning for Coders](https://datatalks.club/blog/free-machine-learning-courses.html#fastai-deep-learning) | Fast.ai | Beginner-friendly | ~13.5 hours | No | fast.ai | Fully Free |
| [Kaggle ML Courses](https://datatalks.club/blog/free-machine-learning-courses.html#kaggle-ml-courses) | Kaggle | Beginner-Intermediate | Few hours each | Free completion cert | Kaggle | Fully Free |
| [ML in Python with scikit-learn](https://datatalks.club/blog/free-machine-learning-courses.html#inria-scikit-learn) | Inria | Intermediate | 36 hours | Open Badge | FUN MOOC | Fully Free |
| [Fundamentals of ML and AI](https://datatalks.club/blog/free-machine-learning-courses.html#aws-ml-fundamentals) | AWS | Beginner | 1 hour | Audit free, paid certificate | Coursera | Audit-Only |
| [Introduction to Machine Learning](https://datatalks.club/blog/free-machine-learning-courses.html#duke-intro-ml) | Duke | Intermediate | 3 weeks | Audit free, paid certificate | Coursera | Audit-Only |
How to Choose the Right ML Course for You
-----------------------------------------
With 20+ free online machine learning courses available, choosing the right one can feel overwhelming. This section will help you match your goals, background, and learning style with the perfect course or combination of courses.
Before choosing the right ML course for you, you should consider the following factors:
1. Your learning goals: are you interested in understanding the mathematical foundations and theoretical underpinnings of ML (perhaps for graduate school or research positions), or you want to build and deploy ML models in production environments?
2. Your math background: are you strong in calculus, linear algebra, and probability?
3. Your available time: are you limited on time and need a quick course to get started?
Based on different combinations of these factors, we’ve identified five common learning scenarios to help you choose the right course or combination of courses.
We describe each scenario in the table below:
| Your Goal | Start With | Add/Progress To | Supplement |
| --- | --- | --- | --- |
| **Get a job as an ML engineer** | ML Zoomcamp or Google ML Crash Course | Kaggle courses for competition experience | Fast.ai for deep learning specialization |
| **Understand ML theory deeply** | Mathematics for ML Specialization (if needed) | Stanford CS229 or MIT ML course | Caltech CS156 for complementary perspective |
| **Add ML to my current role (analyst, engineer, etc.)** | Google ML Crash Course or IBM ML with Python | Kaggle courses for hands-on practice | Inria scikit-learn for production-quality code |
| **Explore ML as a beginner** | Microsoft ML for Beginners or AWS Fundamentals | DeepLearning.AI ML Specialization (Andrew Ng) | Kaggle courses for confidence-building wins |
| **Specialize in deep learning** | Fast.ai Practical Deep Learning for Coders | DeepLearning.AI ML Specialization | FreeCodeCamp ML with Python for PyTorch |
1\. ML Zoomcamp by DataTalksClub
--------------------------------
Free practical machine learning course with hands-on projects and deployment skills
1. **Platform:** DataTalks.Club (ML Zoomcamp)
2. **Provider:** DataTalks.Club (instructor: Alexey Grigorev)
3. **Difficulty Level:** Beginner-Intermediate (coding experience required; no prior ML needed)
4. **Format:** Free course offered in two modes—self-paced (full materials) or a 4-month live cohort with weekly assignments, project reviews, and Slack community
5. **Duration:** ~4 months (live cohort starting September 2025) or self-paced
6. **Certificate:** Available for free upon completion for the live cohort; not issued for the self-paced track
[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
is a practical, end-to-end ML engineering course that takes learners from core foundations to production deployment. You’ll cover regression and classification, evaluation and cross-validation, trees and gradient boosting, and deep learning (CNNs, transfer learning). A deployment track focuses on packaging and serving models (FastAPI APIs, Docker, cloud, serverless, ONNX Runtime, Kubernetes, optional KServe) plus monitoring and CI/CD. The program centers on building: a midterm end-to-end project and a capstone production system, emphasizing reproducible code, system design, and documentation, supported by a structured homework cadence and an active community.
Examples of the final projects:
* [Blood Cell Classifier for Cancer Prediction](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
(Computer Vision)
* [Waste Classifier](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
(Computer Vision)
2\. Google Machine Learning Crash Course (MLCC)
-----------------------------------------------
Google's practical introduction to machine learning with interactive demos and coding exercises
1. **Platform:** Google Developers
2. **Provider:** Google
3. **Difficulty Level:** Beginner-Intermediate
4. **Format:** Free, self-paced course
5. **Duration:** Self-paced (module-based; time commitment varies)
6. **Certificate:** Not available
[“Machine Learning Crash Course” by Google](https://developers.google.com/machine-learning/crash-course)
is a practical introduction to ML that combines short videos, interactive demos, and coding exercises. Core modules cover linear and logistic regression, binary classification metrics, and data preparation for numerical and categorical features (including one-hot, feature hashing, mean encoding, and feature crosses), plus generalization and overfitting. Advanced modules introduce neural networks, embeddings, and a primer on large language models (tokens, Transformers, training basics). Real-world tracks address production ML systems, AutoML, and ML fairness, emphasizing deployment considerations and responsible practice.
3\. Microsoft "ML for Beginners"
--------------------------------
Project-based machine learning curriculum by Microsoft with 26 lessons and hands-on assignments
1. **Platform:** GitHub (open-source curriculum; supplemental modules on Microsoft Learn)
2. **Provider:** Microsoft (Cloud Advocates)
3. **Difficulty Level:** Beginner
4. **Format:** Free, self-paced curriculum
5. **Duration:** 12 weeks (self-paced; time per week varies)
6. **Certificate:** Not available
[“Machine Learning for Beginners” by Microsoft](https://github.com/microsoft/ML-For-Beginners)
is a project-based, introductory curriculum that teaches classical ML with Python and scikit-learn—deliberately avoiding deep learning. Organized as 26 lessons over 12 weeks, it blends short readings, knowledge checks, and coding assignments with end-to-end mini-projects (including a simple web app) to reinforce practice. Datasets and examples span global contexts, and many lessons have R equivalents. You’ll progress from fundamentals and regression to classification, clustering, NLP, time series, and reinforcement learning, with frequent quizzes and structured challenges to solidify concepts. The repository includes solutions, a quiz app, translation support, and guidance for both self-learners and instructors.
4\. Machine Learning with Python by IBM
---------------------------------------
Practical machine learning with Python and scikit-learn by IBM with labs and projects
1. **Platform:** Coursera
2. **Provider:** IBM
3. **Difficulty Level:** Intermediate (recommended experience)
4. **Format:** Self-paced online course
5. **Duration:** ~2 weeks at ~10 hours/week
6. **Certificate:** Audit free; optional certificate available (paid)
[“Machine Learning with Python” by IBM](https://www.coursera.org/learn/machine-learning-with-python)
gives a practical introduction to machine learning with Python and scikit-learn. You’ll cover core concepts and roles in ML; implement regression (linear, multiple linear, polynomial, logistic), supervised methods (decision trees, k-nearest neighbors, SVM), and unsupervised techniques (clustering; dimensionality reduction with PCA, t-SNE, UMAP). Labs emphasize model evaluation (metrics, cross-validation, regularization) and pipeline optimization. A rainfall-prediction project and a course-wide exam consolidate skills. Instructors: Joseph Santarcangelo and Jeff Grossman (IBM).
5\. Machine Learning Specialization by DeepLearning.AI
------------------------------------------------------
Beginner-friendly ML specialization taught by Andrew Ng covering modern machine learning fundamentals
1. **Platform:** Coursera
2. **Provider:** DeepLearning.AI & Stanford Online (offered by Stanford University and DeepLearning.AI)
3. **Difficulty Level:** Beginner (recommended experience)
4. **Format:** Self-paced Specialization; 3 courses
5. **Duration:** ~2 months at ~10 hours/week (≈94 hours total)
6. **Certificate:** Audit free; optional certificate available (paid)
[“Machine Learning Specialization” by DeepLearning.AI](https://www.coursera.org/specializations/machine-learning-introduction)
is a beginner-friendly, three-course program (taught by Andrew Ng) covers the fundamentals of modern machine learning and how to apply them in practice. You’ll build models in Python with NumPy and scikit-learn; implement supervised learning for regression and classification (linear/logistic regression, neural networks with PyTorch, decision trees and tree ensembles); apply best practices for evaluation and data-centric improvement; and use unsupervised methods such as clustering and anomaly detection. The final course adds recommender systems (collaborative filtering, content-based deep learning) and an introduction to deep reinforcement learning.
6\. StanfordOnline: Statistical Learning with Python
----------------------------------------------------
Stanford's statistical learning course with Python labs aligned with the ISLP textbook
1. **Platform:** edX
2. **Provider:** StanfordOnline
3. **Difficulty Level:** Introductory
4. **Format:** Self-paced online course
5. **Duration:** ~11 weeks at 3-5 hours/week
6. **Certificate:** Audit free; optional certificate available (paid)
[“Statistical Learning with Python” by StanfordOnline](https://www.edx.org/learn/python/stanford-university-statistical-learning-with-python)
led by Trevor Hastie, Robert Tibshirani, and Jonathan Taylor centers on regression and classification and builds practical intuition for resampling (cross-validation, bootstrap), model selection and regularization (ridge, lasso), nonlinear modeling (splines, GAMs), tree-based methods (trees, random forests, boosting), support-vector machines, and a concise treatment of neural networks/deep learning. It also surveys unsupervised learning (PCA, k-means, hierarchical clustering), survival analysis, and multiple testing. Computing is done in Python with step-by-step labs that implement the methods covered in the lectures, closely aligned with the textbook An Introduction to Statistical Learning with Applications in Python.
7\. MITx: Machine Learning with Python: from Linear Models to Deep Learning
---------------------------------------------------------------------------
In-depth MIT course covering ML theory and implementation from linear models to deep learning
1. **Platform:** edX
2. **Provider:** MITx (Massachusetts Institute of Technology)
3. **Difficulty Level:** Intermediate-Advanced (requires Python, probability, calculus, and linear algebra)
4. **Format:** Instructor-paced online course
5. **Duration:** ~15 weeks at 10-14 hours/week
6. **Certificate:** Audit free; optional certificate available (paid)
[“Machine Learning with Python: from Linear Models to Deep Learning” by MIT](https://www.edx.org/learn/machine-learning/massachusetts-institute-of-technology-machine-learning-with-python-from-linear-models-to-deep-learning)
is an in-depth, instructor-led course that teaches how to turn training data into effective predictive systems. It develops the theory of representation, overfitting, regularization, and generalization (including VC dimension), and applies it to classification, regression, clustering, recommender problems, probabilistic modeling, and reinforcement learning. You’ll implement linear models, kernel methods and SVMs, neural networks (including deep and recurrent nets), and EM-based generative models, while practicing end-to-end project organization from training/validation and hyperparameter tuning to feature engineering. Projects include an automatic review analyzer, digit recognition with neural networks, and a reinforcement-learning task. Instructors include Regina Barzilay, Tommi Jaakkola, and Karene Chu.
8\. Data Science: Machine Learning by Harvard
---------------------------------------------
Harvard's hands-on ML course focused on building a movie recommendation system end-to-end
1. **Platform:** edX
2. **Provider:** HarvardX
3. **Difficulty Level:** Introductory
4. **Format:** Self-paced online course
5. **Duration:** ~8 weeks at 2-4 hours/week
6. **Certificate:** Audit free; optional certificate available (paid)
[“Data Science: Machine Learning” by Harvard](https://www.edx.org/learn/machine-learning/harvard-university-data-science-machine-learning)
teaches core machine learning skills by guiding you through building a movie recommendation system end to end. You’ll work with training data to uncover predictive relationships, practice cross-validation to avoid overfitting, and implement several common algorithms with an emphasis on when and why they work. Along the way, you’ll learn about regularization and principal component analysis and apply them in a practical pipeline. Designed for learners in the HarvardX Professional Certificate in Data Science, it balances fundamentals with hands-on implementation to develop sound intuition for ML model development and evaluation.
9\. Stanford CS229
------------------
Stanford's rigorous ML course covering theory and applications in robotics, data mining, and bioinformatics
1. **Platform:** Stanford University course website
2. **Provider:** Stanford University (CS)
3. **Difficulty Level:** Intermediate-Advanced (requires Python/NumPy, probability, multivariable calculus, and linear algebra)
4. **Format:** Instructor-led university course
5. **Duration:** Summer 2025 (June 24-August 16, 2025)
6. **Certificate:** Not available
[CS229 “Machine Learning”](https://cs229.stanford.edu/syllabus-autumn2018.html)
is a broad, rigorous introduction to machine learning and statistical pattern recognition. The course covers:
* Supervised learning (generative vs. discriminative models; parametric and non-parametric methods; logistic/linear regression; neural networks; support vector machines)
* Unsupervised learning (clustering, dimensionality reduction, kernel methods)
* Elements of learning theory (bias-variance, generalization)
* Reinforcement learning/adaptive control.
Students practice through problem sets and companion lectures, with recent offerings adding topical guest sessions (e.g., agentic systems, sequence models, time-series forecasting) to connect core theory with modern applications in areas like robotics, data mining, bioinformatics, speech, and text/web processing.
10\. UC Berkeley CS189/289A (upper-undergrad/grad)
--------------------------------------------------
Berkeley's comprehensive ML survey balancing theory and implementation with modern topics
1. **Platform:** University course website (UC Berkeley CS 189/289A)
2. **Provider:** University of California, Berkeley (EECS)
3. **Difficulty Level:** Intermediate-Advanced
4. **Format:** Instructor-led university course
5. **Duration:** ~16 weeks
6. **Certificate:** Not available
[CS 189/289A “Introduction to Machine Learning”](https://eecs189.org/)
is Berkeley’s rigorous survey of modern ML, balancing theory and implementation. The course builds from probabilistic modeling and linear/logistic regression into neural networks (backpropagation, CNNs, attention/Transformers) and model evaluation, then broadens to dimensionality reduction (PCA, t-SNE), clustering, nearest neighbors, and tree/ensemble methods. Later units cover graphical models and HMMs, Markov decision processes and reinforcement learning, plus contemporary topics such as graph neural networks, language/vision applications, and causality. Students apply concepts through weekly discussions and programming assignments, with a midterm and final consolidating both mathematical grounding and practical skills.
11\. Introduction to Machine Learning (MIT 6.036)
-------------------------------------------------
MIT's foundational ML course with lectures, labs, and exercises building conceptual understanding
1. **Platform:** MIT Open Learning Library
2. **Provider:** Massachusetts Institute of Technology (MIT)
3. **Difficulty Level:** Intermediate (Python programming, calculus, and linear algebra recommended)
4. **Format:** Free, self-paced online course
5. **Duration:** ~13 weeks
6. **Certificate:** Not available
[“Introduction to Machine Learning” (MIT 6.036)](https://ocw.mit.edu/courses/6-036-introduction-to-machine-learning-fall-2020/)
formulates well-specified learning problems and develops core ideas of representation, overfitting, and generalization. These principles are applied in supervised learning and reinforcement learning, with example applications to images and temporal sequences. Materials include lectures, notes, labs, and graded-style exercises to build both conceptual understanding and practical skills.
12\. Caltech CS156 "Learning from Data"
---------------------------------------
Caltech's ML course blending rigorous theory with practical intuition and implementation
1. **Platform:** Caltech (course website / YouTube)
2. **Provider:** California Institute of Technology (Caltech)
3. **Difficulty Level:** Introductory-Intermediate (requires basic probability, linear algebra with matrices, and calculus)
4. **Format:** Free, self-paced MOOC
5. **Duration:** 18 lectures (~60 minutes each)
6. **Certificate:** Not available
Recorded from a live Caltech broadcast, [Yaser S. Abu-Mostafa’s Machine Learning course](https://work.caltech.edu/telecourse.html)
blends rigorous theory with practical intuition. It develops generalization foundations (training vs. testing, VC dimension, bias-variance), then works through linear and logistic models, neural networks and backpropagation, overfitting, regularization, and validation. The course also covers SVMs, kernel methods, radial basis functions, and practitioner principles (Occam’s razor, sampling bias, data snooping), with a concluding synthesis that situates Bayesian and ensemble ideas. Learners practice via eight homework sets and a final exam, guided by clear mathematical framing and implementation-focused discussion.
13\. Machine Learning by Carnegie Mellon University
---------------------------------------------------
CMU's rigorous semester-long ML course blending formal theory with hands-on implementation
1. **Platform:** Carnegie Mellon University (course website; open materials)
2. **Provider:** Carnegie Mellon University (CMU)
3. **Difficulty Level:** Intermediate-Advanced (upper-level undergraduate / intro graduate)
4. **Format:** University lecture course; recorded lectures + slides; recitations; homeworks; programming project; two exams
5. **Duration:** Spring 2015 semester (~16 weeks; Jan 12-Apr 29, 2015)
6. **Certificate:** Not available
[CMU’s 10-601 “Machine Learning”](http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml)
, taught by Tom Mitchell and Maria-Florina (Nina) Balcan, is a rigorous, semester-long introduction that blends theory with practice. The course builds from core supervised learning and probabilistic modeling into the mathematics of generalization and regularization, then moves through probabilistic graphical models and inference before tackling ensemble methods, kernel methods, and support vector machines. Mid-semester it shifts to semi-supervised and active learning and, later, to representation learning and dimensionality reduction, culminating with modern neural networks, deep learning, and a primer on reinforcement learning and privacy. Lectures, recitations, homeworks, a programming project, and two exams are designed to connect formal guarantees with hands-on implementation, mirroring the perspective of Mitchell’s classic textbook while updating it with contemporary practice.
14\. Mathematics for Machine Learning Specialization by Imperial College London
-------------------------------------------------------------------------------
Three-course math sequence building essential foundations for understanding ML algorithms
1. **Platform:** Coursera
2. **Provider:** Imperial College London
3. **Difficulty Level:** Foundational to Intermediate (math-focused)
4. **Format:** Self-paced Specialization; 3 courses (Linear Algebra, Multivariate Calculus, Principal Component Analysis); video lectures, quizzes, and exercises
5. **Duration:** ~4-6 weeks per course (≈12-18 weeks total), learn at your own pace
6. **Certificate:** Audit free; optional certificate available (paid)
[“Mathematics for Machine Learning” by Imperial College London](https://www.coursera.org/specializations/mathematics-machine-learning)
is a three-course sequence that builds the core math needed to understand and implement ML algorithms. The Linear Algebra course covers vectors, matrices, dot products, basis changes, and eigenvalues/eigenvectors. Multivariate Calculus covers single- and multivariable differentiation, the chain rule, Jacobians, Hessians, Lagrange multipliers, and Taylor series. The PCA course connects linear algebra and calculus to statistics and geometry—orthogonality, projections, and dimensionality reduction—culminating in practical PCA applications. Courses are taught by Imperial College London faculty.
15\. Machine Learning with Python by FreeCodeCamp
-------------------------------------------------
Practical ML certification with PyTorch covering computer vision, NLP, and recommendation systems
1. **Platform:** freeCodeCamp
2. **Provider:** freeCodeCamp
3. **Difficulty Level:** Intermediate (Python experience recommended)
4. **Format:** Self-paced online certification
5. **Duration:** Self-paced (project-based; varies by learner)
6. **Certificate:** freeCodeCamp “Machine Learning with Python” Certification (upon completing required projects)
[“Machine Learning with Python” by FreeCodeCamp](https://www.freecodecamp.org/learn/machine-learning-with-python/#how-neural-networks-work)
teaches practical machine learning with Python and PyTorch. You’ll build several neural networks and explore core and advanced topics including how neural networks work, CNNs, RNNs/LSTMs, natural language processing, and an introduction to reinforcement learning. Instruction combines a PyTorch course by Tim Ruscica (“Tech With Tim”) with conceptual videos by Brandon Rohrer. To earn the certificate, you complete hands-on projects such as Rock-Paper-Scissors, a Cat/Dog image classifier, a KNN book recommender, a linear-regression health-costs calculator, and an SMS text classifier, demonstrating applied skills across computer vision, NLP, recommendation, and regression.
16\. Practical Deep Learning for Coders by Fast.ai
--------------------------------------------------
Beginner-friendly deep learning course by Jeremy Howard with coding-first approach
1. **Platform:** fast.ai
2. **Provider:** fast.ai (Jeremy Howard; co-created with Rachel Thomas)
3. **Difficulty Level:** Beginner-friendly (requires basic coding; minimal math)
4. **Format:** Free, self-paced video course
5. **Duration:** ≈13.5 hours of core video
6. **Certificate:** No formal certificate
[“Practical Deep Learning for Coders” by Fast.ai](https://course.fast.ai/)
teaches you to apply deep learning and machine learning to real problems without heavy prerequisites. Starting from working, state-of-the-art models, you’ll build and train systems for computer vision, NLP, tabular data, and collaborative filtering; create random-forest and regression baselines; and deploy models as simple web apps. Along the way you’ll learn transfer learning, data augmentation, weight decay, embeddings, and the training loop (including SGD) to understand why models work and how to improve them. The course emphasizes coding first, provides free cloud options for training, and is taught by Jeremy Howard.
17\. Kaggle Machine Learning Courses
------------------------------------
Short practical ML courses with hands-on coding in Kaggle notebooks and certificates
1. **Platform:** Kaggle Learn (Micro-Courses)
2. **Provider:** Kaggle
3. **Difficulty Level:** Beginner-Intermediate
4. **Format:** Self-paced micro-courses; short lessons with hands-on coding in Kaggle notebooks
5. **Duration:** A few hours per course (self-paced)
6. **Certificate:** Kaggle Learn certificate of completion
Kaggle has multiple [short courses](https://www.kaggle.com/learn)
that you can take for free. Here are some of them you can start with:
* [Intro to Machine Learning](https://www.kaggle.com/learn/intro-to-machine-learning)
: Covers the core supervised-learning workflow: how ML models work, basic data exploration, model validation, underfitting vs. overfitting, and training a first tree-based model (random forests) directly in Kaggle notebooks.
* [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning)
: Builds production-minded skills: handling missing values, working with categorical variables, building ML pipelines, applying cross-validation, training XGBoost models, and preventing data leakage.
* [Feature Engineering](https://www.kaggle.com/learn/feature-engineering)
: Focuses on improving signal: using mutual information to assess feature relevance, creating features, and applying unsupervised feature methods (k-means clustering, PCA) plus target encoding.
Recommendation: take them in the order listed to ensure prerequisites are covered.
18\. Machine learning in Python with scikit-learn by Inria
----------------------------------------------------------
In-depth scikit-learn course taught by library's core developers with Jupyter notebooks
1. **Platform:** FUN MOOC (France Université Numérique)
2. **Provider:** Inria (by scikit-learn core developers)
3. **Difficulty Level:** Intermediate (accessible with basic Python; NumPy/Pandas/Matplotlib helpful)
4. **Format:** Self-paced MOOC; Jupyter notebooks; quizzes + programming exercises; hands-on code labs
5. **Duration:** ~36 hours (self-paced)
6. **Certificate:** Open Badge upon ≥60% overall on quizzes and programming exercises (issued on request)
[“Machine learning in Python with scikit-learn” by Inria](https://www.fun-mooc.fr/en/courses/machine-learning-python-scikit-learn/)
is an in-depth, practical introduction to predictive modeling with scikit-learn, taught by members of the library’s core team. You’ll build end-to-end pipelines and develop sound intuition for model design, selection, and evaluation. The syllabus covers the predictive modeling pipeline, selecting the best model, hyperparameter tuning, linear models, decision trees, ensembles, and performance evaluation, with all materials freely available online. Instructors include Gaël Varoquaux, Olivier Grisel, Guillaume Lemaître, Loïc Estève, and colleagues from Inria.
19\. Fundamentals of Machine Learning and Artificial Intelligence by AWS
------------------------------------------------------------------------
Foundational AI/ML course by AWS covering terminology, algorithms, and cloud ML services
1. **Platform:** Coursera
2. **Provider:** Amazon Web Services (AWS)
3. **Difficulty Level:** Beginner / Foundational
4. **Format:** Self-paced online course; 1 module; 1 assignment; taught in English; flexible schedule
5. **Duration:** ~1 hour
6. **Certificate:** Audit free; optional certificate available (paid)
[“Fundamentals of Machine Learning and Artificial Intelligence” by AWS](https://www.coursera.org/learn/fundamentals-of-machine-learning-and-artificial-intelligence)
explains the foundations of AI and ML, the relationships between AI, machine learning, deep learning, and generative AI, and where these techniques are applied. You’ll learn core terminology and survey common machine learning algorithms and neural networks at a high level. The course also highlights selected AWS services that provide AI/ML capabilities and shows how they can be used to solve practical problems across industries. Skills covered include AI/ML fundamentals, generative AI, artificial neural networks, data analysis, and machine learning algorithms. Instruction is delivered by an AWS instructor.
20\. Introduction to Machine Learning by Duke University
--------------------------------------------------------
Hands-on ML course by Duke University with PyTorch for medical diagnostics and computer vision
1. **Platform:** Coursera
2. **Provider:** Duke University
3. **Difficulty Level:** Intermediate (some related experience required)
4. **Format:** Self-paced online course; 6 modules; 24 assignments; hands-on practice in PyTorch; taught in English; flexible schedule
5. **Duration:** ~3 weeks (≈10 hours/week)
6. **Certificate:** Audit free; optional certificate available (paid)
[“Introduction to Machine Learning” by Duke University](https://www.coursera.org/learn/machine-learning-duke)
introduces core machine learning models and where they are useful, covering logistic regression, multilayer perceptrons, convolutional neural networks, and basic natural language processing. Learners practice implementing models on real datasets with PyTorch, with applications spanning medical diagnostics, image recognition, and text prediction. Skills developed include supervised learning, neural networks, reinforcement learning fundamentals, computer vision, data validation, and applied ML. Instructors are Lawrence Carin, David Carlson, and Timothy Dunn from Duke University.
Conclusion
----------
Learning machine learning doesn’t have to be expensive. With so many high-quality courses available for free or free-to-audit, you can build a solid foundation without spending a cent.
The key to free online learning is consistency and practice. Pick one course that matches your current level, stick with it, and apply what you learn through projects. Over time, you’ll gain the understanding of the core machine learning concepts and develop the practical skills.
Frequently Asked Questions
--------------------------
Do I need to know programming before taking these courses?
Most courses require basic Python knowledge. If you’re new to programming, start with Microsoft ML for Beginners or Google ML Crash Course, which include gentler introductions. For intensive programming preparation, consider pairing ML study with foundational Python courses.
Which course is best for someone with no math background?
Start with Google Machine Learning Crash Course, Microsoft ML for Beginners, or Fast.ai Practical Deep Learning for Coders. These courses teach ML concepts with minimal math prerequisites. Once you’re comfortable, you can take the Mathematics for ML Specialization to deepen your understanding.
Can I really learn ML for free, or do I need to pay for certificates?
Absolutely! All courses listed offer free access to materials. Certificates are optional and mainly useful for LinkedIn/resume credentials. Your portfolio projects (especially from ML Zoomcamp or FreeCodeCamp) will be more valuable to employers than certificates.
How long does it take to learn machine learning?
For basic competency: 2-3 months with consistent study. For job readiness: 4-6 months including projects. For advanced expertise: 1-2 years of continuous learning and practice. Start with one course (like ML Zoomcamp or DeepLearning.AI Specialization) and build from there.
Should I take multiple courses simultaneously?
Generally no. Focus on one primary course and one supplementary resource (like Kaggle for practice). Exception: You can pair a math course (Imperial College) with a practical course if you need math foundations.
What's the difference between ML and AI courses?
Machine Learning is a subset of AI focused on algorithms that learn from data. These ML courses teach core algorithms (regression, neural networks, decision trees). For broader AI topics including agents, LLMs, and automation, check out our [AI Dev Tools Zoomcamp](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
.
Which courses offer job placement support?
Most free courses don’t offer formal placement services, but ML Zoomcamp includes an active Slack community where members share job opportunities, and the capstone project creates a strong portfolio piece. Focus on building projects and networking within course communities.
Can I get college credit for these courses?
University OpenCourseWare (MIT 6.036, Stanford CS229, etc.) don’t offer credit, but platforms like Coursera and edX sometimes offer credit-eligible versions for a fee. Check each platform’s credit policies if this is important to you.
Related Posts
-------------
[### Data Engineering Zoomcamp: Free Data Engineering Course and Certification\
\
Become a Data Engineer: Master Modern Data Engineering with Hands-On Training\
\
Read more](https://datatalks.club/blog/data-engineering-zoomcamp.html)
[### Building Discipline in Machine Learning with ML Zoomcamp\
\
A disciplined path to data prep, validation, and deployment, plus how public sharing accelerated my skills, connections, and offers.\
\
Read more](https://datatalks.club/blog/building-discipline-in-machine-learning-with-ml-zoomcamp.html)
[### How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp\
\
Learn what you can build after ML Zoomcamp based on a real project from one of the graduates\
\
Read more](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Apurva Misra – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Apurva Misra
Apurva Misra is an AI Consultant at Sentick, focusing on assisting startups with their AI strategy and building solutions. She leverages her extensive experience in machine learning and a Master’s degree from the University of Waterloo, where her research bridged driving and machine learning, to offer valuable insights. Apurva’s keen interest in the startup world fuels her passion for helping emerging companies incorporate AI effectively. In her free time, she is learning Spanish and she also enjoys exploring hidden gem eateries, always eager to hear about new favourite spots!
[](https://linkedin.com/in/https://www.linkedin.com/in/apurva-misra/)
[](https://apurvamisra.com/)
### Events
* Elevating RAG Systems ([watch on youtube](https://www.youtube.com/watch?v=RexNgpgwmkk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Antonis Stellas – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Antonis Stellas
I am a Freelance Data Scientist currently working at Nanometrisis, a startup focused on providing software for nanoproduct inspection. In addition, recently I joined a freelance platform called Upwork where I offer a range of data solutions to clients.
[](https://twitter.com/StellasAntonis)
[](https://linkedin.com/in/antonisstellas)
[](https://github.com/AntonisCSt)
### Events
* Lessons Learned from Freelancing and Working in a Start-up ([watch on youtube](https://www.youtube.com/watch?v=-Gj7SaI-QW4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anna Hannemann – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anna Hannemann
As a Domain Owner for Data Science at Metro.digital, Anna drives the story of data science and AI within Metro business. In her previous positions, Anna led product teams in areas of recommender systems and robotics/smart logistics. Prior to that, Anna gained several years of experience in software development followed by a PhD in Data Science. Additionally, Anna contributes proactively to a range of initiatives focused on enablement and empowerment of women in tech.
[](https://linkedin.com/in/anna-hannemann)
### Events
* Product Owners in Data Science ([watch on youtube](https://www.youtube.com/watch?v=rTRTjB6cGng)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ankur A. Patel – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ankur A. Patel
Ankur A. Patel is an AI entrepreneur, thought leader, and author. He is currently the cofounder and head of data at Glean and the cofounder of Mellow. Glean uses natural language processing to deliver vendor spend intelligence within an accounts payable solution. Mellow develops easy-to-use natural language processing APIs for developers to use as part of their product build.
He is the author of Hands-on Unsupervised Learning Using Python and Applied Natural Language Processing in the Enterprise, both O’Reilly Media publications.
[](https://linkedin.com/in/ankur-patel-b44a4516)
[](https://github.com/aapatel09)
[](https://www.ankurapatel.io/)
### Books
* [Applied Natural Language Processing in the Enterprise](https://datatalks.club/books/20210726-applied-natural-language-processing-in-the-enterprise.html)
(the book of the week from 26 Jul 2021 to 30 Jul 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Aparna Dhinakaran – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Aparna Dhinakaran
Co-Founder and CPO of Arize AI. Formerly Computer Vision PhD at Cornell, Uber Machine Learning, UC Berkeley AI Research.
[](https://twitter.com/aparnadhinak)
[](https://linkedin.com/in/aparnadhinakaran)
[](https://github.com/AparnaDhinakaran)
[](http://www.aparnadhinakaran.com/)
### Events
* Machine Learning Observability ([watch on youtube](https://datatalks.club/people/aparnadhinakaran.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Deploy ML Models on AWS Lambda with Docker Containers and SAM – DataTalks.Club
"build, train, and serve with Docker, ECR, and SAM, plus CI/CD via GitHub Actions. Follow this proven guide."}"> "build, train, and serve with Docker, ECR, and SAM, plus CI/CD via GitHub Actions. Follow this proven guide."}"> "build, train, and serve with Docker, ECR, and SAM, plus CI/CD via GitHub Actions. Follow this proven guide."}">
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Deploy ML Models on AWS Lambda with Docker Containers and SAM
Deploy ML Models on AWS Lambda with Docker Containers and SAM
=============================================================
### End-to-end guide to train, containerize, and serve ML via Lambda, ECR, SAM, and GitHub Actions
01 Jan 2021 by [Sejal Vaidya](https://datatalks.club/people/sejalvaidya.html)
This post will demonstrate how you can deploy a Machine Learning model on a Serverless API (AWS Lambda), using ECR with Docker as runtime.
This is a companion article to the online workshop I conducted for **[DataTalks.Club](https://datatalks.club/)
.**
The repository for the workshop can be found here: [https://github.com/sejalv/serverless-ml-workshop](https://github.com/sejalv/serverless-ml-workshop)
Background
----------
Deploying machine learning models into production can sometimes be a stumbling block for Data Science teams. A common mode of deployment is to find somewhere to host your models and expose them via APIs. In practice, this can make it easy for your end users to integrate your model outputs directly into their applications and business processes.
Moreover, Serverless Computing has gone long way towards abstracting away the complexity of deploying applications into production environments by largely doing away with the need to manually configure servers, load balancers, containers etc. Provision of such auto-scaling and fault-tolerant functionalities, is what can act as a leverage for Data Scientists for simple model deployments, without any operational overhead.
New features in AWS
-------------------
As of 01.12.2020, re:Invent announced that AWS Lambda can now support up to **[10 GB of memory and 6 vCPU cores](https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-supports-10gb-memory-6-vcpu-cores-lambda-functions/)
**. This means:
* **[Container tooling for workflows](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/)
:**
* You can now package and deploy Lambda functions as container images (Docker/OCI) of up to 10 GB in size and push those images to AWS ECR.
* **[Images](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html)
:**
* Provision of base images for all the supported Lambda runtimes (Python, Node.js, Java, .NET, Go, Ruby) to easily add your code and dependencies
* Extendible base images for custom runtimes based on Amazon Linux to include runtime implementing the Lambda Runtime API
* Deployment of your own arbitrary base images to Lambda, for example- images based on Alpine or Debian Linux.
* **Computation:**
* Proportional to memory, increased CPU power to 6vCPUs
* Easily build and deploy compute-intensive workloads that rely on sizeable dependencies, such as use-cases in machine learning or high performance computing to perform faster
Many more updates related to AWS Lambda and other resources are available on the AWS blog or on **re:Invent** pages.
### Benefits of new updates in AWS Lambda
* We can build larger and faster runtimes with 10GB memory (than the previous 250 MB limit), to serve heavier processes on AWS Lambda, such as ML models using libraries such as pandas, scikit-learn, and PyTorch.
* We can now build and deploy the AWS Lambda function with a docker container as runtime using Amazon ECR, instead of the custom packaging with .zip files, and also overcome the limitations of Lambda Layers. All the libraries and the weights of the model can now be packaged and published to a docker registry or an ECR registry.
* Using container images with version tags will give us some leverage on model versioning
Workflow
--------
The goal is to produce a simple build-train-deploy workflow that works in any CI/CD tool. In addition, we want to:
* Use a docker container, to build/serve anywhere containers can run
* Expose a REST API so that others can consume it
* Decouple the training and serving, to minimise the complexity and attack surface of the operational container
The result is a Docker-based workflow that should look familiar to any software engineer.
Ref: [https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/](https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/)
To adapt this in a serverless context (along with the new updates from AWS), we will:
* Train and serialize a model inside a Docker container
* Package the service using Docker and AWS ECR
* Serve the model with serverless API (AWS Lambda)
* Build, Test, and Deploy, with a CI/CD workflow using GitHub Actions
In addition, we can also use the SAM (Serverless Application Model) framework, in order to use an infrastructure-as-code (IaC) style to set up and configure resources on AWS. This makes it more maintainable, reproducible, and allows version control over IaC.
### Setup
You can lookup the prerequisites and configuration needed to continue with the workshop [here](https://github.com/sejalv/serverless-ml-workshop)
.
We will use the **SAM** framework to develop our serverless application. SAM, or Serverless Application Model, is an open-source framework (provided by AWS) for provisioning AWS services via code, and resources such as functions, triggers, and APIs.
**Advantages of SAM**:
* acts as a wrapper to define the project’s AWS CloudFormation stack
* extends the AWS CLI to functionality for building and testing Lambda applications.
* now uses Docker to run functions in an Amazon Linux environment that matches Lambda
* emulates application’s build environment and API
* IaC benefits like source control, parameterized deployment
Once you have the pre-reqs installed, you can generate a new project using the SAM framework:
$ sam init
Initialize a new serverless project with AWS SAM to scaffold code and templates
The project structure generated as a result:
|-- service # name of the service/project, eg. sam-app
|-- app.py # lambda handler
|-- Dockerfile # Image
|-- requirements.txt # Dependencies
|-- tests
|-- unit
|--test_handler.py
|-- template.yaml # SAM template configuration
`template.yaml` : SAM pre-configures all the necessary info in this file for you to have vanilla functioning app:
* `PackageType`: Image/Zip, filled during the `sam init` step
* `Metadata` \[`Dockerfile`, `DockerContext`, `DockerTag`\]: filled during the `sam init` step
### Containerization
> There are many different ways that you can leverage Docker to deploy your models. My approach here involves training and serializing a model during the image build process. Specifically, we will _embed_ the trained model into the Docker image. Then, whenever we want to run inference or anything, we simply need to run a container from that image, deserialize the model, and generate our predictions.
> This architecture provides _simplicity_: since the only moving pieces are Docker and the model training code. It allows us to leverage Docker’s image tagging system to store and version control our models. It also allows us to use a container registry service, like Amazon’s [Elastic Container Registry](https://aws.amazon.com/ecr/)
> , to store and manage our models. Rather than worry about persisting individual model components, we can just store entire Docker images which contain all of the necessary model artifacts!
Ref: [MLinProduction’s Docker for Machine Learning series](https://mlinproduction.com/docker-for-ml-part-1/)
by Luigi Patruno
As explained here, **our deployment pipeline will be directly integrating the serialized model into the API.** We choose this approach to leverage the large container memory provided to us, and because the scale of the model and our application is pretty small for this workshop.
**NOTE**: However, in most production workflows, it is advisable to keep the training and deployment pipelines decoupled. More info in Conclusion.
### Training
As the focus of this workshop is on deployment, we won’t concern ourselves with training an accurate model. But you can easily replace this with your own more complex model, or convert your Jupyter Notebook to create a `train.py` file.
The contents of this training script can be as simple as:
1. Load and parse data
2. Clean and prepare data
3. Train model
4. Save model parameters results
[Our training script](https://github.com/sejalv/serverless-ml-workshop/blob/dev/service/train.py)
will load a training data set, prepare the data into split into train & test sets, train the model, generate evaluation metrics for the model, and serialize both the model and the evaluation metrics to a specific location.
The `Dockerfile` (located inside the `service` directory, if you have used SAM) should look something like:
FROM public.ecr.aws/lambda/python:3.8 as base
FROM base AS train
COPY requirements.txt .
RUN pip install -r requirements.txt
ENV MODEL_LOCAL_PATH=pickled_model.pkl
COPY train.py .
RUN python3 train.py
Where `base` is the Lambda base image pulled from [AWS ECR Public](https://gallery.ecr.aws/lambda/python)
. You can either use this or one from [DockerHub](https://hub.docker.com/r/amazon/aws-lambda-python)
, or build your custom one.
`MODEL_LOCAL_PATH` is the output file of your serialized model.
To train your model, all you need to do is: `docker build -t .` and wait for it to finish, then you have a container with trained parameters or results ready to serve.
### Serving
[`app.py`](http://app.py/)
: This is where our Lambda code lives.
In a nutshell, this imports libraries, pulls your model from the image, parses the input request, classifies it, and returns a response.
The serving container should:
1. Load saved model parameters/results
2. Instantiate REST API
3. Define routes and serve model
The same [`Dockerfile`](https://github.com/sejalv/serverless-ml-workshop/blob/dev/service/Dockerfile)
should now look something like:
# multi-stage build
FROM public.ecr.aws/lambda/python:3.8 as base
FROM base AS train
COPY requirements.txt .
RUN pip install -r requirements.txt
ENV MODEL_LOCAL_PATH=pickled_model.pkl
COPY train.py .
RUN python3 train.py
FROM base
RUN pip install scikit-learn==0.22.1
ENV MODEL_LOCAL_PATH=pickled_model.pkl
COPY --from=train ./var/task/pickled_model.pkl pickled_model.pkl
COPY app.py ./
# Command can be overwritten by providing a different command in the template directly.
CMD ["app.lambda_handler"]
This is a **[Multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/)
** which incorporates both our training and serving containers. The serving container copies the training artefacts from the training build and pastes them into a location that is expected by the `app.py` file.
Ref: [https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/](https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/)
### Build, Test, and Deploy
Before we begin, please ensure to set the environment variables to your desired values. Or you can directly copy the commands from the GitHub repo.
export AWS_REGION=eu-central-1
export AWS_ACCOUNT=PUT_VALUE_HERE
ECR_REPO=PUT_VALUE_HERE
DOCKER_IMAGE=serverless-ml
The Build-Test-Deploy flow can be done in two ways:
**1) Using Docker**
To **build** the container image locally:
$ docker build -t ${DOCKER_IMAGE} ./
This will package all the dependencies, kick off the training, copy the model params/results over to your serving container, and serve an API endpoint.
To check if this is working, you can start the container image locally using the Lambda Runtime Interface Emulator, and **test** the function invocation with cURL:
$ docker run -p 8080:8080 ${DOCKER_IMAGE}
$ curl -XPOST "http://localhost:8080/2015-03-31/functions/function/invocations" \
-d '{"body": {"data": ".10"}}'
Output:
{"statusCode": 200, "body": "{\"prediction\": \"1\"}"}
Or run unit-tests using `pytest` on your local machine:
$ docker [run|exec] ${DOCKER_IMAGE} python -m pytest tests/ -v
Finally, to **deploy**, authenticate Docker with ECR, and push the image to the container registry with the `latest` tag:
$ aws ecr get-login-password | \
docker login --username AWS \
--password-stdin ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
$ docker push ${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:latest
**2) Using SAM:**
To add an API endpoint to your Lambda Function, you can update the `Events` section in your `template.yaml` to:
Events:
:
Type: Api
Properties:
Path: /classify # or whatever endpoint name you wish to keep
Method: post # or get, based on your API
Optionally, in the same file, you can also add to the `ImageUri` tag, the ECR repo to where your Docker Image would be pushed:
:
Type: AWS::Serverless::Function
Properties:
FunctionName:
ImageUri: .dkr.ecr..amazonaws.com/:
PackageType: Image
**To build:**
$ sam build
The SAM CLI builds a docker image from the `Dockerfile`, and generates a **CloudFormation stack** in the `.aws-sam/build` directory to provision the AWS resources, based on the config defined in `template.yaml` file.
Output:
SAM builds the Docker image and generates CloudFormation artifacts for deployment
**To test:**
$ sam local start-api
$ curl -XPOST http://127.0.0.1:3000/classify \
-H 'Content-Type: application/json' \
-d '{"data":".10"}'
Output:
{"prediction": 1}
**To deploy:**
$ sam deploy --guided
The guided deployment will walk through all required parameters and will create a `samconfig.toml` for us, that caches the resource-related info.
Output:
Guided deploy walks through parameters and provisions resources via CloudFormation
### CI/CD
The steps we executed manually above, can be integrated into a CI/CD pipeline for automatic build, test and push. For this, we’ll use GitHub Actions, but feel free to pick your choice of tool.
**Steps**
1. Store your `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in **[GitHub Secrets](https://github.com/sejalv/serverless-ml-workshop/settings/secrets/actions)
** of your repository.
2. Create the `.github/workflows` directory in the root folder of your project.
3. Now we add two workflows:
* [`tests.yml`](https://github.com/sejalv/serverless-ml-workshop/blob/dev/.github/workflows/python-app.yml)
: which will install Python dependencies, run tests and lints. This is based on the [Python Application template provided by GitHub Actions](https://github.com/actions/starter-workflows/blob/e9e00b017736d3b3811cedf1ee2e8ceb3c48e3dd/ci/python-app.yml)
* [`deploy.yml`](https://github.com/sejalv/serverless-ml-workshop/blob/dev/.github/workflows/deploy-sam.yml)
: which will build the image and deploy it to Cloud (ECR). This is a custom workflow built on Ubuntu environment and using `bash` to execute the same steps from the section above.
### Cleanup
To delete the sample application that you created, use the AWS CLI:
$ aws cloudformation delete-stack --stack-name
Conclusion
----------
As seen here, deploying ML models as serverless functions is probably the fastest, simplest route to getting a stable, scalable ML API deployed. And the new AWS features on container integration provide for resilient deployments.
However, it should also be noted that serverless is not always a good fit, especially for large ML workloads in production.
**When to use serverless/Lambda in DS & ML?**
* Ideal for building prototypes or new products that use lightweight libraries
* When your model is small. A large model is going to result in slow response times.
* Data vectors are small. Lambda comes with a 10GB memory ceiling. So if your input data is massive, it’s just going to crash.
* Your model will be sporadically accessed. Lambda scales out (not up) to infinite and charges you per request. So it’s well suited to intermittent “on-the-fly” demand.
* Low-cost is important. Lambda charges $0.20 per million requests.
### Future Enhancements and Tradeoffs
**Monitoring/tracing/alerting**
* If there are downstream services that need to be notified, we could add notifications to our Lambda function using SNS or CloudWatch too.
**Workflow design alternatives**
* Use of storage for models
* In terms of design, it makes sense to have decoupled deployment pipelines for training and serving, if the changes in models and the application code are not always in sync, and the deployment cycles need to vary. This is often the case in large production workloads.
* In this case, the training script could run independently and push the serialized model to a storage like s3. And your service (Lambda) can then load the model from s3 into your application code.
* Use of Workers:
* Delegating the loading and prediction work to a worker and not integrating the model into the REST API. This is because a model can take a long time to load and predict. Therefore, we can manage them asynchronously by adding an SQS queue and a DynamoDB table.
* Ref: [https://medium.com/swlh/how-to-deploy-your-scikit-learn-model-to-aws-44aabb0efcb4](https://medium.com/swlh/how-to-deploy-your-scikit-learn-model-to-aws-44aabb0efcb4)
* [https://www.ritchievink.com/blog/2018/09/16/deploy-any-machine-learning-model-serverless-in-aws/](https://www.ritchievink.com/blog/2018/09/16/deploy-any-machine-learning-model-serverless-in-aws/)
**Sagemaker vs. Lambda**
* The pay-per-request model of Lambdas makes it great for AB testing and prototyping. But even when Lambda functions come with a 10GB memory ceiling, they are short-running in their processing capacity (they timeout after _max_ 15 minutes). So if your model might be expensive to load or if the input data is very large, it’s better to opt for Sagemaker or Batch Jobs.
* Ref: [https://towardsdatascience.com/saving-95-on-infrastructure-costs-using-aws-lambda-for-scikit-learn-predictions-3ff260a6cd9d](https://towardsdatascience.com/saving-95-on-infrastructure-costs-using-aws-lambda-for-scikit-learn-predictions-3ff260a6cd9d)
**Fargate vs. Lambda**
* Lambda functions are stateless and short-lived. That means that a lot of container workloads in their current form may still suit the Fargate/ECS/EKS camp better. Fargate will remain useful for more traditional, longer-lived workloads that don’t have a need to scale quickly to 100’s or 1000’s of containers.
* Container Image deployment to Lambda enables Lambda’s incredibly rapid and responsive scaling as well as Lambda’s integrations, error handling, destinations, DLQs, queueing, throttling and metrics.
* Ref: [https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh](https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh)
**AWS vs. Kubernetes**
* Deploying your container is entirely dependent on your tech stack. It could be as simple as a `docker run -d -p 8080:80 myimage`. Or you could use that container in a Kubernetes [KNative manifest](https://knative.dev/)
for full-on serverless machine learning deployments.
**Cost & Benchmarks**
[https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh](https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh)
### Questions (from the workshop)
* How will this workflow be different from deploying a Deep learning model?
* This design can still be used for a Deep Learning model. However, if you have high computational needs for training, then you’ll need to think about **where** you are doing the training.
* [Use Of Workers](https://datatalks.club/blog/ml-deployment-lambda.html#use-of-workers)
* Is there any loading speed difference for large models from S3 vs them being part of the image in ECR?
* I’m unaware of the performance difference at the moment, but we can test it out.
* Why do we need SAM? What are the main benefits?
* [Advantages of SAM](https://datatalks.club/blog/ml-deployment-lambda.html#advantages-of-sam)
* Alternatives to SAM: CloudFormation/Terraform, serverless framework
* What are the benefits of using Docker compared to the old way of deploying lambda functions?
* [Benefits of new updates in AWS Lambda](https://datatalks.club/blog/ml-deployment-lambda.html#benefits-new-aws)
### Resources
Workshops built on new AWS features:
* [AWS Lambda – Container Image tutorial](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/)
* [Deploying a DL Model on AWS](https://github.com/alexeygrigorev/aws-lambda-docker)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
The Docker-based workflow (multi-stage build) is based on the following articles, and adapted to the new AWS features:
* [MLinProduction’s Docker for Machine Learning series](https://mlinproduction.com/docker-for-ml-part-1/)
by [Luigi Patruno](https://mlinproduction.com/about/)
* [A Simple Docker-based Workflow For Deploying A Machine Learning Model](https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/)
by [Phil Winder](https://winderresearch.com/team/phil-winder/)
For more information on the use of Docker in Data Science & ML, check out these excellent posts:
* [MLinProduction’s Docker for Machine Learning series](https://mlinproduction.com/docker-for-ml-part-1/)
by [Luigi Patruno](https://mlinproduction.com/about/)
* [Docker Can Help You Become A More Effective Data Scientist](https://towardsdatascience.com/how-docker-can-help-you-become-a-more-effective-data-scientist-7fc048ef91d5)
by [Hamel Husain](https://hamel.dev/)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Arman Jabbari – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Arman Jabbari
Arman Jabbari is a Staff Data Scientist, specializing in marketplace design and leading Airport marketplaces at Lyft. Holding a Ph.D. in Industrial Engineering and Operations Research from UC Berkeley, he excels at challenging established assumptions and redefining traditional approaches.
[](https://linkedin.com/in/armanjabbari)
### Events
* The Trade-off Between Simplicity And Optimality In Problem-solving ([watch on youtube](https://www.youtube.com/watch?v=iSCd31T8S3M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Arseny Kravchenko – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Arseny Kravchenko
Arseny Kravchenko is a seasoned ML Engineer, mostly interested in computer vision problems. He’s been working in ML since 2015 on both individual contributor and leadership roles. Arseny used to be active in competitive ML and achieved Kaggle Master level.
[](https://linkedin.com/in/arsenyinfo)
[](https://github.com/arsenyinfo)
[](https://arseny.info/)
### Events
* Software Testing for Machine Learning Pipelines ([watch on youtube](https://www.youtube.com/watch?v=zrz6uj6Lr74)
)
* Building Scalable and Reliable Machine Learning Systems ([watch on youtube](https://www.youtube.com/watch?v=i-pIdekjUow)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Arpit Choudhury – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Arpit Choudhury
Arpit Choudhury is the founder of Data-led Academy, a go-to place for anybody, irrespective of their technical know-how, to learn how to work with data and to get expert, unbiased answers to data-related questions!
[](https://twitter.com/icanautomate)
[](https://linkedin.com/in/icanautomate)
[](https://dataled.academy/)
### Events
* Becoming a Data-led Professional ([watch on youtube](https://www.youtube.com/watch?v=8v5KpHWgyYw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-led-Professional---Arpit-Choudhury-e11mkgq)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Artemii Frolov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Artemii Frolov
Artemii Frolov is a TUM Master’s of Informatics graduate. His specialty is deep learning and computer vision and he is working in this field for 3+ years.
### Events
* Fighting Fraud with Triplet Loss ([watch on youtube](https://www.youtube.com/watch?v=1Jabakbryyk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ba Linh Le – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ba Linh Le
Ba Linh leads Frontline’s product offering. As Frontlines Chief Data Officer, she oversees the development of Lizzy, an AI-powered domestic abuse risk assessment tool. She’s got a background in social / political science and statistics, and previously worked as a project manager and data scientist in both the public and private sectors working for government institutions, NGOs and companies. As a social data scientist, her main interest lies in translating complex data into actionable insights to foster innovation and drive positive change.
[](https://twitter.com/balinhle1)
[](https://linkedin.com/in/ba-linh-le-)
### Events
* Building a Domestic Risk Assessment Tool ([watch on youtube](https://www.youtube.com/watch?v=CpWlBAmD9ok)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Atita Arora – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Atita Arora
Atita Arora is a seasoned and esteemed professional shaping the landscape of information retrieval systems. With a robust background spanning from her impactful contributions as a committer in various information retrieval and relevancy workbench projects including Apache OpenNLP and Quepid, to her insightful blogs on leveraging vectors in e-commerce search and an open source implementation called Chorus to pair with. Her commitment to calibrating the essence of “good” in information retrieval systems through OTB metrics and user-centric approaches reflects her dedication to enhancing user experiences. As a fervent believer in diversity and inclusion, Atita actively champions initiatives to foster inclusivity in the domains of search and data science. She’s a proud representative of the Women of Search Group and a member of various DEI forums, striving to create a more inclusive environment in tech. Currently immersed in groundbreaking research on evaluating RAGs, Atita continues to drive innovation and positive change in the realm of information retrieval while advocating for diversity, making her a true trailblazer in technology and inclusivity.
[](https://twitter.com/atitaarora)
[](https://linkedin.com/in/atitaarora)
[](https://github.com/atarora)
### Events
* Searching Beyond the Surface: Navigating Challenges and Innovations in Search Technologies ([watch on youtube](https://www.youtube.com/watch?v=_fbe1QyJ1PY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Barbara Sobkowiak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Barbara Sobkowiak
Data scientist by profession, GIS specialist by education, manager by passion. Analyzing data and requirements, searching for the best solutions, supporting and managing the team, i.e. combining soft and technical skills is my professional daily life.
[](https://linkedin.com/in/barbara-sobkowiak-1a4a9568)
### Events
* Data Science Manager vs Data Science Expert ([watch on youtube](https://www.youtube.com/watch?v=hFmIgaN-F8Y)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Science-Manager-vs-Data-Science-Expert---Barbara-Sobkowiak-e1ah3od)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Barr Moses – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Barr Moses
Barr Moses is CEO & Co-Founder of Monte Carlo, a data reliability company backed by Accel, GGV, Redpoint, and other top Silicon Valley investors. Previously, she was VP Customer Operations at customer success company Gainsight, where she helped scale the company 10x in revenue and among other functions, built the data/analytics team. Prior to that, she was a management consultant at Bain & Company and a research assistant at the Statistics Department at Stanford. She also served in the Israeli Air Force as a commander of an intelligence data analyst unit. Barr graduated from Stanford with a B.Sc. in Mathematical and Computational Science.
[](https://twitter.com/BM_DataDowntime)
[](https://linkedin.com/in/barrmoses)
[](https://www.montecarlodata.com/)
### Events
* Data Observability: The Next Frontier of Data Engineering ([watch on youtube](https://www.youtube.com/watch?v=TrMG1SOqZkQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Observability---Barr-Moses-evghmh)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Antje Barth – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Antje Barth
Antje Barth is a Developer Advocate for AI and Machine Learning at Amazon Web Services (AWS) based in Düsseldorf, Germany. She is co-author of the O’Reilly Book, “Data Science on AWS.”
Antje is also co-founder of the Düsseldorf chapter of Women in Big Data. She frequently speaks at AI and Machine Learning conferences and meetups around the world, including the O’Reilly AI and Strata conferences. Besides ML/AI, Antje is passionate about helping developers leverage Big Data, container and Kubernetes platforms in the context of AI and Machine Learning.
Previously, Antje worked in technical evangelism and solutions engineering at MapR and Cisco where she worked with many companies to build and deploy cloud-based AI solutions using AWS and Kubernetes.
[](https://twitter.com/anbarth)
[](https://linkedin.com/in/antje-barth)
[](https://github.com/antje)
### Books
* [Data Science on AWS](https://datatalks.club/books/20210628-data-science-on-aws.html)
(the book of the week from 28 Jun 2021 to 02 Jul 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Anthony Virtuoso – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Anthony Virtuoso
Anthony Virtuoso works as a Principal Engineer at Amazon and holds multiple patents in distributed systems, software defined networks, and security. In his eight years at Amazon, he has helped launch several Amazon Web Services, the most recent of which was Amazon Managed Blockchain. As one of the original authors of Athena Query Federation, you’ll often find him lurking on the Athena Federation GitHub repository answering questions and shipping bug fixes. When not at work, Anthony obsesses over a different set of customers, namely his wife and two little boys, aged 2 and 5. His kids enjoy doing science experiments with dad, like 3D printing toys, building with Lego, or searching the local pond for tardigrades.
[](https://twitter.com/anthonyvirtuoso)
[](https://linkedin.com/in/https://www.linkedin.com/in/avirtuos)
[](https://github.com/avirtuos)
### Books
* [Serverless Analytics with Amazon Athena](https://datatalks.club/books/20220328-serverless-analytics-with-amazon-athena.html)
(the book of the week from 28 Mar 2022 to 01 Apr 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Bartosz Mikulski – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Bartosz Mikulski
Bartosz Mikulski is an AI and data engineer. He specializes in moving AI projects from the good-enough-for-a-demo phase to production by building a testing infrastructure and fixing the issues detected by tests. On top of that, he teaches programmers and non-programmers how to use AI, contributed one chapter to the book, “97 Things Every Programmer Should Know”, and was a speaker at several conferences, including Data Natives, Berlin Buzzwords, and Global AI Developer Days.
[](https://linkedin.com/in/mikulskibartosz)
[](https://github.com/mikulskibartosz)
[](https://mikulskibartosz.name/blog/)
### Events
* Data Intensive AI ([watch on youtube](https://www.youtube.com/watch?v=BP6w_vKySN0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ashish Patel – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ashish Patel
Ashish Patel is a Senior Data Scientist, AI researcher, and AI Consultant with over seven years of experience in the field of AI, Currently living in Ahmedabad(INDIA). He has a Master of Engineering Degree from Gujarat Technological University and his keen interest and ambition to research in the following domains such as (Machine Learning, Deep Learning, Time series, Natural Language Processing, Reinforcement Learning, Audio Analytics, Signal Processing, Sensor Technology, IoT, Computer Vision). He is currently working as Senior Data Scientist for Cynet infotech Pvt Ltd. He has published more than 15 + Research papers in the field of Data Science with Reputed Publications such as IEEE. He holds Rank 3 as a kernel master in Kaggle. Ashish has immense experience working on cross-domain projects involving a wide variety of data, platforms, and technologies
[](https://twitter.com/imashish2604)
[](https://linkedin.com/in/ashishpatel2604)
[](https://github.com/ashishpatel26)
### Books
* [Hands-on Time Series Analysis with Python](https://datatalks.club/books/20230612-hands-on-time-series-analysis-with-python.html)
(the book of the week from 12 Jun 2023 to 16 Jun 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ben Taylor – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ben Taylor
Chief AI Evangelist @ DataRobot
[](https://linkedin.com/in/bentaylordata)
### Events
* The Essentials of Public Speaking for Career in Data Science ([watch on youtube](https://www.youtube.com/watch?v=wOFvlR9UBxI)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Essentials-of-Public-Speaking-for-Career-in-Data-Science---Ben-Taylor-et0m4p)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Bart Vandekerckhove – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Bart Vandekerckhove
As a co-founder and CEO at Raito, Bart wants to help data teams save time on data access management, so they can focus on innovation. As the former PM of Privacy at Collibra, Bart has seen first hand how slow data access management processes can harm innovation.
[](https://twitter.com/Bart_H_VDK)
[](https://linkedin.com/in/bartvandekerckhove)
### Events
* Data Access Management ([watch on youtube](https://www.youtube.com/watch?v=IiPOIiUy5b4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Bastien Boutonnet – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Bastien Boutonnet
Bastien Boutonnet is the Lead Data Scientist at Soda
Bastien Boutonnet is a neuroscientist by training who turned full-time data science and developer tooling nerd for the last 6 years. At Soda, he leads product domains around automated monitoring and data incidents resolution powered by machine learning and other if/elses machines. When he’s not working he likes to think about design and architecture, music production, and streaming his open-source contributions on Twitch.
[](https://twitter.com/b_superhero)
[](https://linkedin.com/in/bastienboutonnet)
[](https://github.com/bastienboutonnet)
[](https://www.bastienboutonnet.com/)
### Events
* Data Incident Management with Soda and dbt ([watch on youtube](https://www.youtube.com/watch?v=I3Fwhx1fw6M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Bela Wiertz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Bela Wiertz
Bela Wiertz is working for a German family office investing in VC Funds and early-stage startups with a focus on open-source Data, AI & Developer Tooling. In his work he is sourcing, evaluating and working with many different open-source companies in the early stages of finding their product-community-fit.
[](https://twitter.com/goldjuunge)
[](https://linkedin.com/in/bela-wiertz-a48710175)
### Events
* Investing in Open-Source Data Tools ([watch on youtube](https://www.youtube.com/watch?v=7Bg1JQLnCao)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Bhavani Ravi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Bhavani Ravi
Bhavani Ravi is a Software Engineer and AI Consultant who builds intelligent automation systems and data platforms. As a freelancer, she specializes in AI agents, workflow orchestration, and data pipelines, working with tools like Apache Airflow, MCP, n8n, and Prefect. Bhavani is also an Apache Airflow Champion, international tech speaker, and LinkedIn Learning instructor, teaching Google Cloud data engineering and advocating for open-source education. She leads TheLearningDev, a community helping developers deepen their expertise in data and AI.
[](https://linkedin.com/in/bhavanicodes)
### Events
* Building with MCP: Tools, Workflows, and Real Examples ([watch on youtube](https://www.youtube.com/watch?v=0IhZdcjddo4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Carmine Paolino – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Carmine Paolino
Carmine is the CTO and a co-founder of FreshFlow.
[](https://twitter.com/paolino)
[](https://linkedin.com/in/carminepaolino)
[](https://github.com/crmne)
[](https://paolino.me/)
### Events
* Launching a Startup: From Idea to First Hire ([watch on youtube](https://www.youtube.com/watch?v=s-w8_GDgIlU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Launching-a-Startup-From-Idea-to-First-Hire---Carmine-Paolino-e15sk4i)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Caitlin Moorman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Caitlin Moorman
Caitlin is the VP of Data and Business Operations at Trove Recommerce, where she helps brands buy back and resell their products at scale, to extend their useful life. She previously led data teams in crowdfunding and self-publishing.
[](https://linkedin.com/in/caitlin-moorman)
[](https://locallyoptimistic.com/post/author/caitlin/)
### Events
* Conquering the Last Mile in Data ([watch on youtube](https://www.youtube.com/watch?v=HfMpG2zpa2I)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Conquering-the-Last-Mile-in-Data---Caitlin-Moorman-e1958c1)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Boyan Angelov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Boyan Angelov
Boyan Angelov is a data strategist with a decade of experience in various academic and commercial environments such as bioinformatics, clinical trials, HRTech, LegalTech, and management consulting. He is the author of “Elements of Data Strategy” and “Python and R for the Modern Data Scientist”. He is also a researcher on the topics of XAI and progress studies and regularly speaks at conferences, meetups, and podcasts. Currently, he is leading the data strategy function at Exxeta AG, a large DACH technology and digital transformation consultancy.
[](https://twitter.com/thinking_code)
[](https://linkedin.com/in/angelovboyan)
[](https://github.com/boyanangelov)
[](https://boyanangelov.com/)
### Events
* Data Strategy: Key Principles and Best Practices ([watch on youtube](https://www.youtube.com/watch?v=jGbfeYdlCiQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ben Wilson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ben Wilson
Ben’s a Practice Lead Resident Solutions Architect at Databricks, based in North Carolina, USA. He’s been doing Data Science work for the past 12 years in companies ranging from semiconductor manufacturing to fashion companies. He’s working on a book for Manning titled “Machine Learning Engineering in Action” that focuses on how to get ML projects into production and help them stay there.
[](https://linkedin.com/in/benjamin-wilson-arch)
[](https://github.com/BenWilson2)
### Events
* DataTalks.Club Summer Marathon: Machine Learning in Production ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* Running from Complexity ([watch on youtube](https://www.youtube.com/watch?v=sMy8NYZnsy8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Running-from-Complexity---Ben-Wilson-e14np51)
)
### Books
* [Machine Learning Engineering in Action](https://datatalks.club/books/20210301-ml-engineering.html)
(the book of the week from 01 Mar 2021 to 05 Mar 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Starting a Career as a Data Scientist – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Starting a Career as a Data Scientist
Starting a Career as a Data Scientist
=====================================
### Tips on what your Data Science career could look like
23 Jul 2022 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
Photo by [Razvan Chisu](https://unsplash.com/@nullplus?utm_source=medium&utm_medium=referral)
on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)
Data science has become one of the most in-demand fields in recent years. As companies increasingly collect and generate data, they need individuals who can sift through this information to identify trends and make predictions.
While a data scientist’s job may vary depending on the industry, their responsibilities usually fall into one of three categories: Type A, Type B, and Type C. In this article, we’ll explore the three main areas that data scientists should focus on. In addition, we’ll describe a possible strategy to start a career in Data Science and some tips to learn new skills.
The article is organized as follows:
* Type A — the Analyst
* Type B — the Builder
* Type C — the Consultant
* Starting a career in Data Science
* Learning new skills
Type A — the Analyst
--------------------
Type A stands for Analyst. As the name suggests, these individuals excel at analyzing data. They’re great at **taking large amounts of data and making sense of it all.**
Analysts are often thought of as “traditional” data scientists. They are highly skilled in statistics and modeling, and they use these skills to extract insights from data. Analysts are often very good at communicating their findings to non-technical audiences since they understand both the technical details and the business implications of their work.
### Skills
Initially, data analysts came from very heavy research backgrounds. Over time, companies realized that these people were really rare. For this reason, their actual study background doesn’t matter as much today — it’s more about their work experience and practical skills. However, you can still find people with technical degrees in computer science or statistics working as data analysts, although in most cases, these people fall into the Type B category.
A data analyst performs **data analysis**, being able to look at insights, do data visualization, make some dashboards, and try and figure out what’s going on with the data. That’s a very large component of data science.
Another important skill for analysts is **programming**. Programming languages like R and Python are designed specifically for working with data. If you want to be able to manipulate data effectively, you need to know how to program.
Third, you need to have an **open mind**. Data science can be challenging, and it is important to be willing to learn new things. If you are closed-minded, you will likely find yourself struggling with data science.
Last but not least, a good data analyst should **be curious**. Once you have collected data, you should be curious about the data and start asking questions. Questions can lead to more questions, and this is how hypotheses are formed. For anyone who wants to improve their curiosity and keep learning more different things, you just have to make the decision that you want to learn more things. It’s as simple as that. Everyone can do their own version of curiosity.
Type B — the Builder
--------------------
Type B stands for Builder. Builders are essentially software engineers, focused on the realm of data science applications. They work in the cloud, set up the infrastructure, manage workflows, collaborate with the data engineering team to ensure all the pipelines are solid, and help other team members write clean code, unit tests, and functional tests to make sure that the code will run smoothly in production.
You can also think of them as the “new age machine learning engineers”. They build those systems and the Machine Learning ops, automated machine learning programs, and different things like that.
One of the key differences between the Building type and the Analyst type is that **the Building type always wants to work in the production environment**. There’s almost a different mindset.
### Skills
As a Type B scientist, you should know frameworks like Git, Docker, and cloud computing platforms. These are important tools for managing and analyzing data.
**Git** is a version control system that lets you track changes to your code. **Docker** is a tool that helps you package up your code so it can run in different environments. **Cloud computing platforms** like Amazon Web Services and Google Cloud Platform provide scalable resources for storing and processing data.
### Differences between Type A and Type B
Type A and Type B have a quite different mindset. Type A is more exploratory, and Type B focuses more on production.
The type A data scientist will be very happy to play around in Jupyter Notebooks: take data from a CSV file, explore and see what’s there, and try to apply some machine learning to it straight away — after figuring out what the problem is, of course. But the Type B data scientist doesn’t want to do that. They think of all of that stuff as a waste of time. They’re going to refactor it, to change it somehow.
The following figure shows the technical skills required for Types A and B:
Type C — the Consultant
-----------------------
Type C stands for “Consultant”. In the past, a data scientist was the intersection of the programmer, mathematician, and business domain expert. This type C is essentially that person in the middle, but with very strong consulting skills.
As the name suggests, consultants are all about giving advice. But unlike other types of data scientists, who tend to focus on either the technical aspects of data analysis or the business applications, consultants have really **strong stakeholder engagement and communication** to make sure that the business problems will be solved adequately with the right solutions.
Consultants are also experienced in working with a variety of different businesses, so they can quickly adapt to your company’s needs.
### Skills
Usually, type C scientists are either **people leaders** or **project managers**. Or they’re very good at persuading and talking to business. If you’re a strong talker, you’re more likely to be a people leader. Usually, the type C data scientist will be your manager of data science, your head of data science, or your chief data scientist.
There are different types of consultants as well. You can think of them as true consultants. They aren’t working very deep into the data. They’re talking about the **process to solve problems**, working closely with the stakeholders to figure out what the problems are and making sure that their concerns and issues are all addressed. But often we see that some leaders are not very technical.
The most important skills for the Type C data scientist include a **commercial mindset** with strong business acumen for making strategic decisions, people leadership abilities, effective communication with business stakeholders, the ability to influence and convince leadership, as well as strong storytelling capabilities.
Starting a career in Data Science
---------------------------------
Here are a few things to keep in mind if you’re wondering if a career in data science is right for you:
* **Data science is a team sport** — No matter how talented you are, you won’t be able to do everything on your own. Data science is a collaborative field, so you’ll need to be able to work well with others. This means being able to communicate effectively and being open to different perspectives.
* **The field is constantly changing** — If you’re the type of person who likes stability and hates change, data science may not be the right fit for you. The field is constantly evolving, so you’ll need to be comfortable with change. This means being open to new ideas and technologies and staying adaptable to new methodologies and tools.
In data science, it is often said that domain expertise is key. This is true to some extent — if you want to be a data scientist working on medical data, it helps to have a background in medicine. However, domain expertise is not always necessary to get started in data science.
### Where to start
As a fresher, breaking into the data science field can seem daunting. However, with the right attitude and some hard work, it is definitely possible to make a name for yourself in this exciting and ever-growing industry. Here are a few tips on how to get started:
1. **Learn Python or R.** These are the two most popular programming languages for data science and are relatively easy to learn compared to other languages.
2. **Get familiar with basic statistical concepts.** Understanding means, median, mode, variance, etc. will be helpful in your journey to becoming a data scientist.
3. **Start playing around with data!** Use public datasets to practice your new skills or try out different data analysis techniques. Kaggle is a great resource for finding datasets and participating in competitions.
4. **Learn Machine Learning.** This is a huge field of study within data science and can seem daunting at first, but there are plenty of resources available to help you get started (including our own course!).
5. **Network with the community.** Attend meetups and conferences, and connect with people already working in the field — they may be able to offer advice or even help you land your first job.
6. **Stay up to date with the latest news** and advancements in data science. Read blogs, listen to podcasts, and watch talks online — there are lots of ways to stay informed about what’s going on in the world of data science.
Learning new skills
-------------------
There are many skills that are needed in order to work as a data scientist. However, there are also many skills that can be learned in order to become a data scientist. Many people who are interested in data science may not have the opportunity to learn these skills at their job. However, there are many ways to learn the skills outside of work.
One way to learn the skills needed to become a data scientist is through **online courses**. There are many online courses that teach the basics of data science. These courses can be found on websites such as Coursera and Udacity. In addition, there are **many blog posts and articles** that can be found online that can help someone learn the basics of data science.
Another way to learn the skills needed to become a data scientist is through **conferences and meetups**. There are many conferences and meetups that discuss data science topics. These events can be found in cities all over the world. Attendees of these events can learn from presentations and network with other people who are interested in data science.
Finally, another way to learn the skills needed to become a data scientist is through **books**. There are many books that have been written about data science.
Summary
-------
In this article, we’ve explored the three main types of data scientists and how they differ in their approach and skillsets:
* **Type A (Analyst)** focuses on data analysis, visualization, and extracting insights
* **Type B (Builder)** focuses on production systems, infrastructure, and building scalable solutions
* **Type C (Consultant)** focuses on stakeholder engagement, business strategy, and leadership
We’ve also covered practical strategies for starting your data science career and various ways to continue learning and growing in this dynamic field.
If you’re interested in pursuing a career in data science, remember that it’s a team sport that requires continuous learning. Choose the path that aligns best with your strengths and interests, and don’t be afraid to explore different aspects of the field as you grow!
The content of this article has been inspired by the podcast episode [The ABC’s of Data Science](https://datatalks.club/podcast/s02e07-abc-data-science.html)
with Danny Ma at DataTalks.Club.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# The Essentials of Public Speaking for a Career in Data Science – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
The Essentials of Public Speaking for a Career in Data Science
The Essentials of Public Speaking for a Career in Data Science
==============================================================
### Some tips on how to boost your public speaking skills as a data scientist
31 Jul 2022 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
Photo by [Kane Reinholdtsen](https://unsplash.com/@kanereinholdtsen?utm_source=medium&utm_medium=referral)
on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral)
Data science is a relatively new field, and public speaking is an essential skill for anyone looking to build a career in this field. The ability to communicate complex data concepts clearly and effectively is essential for success in data science.
If you’re data-driven and looking to boost your career in science, public speaking is a great way to get noticed. But how exactly do you do that? We’ve got the essentials of public speaking for data science so you can make a splash at your next conference.
The article is organized as follows:
* Setting up your mindset
* Getting started
* Organizing a talk
* Top public speaking skills to acquire
Setting up your mindset
-----------------------
There are many different ways to get involved with data science, but one of the most important is **AI evangelism**. An AI evangelist is an “AI missionary”, who tries to convince you that your company needs AI. To be a little bit more specific, you can see AI evangelism as something sitting in between product and engineering and marketing.
As a data scientist, you have the opportunity to be a powerful voice in the promotion of artificial intelligence and its potential to change the world. This means that you have a responsibility to educate others about what AI is and how it can be used for good. You can do this by writing blog posts, giving talks, or even just having conversations with people who are interested in learning more.
No matter what platform you use, your goal should be to provide accurate information about AI and its potential implications. It’s also important to be respectful of different perspectives and open to debate. After all, data science is still a young field and there is a lot that we don’t yet understand about its potential.
By engaging in AI evangelism, you can help shape the future of this exciting field and ensure that it is used for the benefit of all.
Getting started
---------------
The most important thing you can do to become a great public speaker is to practice, practice, practice. The more you get up in front of an audience, the more comfortable you will become with speaking. Additionally, it is important to be well-prepared before each speaking engagement. Know your material inside and out, so that you can focus on delivering your message rather than worrying about what you are going to say next. Finally, remember that confidence is key. Radiate confidence and your audience will believe in you and your message.
More in detail, to become a great speaker, you should follow these steps:
* **Identify your niche**. What is it about data science that you’re passionate about? What are you an expert in? Once you know your focus, start pitching your ideas to conferences. Keep your pitch short and sweet-you don’t want to overwhelm the organizers with too much information.
* **Practice**. The more comfortable you are with your material, the better you’ll be able to engage with your audience. If possible, record yourself giving a talk and watch it back-this will help you identify any areas that need improvement.
* **Connect with people**. The more genuine and enthusiastic you are, the more likely you are to make a lasting impression on your listeners. So go out there and show the world what data science is all about!
In addition, you should take care a lot about the audience feedback, because it’s a point of reference. You need to consult with the audience after the talk, to find out. You’re not going to make everyone happy. You’re always going to have some people that didn’t like the talk or they’ve got some criticism.
### Organizing a talk
One of the most successful ways to organize a talk is by transforming it into a story. You can do that, by focusing on the following aspects:
* Getting attention
* Being remembered
* Audience feedback
The following figure summarizes how to organize a talk:
### Getting Attention
When it comes to giving a presentation or speech, the introduction is often the most important part. This is your chance to make a good first impression and set the tone for the rest of your talk.
When you start a talk, you have all the audience’s attention, but they are going to quickly decide if they’re going to pull out their phones, their laptops, or disengage. One of the most important things that make the difference is the first impression. You can think of your audience as being like cold taffy and you have to warm them up. You can warm them up in different ways, such as humor, storytelling, something surprising, and so on. There are a lot of emotions you can play with.
When you introduce yourself to your audience, you should give them a reason to listen to you. One strategy could be doing a resume overview, like, “I worked here, this is my title. A better approach, which is very difficult, is to immediately jump into a story where the audience concludes that you are the hero of that story. If the audience concludes that you are a hero, you have taken them on an emotional journey. If you’re in the hero territory, then the perceptions are very different. You’re perceived as being in the top 1%. You’re not lying. You’re not trying to pull the wool over people’s eyes, but you’re entertaining them with a story. To tell the hero’s journey, for example, you might pick an AI project in the last 6 months or 12 months, where you’ve nearly failed and then you might just dive into the story.
There’s an important alternative that you can do, and that is having someone else introduce you. A lot of times you’ll have a chair at a conference that will introduce you. But in most cases, they’re reading your bio. But if the chair reads your bio through a storytelling lens, and they say, “I’m so excited about the next speaker! He is one of the best,” and goes down the list, It changes everything. You can actually get the audience into hero territory, just with the chair’s introduction of you.
There are a few key things to keep in mind when crafting your introduction:
1. Make it short and sweet. The last thing you want to do is bore your audience with a long-winded intro. Get to the point and give them a taste of what’s to come.
2. Be confident. It’s perfectly normal to be nervous before a big speech, but don’t let it show. Speak slowly and deliberately, and project confidence in your voice and body language.
3. Hook them from the start. Start with a bang! Tell a joke, share an interesting fact, or ask a rhetorical question to engage your audience from the get-go.
4. Set the stage. Give some brief background information on the topic of your talk so that everyone is on the same page.
5. Tease what’s to come. whet your audience’s appetite by giving them a sneak peek of what’s in store for the rest of your presentation.
### Being remembered
You need to give the audience something for free — some learned insight, some key points. If they never see you again, they need to be able to leave and say, “Thank you. That is useful for me.” You need to identify what that is. Identify the warm-up, and then identify the key takeaways that they can have.
There are other tricks you can have in the talk where you can have a call to action, or something at the end where you try to engage people to go to a certain place, watch something or reach out to you specifically for further questions.
When you’re giving a presentation, you want to be remembered for your great ideas and insights, not for your nervous fidgeting or “um”s. Here are a few tips to help you make a lasting impression.
1. **Make eye contact**: It sounds simple, but making eye contact with your audience is one of the best ways to establish a connection with them.
2. **Use body language**: Your body language should convey confidence and ease. Avoid crossing your arms or legs, and try to keep your hands relaxed.
3. **Speak clearly**: Be sure to enunciate your words and speak at a moderate pace. This will help ensure that your audience understands you and doesn’t get lost in trying to decipher what you’re saying.
4. **Smile**: A genuine smile will make you seem approachable and likable — two qualities that will help make your presentation more successful.
5. **Be prepared**: One of the best ways to boost your confidence is to be thoroughly prepared for your presentation. Know your material inside and out so that you can focus on delivering it effectively rather than worrying about what comes next.
### Audience feedback
If someone from the audience asks you a question, it means that your presentation was successful.
During Question & Answer, if someone asks you a question and you don’t know the answer, or if you think the question is embarrassing, the temptation is to try to answer it. That’s really bad. Don’t try to answer a question you don’t know the answer to. If someone asks you a technical question, there’s a good chance that the textbook question is so technical that it’s actually not a lot of value for the audience.
You should just give up. You can say, “does anyone else in the audience know the answer?” or “let’s talk about this after. I’d love to hear your perspective.” And then go on to the next question where you actually can answer it.
### Writing a talk proposal for conferences
If you are starting out your speaking career, it’s easier for you to go land a local meetup, because you might be able to actually meet the meetup organizer. They’re always looking for interesting talks. If you can work with a local meetup organizer, get a talk there, and actually get a recording — such as a YouTube video — that’ll give you something you can share. Then you have a talk about yourself. You can even imagine a scenario where you could give a talk to no one like, “Hey, everyone listening” You’d organize a talk, you could go record a talk and post it to YouTube. You could share that, trying to pitch yourself into a meetup or pitching yourself into a conference. That way they know that you can give a talk. It’s also important to understand who the conference organizers are because they’re really the decision makers.
You need to fight sameness. You need to run away from sameness because humans are novelty-seeking creatures. When you submit your idea, try to be creative. If you’re interested in giving talks at data science conferences, you’ll need to write a talk proposal. This is a document that outlines your talk and why it would be a good fit for the conference.
Your talk proposal should include:
* A brief description of your talk
* The main theme or topics that will be covered
* Why do you think this would be a good fit for the conference?
* Any supporting materials you have (e.g. slides, demos, etc.)
Writing a strong proposal is essential to getting your talk accepted. Take some time to review the conference’s call for proposals and make sure your proposal aligns with their themes and goals. If you have any questions, don’t hesitate to reach out to the conference organizers.
There are often many different topics to choose from, so it’s important to pick one that will be both informative and engaging. Here are some good topics to begin your talks on:
* The current state of data science and where it’s headed
* The importance of data science in today’s world
* How data science can be used to solve real-world problems
* Interesting case studies or examples of data science in action
* Your own personal experiences with data science
Top public speaking skills to acquire
-------------------------------------
There are a lot of important skills to acquire when seeking a career in data science, but public speaking is definitely one of the most important. Being able to effectively communicate your findings and insights is crucial in this field, and it takes practice to hone this skill. Here are a few tips to help you get started:
* **Know your audience**. It’s important to tailor your message to those who will be listening. Consider their level of understanding and what you want them to take away from your presentation.
* **Be prepared**. This may seem obvious, but it’s worth repeating. Make sure you know your material inside and out so that you can deliver a confident and polished presentation.
* **Use simple language**. Data can be complex, but your presentation doesn’t need to be. Use clear and concise language that everyone can understand.
* **Engage your audience**. Keep your audience engaged by using stories, examples, and humor (when appropriate). no one wants to listen to a boring lecture, so make sure you keep things interesting!
* **Practice, practice, practice!** The more you practice, the better you’ll become at public speaking.
Summary
-------
Congratulations! You have just learned the essentials of public speaking for a career in Data Science!
Public speaking is an essential skill for anyone in the field of data science. Data scientists are often called upon to present their findings to clients, managers, and other stakeholders. Being able to deliver a clear and concise presentation can make all the difference in whether or not your audience understands and buys into your work. If you’re looking to advance your career in data science, start by honing your public speaking skills.
The content of this article has been inspired by the podcast episode [The Essentials of Public Speaking for Career in Data Science](https://datatalks.club/podcast/s02e10-public-speaking.html)
with Ben Taylor at DataTalks.Club.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained
DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained
=====================================================================
### Compare DevOps and MLOps across pipelines, drift monitoring, team roles, and maturity levels—and when to automate retraining.
15 May 2022 by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
DevOps fosters collaboration to automate software delivery ([image source](https://pixabay.com/it/photos/devops-attivit%c3%a0-commerciale-3155972/)
).
About ten years ago, the community realized that there was a barrier between the delivery and operations teams. On the one hand, there was the **development team, which was responsible for writing and testing the code**. On the other hand, **the operations team should make the code run in production**. Production literally means that the software is ready to be publicly used.
Moving software from development to production was a manual process that required a lot of time. To reduce this gap and save time while deploying software, a new culture began to spread: the DevOps culture, which aims to make people collaborate together.
**DevOps (Development and Operations) is a set of best practices that try to establish a collaboration between the development and operations teams, with the purpose of automatizing the software deployment process as much as possible.**
In the last ten years, much progress has been made in the DevOps sector, reaching a fairly high level of maturity. Many tools and platforms were born. To mention the most famous platforms, you can remember Docker and Kubernetes.
Recently, a new concept is spreading. It is MLOps, which means Machine Learning Operations. MLOps is a set of best practices that helps people to **deliver** and **maintain** Machine Learning models in production. The interesting aspect is that in MLOps you should not be able to only deliver Machine Learning models, but also maintain them.
At first, one might think that MLOps is just a new term for DevOps, with a special focus on Machine Learning. But in reality, the matter is not as trivial as it seems.
In this article, we try to give an answer to all those who are wondering if DevOps and MLOps are the same things or not.
The article is organized as follows:
* The DevOps and MLOps workflow
* DevOps and MLOps monitoring
* The DevOps and MLOps teams
* Maturity in DevOps and MLOps.
The DevOps and MLOps workflow
-----------------------------
The typical DevOps workflow includes three main steps:
* building the software
* testing the software
* delivering the software.
The idea is to connect the third phase with the first one, to make the process as much as possible automatic. The classical DevOps workflow is represented through the symbol of infinity, as shown in the following figure:
DevOps infinity loop: build, test, deliver, operate—continuously ([image source](https://pixabay.com/it/illustrations/devops-attivit%c3%a0-commerciale-3148393/)
).
The infinity symbol indicates that all the steps should be run endlessly.
The main artifact in a DevOps workflow is the software, usually provided in a container. The software is **static**, in the sense that it does not degrade over time. However, you could discover some bugs in the code, thus you should update it, following the DevOps workflow.
In an MLOps workflow, you do not have only the software as an artifact. **You also have the model and data.** Unfortunately, your data does not last forever, as well as the model. Your model may be subjected to degradation, both in terms of data drift and concept drift:
* **Data drift** means that data used to train your model are not suitable to represent reality because they are too old. In other words, actual data and data used to train the model have different distributions.
* **Concept drift** means that the relationship between the output variable and the input variable of your model is not valid anymore, thus there is a conceptual error in your model.
Both data drift and concept drift should generate **model retraining**.
This means that the MLOps workflow should also consider this aspect. The building step of the DevOps workflow should be extended to also include model training/model retraining.
The challenge of the MLOps workflow is that the final artifact, which includes the software, the model, and the data, is not static, but it changes (degrades) over time. So you should define a strategy to monitor it, and trigger the retraining process whenever it is needed.
The MLOps workflow should consider **model maintenance**, which includes the previously described steps. Thus, organizations should move from data-driven solutions to model-driven solutions.
DevOps and MLOps monitoring
---------------------------
Monitoring in the space of DevOps is very important. But in MLOps is even more important, because it should be able to trigger a retrain action.
In DevOps monitoring, you should have logs, system metrics, and business-specific metrics. The most popular tools to monitor your DevOps workflow are [Prometheus](https://prometheus.io/docs/introduction/overview/)
and [Grafana](https://grafana.com/grafana/?plcmt=footer)
.
In MLOps you continue to monitor the classical metrics already defined for DevOps. But you should also monitor other metrics, including data and concept drift, model accuracy, adversarial attacks to your model, fairness detection, and so on. As a monitoring tool, you could continue to use Prometheus, but you should extend it with new components for the specific task.
As additional (or specific monitoring tools), you can use some experimentation platforms, which permit you to track, monitor, and compare your experiments, as well as choose the best model to send to production. Some of the most popular tools in this field include [Comet](https://comet.ml/)
, [MLflow](https://mlflow.org/)
, and [Neptune](https://neptune.ai/)
.
Monitoring DevOps and MLOps should help to identify abnormal situations that call to action. **In an ideal situation, all actions should be performed automatically.**
The DevOps and MLOps teams
--------------------------
There are three different roles in DevOps:
* the **production operator**, who makes the pipeline work
* the **business manager**, who defines the requirements and the metrics, and
* the **developer**, who implements the code.
They work together to solve the same problem, that is make the software work in production, by guaranteeing continuous integration and continuous delivery.
In MLOps, you have again three roles, but the developer is a more specific figure, named **Machine Learning Engineering**, who is responsible for building the model.
Maturity in DevOps and MLOps
----------------------------
Maturity in both DevOps and MLOps measures the extent to which human intervention in software release and update processes has been replaced with total automation.
You can check at which level of DevOps and MLOps your organization is. DevOps does not define any specific levels of maturity. However, broadly, we can define the following levels of maturity in DevOps, as shown in the following figure:
DevOps maturity model: from initial and managed to measured and optimized.
The levels are:
* **Initial** — the Development and Operations teams are separated.
* **Managed** — intent of collaboration between the two teams, but only in the Operations team, there is some kind of automation.
* **Defined** — there is a collaboration between the two teams and some kind of automation.
* **Measured** — there is a better understanding of the process and automation.
* **Optimized** — the gap between the two teams disappears.
In MLOps, both Google and Microsoft have defined some levels of maturity:
* 3 levels of maturity from Google
* 5 levels from Microsoft
The two models are quite similar. In this article, we describe the Google’s ones, as shown in the following figure:
MLOps maturity: level 0 (manual), level 1 (pipelines, buttons), level 2 (automated triggers).
The levels are:
* **level 0** — you don’t have MLOps, you have a manual process, for deploying and monitoring a model. For instance, you write your code in Jupyter notebooks.
* **level 1** — you start introducing some automation. You introduce a **pipeline**. You have components that are validating the data you are expecting. You have the evaluation module, that controls the criteria you have set. Teams work together. Machine Learning engineers maintain the training job. Jupyter notebooks are moved to scripts. You produce metrics, you have **buttons to trigger retraining**.
* **level 2** —the retraining process is automatic. You do not monitor only system metrics, but also quality metrics. To detect if the model is degrading, you should have **triggers** that trigger the execution of the pipeline.
This triggering problem is still under discussion. There are two possible ways to trigger an event:
* **on a periodic basis** — this is the case of a scheduler, which periodically checks if something changes, and if so, it triggers a specific event.
* **when data changes** — there is not any scheduler. This aspect is still under investigation and remains an open challenge for the next years.
Conclusion
----------
DevOps and MLOps share many things in common, but they also have many differences.
**So to the question: DevOps and MLOps are the same, we can answer: No, they are not the same, although they share the same principles of collaboration between development and operations teams.**
Both in DevOps and in MLOps the mentality is the same and is based on the principle of automating all processes as much as possible, in order to move **from people to technology**.
This text was freely inspired by the interview with Theofilos Papapanagiotou and Alexey Grigorev, entitled The Rise of MLOps, and available in the [DataTalks.Club](https://datatalks.club/podcast/s02e04-mlops.html)
website as a podcast.
If you have read this far, maybe you’d be interested to know that DataTalks.Club is running a free course about MLOps. You can find out more about it [here](https://github.com/DataTalksClub/mlops-zoomcamp)
!
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Cathy Chen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Cathy Chen
Cathy Chen, CPCC, MA specializes in coaching tech leaders enabling development of their own skills in leading teams. She has held the role of technical program manager, product manager, and engineering manager. She has led teams in large tech companies and startups launching product features, internal tools, and operating large systems. Cathy has a BS in Electrical Engineering from UC Berkeley & MA in Organizational Psychology from Teachers College at Columbia University. Currently, Cathy lives with her partner in Pittsburgh, PA and works at Google in SRE.
[](https://linkedin.com/in/cathychen036)
### Books
* [Reliable Machine Learning](https://datatalks.club/books/20221121-reliable-machine-learning.html)
(the book of the week from 05 Dec 2022 to 09 Dec 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps
Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps
======================================================================================
### Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Learning data skills often involves a tradeoff: either pay for an expensive bootcamp or rely on free tutorials that seldom cover how real systems are built.
DataTalks.Club takes a different approach. We run free, fully developed courses, called **zoomcamps**, designed around building production-grade projects from the ground up. Instead of isolated lessons, you work through complete workflows using the same tools and processes practitioners use every day.
Overview of all six free DataTalks.Club zoomcamp courses: Machine Learning, Data Engineering, MLOps, LLM, AI Dev Tools, and Stock Market Analytics
We currently have six free courses:
* **[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
** on end-to-end ML engineering, from training models to packaging, testing, and deployment
* **[Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
** on data engineering, covering aspects like pipelines, warehouses, orchestration, and analytics infrastructure
* **[MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
** on MLOps or ML operations that involve monitoring, automation, and operationalizing ML models and systems in production
* **[LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
** on building AI applications with large language models (LLMs), combining them with RAG and using them to build AI agents
* **[AI Dev Tools Zoomcamp](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
** on using AI tools like AI chatbots, IDE integrations, and AI agents to accelerate the processes that involve working with code
* **[Stock Market Analytics Zoomcamp](https://pythoninvest.com/course)
** on stock market trading, Python programming in Colab, analytics, and data visualisation.
Across all zoomcamps you work with practical tooling like Python, Docker, cloud services, modern frameworks, and deliver portfolio-ready projects.
You can join live cohorts with weekly deadlines, scored homework, peer-reviewed projects, and a certificate, or learn independently at any time. All materials remain freely available, supported by an active global community of thousands of learners.
In the next sections, you’ll find detailed overviews of each course, including prerequisites, structure, tools, and projects.
DataTalks.Club Free Courses Table
---------------------------------
Here’s a table that summarizes the key information about each course. Use it to overview all courses and navigate to the course you’re interested in.
| Course | Level | Prerequisites | Key Topics | Tools/Tech Stack | What You'll Build |
| --- | --- | --- | --- | --- | --- |
| [Machine Learning Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#machine-learning-zoomcamp) | Beginner to Intermediate | 1+ year programming experience, command line basics | Regression, classification, trees, neural nets, deployment | Python, NumPy, Pandas, Scikit-Learn, TensorFlow, PyTorch, FastAPI, Docker, Kubernetes | Production ML models deployed as web services with Docker & cloud |
| [Data Engineering Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#data-engineering-zoomcamp) | Intermediate | Coding skills, command line, basic SQL | Pipelines, warehouses, orchestration, batch & streaming | Docker, Postgres, BigQuery, dbt, Apache Spark, Apache Kafka | Scalable data pipelines processing batch & streaming data |
| [MLOps Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#mlops-zoomcamp) | Intermediate to Advanced | 1+ year programming, ML exposure, Python, Docker | Experiment tracking, pipelines, deployment, monitoring | MLFlow, FastAPI, AWS, Mage, Evidently AI | Automated ML deployment system with monitoring & alerts |
| [LLM Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#llm-zoomcamp) | Intermediate | Python, command line, Docker (no ML required) | LLMs, RAG, vector search, evaluation, production deployment | OpenAI API, LangChain, Hugging Face, Ollama, Qdrant, Elasticsearch | AI chatbot that answers questions from your knowledge base |
| [AI Dev Tools Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#ai-dev-tools-zoomcamp) | Beginner to Intermediate | Basic programming (Python/JavaScript), no AI required | AI assistants, agents, MCP, automation, DevOps integration | GitHub Copilot, Cursor, MCP, Django, FastAPI, n8n, React | AI-powered development toolkit with assistants & agents |
| [Stock Markets Analytics Zoomcamp](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#analytics-in-stock-markets-zoomcamp) | Beginner to Intermediate | Basic Python, analytical mindset, interest in finance | Financial data, modeling, trading strategies, automation | Python, Pandas, NumPy, financial APIs, trading frameworks | Trading system that generates predictions & executes trades |
How DataTalks.Club Zoomcamps Work
---------------------------------
Before diving into the individual courses, it’s helpful to understand how the overall learning experience at DataTalks.Club is organized: what the core components are, how zoomcamps are structured, and what you can expect as a learner.
All zoomcamps follow the same core pattern:
* **Pre-recorded lectures on YouTube** you can watch anytime
* **Open-source materials on GitHub**, including code, notes, and homework
* **Two learning formats**: fully **self-paced** or joining a **live cohort**
* **A final project** you can include in your portfolio
Each course is divided into modules. Every module covers one topic, adds a hands-on task, and gradually prepares you for the final project.
The learning materials are identical across formats, but the experience differs:
* In **self-paced** mode, you start anytime and progress at your own speed.
* In a **live cohort**, you follow a weekly schedule with deadlines, peer-reviewed projects, and the option to earn a certificate.
Machine Learning Zoomcamp
-------------------------
Machine Learning Zoomcamp 2025 curriculum showing the learning progression from foundational ML algorithms (regression, classification, trees, neural networks) using Python, NumPy, Pandas, and Scikit-learn, through deep learning with TensorFlow and PyTorch, to production deployment with Docker, FastAPI, and Kubernetes
| | |
| --- | --- |
| Level | Beginner to Intermediate |
| Duration | 4-month live cohort (September-December) or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing projects and peer reviews |
| Prerequisites | 1+ year programming experience, command line basics |
| Key topics covered | Regression, classification, trees, neural nets, deployment |
| Tools/tech stack | Python, NumPy, Pandas, Scikit-Learn, TensorFlow, PyTorch, FastAPI, Docker, Kubernetes |
| Who it's for | Beginners with programming experience, aspiring ML engineers, career switchers, and software engineers who want practical, end-to-end ML experience |
| Projects | Production ML models deployed as web services with Docker & cloud |
| Outcomes | Hands-on experience with the full ML lifecycle and two portfolio-ready projects demonstrating your ability to build, deploy, and operate production ML systems |
| Next start date | New cohort starts every September |
[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
is a [free machine learning course](https://datatalks.club/blog/ultimate-list-of-20-free-online-courses-on-machine-learning.html)
that teaches you to build and deploy machine learning models in production environments. It bridges the gap between training models in notebooks and running them in real-world applications.
You’ll work with industry-standard tools including Scikit-Learn, TensorFlow, and PyTorch, and complete two portfolio-ready projects that demonstrate your ability to build, deploy, and operate production ML systems.
[Join Machine Learning Zoomcamp →](https://airtable.com/appflP5cuR8bD5MIm/shryxwLd0COOEaqXo)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
Data Engineering Zoomcamp
-------------------------
Data Engineering Zoomcamp 2026 9-week curriculum covering infrastructure setup with Docker and Terraform, workflow orchestration, data warehousing with BigQuery, analytics engineering with dbt, batch processing with Apache Spark, and stream processing with Apache Kafka
| | |
| --- | --- |
| Level | Intermediate |
| Duration | 9-week live cohort (January-March) or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing a project and peer reviews |
| Prerequisites | Coding skills, command line, basic SQL |
| Key topics covered | Pipelines, warehouses, orchestration, batch & streaming |
| Tools/tech stack | Docker, Postgres, BigQuery, dbt, Apache Spark, Apache Kafka |
| Who it's for | People preparing for junior data engineer roles (including beginners and career switchers), plus experienced professionals who want to refresh their skills and expand their network |
| Projects | A portfolio-ready project demonstrating your ability to build, deploy, and operate end-to-end data systems |
| Outcomes | Hands-on experience with production-grade data pipelines |
| Next start date | New cohort starts every January |
[Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
is a free data engineering course that teaches you how to build production-grade data pipelines from start to finish. It follows a clear progression: infrastructure setup, workflow orchestration, data warehousing, analytics engineering, batch processing, streaming, and a final project.
Throughout the 9-week program, you’ll master essential tools like Docker for containerization, PostgreSQL and BigQuery for data warehousing, dbt for analytics engineering, Apache Spark for batch processing, and Apache Kafka for stream processing. The course culminates in a capstone project where you build an end-to-end data pipeline that demonstrates your ability to design, implement, and operate scalable data systems used in production environments.
[Join Data Engineering Zoomcamp →](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
MLOps Zoomcamp
--------------
MLOps Zoomcamp course overview illustrating the complete machine learning operations lifecycle: experiment tracking with MLflow, orchestration and ML pipelines with Mage, model deployment with FastAPI and AWS, and production monitoring with Evidently AI
| | |
| --- | --- |
| Level | Intermediate to Advanced |
| Duration | 3-month live cohort or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing a project and peer reviews |
| Prerequisites | 1+ year programming, ML exposure, Python, Docker |
| Key topics covered | Experiment tracking, pipelines, deployment, monitoring |
| Tools/tech stack | MLFlow, FastAPI, AWS, Mage, Evidently AI |
| Who it's for | Data scientists, ML engineers, and software developers who are interested in understanding MLOps, the process of putting machine learning code in production |
| Projects | Automated ML deployment system with monitoring & alerts |
| Outcomes | Hands-on experience with the full MLOps lifecycle and a portfolio-ready project demonstrating your ability to build, deploy, and monitor production ML systems |
| Next start date | We don't plan to run a new cohort in 2026. You can register to stay updated if this changes |
[MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
is a free MLOps course that covers the entire MLOps lifecycle: from experiment tracking and model management to deployment and monitoring. Designed for data scientists and ML engineers, the course teaches you how to operationalize machine learning models at scale.
You’ll learn to use MLflow for experiment tracking, build automated training pipelines with Mage, deploy models using FastAPI and AWS services, and set up comprehensive monitoring with Evidently AI. The course emphasizes best practices for testing, CI/CD integration, and maintaining ML systems in production, preparing you to build and manage reliable ML infrastructure.
[Join MLOps Zoomcamp →](https://airtable.com/appYdhA23GVZd1iN2/shrCb8y6eTbPKwSTL)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
LLM Zoomcamp
------------
LLM Zoomcamp 10-week program on building AI applications with large language models, covering RAG (retrieval-augmented generation), vector databases, evaluation techniques, and production deployment using OpenAI, LangChain, Hugging Face, and Ollama
| | |
| --- | --- |
| Level | Intermediate |
| Duration | 10-week live cohort (June-August) or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing a project and peer reviews |
| Prerequisites | Python, command line, Docker (no ML required) |
| Key topics covered | LLMs, RAG, vector search, evaluation, production deployment |
| Tools/tech stack | OpenAI API, LangChain, Hugging Face, Ollama, Qdrant, Elasticsearch |
| Who it's for | Python developers and software engineers who want to build AI-powered applications with large language models, from beginners to those looking to add LLM skills to their toolkit |
| Projects | AI chatbot that answers questions from your knowledge base |
| Outcomes | Hands-on experience building production-ready RAG applications and a portfolio-ready project demonstrating your ability to create AI systems that interact with custom knowledge bases |
| Next start date | New cohort starts every June |
[LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
is a free course on building AI systems with large language models (LLMs). Over 10 weeks, you learn how to create an AI application that answers questions from your own knowledge base, from foundational concepts and retrieval-augmented generation (RAG) to evaluation, optimization, and production deployment.
The course covers both commercial APIs (OpenAI) and open-source models (Hugging Face, Ollama), teaches vector database integration with Qdrant and Elasticsearch, and shows you how to build production-ready RAG applications using LangChain. You’ll also learn modern agent patterns, evaluation techniques, and how to integrate LLMs with external tools and APIs for real-world applications.
[Join LLM Zoomcamp →](https://airtable.com/appPPxkgYLH06Mvbw/shr7WtxHEPXxaui0Q)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
AI Dev Tools Zoomcamp
---------------------
AI Dev Tools Zoomcamp 2025 6-week program on integrating AI into developer workflows, including coding assistants (GitHub Copilot, Cursor), Model Context Protocol (MCP), custom AI agents with Django, and no-code automation with n8n
| | |
| --- | --- |
| Level | Beginner to Intermediate |
| Duration | 6-week live cohort or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing a project and peer reviews |
| Prerequisites | Basic programming (Python/JavaScript), no AI required |
| Key topics covered | AI assistants, agents, MCP, automation, DevOps integration |
| Tools/tech stack | GitHub Copilot, Cursor, MCP, Django, FastAPI, n8n, React |
| Who it's for | Developers and engineers who want to explore how AI tools fit into their workflow, boost productivity with coding assistants, agents, and automation, and prefer project-based practice over theory-heavy tutorials |
| Projects | AI-powered development toolkit with assistants & agents |
| Outcomes | Hands-on experience building complete projects with AI tools and a portfolio demonstrating practical skills in applying AI to engineering work |
| Next start date | New cohort starts every November |
[AI Dev Tools Zoomcamp](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
is a free course on using AI for developer workflows. Over six weeks, you’ll go from experimenting with AI coding assistants to creating your own agent that scaffolds real applications.
The course covers the AI tool landscape (ChatGPT, Copilot, Cursor), teaches you to build end-to-end projects with React/TypeScript and FastAPI backends, introduces the Model Context Protocol (MCP) for connecting AI to GitHub and databases, and shows you how to build custom coding agents with Django. You’ll also learn to apply AI across the development lifecycle—from automated testing and PR reviews to CI/CD integration and no-code automation with n8n.
[Join AI Dev Tools Zoomcamp →](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
Analytics in Stock Markets Zoomcamp
-----------------------------------
Analytics in Stock Markets Zoomcamp 8-week data science program on financial markets, covering Python-based analysis with Pandas and NumPy, trading strategy development, backtesting, and building automated trading systems
| | |
| --- | --- |
| Level | Beginner to Intermediate |
| Duration | 8-week live cohort or self-paced anytime |
| Cost | Free |
| Certificate | Yes, after completing a project and peer reviews |
| Prerequisites | Basic Python, analytical mindset, interest in finance |
| Key topics covered | Financial data, modeling, trading strategies, automation |
| Tools/tech stack | Python, Pandas, NumPy, financial APIs, trading frameworks |
| Who it's for | Python developers and data enthusiasts interested in financial markets, trading strategies, and building automated trading systems |
| Projects | Trading system that generates predictions & executes trades |
| Outcomes | Hands-on experience with financial data analysis, trading strategy development, and building a semi-automatic trading system |
| Next start date | New cohort starts every April |
[Analytics in Stock Markets Zoomcamp](https://pythoninvest.com/course)
is a free data science course that covers data-driven decision-making, using popular Python libraries to work with financial data from data types and cleaning to hypothesis testing and making predictions.
You’ll learn to work with financial APIs, analyze market data using Pandas and NumPy, develop and backtest trading strategies, and build predictive models for stock market movements. You’ll also build a semi-automatic trading system that systematically generates predictions and executes trades, giving you hands-on experience with financial data analysis, risk management, and algorithmic trading concepts.
[Join Analytics in Stock Markets Zoomcamp →](https://pythoninvest.com/course)
[↑ Back to the courses table](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html#datatalksclub-free-online-courses)
Which Course Should You Choose?
-------------------------------
Choosing the right Zoomcamp depends on your background, career goals, and the kind of systems you want to build. All courses are project-based and free, but each one develops a different skill set.
### If you’re new to the field and want a broad, practical entry point
Choose between Machine Learning Zoomcamp and Data Engineering Zoomcamp:
* Machine Learning Zoomcamp: you’ll learn hands-on ML engineering experience, from models to deployment
* Data Engineering Zoomcamp: you’ll learn to work with data pipelines, warehouses, batch/streaming, and infrastructure
### If you’re already working with data or ML and want to advance your skills
Choose between MLOps Zoomcamp and LLM Zoomcamp:
* MLOps Zoomcamp: you’ll learn to productionize ML models and learn experiment tracking, orchestration, CI/CD, and monitoring
* LLM Zoomcamp: you’ll learn to build AI applications powered by large language models, using RAG, vector search, evaluation, and agents
### If you’re a developer looking to add AI to your workflow
Choose AI Dev Tools Zoomcamp: you’ll learn to use AI coding assistants, integrate MCP, build agents, and automate development tasks
### If you’re interested in financial data and trading systems
Choose Analytics in Stock Markets Zoomcamp: you’ll learn to analyze markets, test strategies, and build semi-automatic trading systems
### If you’re unsure where to start
If you’re unsure where to start, use this heuristic:
* Want to work with models? → ML Zoomcamp
* Want to work with data pipelines? → DE Zoomcamp
* Want to operationalize models and run ML in production? → MLOps Zoomcamp
* Want to build with LLMs and AI tools? → LLM or AI Dev Tools Zoomcamp
You can also start with one course in self-paced mode and join a live cohort later. Many learners complete multiple Zoomcamps as they grow in their careers.
DTC Zoomcamps vs. Bootcamps
---------------------------
DataTalks.Club Zoomcamps offer a different model compared to traditional tech bootcamps. Both aim to teach practical, job-relevant skills, but the structure, cost, and learning approach vary significantly.
| Category | Zoomcamps | Bootcamps |
| --- | --- | --- |
| Cost | Completely free. No upsells, no paid tiers, no expiring access | Typically $2,000-$12,000+, sometimes with financing or income-share agreements |
| Format | • Pre-recorded lectures on YouTube
• Open-source materials on GitHub
• Self-paced or live cohort options
• Weekly deadlines only in cohort mode | • Live classes, fixed schedules
• Limited access to materials after graduation
• Less flexibility for people working full-time |
| Curriculum | • Deep focus on building real, production-ready systems
• Tools directly used in industry (Docker, dbt, Spark, FastAPI, Kubernetes, etc.)
• Course content updated yearly by the community | • Often broader but shallower
• Less emphasis on production infrastructure
• Content quality varies widely by provider |
| Portfolio & Projects | • Capstone projects designed to simulate real engineering tasks
• Peer review and community feedback
• Projects remain fully yours and live on GitHub | • Projects usually smaller, more guided
• Often not production-grade |
| Community | • Large, active Slack community (80k+)
• Weekly discussions, Q&A bots, and peer support
• Contributions from real practitioners | • Smaller cohort-based communities
• Some vanish after graduation |
| Certificates | Free certificates available only after real project completion | Certificate included but not always backed by strong practical projects |
What Students Say About the Zoomcamps
-------------------------------------
> Machine Learning Zoomcamp was exhaustive, with comprehensive content that covered concepts in depth. You can learn everything from the simplest concepts to preparing and deploying an ML model for production. Additionally, the community behind this course is highly participative and collaborative. I would like to thank Alexey Grigorev for all the knowledge he shared with us and his team for providing the support we needed to solve each problem we faced.
>
> — [Alexander Daniel Rios](https://www.linkedin.com/in/alexander-daniel-rios)
> , ML Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/alexander-daniel-rios_mlzoomcamp-activity-7295527609239584768-TWHh)
> )
> Machine Learning Zoomcamp has been an incredible journey, thanks to the expert guidance of Alexey Grigorev. I’m thankful for this programme, which provided challenging coursework that is taught in a very structured and lucid way. The assignments & hands-on projects instill a sense of delivery, besides equipping us with practical acumen to solve real-life problems.
>
> — [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi)
> , ML Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/activity-7299906113997524994-R-oD?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> Thank you for what you do! The Data Engineering Zoomcamp gave me skills that helped me land my first tech job.
>
> — [Tim Claytor](https://www.linkedin.com/in/claytor/)
> , Data Engineering Zoomcamp course graduate ([Source](https://www.linkedin.com/feed/update/urn:li:activity:7396882073308938240?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7396882073308938240%2C7396889959711793152%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287396889959711793152%2Curn%3Ali%3Aactivity%3A7396882073308938240%29)
> )
> Three months might seem like a long time, but the growth and learning during this period are truly remarkable. It was a great experience with a lot of learning, connecting with like-minded people from all around the world, and having fun. I must admit, this was really hard. But the feeling of accomplishment and learning made it all worthwhile. And I would do it again!
>
> — [Nevenka Lukic](https://www.linkedin.com/in/nevenka-lukic/)
> , Data Engineering Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/nevenka-lukic_data-engineering-zoomcamp-final-project-activity-7181985646033461248-Lc1O?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> Such a fun deep dive into data engineering, cloud automation, and orchestration. I learned so much along the way. Big shoutout to Alexey Grigorev and the DataTalksClub team for the opportunity and guidance throughout the 3 months of the free course.
>
> — [Assitan NIARE](https://www.linkedin.com/in/assitan-niar%C3%A9-data/)
> , Data Engineering Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/activity-7317441554023874561-E3wm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> If you’re serious about breaking into data engineering, start here. The repo’s structure, community, and hands-on focus make it unparalleled.
>
> — [Wady Osama](https://www.linkedin.com/in/wadyosama/)
> , Data Engineering Zoomcamp course graduate ([Source](https://www.linkedin.com/posts/wadyosama_dataengineering-zoomcamp-dezoomcamp-activity-7292126824711520258-puJm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
Frequently Asked Questions
--------------------------
Why DataTalks.Club courses are called Zoomcamps?
Zoomcamp is a term that originated from [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, the founder of DataTalks.Club. It started with his book “ML Bookcamp.” When Alexey decided to create a video course based on the book, he called it “[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
” - a free, cohort-based course in video format. The name “zoomcamp” is a play on “bookcamp,” referring to the video format of the course. The Zoomcamp series has since expanded to include other free courses like the [Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
, [MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
, and [LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
, all following the same community-driven, open-source philosophy.
Are DataTalks.Club Zoomcamps free?
Yes. All DataTalks.Club courses (Zoomcamps) are completely free. You get full access to the GitHub repositories, YouTube playlists, homework, and the Slack community without paying anything. The only investment required is your time and commitment.
Do I need to attend live sessions?
No. All lectures are pre-recorded and available on YouTube, so you can watch them whenever you like. Occasionally, optional live workshops or Q&A sessions are announced in [Slack](https://datatalks.club/slack.html)
and the newsletter, but they are not required to complete the course.
How do I earn a certificate?
Certificates are available only during live cohorts. To earn one, you need to complete the required project(s) and review other students’ submissions before the deadlines. Self-paced learners have full access to the course but are not eligible for certificates.
Are the certificates recognized by employers?
Certificates are not formally accredited, but they validate that you completed a real project using modern tools. Your GitHub portfolio and project work carry the most weight—many learners use their Zoomcamp projects to showcase practical skills in job applications.
How much time should I dedicate per week?
For a live cohort, plan for around 10-15 hours per week to watch lectures, complete homework, and work on the project. Self-paced learners can adjust the workload to their schedule, but more weekly time generally leads to better progress and outcomes.
What programming languages do I need to know?
Python is the main programming language used across all Zoomcamps. The exact skill level depends on the course—each course section in this guide lists its prerequisites.
What's the difference between a DataTalks.Club Zoomcamp and a typical bootcamp?
Zoomcamps offer a similar project-based, structured learning experience but are completely free and open source. Bootcamps charge between $2,000 and $10,000+, often restrict access to materials, and follow strict schedules. Zoomcamps are flexible and community-driven—no content paywalls, no time limits, and no tuition fees.
How do I join the DataTalks.Club Slack community?
You can request an invite via the [Slack signup form](https://datatalks.club/slack.html)
. After confirming your email, you’ll receive an invitation link. Slack is where discussions happen, questions are answered, and announcements for cohorts and workshops are posted.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Christine Cepelak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Christine Cepelak
Christine Cepelak is a writer and researcher of tech and social issues. She’s currently studying Data Science for Public Policy and previously spent years managing social programs and exploring data science for social good.
[](https://twitter.com/CLcep)
[](https://linkedin.com/in/christinecepelak)
[](https://github.com/ccepelak)
[](https://christinecepelak.com/)
### Events
* Data Science for Social Impact ([watch on youtube](https://www.youtube.com/watch?v=xWC1HAfekRk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Chris Fregly – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Chris Fregly
Chris Fregly is a Developer Advocate for AI and Machine Learning at Amazon Web Services (AWS) based in San Francisco, California. He is co-author of the O’Reilly Book, “Data Science on AWS.”
Chris is also the Founder of many AI-focused global meetups including the global “Data Science on AWS” Meetup. He regularly speaks at AI and Machine Learning conferences across the world including O’Reilly AI, Open Data Science Conference (ODSC), and Nvidia GPU Technology Conference (GTC).
Previously, Chris was Founder at PipelineAI where he worked with many AI-first startups and enterprises to continuously deploy ML/AI Pipelines using Spark ML, Kubernetes, TensorFlow, Kubeflow, Amazon EKS, and Amazon SageMaker.
[](https://twitter.com/cfregly)
[](https://linkedin.com/in/cfregly)
[](https://www.datascienceonaws.com/)
### Books
* [Data Science on AWS](https://datatalks.club/books/20210628-data-science-on-aws.html)
(the book of the week from 28 Jun 2021 to 02 Jul 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Chip Huyen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Chip Huyen
I’m Chip Huyen, a writer and computer scientist. I grew up chasing grasshoppers in a small rice-farming village in Vietnam.
I’m a co-founder of a streaming-first platform for real-time machine learning. Previously, I built machine learning tools at NVIDIA, Snorkel AI, Netflix, and Primer.
I graduated from Stanford University, where I currently teach CS 329S: Machine Learning Systems Design. I’m also the author of the book Designing Machine Learning Systems (O’Reilly, 2022).
LinkedIn included me among Top Voices in Software Development (2019) and Top Voices in Data Science & AI (2020).
In my free time, I travel and write. After high school, I went to Brunei for a 3-day vacation which turned into a 3-year trip through Asia, Africa, and South America. During my trip, I worked as a Bollywood extra, a casino hostess, and a street performer.
[](https://twitter.com/chipro)
[](https://linkedin.com/in/chiphuyen)
[](https://github.com/chiphuyen)
[](https://huyenchip.com/)
### Books
* [Designing Machine Learning Systems](https://datatalks.club/books/20220627-designing-machine-learning-systems.html)
(the book of the week from 27 Jun 2022 to 01 Jul 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Alexey Grigorev – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Alexey Grigorev
Alexey Grigorev is the founder of DataTalks.Club
[](https://twitter.com/Al_Grigor)
[](https://linkedin.com/in/agrigorev)
[](https://github.com/alexeygrigorev)
[](https://alexeygrigorev.com/)
### Articles
* [20+ Best Data Science Slack Communities to Join in 2025](https://datatalks.club/blog/slack-communities.html)
* [Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities](https://datatalks.club/blog/data-roles.html)
* [Interview with Valerii Chetvertakov](https://datatalks.club/blog/interview-with-valerii-chetvertakov.html)
* [Interview with Ken Wu](https://datatalks.club/blog/interview-with-ken-wu.html)
* [MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices](https://datatalks.club/blog/mlops-10-minutes.html)
### Events
* Roles in a Data Team ([watch on youtube](https://www.youtube.com/watch?v=2ZOnA19sDpM)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Roles-in-a-data-team---Alexey-Grigorev-emqcft)
)
* Processes in a Data Science Project ([watch on youtube](https://www.youtube.com/watch?v=SesVTDklFYQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Processes-in-a-Data-Science-Project---Alexey-Grigorev-encdlg)
)
* DataTalks.Club Behind the Scenes ([watch on youtube](https://www.youtube.com/watch?v=IxTyq96juVE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/DataTalks-Club-Behind-the-Scenes---Eugene-Yan--Alexey-Grigorev-e1d4567)
)
* Bring Your Own Data ([watch on youtube](https://www.youtube.com/watch?v=POGiLFWxQWQ)
)
* Introduction to MLOps and MLOps Zoomcamp ([watch on youtube](https://www.youtube.com/watch?v=o34Q_61iA4Y)
)
* Introduction to ML Engineering and ML Zoomcamp ([watch on youtube](https://www.youtube.com/watch?v=a7phcSmuNY0)
)
* DataTalks.Club Anniversary Interview ([watch on youtube](https://www.youtube.com/watch?v=nCqwZT9zA0M)
)
* Introduction to Data Engineering Zoomcamp ([watch on youtube](https://www.youtube.com/watch?v=91b8u9GmqB4)
)
* Stock Market Analytics Zoomcamp Pre-Launch Q&A ([watch on youtube](https://www.youtube.com/watch?v=oswTLnjkRUg)
)
* Chat with Your Own Data: Introduction to the LLM Zoomcamp ([watch on youtube](https://www.youtube.com/watch?v=q-p36Ak6YI8)
)
* MLOps Zoomcamp 2024 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=YmllO3ld5LE)
)
* Implement a Search Engine ([watch on youtube](https://www.youtube.com/watch?v=nMrGK5QgPVE)
)
* LLM Zoomcamp 2024 Pre-Course Live Q&A ([watch on youtube](https://datatalks.club/people/alexeygrigorev.html)
)
* ML Zoomcamp 2024 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=htsvF-QUJDc)
)
* DataTalks.Club Anniversary Podcast ([watch on youtube](https://www.youtube.com/watch?v=GHbeXIKnkLQ)
)
* Using GenAI to Create Horror Stories ([watch on youtube](https://www.youtube.com/watch?v=DvhdJWqE47gyoutube)
)
* Stock Market Analytics Zoomcamp 2025 - Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=1N1jIwa1uag)
)
* MLOps Zoomcamp 2025 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=rv43YJQsZIw)
)
* LLM Zoomcamp 2025
Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=8lgiOLMMKcY)
)
* LLM Zoomcamp Course Launch ([watch on youtube](https://www.youtube.com/watch?v=FgnelhEJFj0)
)
* MLOps Zoomcamp Competition - Bot or Not? ([watch on youtube](https://www.youtube.com/live/ZxUVBG4z5uE)
)
* Open-Source LLM Zoomcamp 2025
Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=mmUyeyAPVnU)
)
* From RAG to Agents: Making Smart AI Assistants (LLM Zoomcamp bonus module) ([watch on youtube](https://www.youtube.com/watch?v=GH3lrOsU3AU)
)
* DataTalks.Club Summer 2025 AI Hackathon Launch ([watch on youtube](https://www.youtube.com/watch?v=APtJ1SDGdw0)
)
* Introduction to Vibe Coding: Build a Game with AI ([watch on youtube](https://www.youtube.com/watch?v=NSMXQk4Axig)
)
* Build Agentic Assistants with OpenAI Function Calling: Part 2 ([watch on youtube](https://www.youtube.com/watch?v=yS_hwnJusDk)
)
* Deploying ML Models with FastAPI and uv ([watch on youtube](https://www.youtube.com/watch?v=jzGzw98Eikk)
)
* Build an AI Coding Agent ([watch on youtube](https://www.youtube.com/watch?v=-XLgk1O421I)
)
* Deploying ML Models with AWS Lambda (Serverless) ([watch on youtube](https://www.youtube.com/watch?v=sHQaeVm5hT8)
)
* ML Zoomcamp 2025 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=ph1PxZIkz1o)
)
* ML Zoomcamp 2025 Course Launch ([watch on youtube](https://www.youtube.com/watch?v=z064DoidiKg)
)
* Deploying ML Models with Kubernetes ([watch on youtube](https://www.youtube.com/watch?v=c_CzCsCnWoU)
)
* Deep Learning with PyTorch ([watch on youtube](https://www.youtube.com/watch?v=Ne25VujHRLA)
)
* AI Dev Tools Zoomcamp 2025 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=sUwrCnP2iGU)
)
* AI Dev Tools Zoomcamp 2025 Course Launch ([watch on youtube](https://www.youtube.com/watch?v=58pn873XO04)
)
* Docker for Data Engineering: Postgres, Docker Compose, and Real-World Workflows ([watch on youtube](https://www.youtube.com/watch?v=lP8xXebHmuE)
)
* Data Engineering Zoomcamp 2026 Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=WB6b1lcguaA)
)
* Durable Agentic Workflows with Temporal.io ([watch on youtube](https://www.youtube.com/watch?v=N1gaI3Qz6vw)
)
* Data Engineering Zoomcamp 2026 Course Launch ([watch on youtube](https://datatalks.club/people/alexeygrigorev.html)
)
### Books
* [Machine Learning Bookcamp](https://datatalks.club/books/20201214-ml-bookcamp.html)
(the book of the week from 14 Dec 2020 to 18 Dec 2020)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Christiaan Swart – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Christiaan Swart
Chris Swart has 6 years of experience delivering Natural Language Processing (NLP) services across the email, complaint, pharma, and sales industries. He has an interest in cost effective dataset creation with distant supervision and building semi-supervised datasets to get the best bang for buck for models. He cofounded Comtura on a mission to help sales teams weponise their customer’s voices to sell more. At Comtura he leads a machine learning team of 3.
[](https://twitter.com/swartchris8)
[](https://linkedin.com/in/christiaan-swart-51a68967)
[](https://github.com/swartchris8)
[](https://swartchris8.github.io/)
### Events
* Dataset Creation and Curation ([watch on youtube](https://www.youtube.com/watch?v=QggWydGrWoo)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Christopher Bergh – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Christopher Bergh
Christopher Bergh is a veteran data and technology leader with more than 25 years of experience spanning research, software engineering, analytics, and executive management. As CEO and Head Chef at DataKitchen, Chris has helped define and popularize the field of DataOps—a methodology that streamlines data engineering, analytics, and AI delivery through automation and agile principles. Under his leadership, DataKitchen has become a trusted name in data observability, data quality, and continuous data operations, empowering enterprises to accelerate insight delivery while reducing risk and cost.
Chris is the co-author of the DataOps Cookbook and the DataOps Manifesto, foundational works that have shaped how organizations approach data pipelines and analytics governance. A recognized industry thought leader, he frequently speaks at major conferences and contributes to advancing best practices in data management, observability, and DevOps-inspired analytics workflows.
Earlier in his career, Chris held senior technology and leadership roles including COO, CTO, and VP of Engineering. He began his career at MIT Lincoln Laboratory and NASA Ames Research Center, where he developed algorithms that optimized aircraft arrivals at major U.S. airports. He also served as a Peace Corps volunteer math teacher in Botswana—an experience that continues to inform his people-first leadership philosophy. Chris holds an M.S. from Columbia University and a B.S. from the University of Wisconsin–Madison.
Through DataKitchen and his advocacy, Chris continues to inspire data professionals worldwide to embrace DataOps as the foundation for fast, reliable, and collaborative analytics.
[](https://twitter.com/ChrisBergh)
[](https://linkedin.com/in/chrisbergh)
### Events
* Storytime for DataOps ([watch on youtube](https://www.youtube.com/watch?v=0Fx5PCoLkf4)
)
* DataOps, Observability, and The Cure for Data Team Blues ([watch on youtube](https://www.youtube.com/watch?v=HzGpIxV8HtA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dânia Meira – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dânia Meira
Dania holds a Master’s degree in Computer Science with a specialisation in Artificial Intelligence from the Unversidade Federal Fluminense in Sao Paolo, Brazil. She has since worked as a data scientist and in teaching roles, and is a founding member and director of the AI guild. She is also a recognized speaker in the field of data science, machine learning and AI.
[](https://twitter.com/meiradania)
[](https://linkedin.com/in/meiradania)
[](https://github.com/meiradania)
### Events
* Accelerating the Adoption of AI through Diversity ([watch on youtube](https://www.youtube.com/watch?v=SRUwwvk_YCk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# CJ Jenkins – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 CJ Jenkins
I am a self-driven PhD currently working as a data scientist lead. I’m focused on credit risk modeling and integrating data-driven decisions into our product to reduce risk and enhance customer experience. With a strong background in statistical analyses, I utilize my expertise across programming languages (SQL, R, and Python). I have excellent communication skills and have been invited to present my research and represent my Universities at senior academic conferences; I have also written a number of well-cited peer-reviewed papers and a textbook currently used in academic curriculum. I am enthusiastic and committed to turning questions into answers using innovative solutions with a commercial approach
[](https://linkedin.com/in/christina-jenkins)
### Events
* Moving from Academia to Industry ([watch on youtube](https://www.youtube.com/watch?v=m4F651BpUFk)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Moving-from-Academia-to-Industry---CJ-Jenkins-e1bh84o)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Christian Winkler – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Christian Winkler
Christian Winkler is a Data Scientist and Machine Learning Architect. He holds a PhD in theoretical physics and has been working in the field of large data volumes and artificial intelligence for 20 years, with particular focus on scalable systems and intelligent algorithms for mass text processing. He is founder of datanizing GmbH, speaker at conferences and author of Machine Learning / Text Analytics articles.
[](https://linkedin.com/in/drchristianwinkler)
[](https://datanizing.com/)
### Books
* [Blueprints for Text Analytics Using Python](https://datatalks.club/books/20211018-blueprints-for-text-analytics-using-python.html)
(the book of the week from 18 Oct 2021 to 22 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Cristian Martinez – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Cristian Martinez
Cristian Martinez works as Lead Data Scientist at OLX Group, mainly focused on Search and Recommenders, and has been working for more than a decade in different companies solving business problems with Machine Learning.
[](https://twitter.com/mac2bua)
[](https://linkedin.com/in/cristian-javier-martinez-09bb7031)
### Events
* Deep Learning Recommender Systems ([watch on youtube](https://www.youtube.com/watch?v=LWAQUgJOYm0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Daliana Liu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Daliana Liu
Daliana is a senior data scientist at Amazon AI with over 7 years of experience.
She writes about data science career advice, interview prep, and study materials she enjoys. She likes to share her success stories and her embarrassing mistakes so we can avoid them.
[](https://twitter.com/dalianaliu)
[](https://linkedin.com/in/dalianaliu)
[](https://dalianaliu.com/)
### Events
* DataTalks.Club Summer Marathon: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dan Becker – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dan Becker
Dan started his data science career by finishing 2nd (out of 1353 teams) in a kaggle competition with a $500,000 grand prize. Since then, he’s worked as a data scientist at Google and was Product Director in the early days of DataRobot.
Dan has contributed to both the TensorFlow and Keras deep learning libraries, and over 250,000 people have taken his online deep learning courses. Dan has led data science consulting projects for 6 companies in the Fortune 100, and he is now the Founder and CEO of Decision AI.
[](https://twitter.com/dan_s_becker)
[](https://linkedin.com/in/dansbecker)
[](https://github.com/dansbecker)
[](https://www.decision.ai/)
### Events
* DataTalks.Club Conference: Product and Process ([watch on youtube](https://www.youtube.com/watch?v=dvzPU43tqFM)
)
* Translating ML Predictions Into Better Real-World Results with Decision Optimization ([watch on youtube](https://www.youtube.com/watch?v=SJuzQ4bcU2c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Translating-ML-Predictions-Into-Better-Real-World-Results-with-Decision-Optimization---Dan-Becker-eqk0b1)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Daniel Egbo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Daniel Egbo
Daniel Egbo is an astrophysicist turned machine learning engineer and AI ambassador (Arize, Tavily). A PhD candidate at the University of Cape Town, he builds end-to-end ML and LLM applications with a focus on reliability and learning in public. His work spans knowledge-retrieval assistants, practical evaluation, and applying data science to astronomy.
[](https://linkedin.com/in/egbodaniel)
[](https://github.com/Danselem)
### Events
* From Astronomy to Applied ML ([watch on youtube](https://www.youtube.com/watch?v=b92gwrsVQtg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Daniel Svonava – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Daniel Svonava
Daniel is an entrepreneurial technologist with a 20-year career starting with competitive programming and web development in high school, algorithm research, and Google & IBM Research internships during university, first entrepreneurial steps with his computational photography startup, and a 6-year tenure as a tech lead for ML infrastructure at YouTube Ads, where his ad performance forecasting engine powers the purchase of $10B of ads per year.
Presently, Daniel is a co-founder of Superlinked.com - an ML infrastructure startup that makes it easier to build information-retrieval-heavy systems - from Recommender Engines to Enterprise-focused LLM apps.
[](https://linkedin.com/in/svonava)
[](https://github.com/svonava)
### Events
* Building Production Search Systems ([watch on youtube](https://www.youtube.com/watch?v=gEmSrknGKDE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers
AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers
====================================================================
### Learn how to integrate AI into real developer workflows, from AI coding assistants to agents, CI/CD, DevOps, and no-code automation.
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
AI development tools have moved into everyday engineering work. Chat applications like ChatGPT and Claude, coding assistants like GitHub Copilot and Cursor, project bootstrappers like Bolt and Lovable, agents, and automation platforms like n8n have shifted the way we code and automate our workflows. The ecosystem is broad, and it’s not always clear which tools to adopt or how to use them reliably in real projects.
AI Dev Tools Zoomcamp 2025 course cover
AI Dev Tools Zoomcamp is a free, project-based course that helps you build a practical toolkit for this stack. Over six modules, you’ll explore vibe coding and the AI tool landscape, ship a simple end-to-end project with React and FastAPI, extend assistants with the Model Context Protocol (MCP), build your own Django coding agent, apply AI to testing and DevOps, and automate workflows with n8n.
By the end, you’ll have hands-on experience applying AI to everyday engineering tasks, plus a small portfolio and a certificate that shows how you work with modern AI dev tools.
[Join the next cohort (starts November 18, 2025) →](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
Table of Contents
-----------------
* [Who the Course Is For](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#who-the-course-is-for)
* [Course Curriculum](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#course-curriculum)
* [How AI Dev Tools Zoomcamp Works](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#how-ai-dev-tools-zoomcamp-works)
* [What is the DataTalks.Club Community?](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#what-is-the-datatalksclub-community)
* [How to Join AI Dev Tools Zoomcamp](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#how-to-join-ai-dev-tools-zoomcamp)
* [Frequently Asked Questions](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#frequently-asked-questions)
Who the Course Is For
---------------------
AI Dev Tools Zoomcamp is designed for people who want to work confidently with today’s AI development stack. It’s a good fit if you want to understand how coding assistants, agents, project generators, and automation tools actually fit into real engineering work.
This course is great for:
* **Software developers** who want to apply AI tools to everyday tasks like coding, debugging, testing, and project setup.
* **Engineers** looking to integrate assistants, agents, or workflow automation into their existing stack.
* **Learners who prefer building** rather than reading documentation, and who want guided, project-based practice with modern AI tooling.
You don’t need any previous experience with AI tools. A basic ability to program (Python, JavaScript, or similar) is enough to follow the materials and complete the projects.
Course Curriculum
-----------------
### What We’ll Cover
| Module | Topic | Focus | Tools You'll Use |
| --- | --- | --- | --- |
| 1 | Introduction to Vibe Coding / AI Tools Overview | AI-assisted development with Snake game example (React + JS) | ChatGPT, Claude, GitHub Copilot, Cursor, Bolt, Lovable |
| 2 | End-to-End Project (Snake) | Use a coding assistant for an end-to-end project: build Snake in React/TS, define API with OpenAPI, generate FastAPI server, add CI/CD, and deploy | React/TypeScript, OpenAPI, FastAPI, CI/CD tools |
| 3 | Model-Context Protocol | Enhance AI assistants with tools such as repo triage, PR summarization, scripted edits, and data queries | MCP servers (GitHub, filesystem, DB/SQL, HTTP/API, CI) |
| 4 | Build an AI Coding Agent (for Django) | Build your own coding agent that can scaffold and extend projects using Django templates and agent orchestration frameworks | Django, agent orchestration frameworks |
| 5 | AI for Testing, CI/CD & DevOps | AI-assisted PR reviews, automated test generation, release notes, changelog drafting, incident postmortems, and on-call copilots | CI/CD tools, LLM evaluation frameworks |
| 6 | Automation with Low-Code and No-Code AI (n8n) | Build automation workflows with composable AI tasks and ship lightweight assistants without maintaining servers | n8n, webhooks, connectors (email, Slack, GitHub, Jira, Notion, databases) |
### Projects
Every module includes a project that applies the concepts in practice.
At the end of the course, you’ll build a complete application using AI tools throughout the entire development lifecycle.
* Start with AI-assisted project planning and architecture
* Use coding assistants for implementation (React/TypeScript frontend, FastAPI backend)
* Implement AI agents for code reviews and testing
* Set up automated CI/CD pipeline with AI-enhanced workflows
* Deploy and monitor using modern DevOps practices
* Document your AI-powered development journey
How AI Dev Tools Zoomcamp Works
-------------------------------
### GitHub Repository
All lessons, homework, and cohort updates live in the [AI Dev Tools Zoomcamp GitHub repository](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
. The structure mirrors our other Zoomcamps, so you can quickly find weekly folders, homework forms, and project guidelines.
AI Dev Tools Zoomcamp [course materials on GitHub](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
### Homework Assignments
Each module includes homework to practice what you’ve learned and track your progress. You can also compete on the course leaderboard, which adds a fun, competitive element.
Track your progress on the anonymous leaderboard
### Learning in Public
A unique feature of our Zoomcamps is our **“learning in public”** approach, inspired by [Shawn Wang](https://www.youtube.com/watch?v=tkBCPqWKCL8&list=PL7NIGf5_PlM-Dk3lgPsZFT94Ng7PpRQEh&index=5&t=195s)
’s (@swyx) [article](https://www.swyx.io/learn-in-public)
on the topic. Instead of keeping your progress private, you’re encouraged to share assignments, reflections, and projects online.
Extract from Shawn @swyx Wang's article explaining the benefits of learning in public
Explaining what you’ve learned helps you understand it better, builds confidence, and creates visible proof of your skills. To encourage this, we award bonus points on the leaderboard for public posts about your work.
Course leaderboard highlighting bonus points earned through public learning activities
This practice has led to real opportunities for learners. For example, [**Pastor Soto**](https://datatalks.club/podcast/s21e03-from-medicine-to-machine-learning-how-public-learning-turned-into-career.html)
, who joined ML Zoomcamp without a LinkedIn account, began posting his projects publicly. These posts not only accelerated his learning but also attracted attention from recruiters at Meta and DeepLearning.AI. Similarly, [**Daniel Egbo**](https://youtu.be/b92gwrsVQtg?si=ziRFIcrYN3GQiVhV)
shared his progress on LinkedIn, which opened the door to collaborations on projects such as deploying models with Intel’s OpenVINO toolkit.
### Certificate
An example of the certificate for the [Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
, another free course at DataTalks.Club
To earn your free certificate, you need to complete a final project and review at least 3 other students’ projects. This ensures you demonstrate practical application of the concepts.
What is the DataTalks.Club Community?
-------------------------------------
Active discussions and support in the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
DataTalks.Club has a supportive community of like-minded individuals in [our Slack](https://datatalks.club/slack.html)
. It’s the perfect place to enhance your skills, deepen your knowledge, and connect with peers who share your passion. You’ll be supported by thousands of learners who exchange ideas, ask questions, and provide feedback, making it easier to stay accountable and find study partners. Many learners have turned these connections into collaborations and long-term professional relationships.
How to Join AI Dev Tools Zoomcamp
---------------------------------
You can join AI Dev Tools Zoomcamp either by **following a live cohort** or **learning at your own pace**.
All materials are freely available in the [AI Dev Tools Zoomcamp GitHub repository](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
. Each module has its own folder, and cohort-specific homework and deadlines are in the `cohorts` directory. Lectures are pre-recorded and available in this [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hLuyafXPyhTdbF4s_uNhc43)
, so you can follow the live cadence or binge-watch at your own pace.
### Option 1: Self-Paced Learning
You can start anytime and move at your own speed. You get full access to materials and community support on Slack.
You can complete homework assignments on the [course platform](https://courses.datatalks.club/)
and build a project for your portfolio, even outside a live cohort.
> With self-paced learning, homework isn’t scored, your project isn’t peer-reviewed, and you can’t earn a certificate.
### Option 2: Live Cohort
AI Dev Tools Zoomcamp runs once per year and typically starts in November.
When you join a live cohort, you get:
* Updated homework
* Automatic homework scoring and a leaderboard
* Project peer review
* Eligibility for a certificate after meeting all requirements
> Even if you join after the official start date, you can still follow along. Note that some homework forms may already be closed. All active deadlines are listed on the [course platform](https://courses.datatalks.club/)
> .
Frequently Asked Questions
--------------------------
What is the AI Dev Tools Zoomcamp?
The AI Dev Tools Zoomcamp is a free, community-driven program by [DataTalks.Club](https://datatalks.club/)
that teaches practical applications of AI tools in software development through hands-on project work.
This 6-module course covers a comprehensive [curriculum](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#what-youll-learn-in-ai-dev-tools-zoomcamp-2025)
with all materials open and available anytime on [GitHub](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
. You’ll work with industry-standard tools including GitHub Copilot, Cursor, Model Context Protocol (MCP), n8n, and various AI coding assistants, and earn a [certificate](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#can-i-get-a-certificate)
.
What does zoomcamp mean?
“Zoomcamp” is a term that originated from [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, the founder of DataTalks.Club. It started with his book “ML Bookcamp.” When Alexey decided to create a video course based on the book, he called it “[Machine Learning Zoomcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html)
” - a free, cohort-based course in video format. The name “zoomcamp” is a play on “bookcamp,” referring to the video format of the course. The Zoomcamp series has since expanded to include other free courses like the [Data Engineering Zoomcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html)
, [MLOps Zoomcamp](https://datatalks.club/blog/mlops-zoomcamp.html)
, [LLM Zoomcamp](https://datatalks.club/blog/llm-zoomcamp.html)
, and [AI Dev Tools Zoomcamp](https://datatalks.club/blog/ai-dev-tools-zoomcamp.html)
, all following the same community-driven, open-source philosophy.
Is AI Dev Tools Zoomcamp 2025 free?
Yes! The AI Dev Tools Zoomcamp is completely free. There are no hidden costs, no tuition fees, and no paid tiers. All course materials, videos, homework assignments, and access to the [Slack community](https://datatalks.club/slack.html)
are provided at no cost. Unlike traditional bootcamps that charge $10,000-$20,000+, this course is entirely community-driven and open source.
What programming experience do I need for AI Dev Tools Zoomcamp?
Basic programming knowledge in Python, JavaScript, or similar languages is recommended. No prior experience with AI tools is required - we’ll teach you everything you need to know about using AI in development workflows. The course is designed for developers who want to explore how AI tools fit into their workflow, engineers aiming to boost productivity with coding assistants, agents, and automation, and learners who prefer project-based practice over theory-heavy tutorials.
How long does AI Dev Tools Zoomcamp 2025 take to complete?
Each module includes hands-on projects, so the time commitment depends on your learning style and how much time you can dedicate. Most learners spend around 2 weeks to complete each module. With 6 modules total, expect to spend approximately 5-15 hours per week, depending on your background. This includes watching videos, completing homework, and working on [the final project](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#building-your-project-portfolio)
. More time may be needed during the final project weeks.
Do I need to install specific AI tools for this course?
We’ll guide you through setting up and using various AI tools throughout the course, including GitHub Copilot, Cursor, Model Context Protocol (MCP), and n8n. Some tools may require subscriptions, but we’ll focus on free or low-cost options where possible. The course covers essential AI development tools and platforms that you’ll learn to integrate into real developer workflows.
How do I get the AI Dev Tools Zoomcamp certificate?
To earn a certificate, you’ll need to complete a final project that demonstrates practical application of the concepts learned throughout the course. The project is submitted for peer review to ensure quality and understanding. After submitting your project, you must also review at least 3 other students’ projects by the deadline and provide constructive feedback. [Learn more about the final project](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#building-your-project-portfolio)
and [certificate requirements](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#can-i-get-a-certificate)
.
Can I get help if I'm stuck during the course?
Absolutely! The course is supported by the [DataTalks.Club community on Slack](https://datatalks.club/slack.html)
, where thousands of learners exchange ideas, ask questions, and provide feedback. You’ll have access to a supportive community throughout your learning journey. We also have an [FAQ repository](https://github.com/DataTalksClub/faq)
with answers to common questions and a @ZoomcampQABot in Slack for quick help.
What makes AI Dev Tools Zoomcamp different from other AI courses?
This course focuses specifically on integrating AI tools into real developer workflows, not just theory. You’ll build complete projects using AI assistants, create your own coding agents, and automate workflows - giving you practical, job-relevant skills. The course is designed with developer workflows in mind. By the end, you won’t just know what Copilot, Cursor, or n8n are; you’ll have used them to build complete projects and integrate them into real processes. Unlike many AI courses that focus on theory or chatbot development, AI Dev Tools Zoomcamp teaches you to use AI tools to actually ship software faster.
When does the next cohort of the AI Dev Tools Zoomcamp start?
The next cohort of the AI Dev Tools Zoomcamp starts on November 18, 2025. Register here: [https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
before the course starts.
Who runs the AI Dev Tools Zoomcamp?
The AI Dev Tools Zoomcamp is run by [DataTalks.Club](https://datatalks.club/)
, a global online community of data professionals and learners. While the initial idea and most of the content were created by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, members of the DataTalks.Club community contribute as instructors and maintainers.
DataTalks.Club is often referred to as “the DataTalks Club”, “data talks club”, or “datatalks club”.
Can I take the course in self-paced mode?
Yes! All course materials, videos, and recordings remain available after the cohort ends, and you can learn at your own pace. You’ll have access to the [Slack community](https://datatalks.club/slack.html)
for support. However, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a [certificate](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html#can-i-get-a-certificate)
. To receive a certificate, you need to join an active cohort.
Can I get a certificate in self-paced mode?
No, certificates are only available when completing the course with a live cohort. Self-paced mode does not include homework submissions, project evaluations, or certificates. This is because the certification process requires you to review three projects, and these peer reviews only happen during the active course period. Additionally, the submission form closes after the peer-review list is compiled. Self-paced learners can access all course materials and the Slack community, but must join a live cohort to earn a certificate.
Where is the GitHub repository?
The GitHub repository is [https://github.com/DataTalksClub/ai-dev-tools-zoomcamp](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
.
Where can I find the course videos?
Course videos are available on the [DataTalks.Club playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hLuyafXPyhTdbF4s_uNhc43)
. For easier navigation, refer to the [GitHub repository](https://github.com/DataTalksClub/ai-dev-tools-zoomcamp)
for course materials and links to video content. We also maintain [year-specific playlists](https://www.youtube.com/@DataTalksClub/playlists)
for updates.
What tools and technologies will I learn?
The course covers essential AI development tools including GitHub Copilot for code generation, Cursor as an AI-first IDE, OpenAI ChatGPT and Anthropic Claude for conversational AI assistance, Model Context Protocol (MCP) for connecting AI to external systems, n8n for workflow automation, and tools for CI/CD, testing, and DevOps. You’ll also learn to build your own coding agents and integrate AI into the complete software development lifecycle.
What is the DataTalks.Club AI Dev Tools community?
The DataTalks.Club AI Dev Tools community is a supportive network of 80,000+ data professionals and learners. As part of the AI Dev Tools Zoomcamp, you’ll have access to a dedicated course channel in [Slack](https://datatalks.club/slack.html)
where you can ask questions, get help from instructors and peers, share your progress, and connect with like-minded individuals. The community provides technical support, peer learning opportunities, and networking that can lead to collaborations and career opportunities. This active community is one of the key differentiators of the course experience.
Is this a free AI Dev Tools course with certificate?
Yes! This is a completely free AI Dev Tools course, with a certificate available when you complete the course with a live cohort. There are no hidden costs or tuition fees. To earn your certificate, you’ll need to complete the technical modules, build one final project demonstrating practical application of AI tools in development workflows, participate in peer reviews, and follow best practices. This free course provides the same quality training as paid bootcamps but at no cost. Certificates, homework submissions, and project evaluations are only available when participating in a live cohort, not in self-paced mode.
Can I get a certificate?
Yes, certificates are available when completing the course with a live cohort. Requirements include completing the technical modules, building one final project that demonstrates practical application of AI tools in development workflows, participating in peer reviews (reviewing at least 3 other students’ projects), and following best practices. Certificates, homework submissions, and project evaluations are not available in self-paced mode.
[Join AI Dev Tools Zoomcamp 2025 (starts November 18, 2025) →](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
Related Posts
-------------
[### Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps\
\
Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics\
\
Read more](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
[### Data Engineering Zoomcamp: Free Data Engineering Course and Certification\
\
Become a Data Engineer: Master Modern Data Engineering with Hands-On Training\
\
Read more](https://datatalks.club/blog/data-engineering-zoomcamp.html)
[### MLOps Zoomcamp: Free MLOps Course and Certification\
\
Learn to deploy, monitor, and maintain ML models in production with MLflow, Docker, AWS, and monitoring tools\
\
Read more](https://datatalks.club/blog/mlops-zoomcamp.html)
[### LLM Zoomcamp: Free LLM Engineering Course and Certification\
\
Master LLM Engineering: Build Production-Ready AI Applications from Scratch\
\
Read more](https://datatalks.club/blog/llm-zoomcamp.html)
[### ML Zoomcamp: Free Machine Learning Engineering Course and Certification\
\
Master Machine Learning Engineering with Python in 4 Months\
\
Read more](https://datatalks.club/blog/machine-learning-zoomcamp.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Christoph Molnar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Christoph Molnar
My name is Christoph Molnar, I’m a statistician and a machine learner. My goal is to make machine learning interpretable.
[](https://twitter.com/ChristophMolnar)
[](https://linkedin.com/in/christoph-molnar)
[](https://github.com/christophM)
[](https://christophmolnar.com/)
### Events
* Cracking the Code: Machine Learning Made Understandable ([watch on youtube](https://www.youtube.com/watch?v=LBuGzyOkx7c)
)
### Books
* [Interpretable Machine Learning](https://datatalks.club/books/20220411-interpretable-machine-learning.html)
(the book of the week from 11 Apr 2022 to 15 Apr 2022)
* [Modeling Mindsets](https://datatalks.club/books/20230529-modeling-mindsets.html)
(the book of the week from 29 May 2023 to 02 Jun 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Engineering Zoomcamp: Free Data Engineering Course and Certification – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Engineering Zoomcamp: Free Data Engineering Course and Certification
Data Engineering Zoomcamp: Free Data Engineering Course and Certification
=========================================================================
### Become a Data Engineer: Master Modern Data Engineering with Hands-On Training
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Breaking into data engineering takes real, hands-on experience with production tools, but most courses stop at theory.
The Data Engineering Zoomcamp changes that. It’s a free data engineering course that teaches you how to build production-grade data pipelines from start to finish. You’ll work with Docker, Terraform, BigQuery, dbt, Spark, and Kafka, and graduate with a portfolio project and a certificate.
Complete Data Engineering Zoomcamp curriculum: from infrastructure setup to stream processing
It’s ideal for beginners and career switchers preparing for junior data engineer roles, as well as experienced professionals who want to refresh their knowledge, expand their network, or test themselves as a mentor for less experienced professionals.
Unlike most courses, DE Zoomcamp helps you build your public portfolio, share your work confidently, and connect with a [global community](https://datatalks.club/slack.html)
for feedback, mentorship, and career opportunities.
> **TL;DR:** Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. [Join the course here](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
> .
Table of Contents
-----------------
* [What is the Data Engineering Zoomcamp?](https://datatalks.club/blog/data-engineering-zoomcamp.html#what-is-the-data-engineering-zoomcamp)
* [Why Learn Data Engineering?](https://datatalks.club/blog/data-engineering-zoomcamp.html#why-learn-data-engineering)
* [Who Can Learn Data Engineering?](https://datatalks.club/blog/data-engineering-zoomcamp.html#who-can-learn-data-engineering-and-what-roles-it-leads-to)
* [Course Prerequisites](https://datatalks.club/blog/data-engineering-zoomcamp.html#course-prerequisites)
* [Course Curriculum: What You’ll Learn](https://datatalks.club/blog/data-engineering-zoomcamp.html#course-curriculum-what-youll-learn-in-the-data-engineering-zoomcamp)
* [How the Data Engineering Zoomcamp Works](https://datatalks.club/blog/data-engineering-zoomcamp.html#how-the-data-engineering-zoomcamp-works)
* [Zoomcamp vs. Bootcamp](https://datatalks.club/blog/data-engineering-zoomcamp.html#zoomcamp-vs-bootcamp-whats-the-difference)
* [How to Get Started](https://datatalks.club/blog/data-engineering-zoomcamp.html#how-to-get-started-with-the-data-engineering-zoomcamp)
* [Testimonials](https://datatalks.club/blog/data-engineering-zoomcamp.html#testimonials)
* [Frequently Asked Questions](https://datatalks.club/blog/data-engineering-zoomcamp.html#frequently-asked-questions)
What is the Data Engineering Zoomcamp?
--------------------------------------
Data Engineering Zoomcamp is a 9-week program that follows a clear progression: infrastructure setup, workflow orchestration, data warehousing, analytics engineering, batch processing, streaming, and a final capstone project.
Data Engineering Zoomcamp GitHub repository showing the course materials
What makes it different is the community. You’ll join an active Slack workspace where thousands of learners troubleshoot together, share progress, and connect for jobs and collaborations. The course encourages learning in public: sharing your work earns bonus points and builds your online presence.
The final three weeks focus on your capstone that gets peer-reviewed, and you graduate with a polished GitHub portfolio that proves you can ship real data systems.
Why Learn Data Engineering?
---------------------------
Typically, data science teams are comprised of data analysts, data scientists, and data engineers. Among different [data roles](https://datatalks.club/blog/data-roles)
, data engineers are the specialists who connect all the pieces of the data ecosystem within a company or institution.
Data engineers are the specialists who connect all the pieces of the data ecosystem.
But why would you want to become one? Here are some of the main reasons that make data engineering roles rewarding and valuable:
1. You become the builder behind every data product that keeps information flowing.
2. You increase your earning potential by joining a smaller, high-value pool of professionals.
3. You develop transferable skills that are valuable across industries and provide long-term career flexibility. These skills are also foundational for roles in [machine learning](https://datatalks.club/blog/machine-learning-zoomcamp.html)
and [MLOps](https://datatalks.club/blog/mlops-zoomcamp.html)
.
4. You future-proof your career by building a mindset that will stay essential even as tools change and processes get automated.
Who Can Learn Data Engineering (and What Roles It Leads To)
-----------------------------------------------------------
The great thing about data engineering is that you don’t need to be a computer science graduate or have years of experience in data to start.
If you understand basic programming and have curiosity about how data systems work, you can learn data engineering.
If you tick most of the following boxes, data engineering might be a good fit for you:
* Enjoy solving technical, logical problems and building things that work reliably.
* Like Python, SQL, or scripting and want to use those skills for something impactful.
* Want to understand how data moves inside organizations — from raw sources to analysis and AI (including [LLM applications](https://datatalks.club/blog/llm-zoomcamp.html)
).
* Prefer practical, system-level work over purely theoretical or statistical modeling.
* Appreciate clear structure and step-by-step learning (which is how DE Zoomcamp is built).
Course Prerequisites
--------------------
As mentioned earlier, learning data engineering doesn’t require prior data engineering experience.
The course is designed to be accessible to beginners.
The only requirements are:
* Comfort with the command line (basic navigation and file operations)
* Basic SQL knowledge (SELECT, WHERE, JOIN statements)
* Python experience is helpful but not required
If you’re completely new to programming, consider spending a few weeks learning the basics before starting the course.
Course Curriculum: What You’ll Learn in the Data Engineering Zoomcamp
---------------------------------------------------------------------
The course follows a logical progression from infrastructure setup to advanced data processing, culminating in an end-to-end project.
| Module | What You'll Learn | Tools & Technologies |
| --- | --- | --- |
| 1\. Infrastructure & Prerequisites | • Set up your development environment with Docker and PostgreSQL
• Learn cloud basics with GCP
• Master infrastructure-as-code using Terraform | Docker, PostgreSQL, GCP, Terraform |
| 2\. Workflow Orchestration | • Master data pipeline orchestration with Mage.AI
• Implement and manage Data Lakes using Google Cloud Storage
• Build automated, reproducible workflows | Mage.AI, Google Cloud Storage |
| 3\. Data Warehouse | • Deep dive into BigQuery for enterprise data warehousing
• Learn optimization techniques like partitioning and clustering
• Implement best practices for data storage and retrieval | BigQuery |
| 4\. Analytics Engineering | • Transform raw data into analytics-ready models using dbt
• Develop testing and documentation strategies
• Create impactful visualizations with modern BI tools | dbt, BI tools |
| 5\. Batch Processing | • Process large-scale data with Apache Spark
• Master Spark SQL and DataFrame operations
• Optimize batch processing workflows | Apache Spark, Spark SQL |
| 6\. Stream Processing | • Build real-time data pipelines with Kafka
• Develop streaming applications using KSQL and Faust
• Implement stream processing patterns | Kafka, KSQL, Faust |
| Final Project | • Build an end-to-end data pipeline from ingestion to visualization
• Apply all learned concepts in a real-world project
• Create a portfolio-ready project with documentation | Cloud platforms (GCP/AWS/Azure), Terraform, Spark, Kafka, dbt, BigQuery |
### Capstone Project
The final three weeks are dedicated to applying your knowledge in a real-world project that showcases everything you’ve learned throughout the course.
Data Engineering Zoomcamp capstone project of one of the course graduates, Maddie Zheng, showing project architecture: extract, load, transform, and visualize data. Source: [Maddie's project](https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/cohorts/2025/projects/maddie-zheng)
You’ll build an end-to-end data pipeline using a dataset of your choice, implementing both data lake and warehouse solutions with proper documentation.
Your project will be peer-reviewed by fellow participants and you’ll peer-review at least three other projects.
| Project Requirements | Deliverables | Evaluation Criteria |
| --- | --- | --- |
| • Select and process a dataset that interests you
• Build end-to-end data pipelines (batch or streaming)
• Implement both data lake and warehouse solutions
• Create analytical dashboards | • Production-ready data pipeline
• Documented data models
• Interactive dashboard
• Project presentation | • Peer review of at least three other projects
• Technical implementation quality
• Documentation completeness
• Solution architecture design |
[Join the next cohort →](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
How the Data Engineering Zoomcamp Works
---------------------------------------
The course runs for 9 weeks in cohort format, providing structure and community support throughout your learning journey.
While you can access all course materials at your own pace without joining a cohort, participating in the structured program offers significant advantages: graded homework assignments, project submission and evaluation, peer interaction, and the opportunity to earn a certificate.
Below, we list the key features of the course and how they work.
### GitHub Repository: The Central Hub of the Course
Data Engineering Zoomcamp GitHub repository showing the course materials
All course materials live in the [GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp)
. The lectures are pre-recorded and available on YouTube, so you can watch at your own pace.
Data Engineering Zoomcamp YouTube playlist with pre-recorded lectures
### Homework Assignments
To reinforce your learning, we release homework assignments for each week of the course. You can submit a homework assignment at the end of each week.
Data Engineering Zoomcamp schedule showing the course schedule and submission deadlines
It doesn’t count toward your certificate, but it helps you practice and appears on an optional anonymous leaderboard.
Your scores are added to an anonymous leaderboard, creating friendly competition among learners and motivating you to do your best.
Course leaderboard displaying student progress and achievements anonymously
You can earn bonus points by learning in public — sharing your work on blogs, YouTube, or social media. The next section explains how it works in practice.
### Learning in Public: Build Your Online Presence
A unique feature is our “learning in public” approach, inspired by [Shawn @swyx Wang](https://datatalks.club/podcast/s03e07-market-yourself.html)
’s [article](https://www.swyx.io/learn-in-public)
. We believe that everyone has something valuable to contribute, regardless of their expertise level.
An extract from Shawn @swyx Wang's article about learning in public
Throughout the course, we actively encourage and incentivize learning in public. By sharing your progress, insights, and projects online, you earn additional points for your homework and projects.
Previous cohort's leaderboard highlighting bonus points earned through learning in public activities
This not only demonstrates your knowledge but also builds a portfolio of valuable content. Sharing your work online also helps you get noticed by social media algorithms, reaching a broader audience and creating opportunities to connect with individuals and organizations you may not have encountered otherwise.
Many of our graduates have shared that their social media presence has helped them attract [job offers](https://www.linkedin.com/feed/update/urn:li:share:7380134597839187968/)
and [collaborations](https://www.linkedin.com/feed/update/urn:li:share:7376510719581646849/)
.
### How to Get a Certificate
Data Engineering Zoomcamp certificate showing the certificate requirements
To earn your certificate, you must complete the course with a live cohort and fulfill three key requirements:
1. **Build a capstone project**: Create an end-to-end data pipeline that demonstrates your mastery of the course concepts
2. **Submit on time**: Meet the project submission deadline to qualify for certification
3. **Peer review**: Evaluate and provide feedback on 3 fellow students’ projects during the peer review process
### What is DataTalks.Club Community? A Place to Connect and Learn with Other Data Professionals
Active discussions and peer support in our dedicated Slack community channel
[DataTalks.Club](https://datatalks.club/)
is a global community of 80,000+ data professionals who connect on [Slack](https://datatalks.club/slack.html)
to share knowledge, ask career questions, and discuss everything from analytics and visualization to machine learning and data engineering. As one of the largest digital groups dedicated to data, it’s where you’ll find data scientists, ML engineers, data analysts, and enthusiasts at all career stages.
When you join a cohort, the dedicated course channel becomes your home base. Here, you’ll troubleshoot problems with peers working through the same challenges and share your progress and insights.
DataTalks.Club FAQ repository showing common questions and technical issues
Beyond peer support, there are two ways to get help: our [FAQ repository](https://datatalks.club/faq/data-engineering-zoomcamp.html)
covers common questions and technical issues, and the @ZoomcampQABot in Slack provides quick answers when you need them.
DataTalks.Club Zoomcamp QABot in Slack providing quick answers when you need them
Zoomcamp vs. Bootcamp: What’s the Difference?
---------------------------------------------
We often get asked what the difference is between the Data Engineering Zoomcamp and paid bootcamps.
Below, we list the key features of the Data Engineering Zoomcamp and how they compare to paid bootcamps.
| Feature | DE Zoomcamp (Cohort) | DE Zoomcamp (Self-paced) | Paid Bootcamps |
| --- | --- | --- | --- |
| Cost | Free | Free | $2,000–$10,000+ |
| Format | 9-week cohort with fixed schedule | Learn anytime at your own pace | Fixed schedule, instructor-led |
| Homework | Weekly scored assignments | Available but no scoring | Weekly with instructor feedback |
| Projects | Capstone project with peer review and scoring | Build on your own, no evaluation and scoring | Instructor-reviewed projects |
| Certificate | Yes, after completing project + peer reviews | No certificate | Certificate of completion |
| Community Support | Active Slack + optional live Q&A sessions | Slack community only | Instructor-led, 1:1 or group mentorship |
| Learning in Public | Encouraged with bonus points | Optional | Rarely emphasized |
| Timeline | 9 weeks (Jan–Mar 2026) | Flexible, self-paced | Typically 12–24 weeks |
| Best For | Career switchers or experienced data engineers wanting community and accountability | Self-motivated learners exploring data engineering | Those needing intensive structured mentorship and guidance |
How to Get Started with the Data Engineering Zoomcamp
-----------------------------------------------------
To get started with the Zoomcamp, you can either join a live cohort or learn at your own pace.
All materials are freely available in the [Data Engineering Zoomcamp GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp)
. Each module has its own folder (e.g., `01-docker`, `03-data-warehouse`), and cohort-specific homework and deadlines are in the `cohorts` directory. Lectures are pre-recorded and available in our official [Data Engineering Zoomcamp YouTube Playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb)
, so you can watch at your own pace.
### Learn at Your Own Pace
The self-paced mode lets you start immediately and progress on your own schedule. You’ll get access to the course materials and the community and can complete the course at a pace that works for you.
All you need is to go to the [Data Engineering Zoomcamp GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp)
and start learning. It serves as a central hub for the course for easier navigation through the course materials. All the lectures are pre-recorded and available on YouTube in our official [Data Engineering Zoomcamp YouTube Playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb)
, so you can watch whenever it suits you.
You can also join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
to get help and support from the community in the `#course-data-engineering` channel. Homework and solutions are available on the [course platform](https://courses.datatalks.club/)
, and you can build a project for your portfolio.
> Remember, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a certificate. To receive certification, you need to join an active cohort.
### Join a Live Cohort
> **2026 Cohort:** Starts January 2026. Register here: [Fill in this form](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
When you join a live cohort, you’ll work through the same materials as self-paced learners, but with the added structure of a published schedule and the energy of hundreds of peers progressing alongside you.
The course runs once per year (starts around January). Each module typically spans one week: you watch the lectures, complete hands-on exercises, and submit a homework assignment. Your submissions get scored and appear on an anonymous leaderboard.
After completing all six modules, the capstone project phase begins. You’ll build your own end-to-end pipeline, submit it through our form, and peer-review at least three other students’ projects while yours gets reviewed by your peers. This reciprocal process gives you valuable feedback on your work and exposes you to different approaches and solutions you might not have considered.
Even if you join after the official start date, you can still follow along — but note that some homework forms may already be closed. All active deadlines are listed on the [course platform](https://courses.datatalks.club/)
.
> To earn a [certificate](https://datatalks.club/blog/data-engineering-zoomcamp.html#how-to-get-a-certificate)
> , you’ll need enough time to complete one [final project](https://datatalks.club/blog/data-engineering-zoomcamp.html#capstone-project)
> and the required peer reviews. Details are in the Projects and Certificate sections.
**Ready to join DE Zoomcamp?** Here’s how it works:
1. [Register for the course](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
, you’ll be automatically accepted into the next cohort
2. Join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
and the `#course-data-engineering` channel for updates, questions, and peer support
3. (Optional) Get a head start by exploring the [GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp)
and watching lectures before the cohort officially begins
4. When the cohort starts, you’ll receive an email with the full schedule and submission deadlines
5. Follow the weekly rhythm: watch lectures, complete exercises, submit homework
6. During the final three weeks, build and submit your capstone project, then peer-review three other projects
7. Receive your certificate once your project passes peer review
The entire journey takes 9 weeks from start to certificate, and you’ll be part of a global cohort tackling the same challenges at the same time.
### Comparison
We summarized the key differences between the two joining options in this table:
| Feature | Self-Paced | Live Cohort |
| --- | --- | --- |
| **Timing** | Learn at your own pace, start anytime | Fixed 9-week schedule (January–March each year) |
| **Course Materials** | Full access to GitHub repository and YouTube lectures | Full access to GitHub repository and YouTube lectures |
| **Community** | Access to Slack community (`#course-data-engineering`) | Access to Slack community (`#course-data-engineering`) |
| **Homework** | Available but not scored | Scored automatically, appears on leaderboard |
| **Projects** | Build on your own, no evaluation | Submit 1 final project (capstone) with peer review |
| **Certificate** | Not available | Available after completing project and peer reviews |
| **Structure** | Flexible, no deadlines | Weekly rhythm with deadlines and peer accountability |
[Join the next cohort →](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
Testimonials
------------
We’ve collected some testimonials from our students who have completed the Data Engineering Zoomcamp.
> Thank you for what you do! The Data Engineering Zoomcamp gave me skills that helped me land my first tech job.
>
> — [Tim Claytor](https://www.linkedin.com/in/claytor/)
> ([Source](https://www.linkedin.com/feed/update/urn:li:activity:7396882073308938240?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7396882073308938240%2C7396889959711793152%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287396889959711793152%2Curn%3Ali%3Aactivity%3A7396882073308938240%29)
> )
> Three months might seem like a long time, but the growth and learning during this period are truly remarkable. It was a great experience with a lot of learning, connecting with like-minded people from all around the world, and having fun. I must admit, this was really hard. But the feeling of accomplishment and learning made it all worthwhile. And I would do it again!
>
> — [Nevenka Lukic](https://www.linkedin.com/in/nevenka-lukic/)
> ([Source](https://www.linkedin.com/posts/nevenka-lukic_data-engineering-zoomcamp-final-project-activity-7181985646033461248-Lc1O?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> One of the significant things I inferred from the Zoomcamp is to prioritize fundamentals and principles over ever-evolving tools and tech stacks. Hugely grateful to Alexey Grigorev for putting together this incredible course and offering it for free.
>
> — [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi/)
> ([Source](https://www.linkedin.com/posts/activity-7325692407675604992-XSKI?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> Such a fun deep dive into data engineering, cloud automation, and orchestration. I learned so much along the way. Big shoutout to Alexey Grigorev and the DataTalksClub team for the opportunity and guidance throughout the 3 months of the free course.
>
> — [Assitan NIARE](https://www.linkedin.com/in/assitan-niar%C3%A9-data/)
> ([Source](https://www.linkedin.com/posts/activity-7317441554023874561-E3wm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> If you’re serious about breaking into data engineering, start here. The repo’s structure, community, and hands-on focus make it unparalleled.
>
> — [Wady Osama](https://www.linkedin.com/in/wadyosama/)
> ([Source](https://www.linkedin.com/posts/wadyosama_dataengineering-zoomcamp-dezoomcamp-activity-7292126824711520258-puJm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
Frequently Asked Questions
--------------------------
Is the Data Engineering Zoomcamp a free data engineering course?
Yes. The Data Engineering Zoomcamp is a free, project-based data engineering course that covers pipelines, data warehouses, batch and streaming, and orchestration—without any tuition fees.
Whats the difference between a DataTalks.Club Zoomcamp and a typical data engineering bootcamp?
Bootcamps usually charge tuition, offer full-time schedules, and sometimes include dedicated career services. Zoomcamps are part-time, community-run, open-source, and free, with an emphasis on self-directed learning and peer support.
Does the Data Engineering Zoomcamp offer a certificate?
During an active cohort, you can earn a certificate by completing the final project, reviewing peers’ projects, and meeting all deadlines. Self-paced learners don’t receive certificates but still build the same projects.
How does the Data Engineering Zoomcamp certificate work?
To earn a [certificate](https://datatalks.club/blog/data-engineering-zoomcamp.html#how-to-get-a-certificate)
, you need to complete one [capstone project](https://datatalks.club/blog/data-engineering-zoomcamp.html#capstone-project)
by building an end-to-end data pipeline. After submitting your project, you must also review at least 3 other students’ projects by the deadline and provide constructive feedback.
When does the next cohort of the DE Zoomcamp start?
The next cohort of the DE Zoomcamp starts around January each year. Register here: [https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
before the course starts.
Can I join the Data Engineering Zoomcamp 2025 cohort?
No, the 2025 cohort is closed. You can join the 2026 cohort when it starts in January 2026. [Register here](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
before the course starts.
Who runs the Data Engineering Zoomcamp?
DE Zoomcamp is run by [DataTalks.Club](https://datatalks.club/)
, a global online community of data professionals and learners. While the initial idea and most of the content were created by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, members of the DataTalks.Club community contribute as instructors and maintainers.
DataTalks.Club is often referred to as “the DataTalks Club”, “data talks club”, or “datatalks club”.
What are the prerequisites for the Data Engineering Zoomcamp?
You should be comfortable with basic coding (Python or similar), the command line, and basic SQL. No prior data engineering experience is required.
How much time does it take?
Expect to spend 5-15 hours per week, depending on your background. This includes watching videos, completing homework, and working on [the capstone project](https://datatalks.club/blog/data-engineering-zoomcamp.html#capstone-project)
. More time may be needed during the final project weeks.
Can I learn at my own pace?
Yes! All course materials, videos, and recordings remain available after the cohort ends, and you can learn at your own pace. You’ll have access to the [Slack community](https://datatalks.club/slack.html)
for support. However, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a [certificate](https://datatalks.club/blog/data-engineering-zoomcamp.html#how-to-get-a-certificate)
. To receive a certificate, you need to join an active cohort.
What if I get stuck?
You have multiple support channels available. Join the [Slack community](https://datatalks.club/slack.html)
where you can ask questions and get help from instructors and fellow students. We also have an [FAQ repository](https://github.com/DataTalksClub/faq)
with answers to common questions and a @ZoomcampQABot in Slack for quick help.
Where is the GitHub repository for the Data Engineering Zoomcamp?
The GitHub repository is [https://github.com/DataTalksClub/data-engineering-zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
.
Where can I find the Data Engineering Zoomcamp syllabus?
You can find the full syllabus in the readme of the [Data Engineering Zoomcamp GitHub repository](https://github.com/DataTalksClub/data-engineering-zoomcamp)
.
Is it the same as DE Zoomcamp, Data Engineer Zoomcamp, or Zoomcamp Data Engineering?
Yes, people use these names interchangeably. Throughout this page we’ll use “Data Engineering Zoomcamp” as the canonical name.
Is the Data Engineering Zoomcamp good preparation for data engineering jobs?
The course focuses on building real-world pipelines and infrastructure with tools like Docker, BigQuery, Spark, and Kafka. Many learners have used the final project and GitHub portfolio as part of their data engineering job search.
How do I join the DataTalks.Club Slack community?
You can request an invite via the [Slack signup form](https://datatalks.club/slack.html)
. After confirming your email, you’ll receive an invitation link to join the workspace.
Is there a Discord or Telegram channel for DataTalks.Club?
The main community hub is [Slack](https://datatalks.club/slack.html)
. Data Engineering Zoomcamp also has a [Telegram announcement channel](https://t.me/dezoomcamp)
, but all core discussion happens in Slack.
[Join the next cohort →](https://airtable.com/appzbS8Pkg9PL254a/shr6oVXeQvSI5HuWD)
Related Posts
-------------
[### Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps\
\
Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics\
\
Read more](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
[### ML Zoomcamp: Free Machine Learning Engineering Course and Certification\
\
Master Machine Learning Engineering with Python in 4 Months\
\
Read more](https://datatalks.club/blog/machine-learning-zoomcamp.html)
[### MLOps Zoomcamp: Free MLOps Course and Certification\
\
Learn to deploy, monitor, and maintain ML models in production with MLflow, Docker, AWS, and monitoring tools\
\
Read more](https://datatalks.club/blog/mlops-zoomcamp.html)
[### LLM Zoomcamp: Free LLM Engineering Course and Certification\
\
Master LLM Engineering: Build Production-Ready AI Applications from Scratch\
\
Read more](https://datatalks.club/blog/llm-zoomcamp.html)
[### AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers\
\
Learn how to integrate AI into real developer workflows, from AI coding assistants to agents, CI/CD, DevOps, and no-code automation.\
\
Read more](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Danny Leybzon – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Danny Leybzon
Danny D. Leybzon has worn many hats, all of them related to data. He studied computational statistics at UCLA, before becoming first an analyst and then a product manager at a big data platform named Qubole. He went on to be the primary field engineer for data science and machine learning at Imply, before taking on his current role as MLOps Architect at WhyLabs. He has worked to evangelize machine learning best practices, talking on subjects such as distributed deep learning, productionizing machine learning models, automated machine learning, and lately has been talking about AI observability and data logging. When Danny’s not researching, practicing, or talking about data science, he’s usually doing one of his numerous outside hobbies: rock climbing, backcountry backpacking, skiing, etc.
[](https://twitter.com/dleybz)
[](https://linkedin.com/in/dleybz)
[](https://github.com/dleybz)
[](http://web.dleybz.co/)
### Events
* Hands-On Data Monitoring with whylogs ([watch on youtube](https://www.youtube.com/watch?v=b6yk9b7A4CQ)
)
* MLOps Architect ([watch on youtube](https://www.youtube.com/watch?v=p1gVaS4Zx5M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Angelica Lo Duca – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Angelica Lo Duca
Angelica Lo Duca is a Researcher at the Institute of Informatics and Telematics at the National Research Council, Italy. Her research interests include Data Science, Machine Learning, Text Analytics, Data Visualisation, Data Journalism, and Web Applications. She is also a professor at the University of Pisa, where she teaches Data Journalism.
[](https://twitter.com/alod83)
[](https://linkedin.com/in/angelicaloduca)
[](https://github.com/alod83)
[](https://alod83.medium.com/)
### Articles
* [How to Build a Data Science Team from Scratch: Complete Hiring Guide](https://datatalks.club/blog/building-data-science-team.html)
* [Data Science Manager vs Expert: Which Role Does Your Company Need?](https://datatalks.club/blog/data-science-manager-vs-data-science-expert.html)
* [A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won!](https://datatalks.club/blog/summary-of-kitchenware-competition.html)
* [DataOps: Similarities and Differences with Data Engineering and Data Science](https://datatalks.club/blog/dataops-similarities-and-differences-with-data-engineering-and-data-science.html)
* [The Essentials of Public Speaking for a Career in Data Science](https://datatalks.club/blog/essentials-of-public-speaking-for-career-in-data-science.html)
* [Starting a Career as a Data Scientist](https://datatalks.club/blog/starting-career-as-data-scientist.html)
* [What is DataOps exactly?](https://datatalks.club/blog/what-dataops-exactly.html)
* [DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained](https://datatalks.club/blog/devops-and-mlops-same-thing.html)
### Events
* Technical Writing and Data Journalism ([watch on youtube](https://www.youtube.com/watch?v=uO_lk12q02A)
)
* Data Storytelling in Python Altair ([watch on youtube](https://www.youtube.com/watch?v=WqcuYXTEeUo)
)
### Books
* [Comet for Data Science](https://datatalks.club/books/20221107-comet-for-data-science.html)
(the book of the week from 24 Oct 2022 to 28 Oct 2022)
* [Data Storytelling with Altair and AI](https://datatalks.club/books/20240902-data-storytelling-with-altair-and-ai.html)
(the book of the week from 02 Sep 2024 to 06 Sep 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Danny Ma – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Danny Ma
Danny is a recovering data scientist who has moved over to the dark side of ML/Data engineering in the past 2 years. His core expertise is in data analytics, supervised ML algorithms, data architecture and designing digital customer experiments for retail and banking sectors.
Danny’s passion is to guide businesses and individuals on their data & machine learning journey. He currently runs the #DataWithDanny community with over 4,500 aspiring data professionals and is working towards his vision of creating a scaleable virtual data apprenticeship program to further spread his knowledge and experience.
[](https://linkedin.com/in/datawithdanny)
[](https://datawithdanny.com/)
### Events
* DataTalks.Club Conference: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=ltFkvoiA57M)
)
* The ABC’s of Data Science ([watch on youtube](https://www.youtube.com/watch?v=HVQ0DZOQcts)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-ABCs-of-Data-Science---Danny-Ma-er33oa)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dashel Ruiz Perez – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dashel Ruiz Perez
Dashel Ruiz Perez is a data analyst, ML engineer, and educator based in Oregon, USA. He spent nearly a decade at Microchip Technology, where he worked across production, process, yield, and software engineering roles. Today, he teaches programming and data skills at major U.S. universities through ThriveDX and continues to deepen his expertise in machine learning and AI. Dashel holds degrees in computer science and data analytics from Western Governors University and is a graduate of ML Zoomcamp.
[](https://linkedin.com/in/dashel-ruiz-perez-2b036172)
### Events
* From Semiconductors to Machine Learning: A Career in Data and Teaching ([watch on youtube](https://www.youtube.com/watch?v=B2tzuUg5uZs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dat Tran – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dat Tran
Dat Tran is the Partner and CTO at DATANOMIQ, a leading AI and data consulting company based in Berlin. He helps organizations build real-world AI systems, scale data science teams, and accelerate digital transformation—from the first prototype to full production.
With over a decade of experience in machine learning, artificial intelligence, and data strategy, Dat has led AI initiatives at major companies including Axel Springer SE, idealo, and Pivotal Software, and co-founded multiple startups such as LegalLint and Priceloop. His work spans industries from media and retail to aviation, legal tech, and finance.
As an advisor, investor, and keynote speaker, Dat is known for his no-nonsense approach to applied AI—focusing on value creation over hype. He regularly speaks at international conferences such as PyData, ODSC, WeAreDevelopers, and the Data Festival, where he shares insights on AI engineering, data infrastructure, and machine learning in production.
[](https://twitter.com/datitran)
[](https://linkedin.com/in/dat-tran-a1602320)
[](https://dat-tran.com/)
### Events
* Building a Data Science Team ([watch on youtube](https://www.youtube.com/watch?v=ScDIB-3O77A)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-a-Data-Science-Team---Dat-Tran-enlmef)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dave Flynn – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dave Flynn
Dave is a Technical Advocate at InfuseAI, the company behind open-source projects such as PrimeHub, ArtiVC, and PipeRider. Dave’s responsibilities include technical writing, documentation, and marketing. Before joining InfuseAI, Dave was a full-stack web developer and Linux system administrator.
[](https://twitter.com/infuseai)
[](https://linkedin.com/in/daveflynn81)
### Events
* Maximizing Confidence in Your Data Model Changes with dbt and PipeRider ([watch on youtube](https://www.youtube.com/watch?v=O-tyUOQccSs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Bader – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Bader
David A. Bader is the Director of the Institute for Data Science at the New Jersey Institute of Technology and a Distinguished Professor and a founder of the Department of Data Science in the Ying Wu College of Computing. Prior to this, he served as founding Professor and Chair of the School of Computational Science and Engineering, College of Computing, at Georgia Institute of Technology. He is a Fellow of the IEEE, ACM, AAAS, and SIAM, a recipient of the IEEE Sidney Fernbach Award, and advises the White House, most recently on the National Strategic Computing Initiative (NSCI) and Future Advanced Computing Ecosystem (FACE).
Dr. Bader is a leading expert in solving global grand challenges in science, engineering, computing, and data science. His interests are at the intersection of data science, big data, high-performance computing, and real-world applications, including cybersecurity, massive-scale analytics, and computational genomics. He has also co-authored over 300 articles in peer-reviewed journals and conferences.
[](https://twitter.com/prof_davidbader)
[](https://linkedin.com/in/dbader13)
[](https://github.com/dbader13)
[](https://davidbader.net/)
### Events
* Leading Data Research ([watch on youtube](https://www.youtube.com/watch?v=vZLlpsUlchQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dave Bechberger – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dave Bechberger
Dave Bechberger is known for his expertise in distributed data architecture as well as being a Graph Database SME and is curently a Sr. Graph Architect on the Amazon Neptune team. He has spent years architecting and building distributed data architectures and delivering full-stack software solutions to complex data problems. He prides himself on taking a pragmatic approach to solving data problems for applications, data analysis, and data science workflows using a variety of SQL and NoSQL data technologies. Dave has previously spoken at a variety of national and international technical conferences including NDC Oslo, NDC London, as well as previous GraphDay conferences in Texas, San Francisco and Seattle. Dave is author of [Graph Databases in Action](https://www.manning.com/books/graph-databases-in-action?a_aid=bechberger)
from Manning publications
[](https://twitter.com/bechbd)
[](https://linkedin.com/in/davebechberger)
[](https://github.com/bechbd)
[](http://www.bechberger.com/)
### Books
* [Graph Databases in Action](https://datatalks.club/books/20210614-graph-databases-in-action.html)
(the book of the week from 14 Jun 2021 to 18 Jun 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Bednar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Bednar
David Bednař loves data and its presentation, and he has spent most of his career in the analytics area. He benefits from experience from both the academic and the business spheres, where he works on projects that further develop his robust technological background. He is a person who likes to develop anything new and innovative, who can define new architectures and patterns, or lead young talents to growth. David graduated from the Technical University of Ostrava focusing on data management and custom data structures, where he helped with research of multidimensional data structures as a member of the Database Research Group.
[](https://linkedin.com/in/davidbednarcz)
### Books
* [Data Analytics Initiatives](https://datatalks.club/books/20220801-data-analytics-initiatives.html)
(the book of the week from 01 Aug 2022 to 05 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Mertz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Mertz
David is founder of KDM Training, a partnership dedicated to educating developers and data scientists in machine learning and scientific computing. He created the data science training program for Anaconda Inc. and was a senior trainer for them. With the advent of deep neural networks he has turned to training our robot overlords as well.
He was honored to work for 8 years with D. E. Shaw Research, who have built the world’s fastest, highly-specialized (down to the ASICs and network layer), supercomputer for performing molecular dynamics.
David was a Director of the PSF for six years, and remains co-chair of its Trademarks Committee and of its Scientific Python Working Group. His columns, Charming Python and XML Matters, written in the 2000s, were the most widely read articles in the Python world. He has written previous books for Packt, O’Reilly and Addison-Wesley, and has given keynote addresses at numerous international programming conferences.
[](https://twitter.com/mertz_david)
[](https://linkedin.com/in/dmertz)
[](https://github.com/DavidMertz)
### Books
* [Cleaning Data for Effective Data Science](https://datatalks.club/books/20210621-cleaning-data-for-effective-data-science.html)
(the book of the week from 21 Jun 2021 to 25 Jun 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Gates – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Gates
David Gates is the founder of [Accents Welcome](https://accentswelcome.com/)
, an English language school dedicated to helping data professionals develop exceptional communication skills.
[](https://linkedin.com/in/david-gates-a84750b)
[](https://accentswelcome.com/)
### Articles
* [Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals](https://datatalks.club/blog/data-narrative.html)
* [Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences](https://datatalks.club/blog/simplifying-concepts.html)
### Events
* Essential Communication Skills for Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=ZRtVBflBkuc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Stephenson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Stephenson
David Stephenson is faculty at the University of Amsterdam Business School and founder of dsianalytics, specializing in launching data science projects and providing trainings for analytics professionals. David has consulted and led global analytics projects for large and small companies across a wide range of sectors, including well-known multi-nationals such as eBay, adidas, ABN AMRO, and IKEA. He has also served as expert advisor to leading investment, private equity and management consulting firms. In addition to advising and leading projects, he has conducted trainings for professionals in numerous companies across the globe.
[](https://linkedin.com/in/dmstephenson)
### Books
* [Business Skills for Data Scientists](https://datatalks.club/books/20210823-business-skills-for-data-scientists.html)
(the book of the week from 23 Aug 2021 to 27 Aug 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Demetrios Brinkmann – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Demetrios Brinkmann
Demetrios is constantly creating as he tries to navigate this thing we call life. He has recently moved to the countryside in Germany after spending the last 10 years in the Basque Country.
In April 2020 he fell into leading the MLOps community which aims to bring clarity around the operational side of Machine Learning. Since diving into the nitty gritty of Machine Learning he has felt a strong calling to explore the ethical issues surrounding the new tech he covers.
The MLOps community is the largest and most active community boasting over 2500 slack members and 25k youtube views of videos on various ML Operations themes. Various experts from all over the world come together daily to assist one another on current obstacles or share their vision of the landscape at large.
[](https://twitter.com/dpbrinkm)
[](https://linkedin.com/in/dpbrinkm)
[](https://mlops.community/)
### Events
* Building and Growing Online Communities ([watch on youtube](https://www.youtube.com/watch?v=ByCE1vSrIr8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-Online-Tech-Communities---Demetrios-Brinkmann-eu35fo)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# ML Zoomcamp: Free Machine Learning Engineering Course and Certification – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
ML Zoomcamp: Free Machine Learning Engineering Course and Certification
ML Zoomcamp: Free Machine Learning Engineering Course
=====================================================
### Master Machine Learning Engineering with Python in 4 Months
25 Nov 2025 by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
Most machine learning courses teach you to build models in Jupyter notebooks and stop there. You learn regression, classification, and neural networks, yet when it’s time to deploy your model to production, you’re left figuring it out alone. The gap between training a model and running it in a real application can derail your ML career before it starts.
Complete ML Zoomcamp curriculum: from machine learning fundamentals to production deployment
ML Zoomcamp solves this by teaching you complete machine learning engineering, covering the entire pipeline: from building models with Python to deploying them in production environments. You’ll master key ML algorithms like linear regression, logistic regression, decision trees, and deep learning with TensorFlow and PyTorch, then learn to containerize with Docker, build APIs with FastAPI, and scale with Kubernetes and AWS Lambda.
With this machine learning course, you’ll get hands-on experience with the entire ML workflow and two portfolio-ready projects showing you can build, deploy, and maintain production machine learning systems.
> **TL;DR:** Machine Learning Zoomcamp is a free 4-month course teaching machine learning engineering. You’ll learn Python ML basics through to production deployment, build real projects, and join a supportive community. The next cohort starts on September 15, 2025. [Join the course here](https://airtable.com/shryxwLd0COOEaqXo)
> .
Table of Contents
-----------------
* [What is Machine Learning Zoomcamp?](https://datatalks.club/blog/machine-learning-zoomcamp.html#what-is-machine-learning-zoomcamp)
* [Why Learn Machine Learning?](https://datatalks.club/blog/machine-learning-zoomcamp.html#why-learn-machine-learning)
* [Who Is This Course For (and What Are the Prerequisites)?](https://datatalks.club/blog/machine-learning-zoomcamp.html#who-is-this-course-for-and-what-are-the-prerequisites)
* [Course Curriculum: What You’ll Learn](https://datatalks.club/blog/machine-learning-zoomcamp.html#course-curriculum-what-youll-learn-in-the-machine-learning-zoomcamp)
* [How the Machine Learning Zoomcamp Works](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-the-machine-learning-zoomcamp-works)
* [DataTalks.Club Community](https://datatalks.club/blog/machine-learning-zoomcamp.html#what-is-datatalksclub-community-a-place-to-connect-and-learn-with-other-machine-learning-professionals)
* [Machine Learning Zoomcamp vs. Bootcamp](https://datatalks.club/blog/machine-learning-zoomcamp.html#machine-learning-zoomcamp-vs-bootcamp-whats-the-difference)
* [How to Get Started](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-to-get-started-with-the-machine-learning-zoomcamp)
* [Testimonials](https://datatalks.club/blog/machine-learning-zoomcamp.html#testimonials)
* [Frequently Asked Questions](https://datatalks.club/blog/machine-learning-zoomcamp.html#frequently-asked-questions)
What is Machine Learning Zoomcamp?
----------------------------------
Machine Learning Zoomcamp is a free machine learning course that teaches you to build and deploy machine learning models in production environments—bridging the gap between training models in notebooks and running them in real-world applications.
Machine Learning Zoomcamp GitHub repository showing the course materials
While ML Zoomcamp covers the fundamental machine learning algorithms you need to know, it focuses heavily on the practical skills that most courses skip. Instead of stopping at model training, you’ll learn the complete production workflow:
* **Model Persistence and Containerization:** Learn how to save trained models properly and package them using Docker containers, making your applications portable and reproducible. You’ll understand why containerization is essential for production deployments and how tools like Docker and Kubernetes solve real deployment challenges.
* **API Development and Service Integration:** Discover how to transform your models into web services using FastAPI, enabling other applications to make predictions via HTTP requests. You’ll learn to integrate your models into existing systems and build complete end-to-end applications with tools like FastAPI, Docker, Kubernetes, and AWS Lambda.
* **Deployment Strategies:** Understand the fundamental differences between traditional server-based deployments (using virtual machines or Kubernetes clusters) and serverless architectures (like AWS Lambda). You’ll learn the trade-offs, cost implications, and when to choose each approach for your specific use case.
Why Learn Machine Learning?
---------------------------
Machine learning has become a foundational technology across industries, driving the rapid expansion of the global AI market valued at [$184 billion in 2024 and projected to reach $826 billion by 2030](https://www.statista.com/forecasts/1474143/global-ai-market-size)
.
As a subset of artificial intelligence, machine learning is essential for modern AI applications. It enables systems to learn from data and make predictions without hard-coded rules. This technology powers many applications you interact with daily: language models like ChatGPT that process and generate text, image generation tools like DALL-E, computer vision systems in autonomous vehicles, and recommendation engines used by streaming and e-commerce platforms.
This widespread adoption has created significant demand for skilled professionals. According to the [World Economic Forum’s Future of Jobs Report 2025](https://reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf)
, AI and machine learning specialists are among the top three fastest-growing roles between 2025 and 2030, with an expected global net growth of 82%.
Who Is This Course For (and What Are the Prerequisites)?
--------------------------------------------------------
Machine learning is a technical field that requires understanding of programming and mathematical concepts. While you don’t need a PhD in mathematics or computer science to start learning, you should be prepared to work with concepts from linear algebra, calculus, and statistics as you progress. You can start with basic understanding, but you’ll eventually need to grasp these concepts to debug models and make informed decisions.
The learning curve can be steep. Understanding why your model isn’t working, dealing with messy real-world data, and troubleshooting deployment issues takes time and patience. You’ll need to be willing to struggle through problems and dedicate time to consistent practice.
Programming Skills
**Essential.** Daily coding to implement algorithms, process data, and build models. Python is most common.
Mathematical Foundation
**Important.** Linear algebra, calculus, and statistics. Start with basics, deepen over time.
Problem-Solving Mindset
**Critical.** Debugging models and troubleshooting require patience and systematic thinking.
Time Commitment
**Significant.** Building competence takes several months of consistent practice.
Four key elements needed for success in machine learning. You don't need to master everything before starting, but having these in place will help you progress effectively.
If this sounds like you, you’re in the right place:
* You are comfortable with programming (1+ year)
* You are comfortable with mathematical thinking
* You are willing to struggle through problems
* You are willing to dedicate time to consistent practice (expect several months to build competence)
People from various backgrounds enter the field: software developers, data analysts, researchers, and career changers with technical experience. Success typically depends on having a foundation in programming, dedicating time to study and practice, and working through problems systematically.
### Course Prerequisites
This course is designed for anyone with programming experience who wants to learn machine learning engineering. You don’t need a background in mathematics, statistics, or data science to get started.
The only requirement for this course is prior programming experience (1+ year) and familiarity with the command line. If you’re completely new to programming, consider spending a few weeks learning the basics before starting the course.
Software Developers
Adding ML skills
Data Analysts
Moving to ML
Students
Practical skills
Career Changers
With coding background
This course bridges the gap between theoretical ML knowledge and practical engineering skills, making it ideal for professionals who want to deploy models in real-world applications.
The course is particularly suited for:
* Software developers and engineers who want to add machine learning to their skillset
* Data analysts looking to move into machine learning roles
* Students in computer science or related fields seeking practical ML skills
* Career changers with programming experience interested in AI and data science
* Technical professionals who want to understand how to deploy ML models in production
While having experience with Python is helpful, the course covers the essential libraries you’ll need. The key requirement is being comfortable with programming concepts and willing to commit time to learning and completing projects.
Course Curriculum: What You’ll Learn in the Machine Learning Zoomcamp
---------------------------------------------------------------------
The course is divided into two main parts, carefully designed to build both your theoretical knowledge and practical skills.
### Part 1: Machine Learning Foundations
Key technologies and libraries covered in Part 1 of ML Zoomcamp
Part 1 focuses on core machine learning concepts and their practical implementation using Python.
| Topic | Description | Tools |
| --- | --- | --- |
| **Linear Regression and Feature Engineering** | Master feature creation, categorical variable handling, and regularization techniques | NumPy, Pandas, Scikit-Learn |
| **Classification with Logistic Regression** | Learn feature importance and model evaluation | Scikit-Learn, Matplotlib |
| **Decision Trees and Ensemble Methods** | Explore gradient boosting and XGBoost implementation | XGBoost, Scikit-Learn |
| **Neural Networks and Deep Learning** | Build CNNs and implement transfer learning | TensorFlow, PyTorch, Keras |
We’ll also use Jupyter Notebooks for development, and Matplotlib and Seaborn for visualization.
### Part 2: Production Deployment
Deployment technologies used in Part 2 of ML Zoomcamp for putting ML models into production
Part 2 focuses on model deployment, which involves putting machine learning models into production:
| Topic | Description | Tools |
| --- | --- | --- |
| **Model Deployment** | Move models from notebooks to services and applications | FastAPI, Pipenv, Docker |
| **Serverless Deep Learning** | Deploy models efficiently using serverless architecture | AWS Lambda, ONNX Runtime |
| **Container Orchestration** | Automate deployment, scaling, and management of containerized applications | Kubernetes, TensorFlow Serving |
| **KServe (optional)** | Advanced deployment capabilities for production ML systems | KServe |
### Capstone Project
During ML Zoomcamp, you’ll finalize and submit two capstone projects:
* [Midterm project](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#midterm-project)
in the middle of the course
* [Capstone project 1](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#capstone-project-1)
and/or [Capstone project 2](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#capstone-project-2-optional)
at the end of the course
These projects allow you to apply everything you’ve learned and make a great addition to your GitHub profile. You’ll choose a problem that interests you, find a suitable dataset, develop your model, and deploy it as a web service.
A local deployment architecture using Kubernetes with Kind from one of the students' projects
Students from previous cohorts built a wide range of projects. For example, one graduate created a [blood cell classifier for cancer prediction](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
and another built a [waste classifier](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
with over 15,000 images, achieving 93% accuracy.
A snapshot of the ImageNet data used in the waste classifier project
How the Machine Learning Zoomcamp Works
---------------------------------------
### Theory and Practice
Our lectures make machine learning theory accessible and engaging through real-world examples. We provide code demonstrations directly in the lectures to illustrate the implementation of concepts, making it easier to apply them in your projects.
Example from ML Zoomcamp lecture: bridging theory and practice with hands-on Python implementation
### GitHub Repository: The Central Hub of the Course
All course materials live in the [GitHub repository](https://github.com/DataTalksClub/machine-learning-zoomcamp)
. It serves as a central hub for the course, providing easier navigation through all course materials.
Machine Learning Zoomcamp GitHub repository: your central hub for all course materials
The lectures are pre-recorded and available on YouTube in our official [Machine Learning Zoomcamp YouTube Playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, so you can watch at your own pace.
Machine Learning Zoomcamp YouTube playlist with all course lectures
### Homework Assignments
To strengthen your understanding of the course material, we provide homework assignments for each module.
ML Zoomcamp schedule showing the course schedule and submission deadlines
While these assignments don’t contribute to your certification, they serve as valuable practice opportunities.
Your performance is recorded on an optional anonymous leaderboard, fostering healthy competition among participants and encouraging you to strive for excellence.
Course leaderboard displaying student progress and achievements anonymously
You can earn additional points through learning in public, the process of documenting and sharing your journey on blogs, YouTube, or social media platforms. Our upcoming section explores this concept in detail.
### Learning in Public
A unique feature is our “learning in public” approach, inspired by [Shawn @swyx Wang](https://datatalks.club/podcast/s03e07-market-yourself.html)
’s [article](https://www.swyx.io/learn-in-public)
. We believe that everyone has something valuable to contribute, regardless of their expertise level.
Extract from Shawn @swyx Wang's article explaining the benefits of learning in public
Throughout the course, we actively encourage and incentivize learning in public. By sharing your progress, insights, and projects online, you earn additional points for your homework and projects.
Course leaderboard highlighting bonus points earned through public learning activities
This not only demonstrates your knowledge but also builds a portfolio of valuable content. Sharing your work online also helps you get noticed by social media algorithms, reaching a broader audience and creating opportunities to connect with individuals and organizations you may not have encountered otherwise.
Many of our graduates have shared that their social media presence has helped them attract [job offers](https://www.linkedin.com/feed/update/urn:li:share:7380134597839187968/)
and [collaborations](https://www.linkedin.com/feed/update/urn:li:share:7376510719581646849/)
.
### How to Get a Certificate
Machine Learning Zoomcamp certificate awarded upon successful completion
To receive a certificate, you’ll need to complete and submit 2 projects:
1. **Complete 2 projects**: Submit either a midterm project and a capstone project, OR two capstone projects
2. **Submit on time**: Meet the project submission deadlines to qualify for certification
3. **Peer review**: Evaluate and provide feedback on 3 fellow students’ projects during the peer review process
What is DataTalks.Club Community? A Place to Connect and Learn with Other Machine Learning Professionals
--------------------------------------------------------------------------------------------------------
[DataTalks.Club](https://datatalks.club/)
is a global community of 80,000+ data professionals who connect on [Slack](https://datatalks.club/slack.html)
to share knowledge, ask career questions, and discuss everything from analytics and visualization to machine learning and data engineering. As one of the largest digital groups dedicated to data, it’s where you’ll find data scientists, ML engineers, data analysts, and enthusiasts at all career stages.
When you join a cohort, the dedicated course channel becomes your home base. Here, you’ll troubleshoot problems with peers working through the same challenges and share your progress and insights.
Active discussions and support in the ML Zoomcamp Slack community channel
Beyond peer support, there are two ways to get help: our [FAQ repository](https://datatalks.club/faq/machine-learning-zoomcamp.html)
covers common questions and technical issues, and the @ZoomcampQABot in Slack provides quick answers when you need them.
Machine Learning Zoomcamp QABot in Slack providing quick answers when you need them
Machine Learning Zoomcamp vs. Bootcamp: What’s the Difference?
--------------------------------------------------------------
We often get asked what the difference is between the Machine Learning Zoomcamp and paid bootcamps.
Below, we list the key features of the Machine Learning Zoomcamp and how they compare to paid bootcamps.
| Feature | ML Zoomcamp (Cohort) | ML Zoomcamp (Self-paced) | Paid Bootcamps |
| --- | --- | --- | --- |
| Cost | Free | Free | $2,000–$10,000+ |
| Format | 4-month cohort with fixed schedule | Learn anytime at your own pace | Fixed schedule, instructor-led |
| Homework | Weekly scored assignments | Available but no scoring | Weekly with instructor feedback |
| Projects | Capstone project with peer review and scoring | Build on your own, no evaluation and scoring | Instructor-reviewed projects |
| Certificate | Yes, after completing project + peer reviews | No certificate | Certificate of completion |
| Community Support | Active Slack + optional live Q&A sessions | Slack community only | Instructor-led, 1:1 or group mentorship |
| Learning in Public | Encouraged with bonus points | Optional | Rarely emphasized |
| Timeline | 4 months (Sep–Dec 2025) | Flexible, self-paced | Typically 12–24 weeks |
| Best For | Career switchers or experienced engineers wanting community and accountability | Self-motivated learners exploring machine learning | Those needing intensive structured mentorship and guidance |
How to Get Started with the Machine Learning Zoomcamp
-----------------------------------------------------
To get started with ML Zoomcamp, you can either join a live cohort or learn at your own pace.
All course materials are freely available in the [Machine Learning Zoomcamp GitHub repository](https://github.com/DataTalksClub/machine-learning-zoomcamp)
, where each module has its own folder (e.g., `01-intro`, `03-classification`). Cohort-specific homework and deadlines are organized in the `cohorts` directory. All lectures are pre-recorded and available in our official [Machine Learning Zoomcamp YouTube Playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
.
### Learn at Your Own Pace
The self-paced mode lets you start anytime and learn at your own pace. You’ll get full access to all course materials and can join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
for support on the `#course-ml-zoomcamp` channel. Homework assignments and solutions are available on the [course platform](https://courses.datatalks.club/)
, and you can build projects for your portfolio.
> Remember, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a certificate. To receive certification, you need to join an active cohort.
### Join a Live Cohort
> **2025 Cohort:** Starts September 15, 2025. [Register here](https://airtable.com/shryxwLd0COOEaqXo)
The live cohort runs once per year from September through December. When you join a live cohort, you’ll work through the same materials as self-paced learners, but with the added structure of a published schedule and the energy of hundreds of peers progressing alongside you.
Each module typically spans one week. You’ll watch the lectures, complete hands-on exercises, and submit a homework assignment. Your submissions get scored automatically and appear on an anonymous leaderboard, creating friendly competition and helping you track your progress.
After completing all modules, the project phase begins. You’ll build your own end-to-end machine learning project, submit it through our form, and peer-review at least three other students’ projects while yours gets reviewed by your peers. This reciprocal process gives you valuable feedback on your work and exposes you to different approaches and solutions you might not have considered.
Even if you join after the official start date, you can still follow along—though some homework forms may already be closed. All active deadlines are listed on the [course platform](https://courses.datatalks.club/)
.
> To earn a certificate, you’ll need enough time to complete 2 projects (either a midterm project and a capstone project, OR two capstone projects) and the required peer reviews. Details are in the [How to Get a Certificate section](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-to-get-a-certificate)
> .
**Ready to join ML Zoomcamp?** Here’s how it works:
1. [Register for the course](https://airtable.com/shryxwLd0COOEaqXo)
, you’ll be automatically accepted into the next cohort
2. Join the [DataTalks.Club Slack community](https://datatalks.club/slack.html)
and the `#course-ml-zoomcamp` channel for updates, questions, and peer support
3. (Optional) Get a head start by exploring the [GitHub repository](https://github.com/DataTalksClub/machine-learning-zoomcamp)
and watching lectures before the cohort officially begins
4. When the cohort starts, you’ll receive an email with the full schedule and submission deadlines
5. Follow the weekly rhythm: watch lectures, complete exercises, submit homework
6. Complete and submit 2 projects (either midterm + capstone OR two capstones), then peer-review three other projects
7. Receive your certificate once your projects pass peer review
The entire journey takes 4 months from start to certificate, and you’ll be part of a global cohort tackling the same challenges at the same time.
[Register for ML Zoomcamp →](https://airtable.com/shryxwLd0COOEaqXo)
Testimonials
------------
> Machine Learning Zoomcamp was exhaustive, with very comprehensive content that covered concepts in depth. You can learn everything from the simplest concepts to preparing and deploying an ML model for production. Additionally, the entire community behind this course is highly participative and collaborative. I would like to thank Alexey Grigorev for all the knowledge he shared with us and his team for providing the support we needed to solve each problem we faced.
>
> [Alexander Daniel Rios](https://www.linkedin.com/in/alexander-daniel-rios)
> ([Source](https://www.linkedin.com/posts/alexander-daniel-rios_mlzoomcamp-activity-7295527609239584768-TWHh)
> )
> Machine Learning Zoomcamp has been an incredible journey, thanks to the expert guidance of Alexey Grigorev. Hugely grateful to Alexey, Timur, and the entire DataTalksClub team for this course, and to my cohort batchmates for the invaluable support that enriched my learning experience. I’m thankful for this programme, which provided challenging coursework that is taught in a very structured and lucid way. The timely assignments & hands-on projects instill the sense of timely delivery, besides equipping us with practical acumen to solve real-life problems.
>
> [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi)
> ([Source](https://www.linkedin.com/posts/activity-7299906113997524994-R-oD?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> Balancing the intensive Machine Learning Zoomcamp with my other engagements was no easy task, but the experience deepened my expertise in machine learning engineering, reinforced my passion for ML deployment and cloud technologies, and strengthened my resilience in handling real-world ML challenges. Thank you, Alexey Grigorev, for this course!
>
> [Patrick Edosoma](https://www.linkedin.com/in/patrickedosoma)
> ([Source](https://www.linkedin.com/posts/patrickedosoma_machinelearning-mlzoomcamp-datascience-activity-7299090071201193985-JyuC)
> )
> Highly recommend the ML Zoomcamp for anyone wanting a structured path to production-ready machine learning. A big thank you - Alexey Grigorev and to the team at DataTalksClub for providing such a well-structured and engaging course.
>
> [Abdiaziz Mohamed](https://www.linkedin.com/in/abdiaziz-mohamed)
> ([Source 1](https://www.linkedin.com/posts/abdiaziz-mohamed_machinelearning-deployment-docker-activity-7257086439333523456-CyK4)
> , [Source 2](https://www.linkedin.com/posts/abdiaziz-mohamed_machinelearningzoomcamp-machinelearning-kubernetes-activity-7277039208072904704-OAiY?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)
> )
> A huge thank you to Alexey Grigoriev for creating such an amazing course—and making it free! It’s truly inspiring.
>
> [Guilherme Pereira](https://www.linkedin.com/in/guilherme-torres-pereira)
> ([Source](https://www.linkedin.com/posts/guilherme-torres-pereira_alexeygrigoriev-mlzoomcamp-machinelearning-activity-7396336012018356224-sK27)
> )
> Huge thanks to Alexey Grigorev and the DataTalksClub community for the incredible support and clarity throughout. The open-source spirit and collaborative notes made the learning experience even richer.
>
> [Rajendra Rawale](https://www.linkedin.com/in/rajendra1x)
> ([Source](https://www.linkedin.com/posts/rajendra1x_machinelearning-mlzoomcamp-datatalksclub-activity-7378450260999852032-V5Z1)
> )
Frequently Asked Questions
--------------------------
How is ML Zoomcamp structured?
The course runs for 4 months (September - December) and is divided into two main parts: Part 1 covers ML foundations (regression, classification, decision trees, neural networks with TensorFlow and PyTorch), and Part 2 focuses on production deployment (FastAPI, Docker, Kubernetes, AWS Lambda, serverless architectures). You’ll complete weekly homework assignments and 2 projects (either midterm + capstone OR two capstones). Expect to dedicate around 10 hours per week for coursework and projects. [Learn more about the curriculum](https://datatalks.club/blog/machine-learning-zoomcamp.html#course-curriculum-what-youll-learn-in-the-machine-learning-zoomcamp)
.
What's new in the 2025 ML Zoomcamp edition?
The 2025 edition features several major updates: 1) Deployment module updated to FastAPI (replacing Flask) with modern deployment tools, 2) Neural networks are taught with PyTorch (theory videos in Keras are retained, with additional PyTorch implementation videos), and 3) Deep learning deployment uses ONNX Runtime on AWS Lambda (replacing TensorFlow Lite). These updates reflect current industry best practices and tools.
How do I get started with ML Zoomcamp?
You can choose between two learning paths: self-paced learning, where you can start immediately with pre-recorded materials freely available on [GitHub](https://github.com/DataTalksClub/machine-learning-zoomcamp)
and learn at your own pace with [Slack community](https://datatalks.club/slack.html)
support, or [joining our live cohort](https://airtable.com/shryxwLd0COOEaqXo)
starting on September 15, 2025, to learn alongside peers with structured deadlines, submit homework for automatic scoring, complete 2 peer-reviewed projects (either midterm + capstone OR two capstones), and earn a certificate. Certificates are only available for cohort participants. [Learn more about getting started](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-to-get-started-with-the-machine-learning-zoomcamp)
.
What's included in the live cohort?
The live cohort includes a structured learning path with deadlines, automatically scored homework assignments, peer interaction and community support through [Slack](https://datatalks.club/slack.html)
, the opportunity to earn a certificate through submitting 2 projects (either midterm + capstone OR two capstones) and peer review, and access to all pre-recorded course materials on [YouTube](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
. [See how it works](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-the-machine-learning-zoomcamp-works)
.
How do I get certified?
To earn a certificate, you’ll need to complete and submit 2 projects: either a [midterm project](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#midterm-project)
and a [capstone project](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#capstone-project-1)
, OR two [capstone projects](https://github.com/DataTalksClub/machine-learning-zoomcamp/tree/master#capstone-project-1)
. You’ll also need to review 3 peers’ projects by the deadline. Projects must be completed individually, and you must be part of a cohort to be eligible.
How is homework scored?
All homework assignments are automatically scored when you submit your answers through the homework form. You’ll see your results on an anonymous leaderboard, which fosters healthy competition among participants. You can earn additional bonus points through “learning in public”—sharing your progress, insights, and projects on social media, blogs, or YouTube. While homework doesn’t count toward certification, it provides valuable practice and helps you track your progress throughout the course.
Can I join ML Zoomcamp after the course has started?
Yes! While you might miss some homework deadlines, you can still join and get certified by completing 2 projects (either midterm + capstone OR two capstones) and the required peer reviews. All course materials remain accessible.
Why should I learn machine learning in 2025?
Machine learning has become a foundational technology driving the global AI market, valued at [$184 billion in 2024 and projected to reach $826 billion by 2030](https://www.statista.com/forecasts/1474143/global-ai-market-size)
. ML powers applications you use daily: ChatGPT, DALL-E, autonomous vehicles, and recommendation engines on streaming platforms. According to the [World Economic Forum’s Future of Jobs Report 2025](https://reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf)
, AI and machine learning specialists are among the top three fastest-growing roles between 2025 and 2030, with an expected global net growth of 82%. The demand for skilled ML professionals continues to grow across all industries. [Learn more about why learn machine learning](https://datatalks.club/blog/machine-learning-zoomcamp.html#why-learn-machine-learning)
.
What Python skills do I need?
You should be familiar with basic Python concepts like variables, libraries, and Jupyter notebooks. If you need to brush up, we recommend taking our Introduction to Python course first.
How do I join the Slack community?
Join our active [Slack community](https://datatalks.club/slack.html)
of 80,000+ data professionals on the #course-ml-zoomcamp channel. You can troubleshoot problems with peers, share your progress, ask questions, and get help from the @ZoomcampQABot. The course also has a comprehensive [FAQ repository](https://datatalks.club/faq/machine-learning-zoomcamp.html)
covering common questions and technical issues. Sharing your journey on social media with #mlzoomcamp earns you bonus points and helps build your online portfolio. [Learn more about the community](https://datatalks.club/blog/machine-learning-zoomcamp.html#what-is-datatalksclub-community-a-place-to-connect-and-learn-with-other-machine-learning-professionals)
.
Is ML Zoomcamp suitable for beginners?
The course is suitable for anyone with programming experience (1+ year) and command line familiarity. You don’t need a PhD in mathematics or prior ML experience—the course starts from first principles with a gentle math refresher. However, machine learning is technical and requires dedication. You’ll work with concepts from linear algebra, calculus, and statistics. The learning curve can be steep, and you’ll need patience to debug models, handle messy data, and troubleshoot deployment issues. The course is ideal for software developers, data analysts, CS students, and career changers with technical backgrounds. [Learn more about prerequisites](https://datatalks.club/blog/machine-learning-zoomcamp.html#course-prerequisites)
and [who can learn machine learning](https://datatalks.club/blog/machine-learning-zoomcamp.html#who-can-learn-machine-learning)
.
What do I need to get started?
You’ll need a laptop with an internet connection and basic tools installed (Python, Git, Docker). The course uses Jupyter Notebooks for development, and you’ll work with Python libraries including NumPy, Pandas, scikit-learn, TensorFlow, PyTorch, and XGBoost. For deployment, you’ll use Docker, FastAPI, and optionally cloud platforms like AWS Lambda and Kubernetes. For deep learning sections, cloud resources may be recommended for intensive computations.
What's the difference between self-paced and cohort learning?
While all course materials are freely available for self-paced learning on [GitHub](https://github.com/DataTalksClub/machine-learning-zoomcamp)
, joining a cohort offers additional benefits: a structured timeline with regular deadlines, the ability to submit homework for automatic scoring and appear on the leaderboard, submit 2 projects (either midterm + capstone OR two capstones) for peer evaluation, active peer learning and discussion in [Slack](https://datatalks.club/slack.html)
, the opportunity to earn a certificate, and a shared learning experience with others facing similar challenges. Self-paced learners can access all materials and the Slack community but cannot submit homework, participate in project evaluations, or earn certificates. Many students find the cohort structure helps them stay motivated and complete the course successfully. [See the comparison](https://datatalks.club/blog/machine-learning-zoomcamp.html#how-to-get-started-with-the-machine-learning-zoomcamp)
.
What support is available for self-paced learners?
All course materials on [GitHub](https://github.com/DataTalksClub/machine-learning-zoomcamp)
, videos on [YouTube](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, and recordings remain available after the cohort ends, and you can learn at your own pace. You’ll have access to the [Slack community](https://datatalks.club/slack.html)
for support. However, self-paced learning does not include homework submissions, project evaluations, or the ability to earn a certificate. To receive a certificate, you need to [join an active cohort](https://airtable.com/shryxwLd0COOEaqXo)
.
Are there office hours or live sessions?
There are no office hours—all lectures are pre-recorded and available in the [YouTube playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hIhxl5Ji8t4O6lPAOpHaCLR)
, so you can watch them whenever it suits you.
All course materials are in the [ML Zoomcamp GitHub repository](https://github.com/DataTalksClub/machine-learning-zoomcamp)
. Each module has its own folder (for example, 01-intro or 03-classification), while cohort-specific homework and deadlines are located in cohorts/2025.
Occasionally, additional workshops or updated implementation videos are released—there will be additional announcements if this happens.
How can ML Zoomcamp help my career?
To maximize the course’s career impact, we recommend starting your project planning early and building portfolio-worthy projects that solve real problems. Stay engaged with the Slack community and share your learning journey on social media using #mlzoomcamp. Take time to review and learn from other students’ projects. When job hunting, use your projects to demonstrate practical skills in applications and interviews—many of our alumni have successfully leveraged their course projects to showcase their machine learning capabilities during the hiring process.
What is ML Zoomcamp?
The Machine Learning Zoomcamp is a free machine learning engineering course by [DataTalks.Club](https://datatalks.club/)
that teaches you to build and deploy ML models in production environments—bridging the gap between training models in notebooks and running them in real-world applications.
This 4-month course covers the complete ML workflow: from fundamental algorithms (linear regression, logistic regression, decision trees, neural networks) to production deployment (Docker, FastAPI, Kubernetes, AWS Lambda). You’ll learn model persistence, containerization, API development, and deployment strategies while building two portfolio-ready projects. All materials are open and available on [GitHub](https://github.com/DataTalksClub/machine-learning-zoomcamp)
. [Learn more](https://datatalks.club/blog/machine-learning-zoomcamp.html#what-is-machine-learning-zoomcamp)
.
What makes ML Zoomcamp different from other ML courses?
Unlike most ML courses that stop at model training in Jupyter notebooks, ML Zoomcamp teaches the complete production workflow. You’ll learn practical deployment skills that most courses skip: model persistence and containerization with Docker, API development with FastAPI for turning models into web services, and deployment strategies including both server-based deployments (Kubernetes) and serverless architectures (AWS Lambda). You’ll understand trade-offs, cost implications, and when to choose each approach. The course emphasizes hands-on learning with real projects that demonstrate you can build, deploy, and maintain production ML systems—not just train models.
Is ML Zoomcamp really free?
Yes, ML Zoomcamp is completely free. There are no hidden costs, no tuition fees, and no paid tiers. All course materials, videos, homework assignments, and access to the [Slack community](https://datatalks.club/slack.html)
are provided at no cost. Unlike traditional bootcamps that charge $10,000-$20,000+, this course is entirely community-driven and open source.
How does ML Zoomcamp compare to bootcamps?
ML Zoomcamp differs from traditional machine learning bootcamps in several key ways:
1. **Cost**: Completely free vs. $2,000-$10,000+ for bootcamps
2. **Format**: 4-month cohort with fixed schedule vs. typically 12-24 weeks for bootcamps
3. **Community**: [80,000+ member community](https://datatalks.club/slack.html)
vs. smaller closed cohorts
4. **Projects**: 2 peer-reviewed projects (midterm + capstone OR two capstones) vs. instructor-reviewed projects
5. **Learning in Public**: Encouraged with bonus points vs. rarely emphasized
6. **Access**: All materials remain on [GitHub](https://github.com/DataTalksClub/machine-learning-zoomcamp)
forever vs. locked behind paywalls
7. **Flexibility**: Can continue self-paced after cohort vs. rigid bootcamp schedules
ML Zoomcamp is best for career switchers and experienced engineers who want community and accountability. Bootcamps suit those who need intensive structured mentorship. [See detailed comparison](https://datatalks.club/blog/machine-learning-zoomcamp.html#machine-learning-zoomcamp-vs-bootcamp-whats-the-difference)
.
[Register for ML Zoomcamp →](https://airtable.com/shryxwLd0COOEaqXo)
Related Posts
-------------
[### Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps\
\
Earn certificates and gain practical experience in ML, data engineering, MLOps, LLMs, AI development tools, and stock market analytics\
\
Read more](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
[### Data Engineering Zoomcamp: Free Data Engineering Course and Certification\
\
Become a Data Engineer: Master Modern Data Engineering with Hands-On Training\
\
Read more](https://datatalks.club/blog/data-engineering-zoomcamp.html)
[### MLOps Zoomcamp: Free MLOps Course and Certification\
\
Learn to deploy, monitor, and maintain ML models in production with MLflow, Docker, AWS, and monitoring tools\
\
Read more](https://datatalks.club/blog/mlops-zoomcamp.html)
[### LLM Zoomcamp: Free LLM Engineering Course and Certification\
\
Master LLM Engineering: Build Production-Ready AI Applications from Scratch\
\
Read more](https://datatalks.club/blog/llm-zoomcamp.html)
[### AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers\
\
Learn how to integrate AI into real developer workflows, from AI coding assistants to agents, CI/CD, DevOps, and no-code automation.\
\
Read more](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Delina Ivanova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Delina Ivanova
Delina has over 10 years of experience across data and analytics, consulting, and strategy with roles spanning financial services, public sector and CPG industries. She is currently the Associate Director, Data & Insights at HelloFresh Canada where she leads a full service data team, including data engineering, data science, and business intelligence and au
[](https://linkedin.com/in/delina-ivanova)
### Events
* Analytics Use Cases Across Non-Automated Operations ([watch on youtube](https://www.youtube.com/watch?v=TolOCxHdVx4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Daynan Crull – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Daynan Crull
Daynan and his co-founders created Karman+ with the mission to mine near-earth asteroids to provide abundant, sustainable water and mineral resources for the space economy. Daynan leads the science and technology effort, including the strategy for detecting and characterizing candidate asteroids and conducting mission design analysis for resource extraction.
Prior to Karman+, Daynan worked as a Sr. Consultant and Data Scientist at New Light Technologies and specialized in applications of machine learning for remote sensing. He led research projects for several organizations, including the World Bank and the U.S. Federal Emergency Management Agency, investigating urban heat island effect and developing novel remote sensing data science packages, such as this tool to harmonize nighttime light satellite data time series. Prior to that he was a Sr. Data Scientist at GeoPhy, a real estate data company, where he contributed to the development of core inference models and data science methodologies.
Previously, he served the City of New York, most recently as Director of Strategy and Performance for the NYC Dept. of I.T. and Telecoms., where he implemented a statistics-based approach to infrastructure monitoring, and before that as a Sr. Policy Advisor in the Mayor’s Office as part of Michael Bloomberg’s special task force for rebuilding and resiliency following Hurricane Sandy.
Daynan’s research and work has centered on informatics (data retrieval and inference related to information systems). He is particularly interested in the application of machine learning and advances in data processing to improve and scale methods of signal processing for remotely sensed data.
Daynan received an M.S. in informatics from New York University and a B.A. from DePauw University. He serves on the board for 100cameras, a non-profit dedicated to teaching photography, and lives in New York City.
[](https://twitter.com/daynancrull)
[](https://linkedin.com/in/daynan)
[](https://github.com/dcrull)
### Events
* Using Data for Asteroid Mining ([watch on youtube](https://www.youtube.com/watch?v=YxijEUoDCfw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dimitri Visnadi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dimitri Visnadi
Dimitri Visnadi is an independent data consultant with a focus on data strategy. He has been consulting companies leading the marketing data space such as Unilever, Ferrero, Heineken, and Red Bull.
He has lived and worked in 6 countries across Europe in both corporate and startup organizations. He was part of data departments at Hewlett-Packard (HP) and a Google partnered consulting firm where he was working on data products and strategy.
Having received a Masters in Business Analytics with Computer Science from University College London and a Bachelor in Business Administration from John Cabot University, Dimitri still has close ties to academia and holds a mentor position in entrepreneurship at both institutions.
[](https://linkedin.com/in/visnadi)
[](https://thedatafreelancer.com/)
### Events
* Become a Data Freelancer ([watch on youtube](https://www.youtube.com/watch?v=R_EnSa9aZtE)
)
* Taking your Freelance Career to the Next Level ([watch on youtube](https://www.youtube.com/watch?v=S93V8RgwBig)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# David Sweet – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 David Sweet
David Sweet has worked as a quantitative trader at GETCO and a machine learning engineer at Instagram, where he used experimental methods to tune trading systems and recommender systems. This book is an extension of his lectures on tuning quantitative trading systems given at NYU Stern over the past three years.
[](https://twitter.com/phinance99)
[](https://linkedin.com/in/dsweet99)
[](https://github.com/tuningup)
[](http://www.andamooka.org/~dsweet/)
### Books
* [Tuning Up](https://datatalks.club/books/20210816-tuning-up.html)
(the book of the week from 16 Aug 2021 to 20 Aug 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Denise Gosnell – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Denise Gosnell
Dr. Denise Gosnell’s passion for examining, applying, and evangelizing the applications of graph data was ignited during her apprenticeship under Dr. Teresa Haynes and Dr. Debra Knisley during her first NSF Fellowship. This group’s work was one of the earliest applications of neural networks and graph theoretic structure in predictive computational biology. Since then, Dr. Gosnell has built, published, patented, and spoke on dozens of topics related to graph theory, graph algorithms, graph databases, and applications of graph data across all industry verticals.
Currently, Dr. Gosnell is with DataStax where she aspires to build upon her experiences as a data scientist and graph architect. Prior to her role with DataStax, she built software solutions for and spoke at over a dozen conferences on permissioned blockchains, machine learning applications of graph analytics, and data science within the healthcare industry.
[](https://twitter.com/denisekgosnell)
[](https://linkedin.com/in/denisekgosnell)
[](https://github.com/denisekgosnell)
### Books
* [The Practitioner's Guide to Graph Data](https://datatalks.club/books/20210405-the-practitioners-guide-to-graph-data.html)
(the book of the week from 05 Apr 2021 to 09 Apr 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Denis Rothman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Denis Rothman
Denis Rothman graduated from Sorbonne University and Paris-Diderot University, patenting one of the very first word2matrix embedding solutions. Denis Rothman is the author of three cutting-edge AI solutions: one of the first AI cognitive chatbots more than 30 years ago; a profit-orientated AI resource optimizing system; and an AI APS (Advanced Planning and Scheduling) solution based on cognitive patterns used worldwide in aerospace, rail, energy, apparel, and many other fields. Designed initially as a cognitive AI bot for IBM, it then went on to become a robust APS solution used to this day.
[](https://linkedin.com/in/denis-rothman-0b034043)
[](https://github.com/Denis2054)
### Books
* [Transformers for Natural Language Processing](https://datatalks.club/books/20210419-transformers-for-natural-language-processing.html)
(the book of the week from 19 Apr 2021 to 23 Apr 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Dmitry Muzalevskiy – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Dmitry Muzalevskiy
Dmitry Muzalevskiy is a Lead Data Scientist with Audibene. He has more than 8 years of experience in different areas of Machine Learning: telecom, retail, banking, fintech and medtech.
[](https://linkedin.com/in/dm-muzalevskiy)
[](https://github.com/ds-muzalevskiy)
### Events
* Deploying models with AWS Sagemaker ([watch on youtube](https://www.youtube.com/watch?v=2ZOnA19sDpM)
)
* AWS Glue DataBrew ([watch on youtube](https://www.youtube.com/watch?v=HQTKAcuCbTc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Don Jones – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Don Jones
Microsoft MVP Don Jones brings his years of experience as a successful IT trainer to this engaging guide.
[](https://twitter.com/concentrateddon)
[](http://donjones.com/)
### Books
* [Own Your Tech Career](https://datatalks.club/books/20211129-own-your-tech-career.html)
(the book of the week from 29 Nov 2021 to 03 Dec 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Eleni Stamatelou – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Eleni Stamatelou
I’m Elena Stamatelou, a machine learning researcher and educator passionate about using Data Science to improve healthcare and save human lives. My expertise includes signal processing, deep learning, and data-driven design.
[](https://linkedin.com/in/elenistamatelou)
[](https://github.com/stamatelou)
### Events
* Bridging Data Science and Healthcare ([watch on youtube](https://www.youtube.com/watch?v=pDOwlulDh0c)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Elena Samuylova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Elena Samuylova
Elena Samuylova, Co-founder & CEO Evidently AI.
Elena is a CEO and Co-founder at Evidently AI, a startup developing open-source tools to analyze and monitor the performance of machine learning models.
She has been active in the applied machine learning space since 2014. Previously, she co-founded and served as a CPO of an industrial AI startup. She worked with global metal and chemical companies to implement machine learning for production optimization. Prior to that, she led business development at Yandex Data Factory, an enterprise AI division of Yandex. She focused on delivering ML-based solutions to retail, banking, telecom, and other industries. In 2018, Elena was named 50 Women in Product Europe by Product Management Festival.
[](https://twitter.com/elenasamuylova)
[](https://linkedin.com/in/elenasamuylova)
### Events
* DataTalks.Club Conference: Product and Process ([watch on youtube](https://www.youtube.com/watch?v=dvzPU43tqFM)
)
* How Your Machine Learning Project Will Fail ([watch on youtube](https://www.youtube.com/watch?v=bVxHQH2-PMo)
)
* DataTalks.Club Summer Marathon: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
* I Want to Build a Machine Learning Startup! ([watch on youtube](https://www.youtube.com/watch?v=DiDs5aMjEWg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/I-Want-to-Build-a-Machine-Learning-Startup----Elena-Samuylova-e139ste)
)
* A Gentle Introduction to LLM Evaluations ([watch on youtube](https://www.youtube.com/watch?v=ac6ZB5QEwGU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Eddy Zulkifly – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Eddy Zulkifly
Eddy is a Staff Data Engineer at Kinaxis, building robust data platforms across Google Cloud, Azure, and AWS. With a decade of experience in data, he actively shares his expertise as a Mentor on ADPList and Teaching Assistant at Uplimit. Previously, he was a Senior Data Engineer at Home Depot, specializing in e-commerce and supply chain analytics. Currently pursuing a Master’s in Analytics at the Georgia Institute of Technology, Eddy is also passionate about open-source data projects and enjoys watching/exploring the analytics behind the Fantasy Premier League.
[](https://twitter.com/eddarief)
[](https://linkedin.com/in/eddyzulkifly)
[](https://github.com/eyzyly)
### Events
* From Supply Chain Management to Digital Warehousing and FinOps ([watch on youtube](https://www.youtube.com/watch?v=7ePp6wuxM5s)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Duygu Altinok – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Duygu Altinok
Duygu Altınok is a senior NLP engineer with 12 years of experience in almost all areas of NLP including search engine technology, speech recognition, text analytics, and conversational AI. She authored several publications in the NLP area at conferences such as LREC and CLNLP. She also enjoys working on open-source projects and is a contributor to the spaCy library. Duygu earned her undergraduate degree in Computer Engineering from METU, Ankara in 2010 and later earned her Master’s degree in Mathematics from Bilkent University, Ankara in 2012. She is currently a senior engineer at German Autolabs with a focus on conversational AI for voice assistants. Originally from Istanbul, Duygu currently resides in Berlin, DE with her cute dog Adele.
[](https://linkedin.com/in/duygu-altinok-4021389a)
[](https://github.com/DuyguA)
[](https://duygua.github.io/)
### Books
* [Mastering spaCy](https://datatalks.club/books/20211213-mastering-spacy.html)
(the book of the week from 13 Dec 2021 to 17 Dec 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Douglas Gray – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Douglas Gray
Douglas A. Gray is a seasoned practitioner, leader, and advisor with over 30 years in data and analytics, IT, e-commerce, and consulting, having driven more than $3 billion in business value through analytics and AI. His teams have received numerous industry awards, including the Teradata EPIC and FICO Decision Management Innovation Awards. A founding CTO of Travelocity and early member of American Airlines Decision Technologies, he pioneered the use of analytics in airline operations. Currently, he is Director of Data Science at Walmart Global Tech, optimizing end-to-end fulfillment. Through his company, Optima Analytics, he advises on digital transformation, and he teaches analytics and AI strategy at SMU. He holds an MBA from SMU, an MS in Operations Research from Georgia Tech, and a BS in Mathematical Sciences from Loyola University Maryland.
[](https://linkedin.com/in/doug-gray-06bb4a4)
### Books
* [Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype (Chapman & Hall/CRC Data Science Series)](https://datatalks.club/books/20241118-why-data-science-projects-fail-harsh-realities-of-implementing-ai-and-analytics-without-hype-chapman-hall-crc-data-science-series.html)
(the book of the week from 18 Nov 2024 to 22 Nov 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Elias Nema – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Elias Nema
Elias leads search and recommendations at the OLX Group. He’s captivated by using data to solve user problems, building analytics-driven engineering culture, and experimenting — to make better and faster decisions.
[](https://twitter.com/eliasnema)
[](https://linkedin.com/in/eliasnema)
[](https://eliasnema.com/)
### Events
* DataTalks.Club Conference: Product and Process ([watch on youtube](https://www.youtube.com/watch?v=dvzPU43tqFM)
)
* Building Data-Intensive Teams ([watch on youtube](https://www.youtube.com/watch?v=5rBE6MO4lac)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Eleni Tzirita Zacharatou – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Eleni Tzirita Zacharatou
Eleni Tzirita Zacharatou is a postdoctoral researcher at the DIMA Group at TU Berlin. She conducts research on spatial big data analytics and is further interested in robust stream processing in IoT environments. Her research results have appeared in premier data management venues (like VLDB, CIDR, ICDE), and her work has received the 2018 ACM SIGMOD best demonstration award. Eleni holds a Ph.D. in Computer Science from EPFL and a Diploma - M.Eng. degree in Electrical and Computer Engineering from the National Technical University of Athens.
[](https://linkedin.com/in/eleni-tzirita-zacharatou)
[](https://github.com/heltzi)
[](https://www.user.tu-berlin.de/tzirita/)
### Events
* Advancing Big Data Analytics: Post-Doctoral Research ([watch on youtube](https://www.youtube.com/watch?v=7jgmIQGMhGE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Advancing-Big-Data-Analytics-Post-Doctoral-Research---Eleni-Tzirita-Zacharatou-e1b6f41)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Doug Turnbull – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Doug Turnbull
In 2012, Doug saw search relevance would be central to user experiences. Sadly, search relevance was a topic clouded esoteric mystery. Doug began to democratize this daunting field through blogging and speaking. In 2016, Doug’s book [Relevant Search](https://www.manning.com/books/relevant-search)
woke the industry up to the importance of search quality. Doug now works at Shopify, working to make great commerce search+discovery possible for everyone. Doug co-created [Elasticsearch Learning to Rank](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/)
with the Wikimedia Foundation and contributed to Trey Grainger’s upcoming [AI Powered Search](http://aipoweredsearch.com/)
. You can find Doug at [his website](http://softwaredoug.com/)
where he blogs about search and data.
[](https://twitter.com/softwaredoug)
[](https://linkedin.com/in/softwaredoug)
[](https://github.com/softwaredoug)
[](https://softwaredoug.com/)
### Events
* DataTalks.Club Summer Marathon: Machine Learning in Production ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* Why Your Search Relevance Project Will Fail ([watch on youtube](https://www.youtube.com/watch?v=Ms9QBBB8MxE)
)
### Books
* [Relevant Search](https://datatalks.club/books/20210712-relevant-search.html)
(the book of the week from 12 Jul 2021 to 16 Jul 2021)
* [AI-Powered Search](https://datatalks.club/books/20211101-ai-powered-search.html)
(the book of the week from 01 Nov 2021 to 05 Nov 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Elle O'Brien – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Elle O'Brien
Elle is a data scientist at Iterative, a startup building open source software tools for machine learning, and a lecturer at the University of Michigan School of Information. She completed her PhD at the University of Washington where she conducted research on speech and hearing using mathematical models. Elle is broadly interested in developing methods, standards, and educational resources for anyone who works with data.
[](https://twitter.com/DrElleOBrien)
[](https://linkedin.com/in/drelleobrien)
[](https://github.com/elleobrien)
[](https://elle-obrien.com/)
### Events
* Developer Advocacy for Data Science ([watch on youtube](https://www.youtube.com/watch?v=jv5W4jXk4P4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Developer-Advocacy-for-Data-Science---Elle-OBrien-epcbak)
)
* DataTalks.Club Conference: ML in Production ([watch on youtube](https://www.youtube.com/watch?v=og1DG1KZ71c)
)
* Continuous Integration for Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=A3OEaaiGPhk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Emeli Dral – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Emeli Dral
Emeli Dral is a Co-founder and CTO at Evidently AI, a startup developing tools to analyze and monitor the performance of machine learning models.
Earlier, she co-founded an industrial AI startup and served as the Chief Data Scientist at Yandex Data Factory. She led over 50 applied ML projects for various industries - from banking to manufacturing. Emeli is a data science lecturer at GSOM SpBU and Harbour.Space University. She is a co-author of the Machine Learning and Data Analysis curriculum at Coursera with over 100,000 students. She also co-founded Data Mining in Action, the largest open data science course in Russia.
[](https://twitter.com/EmeliDral)
[](https://linkedin.com/in/emelidral)
### Events
* Machine Learning Performance Monitoring ([watch on youtube](https://www.youtube.com/watch?v=iiLadbM_It8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Emil Bogomolov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Emil Bogomolov
Research engineer of the ADASE research group at Skoltech, a graduate of the department of System Analysis of the CS MSU and the MailRu Technosphere. An enthusiast in the field of machine learning and computer vision. Contributor of open-source frameworks. Multiple winner of data analysis hackathons.
He is the author of articles at international conferences: WACV on the topic of segmentation and detection of human body parts and posture, and at CVPR on the topic of segmentation and restoration of objects on 3D scans.
Previously, he worked in the field of data analysis in retail. He was engaged in processing big data and detecting anomalies using a model based on gradient boosting.
[](https://linkedin.com/in/emil-bogomolov)
[](https://github.com/zetyquickly)
### Events
* PyTorch Contributor's Guide: How and Why? ([watch on youtube](https://www.youtube.com/watch?v=RCfnWe9VVGM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ellen König – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ellen König
Ellen König is an accomplished engineering leader with deep expertise in data, backend, and machine learning systems. As Head of Engineering at alcemy, she leads two cross-functional teams delivering an ML-powered quality monitoring platform for the concrete industry. Under her leadership, alcemy scaled its customer base from one to over ten B2B clients in less than two years. Ellen has optimized engineering structures, accelerated delivery through agile process improvements, and grown diverse, high-performing teams through mentorship and talent development.
Before joining alcemy, Ellen founded and led the data engineering department at WhereIsMyTransport, building a global data strategy and team supporting products across Latin America and Southeast Asia. She also worked at Thoughtworks, SoundCloud, and Native Instruments, where she developed data platforms, led A/B testing programs, and introduced business intelligence systems that strengthened data literacy across organizations.
With a background spanning software development, data science, and engineering management, Ellen is passionate about building scalable data platforms and responsible machine learning systems. Her technical expertise includes Python, Airflow, BigQuery, Databricks, and GCP. Beyond the technical, she focuses on creating inclusive engineering cultures where data-informed decisions drive product excellence.
[](https://twitter.com/ellen_koenig)
[](https://linkedin.com/in/ellenkoenig)
[](https://github.com/ellenkoenig)
[](https://www.ellenkoenig.de/)
### Events
* From Data Science to Data Engineering ([watch on youtube](https://www.youtube.com/watch?v=3TTu-hYzxeg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ella (Wati) Sahnan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
sahnan.jpg) Ella (Wati) Sahnan
Ella (Wati) Sahnan is an experienced IT professional with a diverse background in teaching, customer support, and data protection. As an IT Executive at All Eights (Singapore) Pte Ltd from 2015 to 2018, she led the digitization of operations, streamlined data systems, and ensured PDPA compliance. Ella’s tech journey spans from early coding in Notepad++ to utilizing AI tools like ChatGPT, yet she remains committed to the core values of ethics and empathy. Passionate about AI’s potential in healthcare, education, and climate, she advocates for responsible data use that enhances human lives.
[](https://linkedin.com/in/wati-sahnan)
### Events
* Developing Your Career in ML by Studying ([watch on youtube](https://datatalks.club/people/ella(wati)sahnan.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Eric Sims – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Eric Sims
Eric is a Sr. Analyst in Strategy & Analytics at LendingTree, where he focuses on supporting and driving improvements in Marketing and Product for Small Business and Investments verticals. Prior to joining LendingTree he studied at the Institute for Advanced Analytics at North Carolina State University. In his free time, he enjoys exploring Python networks, doing freelance ML projects, playing VR paintball, and contributing to the data science community on LinkedIn.
[](https://linkedin.com/in/ericsims2)
[](https://github.com/EricPostMaster)
### Events
* Getting Started with Network Analytics in Python ([watch on youtube](https://www.youtube.com/watch?v=LwSeYUlvvtE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Emmanuel Ameisen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Emmanuel Ameisen
Emmanuel Ameisen, a machine learning engineer at Stripe, implemented and deployed predictive analytics and machine learning solutions for Local Motion and Zipcar. Recently, he led Insight Data Science’s AI program, directing more than a hundred machine learning projects. Emmanuel holds graduate degrees in artificial intelligence, computer engineering, and management from three of France’s top schools.
[](https://twitter.com/mlpowered)
[](https://linkedin.com/in/ameisen)
[](https://github.com/hundredblocks)
### Books
* [Building Machine Learning Powered Applications](https://datatalks.club/books/20211122-building-machine-learning-powered-applications.html)
(the book of the week from 22 Nov 2021 to 26 Nov 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Emmanuel Raj – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Emmanuel Raj
Emmanuel Raj is a Finland-based Lead Machine Learning Engineer with 7+ years of industry experience. He is the author of the “Engineering MLOps” book, Lead Machine Learning Engineer at Relex Solution and an ML researcher at Arcada University of Applied sciences. He is passionate about democratizing AI and bringing research and academia to industry. He is proficient in building data pipelines, Machine learning models and deploying software to production.
[](https://linkedin.com/in/emmanuelraj7)
[](https://github.com/emmanuelraj7)
### Events
* Load Testing ML Microservices for Robustness and Scalability ([watch on youtube](https://www.youtube.com/watch?v=NE9vl4ma3Tk)
)
### Books
* [Engineering MLOps](https://datatalks.club/books/20210705-engineering-mlops.html)
(the book of the week from 05 Jul 2021 to 09 Jul 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Engin Yöyen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Engin Yöyen
I am a Software Engineer, dad, humble home chef (emphasis on humble), motorcycle enthusiast, and currently living the Berlin life with my wife and our two tiny humans.
I did dabble in everything from the Internet of Things to Telecommunications, Smart Classrooms, CMS platforms, and more. So yes — I do build software. I’ve even done it at places you might’ve heard of… like Microsoft and eBay. (Name drop: achieved.)
I do co-founded trustpath.io — we’re in the cybersecurity biz, stopping fraud before it even thinks about happening. We like to think of it as digital crime-fighting, minus the capes.
I have a background with everything but the kitchen sink: Computer Science + Psychology (yes, really), Business Administration, and a Master’s in Embedded Software Engineering.
I like tinkering with tech, writing things (sometimes just to see if anyone’s reading), experimenting with new ideas, and learning weird, wonderful stuff no one asked for — but someone, someday, might need.
[](https://linkedin.com/in/enginyoyen)
[](https://enginyoyen.com/about/)
### Books
* [How Software Fails](https://datatalks.club/books/20250922-how-software-fails.html)
(the book of the week from 22 Sep 2025 to 26 Sep 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ertugrul Mutlu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ertugrul Mutlu
Ertuğrul Mutlu is a Computer Engineering student at RWTH Aachen University and a Werkstudent Researcher at Fraunhofer IAIS (Enterprise Information Systems). His work spans reliable AI systems, agentic workflows, applied LLM engineering, and signal‑processing‑based feature extraction. He focuses on building practical, lightweight AI systems that bridge classical methods with modern LLM‑driven agent architectures. He recently published a preprint on wavelet‑based feature engineering and clustering, writes technical articles on dev.to about ML systems and agentic AI, and actively contributes to the open‑source and data/ML community through prototypes, research notes, and talks.
[](https://linkedin.com/in/ertugrul-mutlu)
[](https://github.com/Ertugrulmutlu)
[](https://ertugrulmutlu.github.io/)
### Events
* From Human-in-the-Loop to Agent-in-the-Loop: A Practical Transition Guide ([watch on youtube](https://www.youtube.com/watch?v=HwCR59VuYn4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Fabiana Clemente – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Fabiana Clemente
Fabiana is a Data Scientist with a background that ranges from Business Intelligence to Big Data Development and IoT architecture. Throughout her professional career, she has been leading state-of-the-art projects not only in global companies but also in startups. She has an academic background in Applied Maths and MSc in Data Management combined with nano degrees in Deep Learning and Secure and Private AI. As YData’s Co-Founder and Chief Data Officer, she combines Data Privacy with Deep Learning as her main field of work and research, with the mission to improve data quality.
[](https://twitter.com/fab_clemente)
[](https://linkedin.com/in/fabiana-clemente)
[](https://medium.com/@fabiana_clemente)
### Events
* DataTalks.Club Summer Marathon: Machine Learning in Production ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* The Importance of Data Quality ([watch on youtube](https://www.youtube.com/watch?v=SQULlY5ZOcw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Eugene Yan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Eugene Yan
Eugene Yan works at the intersection of machine learning & product to build pragmatic ML systems that serve customers. He’s currently an Applied Scientist at Amazon. Previously, he led the data science teams at Lazada and uCare.ai. He writes & speaks about data science, ML in production, and career growth at [eugeneyan.com](https://eugeneyan.com/)
and tweets at [@eugeneyan](https://twitter.com/eugeneyan)
.
[](https://twitter.com/eugeneyan)
[](https://linkedin.com/in/eugeneyan)
[](https://github.com/eugeneyan)
[](https://eugeneyan.com/)
### Events
* The Importance of Writing in a Tech Career ([watch on youtube](https://www.youtube.com/watch?v=vXWGd7olv3c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Importance-of-Writing-in-a-Tech-Career---Eugene-Yan-ep17du)
)
* DataTalks.Club Conference: ML Use Cases ([watch on youtube](https://www.youtube.com/watch?v=jvqS1_GnLsk)
)
* Building an ML System for Southeast Asia’s Largest Hospital Group ([watch on youtube](https://www.youtube.com/watch?v=G5F-L7hdqSQ)
)
* DataTalks.Club Behind the Scenes ([watch on youtube](https://www.youtube.com/watch?v=IxTyq96juVE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/DataTalks-Club-Behind-the-Scenes---Eugene-Yan--Alexey-Grigorev-e1d4567)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Erum Afzal – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Erum Afzal
Erum is an enthusiastic speaker, mentor, and lead ML Engineer with a passion for Data Science and Machine Learning. She holds an MS in Information Technology from NUST, Islamabad, Pakistan. Currently, she is a researcher at Justus Liebig University, Germany, pursuing her PhD in AI solutions for teacher training. Erum is also associated with various international bodies in the field of Data Science and Machine Learning. She serves as a Teaching Expert at Women in AI Academy (Germany), where she instructs courses on Data Science and Machine Learning and leads Omdena Academy at Omdena, contributing to numerous projects and receiving accolades as part of the AI wonder girls team.
Previously, Erum taught a deep learning course at Eskewlab Philippines in collaboration with Omdena. She led the WWCode Data Science track, organizing boot camps and workshops in Data Science. Furthermore, Erum served as a Trainer at AIDA Lab, Prince Sultan University, Kingdom of Saudi Arabia, where she was a master trainer of AI courses and conducted research.
[](https://linkedin.com/in/erum-afzal-64827b24)
### Events
* Community Building and Teaching in AI & Tech ([watch on youtube](https://www.youtube.com/watch?v=7SLd5V7z3xQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Evan Shellshear – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Evan Shellshear
Dr. Evan Shellshear is the Managing Director and Group CEO of Ubidy, a global AI-driven recruitment marketplace. He holds degrees in mathematics from the University of Queensland and Universitaet Bielefeld and earned a PhD in Mathematical Economics from the University of Bielefeld, studying Game Theory. With over 15 years of experience in AI and algorithm development across various sectors, Dr. Shellshear has held key roles, including CEO of Biarri, and worked with the Fraunhofer Society on advanced AI projects. An author of four books and nearly 100 articles, he is also an Adjunct Professor at the University of Queensland and Queensland University of Technology, focusing on AI and business analytics.
[](https://linkedin.com/in/eshellshear)
### Books
* [Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype (Chapman & Hall/CRC Data Science Series)](https://datatalks.club/books/20241118-why-data-science-projects-fail-harsh-realities-of-implementing-ai-and-analytics-without-hype-chapman-hall-crc-data-science-series.html)
(the book of the week from 18 Nov 2024 to 22 Nov 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Faisal Masood – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Faisal Masood
I am a Consulting Architect in Red Hat and focus on bringing business value through IT and collaboration. I also have an interest in Machine Learning and love to bring the discipline of Software Engineering to Machine Learning projects.
I am currently living in Australia and worked in North America, Asia, and Australia. I am a father of two girls, so naturally, unicorns are my favorite creatures.
I blog at https://developers.redhat.com/blog/author/fmasoodredhat-com/ and my GitHub URL is https://github.com/masoodfaisal.
[](https://twitter.com/masoodfaisal)
[](https://linkedin.com/in/faisalmas)
[](https://github.com/masoodfaisal)
### Books
* [Machine Learning on Kubernetes](https://datatalks.club/books/20221107-machine-learning-on-kubernetes.html)
(the book of the week from 07 Nov 2022 to 11 Nov 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Filipa Castro – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Filipa Castro
Filipa Castro has worked as a data scientist for the last three years. She started by developing autonomous solutions for the subsea environment and then moved to the surface, as she is currently working for the automotive industry. Apart from work, she belongs to the Lead Team of Data Science for Social Good Portugal (DSSG PT). DSSG PT is an open community of data scientists, data lovers and data enthusiasts that want to tackle problems that really matter. Together they help non-profit, non-governmental and governmental organizations harness the power of their data to improve its impact on the community.
[](https://twitter.com/filipafcastro)
[](https://linkedin.com/in/filipafcastro)
[](https://github.com/filipafcastro)
### Events
* Data Science for Social Good ([watch on youtube](https://www.youtube.com/watch?v=arwHfVX8_cc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Florian Hoenicke – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Florian Hoenicke
Florian has 10 years of experience in the AI field working at Axel-Springer, Deloitte, and SoundCloud. Currently, he works as a Principal Engineer at Jina AI leading the technical development Prompt Engineering and Embedding Models Technology. Florian is serving as an AI policy advisor, providing explanations and insights to members of the European Parliament to enhance their understanding of artificial intelligence.
[](https://twitter.com/florianhoenicke)
[](https://linkedin.com/in/florian-h%C3%B6nicke-b902b6aa)
### Events
* How to build an Agentic Search Flow ([watch on youtube](https://www.youtube.com/live/Gb-LnXu_Hc0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Fernando Doglio – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Fernando Doglio
Fernando Doglio has been working as a Web Developer for the past 10 years. In that time, he’s come to love the web and has had the opportunity of working with most of the leading technologies at the time, such as PHP, Ruby on Rails, MySQL, Node. js, Angular.js, AJAX, REST APIs, and others.
In his spare time, he likes to tinker and learn new things, which is why his Github account keeps getting new repos every month.
He can be contacted on Twitter at: @deleteman123 or you can read more about him and his work at www.fdoglio.com
When not programming, he can be seen spending time with his family.
[](https://twitter.com/deleteman123)
[](https://linkedin.com/in/fernandodoglio)
[](https://github.com/deleteman)
### Books
* [Skills of a Successful Software Engineer](https://datatalks.club/books/20220912-skills-of-successful-software-engineer.html)
(the book of the week from 12 Sep 2022 to 16 Sep 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Gant Laborde – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Gant Laborde
Gant Laborde is an owner of Infinite Red, mentor, adjunct professor, published author, and award-winning speaker. For 20 years, he has been involved in software development and continues strong today. He is an “open sourcerer”, team leader, and aspires to one day become a mad scientist. He blogs, videos, and maintains popular repositories for the community.
[](https://twitter.com/GantLaborde)
[](https://linkedin.com/in/gant-laborde)
[](https://github.com/GantMan)
[](https://gantlaborde.com/)
### Books
* [Learning Tensorflow.js](https://datatalks.club/books/20210329-learning-tensorflow-js.html)
(the book of the week from 29 Mar 2021 to 02 Apr 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Geo Jolly – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Geo Jolly
Geo Jolly is a product leader with expertise in building Artificial Intelligence, Machine Learning, and Big Data Systems. He works at Glovo as a Technical PM.
[](https://twitter.com/geo_jolly_)
[](https://linkedin.com/in/geojolly)
### Events
* Product Management for Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=PjqjPvHliqg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Product-Management-for-Machine-Learning---Geo-Jolly-e1brpvm)
)
* Machine Learning in Digital Identity ([watch on youtube](https://www.youtube.com/watch?v=YoOWHfPuKhk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Gloria Quiceno – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Gloria Quiceno
Gloria Quiceno is a Senior Analytics Engineer at ICE, where she develops reliable, scalable, and cost-efficient data solutions that power strategic decision-making. She led the rebuild of ICE’s business intelligence platform for market and licensing reporting, achieving over €250K in annual savings and establishing a centralized source of truth for stakeholders. Gloria designs and orchestrates ETL and ELT pipelines using Snowflake, AWS Step Functions, and Docker, develops intuitive dashboards in QuickSight, and documents best practices that enhance collaboration and data literacy across the organization.
Before moving into analytics engineering, Gloria worked as a Business Data Analyst at ICE, where she optimized KPI reporting and automated workflows across Snowflake, Redshift, IBM Db2, Oracle, and AWS. Earlier, she built a foundation in research as a neuroscience scientist at institutions including Otto-von-Guericke University Magdeburg and University College London, developing analytical pipelines in C++, Matlab, and Python for complex data experiments.
With her combined expertise in neuroscience and data engineering, Gloria brings a scientific mindset to analytics—balancing precision, clarity, and business impact. She’s passionate about using data to simplify decision-making, foster transparency, and drive measurable outcomes across teams.
[](https://linkedin.com/in/gloria-quiceno)
[](https://github.com/gdq12)
### Events
* From Academia to Data Analytics and Engineering ([watch on youtube](https://www.youtube.com/watch?v=0wANfIvum4U)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Gonçalo Sequeira – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Gonçalo Sequeira
In 2022 I decided to open Hiire - my first real business - focusing on helping companies to hire better and faster every single day without forgetting an amazing candidate experience. 100% digital because I will be travelling the world while I’m expanding Hiire and helping international companies.
I worked for companies like Feedzai, Mercedes-Benz.io, Adyen and Trengo. Hired +500 IT professionals (from Software Engineers to CTOs), and coached recruitment teams to help them go to their next level in Portugal, Germany, Netherlands and Singapore.
Since 2020 I’ve been actively helping great professionals find better jobs and different career paths with personalised coaching programs.
[](https://linkedin.com/in/gsequeira)
### Events
* How to Land Your First Data Engineer Job ([watch on youtube](https://www.youtube.com/watch?v=1AchMU9xf0Q)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Giuseppe Bonaccorso – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Giuseppe Bonaccorso
Giuseppe Bonaccorso is an experienced data science manager with expertise in machine/deep learning. He got his M.Sc. Eng. in Electronics Engineering in 2005 from the University of Catania, Italy and continued his studies (MBA) at the University of Rome “Tor Vergata,” Italy and the University of Essex, UK. His main interests include machine/deep learning, data science strategy, and digital innovation in the healthcare industry.
[](https://twitter.com/GiuseppeB)
[](https://linkedin.com/in/giuseppebonaccorso)
[](https://github.com/giuseppebonaccorso)
### Books
* [Mastering Machine Learning Algorithms - Second Edition](https://datatalks.club/books/20210125-mastering-ml-algorithms-2ed.html)
(the book of the week from 25 Jan 2021 to 29 Jan 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Gráinne McKnight – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Gráinne McKnight
Data Scientist in Berlin ❤️. Worked for past years in fintech on building data products including a customer service chatbot, a geolocation service and models for money laundering and fraud detection and am now working on exciting new things at Spoke. Bayesian at heart, and always happy to chat about reinforcement learning, network science and ML ops.
[](https://linkedin.com/in/gr%C3%A1inne-mcknight-859630189)
[](https://github.com/grainnemcknight)
### Events
* Algorithmic Fairness ([watch on youtube](https://www.youtube.com/watch?v=CMojXBatk2c)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Greg Coquillo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Greg Coquillo
My passion for solving business use cases, such as Price Optimization for commodity-based products has led me to deep dive into the world of Artificial Intelligence. As a Technology Manager at Amazon, I own a roadmap that drives the adoption of AI products that support Private Brands’ Product Safety and Compliance. I work with Data Scientists, Software Engineers and Business Intelligence Engineers to leverage Machine Learning and Big Data Analytics to design predictive models that enhance business decision-making and performance to accelerate growth, reduce Time to Market of products, monitor cross-geo compliance and import requirements, and ensure customer safety.
[](https://linkedin.com/in/greg-coquillo)
### Events
* Product Management Essentials for Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=p4wg0Vd2uD4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Product-Management-Essentials-for-Data-Professionals---Greg-Coquillo-e1dr8g5)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Guillaume Lemaître – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Guillaume Lemaître
Guillaume is an open-source software engineer working at :probably. He is a core maintainer of the scikit-learn and imbalanced-learn libraries. He holds a PhD in medical imaging.
[](https://linkedin.com/in/guillaume-lemaitre-b9404939)
[](https://github.com/glemaitre)
[](https://glemaitre.github.io/)
### Events
* Working as a Core Developer in the Scikit Learn Universe ([watch on youtube](https://www.youtube.com/watch?v=RR6xaYqKJ3o)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hagop Dippel – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hagop Dippel
Hagop Dippel is an Applied Scientist at Zalando, where he focuses on building demand forecasting and inventory optimisation applications. He particularly enjoys bringing research ideas to end-2-end production systems. He’s passionate about deep learning applied to real-world industry use cases and human centred AI. Hagop studied Data Science and Econometrics at the Aix-Marseille University in France. In his free time, you can find him cycling or running (just contact him if you want to jog and/or chat!)
[](https://linkedin.com/in/hagop-boghazdeklian)
### Events
* Probabilistic Demand Forecasting at Scale ([watch on youtube](https://www.youtube.com/watch?v=it7QyLPi4gU)
)
* Inventory Optimization in E-commerce ([watch on youtube](https://www.youtube.com/watch?v=RlXlJK8OXhY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Guy Adams – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Guy Adams
Snowflake Global #1 Data SuperHero! An experienced CTO and VP, I’m passionate about DataOps. I’ve spent 20+ years running software development organisations and now my focus is bringing the principles and business value from DevOps and CI/CD to data. Cofounder of the truedataops.org movement.
Also Dad, technologist, (over) engineer, amateur inventor, skier, mild eccentric.
[](https://twitter.com/guydadams)
[](https://linkedin.com/in/guydadams)
### Books
* [DataOps for Dummies](https://datatalks.club/books/20210913-dataops-for-dummies.html)
(the book of the week from 13 Sep 2021 to 17 Sep 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hannes Hapke – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hannes Hapke
I love solving on Machine Learning problems. In particular, I am interested in Machine Learning Engineering and Matural Language applications.
[](https://twitter.com/hanneshapke)
[](https://linkedin.com/in/hanneshapke)
[](https://github.com/hanneshapke)
[](http://hanneshapke.github.io/)
### Books
* [Building Machine Learning Pipelines](https://datatalks.club/books/20210607-building-machine-learning-pipelines.html)
(the book of the week from 07 Jun 2021 to 11 Jun 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hayden Liu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hayden Liu
Yuxi (Hayden) Liu is a Software Engineer, Machine Learning at Google. He is developing and improving machine learning models and systems for ads optimization on the largest search engine in the world.
[](https://linkedin.com/in/hayden-liu-80445056)
[](https://github.com/haydenliu)
### Books
* [Python Machine Learning By Example](https://datatalks.club/books/20220620-python-machine-learning-by-example.html)
(the book of the week from 20 Jun 2022 to 24 Jun 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Haziqa Sajid – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Haziqa Sajid
Haziqa Sajid is a data scientist and developer advocate specializing in AI, machine learning, and data infrastructure. She produces in-depth technical content on LLMs, vector databases, RAG systems, MLOps, and computer vision. She is the founder of Disruptivera, where she leads distributed teams delivering whitepapers, case studies, developer documentation, and data-driven marketing for global SaaS and AI companies. Her work focuses on translating complex systems into clear, practical resources for engineers and product teams.
[](https://linkedin.com/in/haziqa-sajid-22b53245)
### Articles
* [Open Source and Free AI Agent Evaluation Tools](https://datatalks.club/blog/open-source-free-ai-agent-evaluation-tools.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hélder Russa – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hélder Russa
Helder Russa is a Data Engineering Lead at Jumia with a background in Information Technologies and Data Science. Has over 10 years of professional experience in computer science, with an emphasis on evolving and maintaining data solutions applied to decision making. Nowadays, he works as a lead data engineer at Jumia where he contributes to the strategy definition, design, and implementation of multiple Jumia data platforms. In similitude, and since 2018, he is a co-founder and data architect of ShopAI, a company specialized in deep learning, that leverages the capabilities of the image for optimization of search channels inside webshops.
[](https://linkedin.com/in/hrussa)
### Books
* [Analytics Engineering with SQL and DBT](https://datatalks.club/books/20231106-analytics-engineering-with-sql-and-dbt.html)
(the book of the week from 06 Nov 2023 to 10 Nov 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hiba Jamal – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hiba Jamal
Hiba is an AI Engineer at dltHub. She works with data pipelines, data modelling, LLMs, and ML tasks in her current role. Before that, she has worked at other startups in data roles.
[](https://linkedin.com/in/hiba-jamal)
### Events
* From REST to reasoning: ingest, index, and query with dlt and Cognee ([watch on youtube](https://www.youtube.com/watch?v=MNt_KK32gys)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Himanshu Upreti – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Himanshu Upreti
Himanshu Upreti is the Co-Founder & CTO at Ai Palette leading the technology and vision at the same. At Ai Palette, he is building the world’s first AI Platform that can generate consumer product concepts leveraging million of data points in different languages with the click of a button for the CPG Industry. Prior to Ai Palette, Himanshu worked at Visa Data Labs where he filed two trade secrets and played a major role in restructuring of Visa’s Tech Stack. Himanshu is a graduate in Mathematics and Computing from IIT Guwahati.
[](https://twitter.com/himanshu_iitg)
[](https://linkedin.com/in/hupreti)
[](https://www.aipalette.com/)
### Events
* DataTalks.Club Conference: ML Use Cases ([watch on youtube](https://www.youtube.com/watch?v=jvqS1_GnLsk)
)
* How to use AI in Consumer Food Product Innovation ([watch on youtube](https://www.youtube.com/watch?v=7DuTkKrOZ1Y)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hugo Bowne-Anderson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Hugo Bowne-Anderson
Hugo Bowne-Anderson is Head of Developer Relations at [Outerbounds](https://outerbounds.com/)
, a company committed to building infrastructure that provides a solid foundation for machine learning applications of all shapes and sizes. He is also host of the industry podcast [Vanishing Gradients](https://vanishinggradients.fireside.fm/)
. Hugo is a data scientist, educator, evangelist, content marketer, and data strategy consultant, with extensive experience at Coiled, a company that makes it simple for organizations to scale their data science seamlessly, and DataCamp, the online education platform for all things data. He also has experience teaching basic to advanced data science topics at institutions such as Yale University and Cold Spring Harbor Laboratory, conferences such as SciPy, PyCon, and ODSC and with organizations such as Data Carpentry. He has developed over 30 courses on the DataCamp platform, impacting over 2 million learners worldwide through his own courses. He also created the weekly data industry podcast DataFramed, which he hosted and produced for 2 years. He is committed to spreading data skills, access to data science tooling, and open source software, both for individuals and the enterprise.
[](https://twitter.com/hugobowne)
[](https://linkedin.com/in/hugo-bowne-anderson-045939a5)
### Events
* Data Developer Relations ([watch on youtube](https://www.youtube.com/watch?v=z7BvslwVRbQ)
)
* How to Build and Evaluate AI systems in the Age of LLMs ([watch on youtube](https://www.youtube.com/watch?v=eC3RNuI6ow0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Igor Demidov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Igor Demidov
Machine Learning Engineer
[](https://linkedin.com/in/igor-demidov)
[](https://github.com/ruzarx)
[](https://medium.com/@ruzarx)
### Articles
* [Naming Variables in Machine Learning](https://datatalks.club/blog/naming-variables-in-machine-learning.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Igor Susmelj – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Igor Susmelj
Igor is a co-founder at Lightly – an ETH Spin-Off based in Zurich that is working on a novel data curation platform. Before, he worked for two years in the innovation lab of a financial institution as a software engineer. Igor holds a degree in electrical engineering from ETH Zurich. During his studies, he developed a lot of experience in machine learning and robotics and had multiple successful publications in the area of deep learning.
[](https://twitter.com/ISusmelj)
[](https://linkedin.com/in/igorsusmelj)
[](https://github.com/IgorSusmelj)
[](https://data-annotation.com/)
### Events
* Active and Self-Supervised Learning for Data Selection ([watch on youtube](https://www.youtube.com/watch?v=US1TTdRRUQQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ilia Ivanov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ilia Ivanov
Ilia Ivanov is a Data Scientist in OLX Europe (online marketplace) with 4 years of experience in DS focusing on recommendations and NLP.
### Events
* Deep Learning Recommender Systems ([watch on youtube](https://www.youtube.com/watch?v=LWAQUgJOYm0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Illia Todor – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Illia Todor
Illia Todor is a Data Engineer. He builds Data Platform @ HRS. He’s a certified AWS Cloud Practitioner and sometimes he contributes to open source.
[](https://linkedin.com/in/iamtodor)
[](https://github.com/iamtodor)
[](https://iamtodor.medium.com/)
### Events
* Ingestion and Historization in the Data Lake ([watch on youtube](https://datatalks.club/people/illiatodor.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ilya Boytsov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ilya Boytsov
Ilya is a deep learning research engineer. He works at Wayfair on GenAI projects, he also co-leads the Oxford LLMs project at the University of Oxford and the Street Smart AI meetup in Berlin, Germany.
[](https://twitter.com/ieBoytsov)
[](https://linkedin.com/in/ieboytsov)
### Events
* AI Math Olympiad: A Technical Debrief of the Competition ([watch on youtube](https://www.youtube.com/watch?v=GH_IK_HK1HA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Articles – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Latest Articles
===============
* [Open Source and Free AI Agent Evaluation Tools](https://datatalks.club/blog/open-source-free-ai-agent-evaluation-tools.html)
by [Haziqa Sajid](https://datatalks.club/people/haziqasajid.html)
* [15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide](https://datatalks.club/blog/free-data-engineering-courses.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [LLM Zoomcamp: Free LLM Engineering Course and Certification](https://datatalks.club/blog/llm-zoomcamp.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [MLOps Zoomcamp: Free MLOps Course and Certification](https://datatalks.club/blog/mlops-zoomcamp.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Data Engineering Zoomcamp: Free Data Engineering Course and Certification](https://datatalks.club/blog/data-engineering-zoomcamp.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [ML Zoomcamp: Free Machine Learning Engineering Course](https://datatalks.club/blog/machine-learning-zoomcamp.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [20+ Best Free Machine Learning Courses](https://datatalks.club/blog/free-machine-learning-courses.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [20+ Best Data Science Slack Communities](https://datatalks.club/blog/slack-communities.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
by [Alexander Daniel Rios](https://datatalks.club/people/alexanderdanielrios.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Building Discipline in Machine Learning with ML Zoomcamp](https://datatalks.club/blog/building-discipline-in-machine-learning-with-ml-zoomcamp.html)
by [Alexander Daniel Rios](https://datatalks.club/people/alexanderdanielrios.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How to Build a Data Science Team from Scratch: Complete Hiring Guide](https://datatalks.club/blog/building-data-science-team.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Key Lessons from ML Zoomcamp: Serena Haidar](https://datatalks.club/blog/key-lessons-from-ml-zoomcamp-serena-haidar.html)
by [Serena Haidar](https://datatalks.club/people/serenahaidar.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How to Build a Waste Classifier: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
by [Serena Haidar](https://datatalks.club/people/serenahaidar.html)
, [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Data Team Roles Explained — Alexey Grigorev (OLX) on Skills and Responsibilities](https://datatalks.club/blog/data-roles.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [Data Science Manager vs Expert: Which Role Does Your Company Need?](https://datatalks.club/blog/data-science-manager-vs-data-science-expert.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [DataTalks.Club Community Demographics](https://datatalks.club/blog/datatalks-club-community-demographics.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How Do Data Professionals Use Data Engineering Tools and Practices?](https://datatalks.club/blog/how-do-data-professionals-use-data-engineering-tools-and-practices.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How Do Data Professionals Use MLOps Tools and Frameworks?](https://datatalks.club/blog/how-do-data-professionals-use-ml-and-mlops-tools-and-practices.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How Do Professionals Use Data Engineering Tools and Practices?](https://datatalks.club/blog/how-do-data-professionals-use-data-engineering-tools-and-practices.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How Do Professionals Use LLM Tools and Frameworks?](https://datatalks.club/blog/how-do-professionals-use-llm-tools-and-frameworks.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [How Do Professionals Use AI Tools for Personal Productivity?](https://datatalks.club/blog/ai-tools-for-personal-productivity.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Building an AI Agent that Thrives in the Real World](https://datatalks.club/blog/building-ai-agent-that-thrives-in-real-world.html)
by [Sally-Ann DeLucia](https://datatalks.club/people/sallyanndelucia.html)
* [Winning Solutions from the LLM Zoomcamp 2024 Competition](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [8 Newsletters for Data Science, AI, and ML Enthusiasts](https://datatalks.club/blog/8-newsletters-for-data-science-ai-and-ml-enthusiasts.html)
by [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html)
* [Prepare for (Better) Structured Data Extraction](https://datatalks.club/blog/prepare-for-better-structured-data-extraction.html)
by [Amber Roberts](https://datatalks.club/people/amberroberts.html)
* [How to run PostgreSQL and PgAdmin with Docker](https://datatalks.club/blog/how-to-run-postgresql-and-pgadmin-with-docker.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [A Summary Of The Kaggle Kitchenware Classification Competition: Find Out Who Won!](https://datatalks.club/blog/summary-of-kitchenware-competition.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Guidelines to Get a Data Engineer Job Against the Odds](https://datatalks.club/blog/guidelines-to-get-data-engineer-job-against-odds.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [DataOps: Similarities and Differences with Data Engineering and Data Science](https://datatalks.club/blog/dataops-similarities-and-differences-with-data-engineering-and-data-science.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Important SQL Fact That Everyone Should Know](https://datatalks.club/blog/important-sql-fact-that-everyone-should-know.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [Naming Variables in Machine Learning](https://datatalks.club/blog/naming-variables-in-machine-learning.html)
by [Igor Demidov](https://datatalks.club/people/igordemidov.html)
* [Interview with Valerii Chetvertakov](https://datatalks.club/blog/interview-with-valerii-chetvertakov.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [Do You Know the Golden Rules While Working With Data?](https://datatalks.club/blog/do-you-know-golden-rules-while-working-with-data.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [Regularization in Regression](https://datatalks.club/blog/regularization-in-regression.html)
by [Ksenia Legostay](https://datatalks.club/people/ksenialegostay.html)
* [How to Setup a Lightweight Local Version for Airflow](https://datatalks.club/blog/how-to-setup-lightweight-local-version-for-airflow.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [Data Engineers Aren't Plumbers](https://datatalks.club/blog/data-engineers-arent-plumbers.html)
by [Luís Oliveira](https://datatalks.club/people/luisoliveira.html)
* [The Hiring Process for Data Professionals](https://datatalks.club/blog/hiring-process-for-data-professionals.html)
by [Pavel Chernetsov](https://datatalks.club/people/pavelchernetsov.html)
* [Interview with Ken Wu](https://datatalks.club/blog/interview-with-ken-wu.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [The Essentials of Public Speaking for a Career in Data Science](https://datatalks.club/blog/essentials-of-public-speaking-for-career-in-data-science.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Starting a Career as a Data Scientist](https://datatalks.club/blog/starting-career-as-data-scientist.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [What Open Source Can Do For Your Data Career](https://datatalks.club/blog/what-open-source-can-do-for-your-data-career.html)
by [Mehdi OUAZZA](https://datatalks.club/people/mehdiouazza.html)
* [What is DataOps exactly?](https://datatalks.club/blog/what-dataops-exactly.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Starting a Career in Data Science at 45](https://datatalks.club/blog/starting-career-in-data-science-at-45.html)
by [Tatyjana Ankudo](https://datatalks.club/people/tatyjanaankudo.html)
* [DevOps vs MLOps: Workflows, Monitoring, and Maturity Models Explained](https://datatalks.club/blog/devops-and-mlops-same-thing.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [MLOps in 10 Minutes: Design, Train, and Operate with Proven Practices](https://datatalks.club/blog/mlops-10-minutes.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [How I Landed a Job As a Product Analyst](https://datatalks.club/blog/landing-product-analyst-job.html)
by [Nishant Mohan](https://datatalks.club/people/nishantmohan.html)
* [Deploy ML Models on AWS Lambda with Docker Containers and SAM](https://datatalks.club/blog/ml-deployment-lambda.html)
by [Sejal Vaidya](https://datatalks.club/people/sejalvaidya.html)
* [Data Storytelling: Characters, Conflict, and Conclusion for Data Professionals](https://datatalks.club/blog/data-narrative.html)
by [David Gates](https://datatalks.club/people/davidgates.html)
* [Simplify Technical Concepts: A 3-Step Framework for Non-Technical Audiences](https://datatalks.club/blog/simplifying-concepts.html)
by [David Gates](https://datatalks.club/people/davidgates.html)
* [NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset](https://datatalks.club/blog/ner-reformers.html)
by [Saurav Maheshkar](https://datatalks.club/people/sauravmaheshkar.html)
* [Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide](https://datatalks.club/blog/practical-guide-better-code.html)
by [Oleg Polivin](https://datatalks.club/people/olegpolivin.html)
* [Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data](https://datatalks.club/blog/segmentation.html)
by [Nishant Mohan](https://datatalks.club/people/nishantmohan.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Irina Brudaru – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Irina Brudaru
Irina works as Head of Data & Analytics at Finlex (https://www.linkedin.com/company/finlex-gmbh/). Irina studied computer science in Romania and Germany, and she has worked at a variety of tech companies in Berlin, Amsterdam and the San Francisco Bay Area in the US. She has spent close to 6 years working at Google. She is also very active as a mentor in the area of data analytics, leadership and she holds a variety of teaching positions with a focus on supporting women in tech.
[](https://linkedin.com/in/irinabrudaru)
### Events
* Teaching and Mentoring in Data Analytics ([watch on youtube](https://www.youtube.com/watch?v=saaRRzgHsmE)
)
* Applied Causal Inference ([watch on youtube](https://datatalks.club/people/irinabrudaru.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ioannis Mesionis – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ioannis Mesionis
Ioannis possesses over 4 years of experience as an accomplished data scientist and trusted leader in easyJet’s Data Science and Analytics team. Since joining easyJet in 2019, he has risen to the role of Lead Data Scientist, committed to supporting easyJet’s ambition of becoming the world’s leading data-driven airline.
In his current position, Ioannis works cross-functionally with Digital, Customer & Marketing to produce robust data products and solve business problems while leading the easyJet’s MLOps team to efficiently operationalise, scale, and govern AI solutions enterprise-wide.
[](https://linkedin.com/in/ioannis-mesionis)
[](https://github.com/ioannismesionis)
[](https://ioannismesionis.github.io/)
### Events
* Collaborative Data Science in Business ([watch on youtube](https://www.youtube.com/watch?v=1pExOVuCF8Q)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Isabella Bicalho – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Isabella Bicalho
Isabella is a Machine Learning Engineer and Data Scientist with three years of hands-on AI development experience. She draws upon her early computational research expertise to develop ML solutions. While contributing to open-source projects, she runs a newsletter dedicated to showcasing women’s accomplishments in data science.
[](https://linkedin.com/in/isabella-frazeto)
### Events
* Career advice, learning, and featuring women in ML and AI ([watch on youtube](https://www.youtube.com/watch?v=GifY8Zn-pnU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Itai Admi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Itai Admi
Itai is a software engineer at Treeverse, the company behind lakeFS, an open source platform that delivers resilience and manageability to object-storage based data lakes. Previously, Itai worked at Microsoft and Ridge on data infrastructure, tooling, and performance. He received his B.S. degree in computer science from Tel Aviv University.
[](https://linkedin.com/in/itai-admi)
### Events
* Data Versioning Explained ([watch on youtube](https://www.youtube.com/watch?v=FU2tQSobPNk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ivan Bilan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ivan Bilan
Engineering Manager at Personio, currently working on Identity and Access Management. Main technical interests include building microservices for data intensive applications, MLOps for NLP, as well as Deep Learning research.
[](https://twitter.com/DemiourgosUA)
[](https://linkedin.com/in/ivan-bilan-6904b4b9)
[](https://github.com/ivan-bilan)
[](https://medium.com/@ivanbilan)
### Events
* Introduction to NLP for Industry Use ([watch on youtube](https://www.youtube.com/watch?v=VRur3xey31s)
)
* Leading NLP Teams ([watch on youtube](https://www.youtube.com/watch?v=RJEf6mzxh1w)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Leading-NLP-Teams---Ivan-Bilan-e1c4929)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ivan Brigida – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ivan Brigida
Ivan is a business intelligence analyst with 10+ years of experience in digital marketing, statistical modeling, forecasting and visualization, database programming, data analysis, econometrics, machine learning, and finance.
[](https://twitter.com/python_invest)
[](https://linkedin.com/in/ivan-brigida-0b961319)
[](https://github.com/realmistic)
### Events
* Stock Market Analysis with Python and Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=NThHAEIazFk)
)
* Stock Market Analytics Zoomcamp Pre-Launch Q&A ([watch on youtube](https://www.youtube.com/watch?v=oswTLnjkRUg)
)
* Economics and Automation Workshop: Building a Data Pipeline for Economic Insights ([watch on youtube](https://www.youtube.com/watch?v=MeDUe75WQaQ)
)
* Predicting Financial Time-Series ([watch on youtube](https://www.youtube.com/watch?v=PxAh08Pcmj4)
)
* Stock Market Analytics Zoomcamp 2025 - Pre-Course Live Q&A ([watch on youtube](https://www.youtube.com/watch?v=1N1jIwa1uag)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ivan Potapov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ivan Potapov
Ivan Potapov is a Research Engineer at Zalando, specializing in search. He has taught workshops on data engineering, AI agents, and LLM alignment, helping practitioners bridge software engineering with applied machine learning.
[](https://linkedin.com/in/ivan-sur)
[](https://github.com/ivan-digital)
[](https://blog.ivan.digital/)
### Events
* Practical guide: Fine-tuning Qwen3 with LoRA ([watch on youtube](https://www.youtube.com/watch?v=cayFaWkI39A)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jack Blandin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jack Blandin
Jack Blandin began his career as a Software Engineer in 2015, transitioning into Data Science and Machine Learning in 2017. He has held various machine learning IC and leadership roles at Fi, Wayfair, Trunk Club, and GoHealth, managing teams ranging from 2 to 15. Currently, he is the VP of Data Science & Machine Learning at Fi. On November 15, 2023, he plans to start his own company, a hiring marketplace for data and machine learning professionals. His goal is to design a system that allows candidates to better showcase their actual ability and not need to rely on a resume. He is also currently completing his PhD in Computer Science, and plans to finish in December 2023. His research focuses on machine learning, reinforcement learning, and algorithmic fairness.
[](https://linkedin.com/in/jack-blandin-19761847)
[](https://github.com/jackblandin)
### Events
* The Unwritten Rules for Success in Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=su2M058m3Lw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jacques Peeters – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jacques Peeters
Jacques Peeters is a lead data scientist at ManoMano an European home improvement marketplace. He tackles various topics such as recsys, sales forecasting, bidding algorithms, customer targeting.
Jacques is also a Data Science competitor where he had some success:
* Kaggle - Estimate the uncertainty distribution of Walmart unit sales - 24th out of 900
* ACM RecSys challenge 2019 - Trivago context-aware recommender system - 6th out of 1560
* Kaggle - Elo Merchant Category Recommendation - top1% (42nd out of 4820)
* DrivenData - Power Laws: Forecasting Energy Consumption - 2nd out of 1030 - 7 000€
* Kaggle - Instacart Market Basket Analysis - top 2% (52nd out of 2620)
[](https://linkedin.com/in/j4cquespeeters)
### Events
* A Framework for Feature Engineering and Machine Learning Pipelines ([watch on youtube](https://www.youtube.com/watch?v=HK-kBkERTBY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jakob Graff – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jakob Graff
Jakob Graff is an experienced data science and analytics leader with a proven track record of driving business growth through data-informed decision-making. As Director of Data Science and Data Analytics at diconium, Jakob oversees end-to-end data strategy, building scalable teams and analytics platforms that empower organizations to unlock value from their data.
Before joining diconium, Jakob led data science and product analytics at Inkitt and Babbel, where he built high-performing analytics teams, defined product KPIs, and launched experimentation platforms that shaped company-wide OKRs. Earlier in his career, he worked at King and NERA Economic Consulting, applying statistical modeling and behavioral analysis to large-scale user and market data.
With a strong background in econometrics and product experimentation, Jakob is passionate about bridging the gap between data science, engineering, and business strategy. Based in Berlin, he also contributes to the data community as a co-organizer of SatRday Berlin, promoting open-source collaboration and knowledge sharing within the R and data science ecosystem.
[](https://linkedin.com/in/jakob-graff-a6113a3a)
### Events
* A/B Testing ([watch on youtube](https://www.youtube.com/watch?v=0Gqx1LtqRZU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# James Phoenix – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 James Phoenix
James Phoenix has a background in building reliable data pipelines and software for marketing teams, including automation of thousands of recurring marketing tasks. He has taught 60+ Data Science bootcamps for General Assembly.
[](https://linkedin.com/in/https://www.linkedin.com/in/jamesphoenix/)
### Books
* [Prompt Engineering for Generative AI](https://datatalks.club/books/20240701-prompt-engineering-for-generative-ai.html)
(the book of the week from 01 Jul 2024 to 05 Jul 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jamie Broomall – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jamie Broomall
Jamie Broomall, Senior Software Development Engineer at Whylabs, Technical lead for whylogs
[](https://linkedin.com/in/jamie-broomall-aa60603)
[](https://github.com/jamie256)
### Events
* Speeding up Python Code 500x ([watch on youtube](https://www.youtube.com/watch?v=LmHdEghy204)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Janna Lipenkova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Janna Lipenkova
Janna Lipenkova holds a Master in Chinese Studies and Economics and a PhD in Computational Linguistics. After many years of work in AI and NLP in both academia and industry, she started her own AI and analytics business. She has acquired and managed projects for global companies, collecting first-hand experience in achieving business success with AI. Presently, she focuses on leveraging AI to generate strategic recommendations for companies in central areas such as innovation, digitalization, and sustainability.
[](https://linkedin.com/in/janna-lipenkova-07137399)
[](https://jannalipenkova.com/)
### Books
* [Creating Intelligent Products](https://datatalks.club/books/20240205-creating-intelligent-products.html)
(the book of the week from 05 Feb 2024 to 09 Feb 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jan Schlicht – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jan Schlicht
Jan has more than ten years of experience developing complex algorithms and distributed systems, both in mechanical engineering and the software industry. In addition to distributed systems, his focus is on decentralized machine learning and statistical models. With Katulu, he seeks to reconcile ML and data sovereignty.
[](https://linkedin.com/in/jan-schlicht-a17950b7)
[](https://github.com/nfnt)
### Events
* Federated Learning to the Rescue ([watch on youtube](https://www.youtube.com/watch?v=GtJsn3es2kA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jan Zawadzki – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jan Zawadzki
Jan is the Head of Artificial Intelligence at CARIAD SE, the central software development company of Volkswagen Group. His team aims to push the boundaries of the software-enabled car of the future with the help of Machine Learning. Jan is passionate about advancing the automotive industry through Machine Learning and sharing his knowledge in the fields of Project Management and AI. He is a top contributor to the “Towards Data Science” Publication on Medium and enjoys supporting the team around Deep Learning Luminary Andrew Ng.
[](https://twitter.com/janmzawa)
[](https://linkedin.com/in/jan-zawadzki)
[](https://medium.com/@janzawadzki)
### Events
* DataTalks.Club Summer Marathon: Machine Learning in Production ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* Setting Up AI Projects for Success ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jeanine Harb – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jeanine Harb
Jeanine is working as a Data Engineer at Ubisoft within the Data Office. Her work is focused on applying software engineering principles to data pipelines and improving the lifecycle of ML products thanks to ML/DataOps. Previously, Jeanine worked on a proof-of-concept for autonomous trains and contributed to open-sourcing a large-scale dataset for vision-based railway traffic light detection and recognition. She holds a diploma in Computer Engineering from Saint Joseph University of Beirut (Lebanon) and a Masters degree in Big Data and Machine Learning from Université Paris-Saclay, Télécom Paris and Ecole Polytechnique (France).
Jeanine also organizes data-related events at Ubisoft and, in her spare time, is a trained facilitator for #IamRemarkable workshops.
[](https://linkedin.com/in/jeanineharb)
[](https://github.com/jeanineharb)
### Events
* Productionizing a Feature Store for Fraud Detection at Ubisoft ([watch on youtube](https://www.youtube.com/watch?v=EmZIYcHGX9U)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jeff Katz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jeff Katz
Jeff Katz is a Machine Learning Engineer with a deep background in applied AI, data engineering, and technical education. At AppFolio, he designs and builds AI agents that guide prospects through the sales lifecycle using LangChain, LangGraph, and OpenAI models. His work combines natural language processing, reinforcement learning, and production engineering to bring intelligent automation into real business workflows. He also contributes to AppFolio’s multi-agent systems in Python and Java using PyTorch and large language models.
Jeff is an active open source contributor at the intersection of agentic AI and developer tooling. At All Hands AI, he enhanced OpenDevin’s autonomous coding agent by implementing multilingual linting and integration with the Tree-sitter library—improving the system’s performance and accuracy on SWEBench benchmarks. He also refactored LangChain integrations for Perplexity AI and Neo4J, optimizing embedding pipelines with Hugging Face models.
Before focusing on ML engineering, Jeff founded Jigsaw Labs, where he built and taught full-stack, data science, and MLOps curricula. His students have gone on to roles at companies like Amazon, Facebook, LinkedIn, and Microsoft. Previously, as Lead Instructor and Curriculum Writer at Flatiron School, he launched the data science program from scratch, growing it into one of the most recognized bootcamps in the field.
Earlier in his career, Jeff developed child welfare software at Case Commons, helping build one of the first web-based social service tools, and worked in corporate law before transitioning into tech.
Today, Jeff combines his technical depth and teaching expertise to advance practical, open, and responsible AI systems—empowering both engineers and learners to build the next generation of intelligent software.
[](https://linkedin.com/in/jeff-katz-5417a65)
### Events
* Teaching Data Engineers ([watch on youtube](https://www.youtube.com/watch?v=dFo10l8B6Go)
)
* Getting a Data Engineering Job ([watch on youtube](https://www.youtube.com/watch?v=yvEWG-S1F_M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jekaterina Kokatjuhha – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jekaterina Kokatjuhha
TBA
[](https://linkedin.com/in/jekaterina-kokatjuhha)
### Events
* The Journey of a Data Generalist: From Bioinformatics to Freelancing ([watch on youtube](https://www.youtube.com/watch?v=FRi0SUtxdMw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jens Albrecht – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jens Albrecht
Jens Albrecht is a full-time professor for Computer Science Department at the Nuremberg Institute of Technology. His work focuses on data management and analytics with a focus on text. He holds a doctorates degree in computer science. Before he rejoined academia in 2012, he has been working for over a decade in the industry as consultant and data architect. He is author of several articles on Big Data management and analysis.
[](https://linkedin.com/in/jens-albrecht)
[](https://www.amazon.com/Jens-Albrecht/e/B093JK81YB/)
### Books
* [Blueprints for Text Analytics Using Python](https://datatalks.club/books/20211018-blueprints-for-text-analytics-using-python.html)
(the book of the week from 18 Oct 2021 to 22 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jesse Anderson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jesse Anderson
Jesse Anderson is a Data Engineer, Creative Engineer and Managing Director of Big Data Institute.
He works with companies ranging from startups to Fortune 100 companies on Big Data. This includes training on cutting edge technologies like Apache Kafka, Apache Hadoop and Apache Spark. He has taught over 30,000 people the skills to become data engineers.
He is widely regarded as an expert in the field and for his novel teaching practices. Jesse is published on Apress, O’Reilly, and Pragmatic Programmers. He has been covered in prestigious publications such as The Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired.
[](https://twitter.com/jessetanderson)
[](https://linkedin.com/in/jessetanderson)
[](https://www.jesse-anderson.com/)
### Books
* [Data Teams](https://datatalks.club/books/20210201-data-teams.html)
(the book of the week from 01 Feb 2021 to 05 Feb 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jessi Ashdown – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jessi Ashdown
Jessi Ashdown is a Senior User Experience Researcher for Google Cloud who conducts user studies with customers from all over the world and uses the findings and feedback from these studies to help inform and shape Google’s data governance products to best serve those users’ needs.
[](https://linkedin.com/in/jashdown)
### Events
* Data Governance ([watch on youtube](https://www.youtube.com/watch?v=tJ3v8h7A7RY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Governance---Jessi-Ashdown--Uri-Gilad-e12jmoo)
)
### Books
* [Data Governance: The Definitive Guide](https://datatalks.club/books/20210524-data-governance-the-definitive-guide.html)
(the book of the week from 24 May 2021 to 28 May 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jessica Greene – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jessica Greene
Jessica Greene is a Senior Machine Learning Engineer at Ecosia, where she leverages artificial intelligence to design eco-conscious search experiences and drive measurable climate impact. Her expertise covers the full machine learning lifecycle and modern backend engineering. Jessica develops and maintains reliable microservices in Python and Go, deploys NLP and classification models to enhance query understanding and results, and actively promotes data ethics and privacy, ensuring user trust grows alongside model performance. At Ecosia, she focuses on relevance, sustainability, and responsible AI, enabling everyday searches to contribute to a healthier planet.
A self-taught and community-supported developer, Jessica transitioned to tech in 2017 after careers in film and specialty coffee. She is passionate about broadening access to technology. Since 2018, she has co-organized PyLadies Berlin, mentoring new speakers, building partnerships, and coaching contributors to become leaders in the Python community. Jessica sits on the board of the Python Software Verband and frequently shares insights on inclusive and ethical digital services. Previously, she contributed to Nextcloud through the Rails Girls Summer of Code, boosting her skills in collaborative engineering and user-centered design.
Jessica combines production readiness, community leadership, and a mission-driven mindset. She is skilled in Python, Go, SQL, data pipelines, model training and evaluation, privacy-conscious analytics, and user-focused machine learning for search. Whether enhancing ranking quality, reducing compute usage, or supporting underrepresented groups in tech, Jessica’s goal remains consistent: to use AI for better decision-making and to advance web sustainability.
[](https://twitter.com/sleepypioneer)
[](https://linkedin.com/in/jessica0greene)
[](https://github.com/sleepypioneer)
### Events
* From Roasting Coffee to Backend Development ([watch on youtube](https://www.youtube.com/watch?v=BKqmNdxsBko)
)
* Observing Python Applications Using Prometheus ([watch on youtube](https://www.youtube.com/watch?v=Jb5ikhhttxc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jessie Yaros – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jessie Yaros
I am a PhD candidate in neurobiology and behavior at UC Irvine studying the neural correlates of facial recognition. I first became interested in neuroscience in high school, realizing it was the marriage of my two favorite subjects – biology and psychology. Since then, I’ve spent a rewarding decade-plus learning about the brain. I started at UC San Diego’s Cognitive Science department, studying the mind through biological, anthropological, philosophical, and computational perspectives. This interdisciplinary training primed my current research on the neurological intersection of facial recognition with social and perceptual aspects of race.
[](https://twitter.com/jlyaros)
[](https://linkedin.com/in/jessie-yaros)
### Events
* Modeling the Human Brain ([watch on youtube](https://www.youtube.com/watch?v=6X7P-zwSi7E)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Joe Reis – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Joe Reis
Joe Reis is a business-minded data nerd who’s worked in the data industry for 20 years, with responsibilities ranging from statistical modeling, forecasting, machine learning, data engineering, data architecture, and almost everything else in between. Joe is the CEO and Co-Founder of Ternary Data, a data engineering and architecture consulting firm based in Salt Lake City, Utah. In addition, he volunteers with several technology groups and teaches at the University of Utah. In his spare time, Joe likes to rock climb, produce electronic music, and take his kids on crazy adventures.
[](https://linkedin.com/in/josephreis)
[](https://github.com/JoeReis)
[](https://josephreis.com/)
### Books
* [Fundamentals of Data Engineering](https://datatalks.club/books/20220815-fundamentals-of-data-engineering.html)
(the book of the week from 15 Aug 2022 to 19 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Johanna Bayer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Johanna Bayer
Johanna has a formal background in psychology and computational neuroscience and is now about to complete her PhD in the field of machine learning in clinical neuroimaging. She is joining us from the University of Melbourne, Australia, where she discovered the field of research software engineering. Tied to research software engineering, Johanna has contributed to several open source projects and is an advocate for open source and open science.
[](https://twitter.com/likeajumprope)
[](https://linkedin.com/in/johanna-bayer/)
[](https://github.com/likeajumprope)
### Events
* Doing Software Engineering in Academia ([watch on youtube](https://www.youtube.com/watch?v=K0PdQITQzVQ)
)
* DataTalks.Club Anniversary Interview ([watch on youtube](https://www.youtube.com/watch?v=nCqwZT9zA0M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Book of the Week – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Book of the Week
================
Each week we have a book author coming to DataTalks.Club to answer your questions about their book and, in general, about the topic of their book.
How it works
------------
* [Register on DataTalks.Club](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel in our Slack
* Ask as many questions as you’d like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
Upcoming books
--------------
Upcoming Book Discussions
-------------------------
Archive
-------
Past Book Discussions
---------------------
* [Software Development at Rocket Speed](https://datatalks.club/books/20251006-software-development-at-rocket-speed.html)
by [Nikolay Smorchkov](https://datatalks.club/people/nikolaysmorchkov.html)
(from 06 Oct 2025 to 10 Oct 2025)
* [How Software Fails](https://datatalks.club/books/20250922-how-software-fails.html)
by [Engin Yöyen](https://datatalks.club/people/enginyoyen.html)
(from 22 Sep 2025 to 26 Sep 2025)
* [Machine Learning Algorithms in Depth](https://datatalks.club/books/20250908-machine-learning-algorithms-in-depth.html)
by [Vadim Smolyakov](https://datatalks.club/people/vadimsmolyakov.html)
(from 08 Sep 2025 to 12 Sep 2025)
* [Production‑Ready Data Science](https://datatalks.club/books/20250728-production-ready-data-science.html)
by [Khuyen Tran](https://datatalks.club/people/khuyentran.html)
(from 28 Jul 2025 to 01 Aug 2025)
* [Machine Learning for Tabular Data](https://datatalks.club/books/20250505-machine-learning-for-tabular-data.html)
by [Mark Ryan](https://datatalks.club/people/markryan.html)
, [Luca Massaron](https://datatalks.club/people/lucamassaron.html)
(from 05 May 2025 to 09 May 2025)
* [Why Data Science Projects Fail: The Harsh Realities of Implementing AI and Analytics, without the Hype (Chapman & Hall/CRC Data Science Series)](https://datatalks.club/books/20241118-why-data-science-projects-fail-harsh-realities-of-implementing-ai-and-analytics-without-hype-chapman-hall-crc-data-science-series.html)
by [Evan Shellshear](https://datatalks.club/people/evanshellshear.html)
, [Douglas Gray](https://datatalks.club/people/douglasgray.html)
(from 18 Nov 2024 to 22 Nov 2024)
* [LLM Engineer's Handbook](https://datatalks.club/books/20241104-llm-engineer-s-handbook.html)
by [Paul Iusztin](https://datatalks.club/people/pauliusztin.html)
, [Maxime Labonne](https://datatalks.club/people/maximelabonne.html)
(from 04 Nov 2024 to 08 Nov 2024)
* [Build a Large Language Model (From Scratch)](https://datatalks.club/books/20241017-build-large-language-model-from-scratch.html)
by [Sebastian Raschka](https://datatalks.club/people/sebastianraschka.html)
(from 14 Oct 2024 to 20 Oct 2024)
* [Data Storytelling with Altair and AI](https://datatalks.club/books/20240902-data-storytelling-with-altair-and-ai.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
(from 02 Sep 2024 to 06 Sep 2024)
* [AI Data Privacy and Protection](https://datatalks.club/books/20240715-ai-data-privacy-and-protection.html)
by [Mario Lazo](https://datatalks.club/people/mariolazo.html)
, [Justin Ryan](https://datatalks.club/people/justinryan.html)
(from 15 Jul 2024 to 19 Jul 2024)
* [Prompt Engineering for Generative AI](https://datatalks.club/books/20240701-prompt-engineering-for-generative-ai.html)
by [Michael Taylor](https://datatalks.club/people/michaeltaylor.html)
, [James Phoenix](https://datatalks.club/people/jamesphoenix.html)
(from 01 Jul 2024 to 05 Jul 2024)
* [Fundamentals of Data Observability](https://datatalks.club/books/20240429-fundamentals-of-data-observability.html)
by [Andy Petrella](https://datatalks.club/people/andypetrella.html)
(from 29 Apr 2024 to 03 May 2024)
* [Understanding ETL](https://datatalks.club/books/20240415-understanding-etl.html)
by [Matt Palmer](https://datatalks.club/people/mattpalmer.html)
(from 15 Apr 2024 to 19 Apr 2024)
* [Data-Centric Machine Learning with Python](https://datatalks.club/books/20240408-data-centric-machine-learning-with-python.html)
by [Nakul Bajaj](https://datatalks.club/people/nakulbajaj.html)
, [Jonas Christensen](https://datatalks.club/people/jonaschristensen.html)
, [Manmohan Gosada](https://datatalks.club/people/manmohangosada.html)
(from 08 Apr 2024 to 12 Apr 2024)
* [Creating Intelligent Products](https://datatalks.club/books/20240205-creating-intelligent-products.html)
by [Janna Lipenkova](https://datatalks.club/people/jannalipenkova.html)
(from 05 Feb 2024 to 09 Feb 2024)
* [Distributed Machine Learning Patterns](https://datatalks.club/books/20240115-distributed-machine-learning-patterns.html)
by [Yuan Tang](https://datatalks.club/people/yuantang.html)
(from 15 Jan 2024 to 19 Jan 2024)
* [Analytics Engineering with SQL and DBT](https://datatalks.club/books/20231106-analytics-engineering-with-sql-and-dbt.html)
by [Rui Machado](https://datatalks.club/people/ruimachado.html)
, [Hélder Russa](https://datatalks.club/people/helderrussa.html)
(from 06 Nov 2023 to 10 Nov 2023)
* [Driving Data Quality with Data Contracts](https://datatalks.club/books/20230807-driving-data-quality-with-data-contracts.html)
by [Andrew Jones](https://datatalks.club/people/andrewjones.html)
(from 07 Aug 2023 to 11 Aug 2023)
* [Hands-on Time Series Analysis with Python](https://datatalks.club/books/20230612-hands-on-time-series-analysis-with-python.html)
by [Ashish Patel](https://datatalks.club/people/ashishpatel.html)
, [Vishwas BV](https://datatalks.club/people/vishwasbv.html)
(from 12 Jun 2023 to 16 Jun 2023)
* [Modeling Mindsets](https://datatalks.club/books/20230529-modeling-mindsets.html)
by [Christoph Molnar](https://datatalks.club/people/christophmolnar.html)
(from 29 May 2023 to 02 Jun 2023)
* [GPT-3](https://datatalks.club/books/20230306-gpt-3.html)
by [Sandra Kublik](https://datatalks.club/people/sandrakublik.html)
, [Shubham Saboo](https://datatalks.club/people/shubhamsaboo.html)
(from 06 Mar 2023 to 10 Mar 2023)
* [Snowflake: The Definitive Guide](https://datatalks.club/books/20230123-snowflake-definitive-guide.html)
by [Joyce Kay Avila](https://datatalks.club/people/joycekayavila.html)
(from 23 Jan 2023 to 27 Jan 2023)
* [Reliable Machine Learning](https://datatalks.club/books/20221121-reliable-machine-learning.html)
by [Todd Underwood](https://datatalks.club/people/toddunderwood.html)
, [Kranti K. Parisa](https://datatalks.club/people/krantik-parisa.html)
, [Cathy Chen](https://datatalks.club/people/cathychen.html)
, [Niall Murphy](https://datatalks.club/people/niallmurphy.html)
(from 05 Dec 2022 to 09 Dec 2022)
* [Machine Learning on Kubernetes](https://datatalks.club/books/20221107-machine-learning-on-kubernetes.html)
by [Ross Brigoli](https://datatalks.club/people/rossbrigoli.html)
, [Faisal Masood](https://datatalks.club/people/faisalmasood.html)
(from 07 Nov 2022 to 11 Nov 2022)
* [Comet for Data Science](https://datatalks.club/books/20221107-comet-for-data-science.html)
by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
(from 24 Oct 2022 to 28 Oct 2022)
* [Managing Machine Learning Projects](https://datatalks.club/books/20221010-managing-machine-learning-projects.html)
by [Simon Thompson](https://datatalks.club/people/simonthompson.html)
(from 10 Oct 2022 to 14 Oct 2022)
* [Graph Algorithms for Data Science](https://datatalks.club/books/20220926-graph-algorithms-for-data-science.html)
by [Tomaz Bratanic](https://datatalks.club/people/tomazbratanic.html)
(from 26 Sep 2022 to 30 Sep 2022)
* [The Kaggle Book](https://datatalks.club/books/20220919-kaggle-book.html)
by [Luca Massaron](https://datatalks.club/people/lucamassaron.html)
, [Konrad Banachewicz](https://datatalks.club/people/konradbanachewicz.html)
(from 19 Sep 2022 to 23 Sep 2022)
* [Skills of a Successful Software Engineer](https://datatalks.club/books/20220912-skills-of-successful-software-engineer.html)
by [Fernando Doglio](https://datatalks.club/people/fernandodoglio.html)
(from 12 Sep 2022 to 16 Sep 2022)
* [Essential Math for Data Science](https://datatalks.club/books/20220829-essential-math-for-data-science.html)
by [Thomas Nield](https://datatalks.club/people/thomasnield.html)
(from 29 Aug 2022 to 02 Sep 2022)
* [Fundamentals of Data Engineering](https://datatalks.club/books/20220815-fundamentals-of-data-engineering.html)
by [Joe Reis](https://datatalks.club/people/joereis.html)
, [Matthew Housley](https://datatalks.club/people/matthewhousley.html)
(from 15 Aug 2022 to 19 Aug 2022)
* [Data Analytics Initiatives](https://datatalks.club/books/20220801-data-analytics-initiatives.html)
by [Ondřej Kubera](https://datatalks.club/people/ondrejkubera.html)
, [David Bednar](https://datatalks.club/people/davidbednar.html)
, [Ondřej Bothe](https://datatalks.club/people/ondrejbothe.html)
, [Martin Potančok](https://datatalks.club/people/martinpotancok.html)
(from 01 Aug 2022 to 05 Aug 2022)
* [Hands-On Data Analysis with Pandas - Second Edition](https://datatalks.club/books/20220718-hands-on-data-analysis-with-pandas.html)
by [Stefanie Molin](https://datatalks.club/people/stefaniemolin.html)
(from 18 Jul 2022 to 22 Jul 2022)
* [Grokking Streaming Systems](https://datatalks.club/books/20220704-grokking-streaming-systems.html)
by [Josh Fischer](https://datatalks.club/people/joshfischer.html)
, [Ning Wang](https://datatalks.club/people/ningwang.html)
(from 04 Jul 2022 to 08 Jul 2022)
* [Designing Machine Learning Systems](https://datatalks.club/books/20220627-designing-machine-learning-systems.html)
by [Chip Huyen](https://datatalks.club/people/chiphuyen.html)
(from 27 Jun 2022 to 01 Jul 2022)
* [Python Machine Learning By Example](https://datatalks.club/books/20220620-python-machine-learning-by-example.html)
by [Hayden Liu](https://datatalks.club/people/haydenliu.html)
(from 20 Jun 2022 to 24 Jun 2022)
* [AI-Powered Business Intelligence](https://datatalks.club/books/20220606-ai-powered-business-intelligence.html)
by [Tobias Zwingmann](https://datatalks.club/people/tobiaszwingmann.html)
(from 06 Jun 2022 to 10 Jun 2022)
* [Practical Fairness](https://datatalks.club/books/20220523-practical-fairness.html)
by [Nielsen Aileen](https://datatalks.club/people/nielsenaileen.html)
(from 23 May 2022 to 27 May 2022)
* [Everyday ML Questions](https://datatalks.club/books/20220516-everyday-ml-questions.html)
by [Santiago Valdarrama](https://datatalks.club/people/svpino.html)
, [Vladimir Haltakov](https://datatalks.club/people/vladimirhaltakov.html)
(from 16 May 2022 to 20 May 2022)
* [Artificial Intelligence with Python](https://datatalks.club/books/20220509-artificial-intelligence-with-python.html)
by [Prateek Joshi](https://datatalks.club/people/prateekjoshi.html)
(from 09 May 2022 to 13 May 2022)
* [Natural Language Processing with Transformers](https://datatalks.club/books/20220425-natural-language-processing-with-transformers.html)
by [Leandro von Werra](https://datatalks.club/people/leandrovonwerra.html)
, [Lewis Tunstall](https://datatalks.club/people/lewistunstall.html)
, [Thomas Wolf](https://datatalks.club/people/thomaswolf.html)
(from 25 Apr 2022 to 29 Apr 2022)
* [Interpretable Machine Learning](https://datatalks.club/books/20220411-interpretable-machine-learning.html)
by [Christoph Molnar](https://datatalks.club/people/christophmolnar.html)
(from 11 Apr 2022 to 15 Apr 2022)
* [Serverless Analytics with Amazon Athena](https://datatalks.club/books/20220328-serverless-analytics-with-amazon-athena.html)
by [Anthony Virtuoso](https://datatalks.club/people/anthonyvirtuoso.html)
(from 28 Mar 2022 to 01 Apr 2022)
* [Data Engineering with Apache Spark, Delta Lake, and Lakehouse](https://datatalks.club/books/20220314-data-engineering-with-apache-spark-delta-lake-and-lakehouse.html)
by [Manoj Kukreja](https://datatalks.club/people/manojkukreja.html)
(from 14 Mar 2022 to 18 Mar 2022)
* [Hands-On Data Preprocessing in Python](https://datatalks.club/books/20220228-hands-on-data-preprocessing-in-python.html)
by [Roy Jafari](https://datatalks.club/people/royjafari.html)
(from 28 Feb 2022 to 04 Mar 2022)
* [A Visual Introduction to Deep Learning](https://datatalks.club/books/20220214-a-visual-introduction-to-deep-learning.html)
by [Meor Amer](https://datatalks.club/people/meoramer.html)
(from 14 Feb 2022 to 18 Feb 2022)
* [Effective Pandas](https://datatalks.club/books/20220131-effective-pandas.html)
by [Matt Harrison](https://datatalks.club/people/mattharrison.html)
(from 31 Jan 2022 to 04 Feb 2022)
* [Machine Learning Engineering with Python](https://datatalks.club/books/20220117-machine-learning-engineering-with-python.html)
by [Andrew McMahon](https://datatalks.club/people/andrewmcmahon.html)
(from 10 Jan 2022 to 21 Jan 2022)
* [Mastering spaCy](https://datatalks.club/books/20211213-mastering-spacy.html)
by [Duygu Altinok](https://datatalks.club/people/duygualtinok.html)
(from 13 Dec 2021 to 17 Dec 2021)
* [Deep Learning with fastai Cookbook](https://datatalks.club/books/20211206-deep-learning-with-fastai-cookbook.html)
by [Mark Ryan](https://datatalks.club/people/markryan.html)
(from 06 Dec 2021 to 10 Dec 2021)
* [Own Your Tech Career](https://datatalks.club/books/20211129-own-your-tech-career.html)
by [Don Jones](https://datatalks.club/people/donjones.html)
(from 29 Nov 2021 to 03 Dec 2021)
* [Building Machine Learning Powered Applications](https://datatalks.club/books/20211122-building-machine-learning-powered-applications.html)
by [Emmanuel Ameisen](https://datatalks.club/people/emmanuelameisen.html)
(from 22 Nov 2021 to 26 Nov 2021)
* [Ace The Data Science Interview](https://datatalks.club/books/20211115-ace-the-data-science-interview.html)
by [Kevin Huo](https://datatalks.club/people/kevinhuo.html)
, [Nick Singh](https://datatalks.club/people/nicksingh.html)
(from 15 Nov 2021 to 19 Nov 2021)
* [Generative AI with Python and TensorFlow 2](https://datatalks.club/books/20211108-generative-ai-with-python-and-tensorflow-2.html)
by Joseph Babcock, [Raghav Bali](https://datatalks.club/people/raghavbali.html)
(from 08 Nov 2021 to 12 Nov 2021)
* [AI-Powered Search](https://datatalks.club/books/20211101-ai-powered-search.html)
by Trey Grainger, [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
, Max Irwin (from 01 Nov 2021 to 05 Nov 2021)
* [Data Analysis with Python and PySpark](https://datatalks.club/books/20211025-data-analysis-with-python-and-pyspark.html)
by [Jonathan Rioux](https://datatalks.club/people/jonathanrioux.html)
(from 25 Oct 2021 to 29 Oct 2021)
* [Blueprints for Text Analytics Using Python](https://datatalks.club/books/20211018-blueprints-for-text-analytics-using-python.html)
by [Jens Albrecht](https://datatalks.club/people/jensalbrecht.html)
, [Sidharth Ramachandran](https://datatalks.club/people/sidharthramachandran.html)
, [Christian Winkler](https://datatalks.club/people/christianwinkler.html)
(from 18 Oct 2021 to 22 Oct 2021)
* [Mastering Transformers](https://datatalks.club/books/20211011-mastering-transformers.html)
by [Savaş Yıldırım](https://datatalks.club/people/savasyildirim.html)
, [Meysam Asgari-Chenaghlu](https://datatalks.club/people/meysamasgarichenaghlu.html)
(from 11 Oct 2021 to 15 Oct 2021)
* [Transfer Learning in Action](https://datatalks.club/books/20211004-transfer-learning-in-action.html)
by Dipanjan Sarkar, [Raghav Bali](https://datatalks.club/people/raghavbali.html)
(from 04 Oct 2021 to 08 Oct 2021)
* [Effective Data Science Infrastructure](https://datatalks.club/books/20210927-effective-data-science-infrastructure.html)
by [Ville Tuulos](https://datatalks.club/people/villetuulos.html)
(from 27 Sep 2021 to 01 Oct 2021)
* [Python Feature Engineering Cookbook](https://datatalks.club/books/20210920-python-feature-engineering-cookbook.html)
by [Soledad Galli](https://datatalks.club/people/soledadgalli.html)
(from 20 Sep 2021 to 24 Sep 2021)
* [DataOps for Dummies](https://datatalks.club/books/20210913-dataops-for-dummies.html)
by Justin Mullen, [Guy Adams](https://datatalks.club/people/guyadams.html)
(from 13 Sep 2021 to 17 Sep 2021)
* [Software Mistakes and Tradeoffs](https://datatalks.club/books/20210906-software-mistakes-and-tradeoffs.html)
by [Tomasz Lelek](https://datatalks.club/people/tomaszlelek.html)
, [Jon Skeet](https://datatalks.club/people/jonskeet.html)
(from 06 Sep 2021 to 10 Sep 2021)
* [Practical MLOps](https://datatalks.club/books/20210830-practical-mlops.html)
by [Noah Gift](https://datatalks.club/people/noahgift.html)
, Alfredo Deza (from 30 Aug 2021 to 03 Sep 2021)
* [Business Skills for Data Scientists](https://datatalks.club/books/20210823-business-skills-for-data-scientists.html)
by [David Stephenson](https://datatalks.club/people/davidstephenson.html)
(from 23 Aug 2021 to 27 Aug 2021)
* [Tuning Up](https://datatalks.club/books/20210816-tuning-up.html)
by [David Sweet](https://datatalks.club/people/davidsweet.html)
(from 16 Aug 2021 to 20 Aug 2021)
* [Grokking Machine Learning](https://datatalks.club/books/20210809-grokking-machine-learning.html)
by [Luis Serrano](https://datatalks.club/people/luisserrano.html)
(from 09 Aug 2021 to 13 Aug 2021)
* [Practical Recommender Systems](https://datatalks.club/books/20210802-practical-recommender-systems.html)
by [Kim Falk](https://datatalks.club/people/kimfalk.html)
(from 02 Aug 2021 to 06 Aug 2021)
* [Applied Natural Language Processing in the Enterprise](https://datatalks.club/books/20210726-applied-natural-language-processing-in-the-enterprise.html)
by [Ankur A. Patel](https://datatalks.club/people/ankurapatel.html)
, Ajay Uppili Arasanipalai (from 26 Jul 2021 to 30 Jul 2021)
* [Interpretable Machine Learning with Python](https://datatalks.club/books/20210719-interpretable-machine-learning-with-python.html)
by [Serg Masis](https://datatalks.club/people/sergmasis.html)
(from 19 Jul 2021 to 23 Jul 2021)
* [Relevant Search](https://datatalks.club/books/20210712-relevant-search.html)
by [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
, John Berryman (from 12 Jul 2021 to 16 Jul 2021)
* [Engineering MLOps](https://datatalks.club/books/20210705-engineering-mlops.html)
by [Emmanuel Raj](https://datatalks.club/people/emmanuelraj.html)
(from 05 Jul 2021 to 09 Jul 2021)
* [Data Science on AWS](https://datatalks.club/books/20210628-data-science-on-aws.html)
by [Chris Fregly](https://datatalks.club/people/chrisfregly.html)
, [Antje Barth](https://datatalks.club/people/antjebarth.html)
(from 28 Jun 2021 to 02 Jul 2021)
* [Cleaning Data for Effective Data Science](https://datatalks.club/books/20210621-cleaning-data-for-effective-data-science.html)
by [David Mertz](https://datatalks.club/people/davidmertz.html)
(from 21 Jun 2021 to 25 Jun 2021)
* [Graph Databases in Action](https://datatalks.club/books/20210614-graph-databases-in-action.html)
by [Dave Bechberger](https://datatalks.club/people/davebechberger.html)
, Josh Perryman (from 14 Jun 2021 to 18 Jun 2021)
* [Building Machine Learning Pipelines](https://datatalks.club/books/20210607-building-machine-learning-pipelines.html)
by [Hannes Hapke](https://datatalks.club/people/hanneshapke.html)
, Catherine Nelson (from 07 Jun 2021 to 11 Jun 2021)
* [Advanced Algorithms and Data Structures](https://datatalks.club/books/20210531-advanced-algorithms-and-data-structures.html)
by [Marcello La Rocca](https://datatalks.club/people/marcellolarocca.html)
(from 31 May 2021 to 04 Jun 2021)
* [Data Governance: The Definitive Guide](https://datatalks.club/books/20210524-data-governance-the-definitive-guide.html)
by Evren Eryurek, [Uri Gilad](https://datatalks.club/people/urigilad.html)
, Valliappa Lakshmanan, Anita Kibunguchy-Grant, [Jessi Ashdown](https://datatalks.club/people/jessiashdown.html)
(from 24 May 2021 to 28 May 2021)
* [Grokking Deep Reinforcement Learning](https://datatalks.club/books/20210517-grokking-deep-reinforcement-learning.html)
by [Miguel Morales](https://datatalks.club/people/miguelmorales.html)
(from 17 May 2021 to 21 May 2021)
* [The Coding Career Handbook](https://datatalks.club/books/20210510-the-coding-career-handbook.html)
by [Shawn Swyx Wang](https://datatalks.club/people/swyx.html)
(from 10 May 2021 to 14 May 2021)
* [Machine Learning Using TensorFlow Cookbook](https://datatalks.club/books/20210503-machine-learning-using-tensorflow-cookbook.html)
by [Alexia Audevart](https://datatalks.club/people/alexiaaudevart.html)
, Konrad Banachewicz, Luca Massaron (from 03 May 2021 to 07 May 2021)
* [Tiny Python Projects](https://datatalks.club/books/20210426-tiny-python-projects.html)
by [Ken Youens-Clark](https://datatalks.club/people/kenyouens-clark.html)
(from 26 Apr 2021 to 30 Apr 2021)
* [Transformers for Natural Language Processing](https://datatalks.club/books/20210419-transformers-for-natural-language-processing.html)
by [Denis Rothman](https://datatalks.club/people/denisrothman.html)
(from 19 Apr 2021 to 23 Apr 2021)
* [AI and Machine Learning for Coders](https://datatalks.club/books/20210412-ai-and-machine-learning-for-coders.html)
by [Laurence Moroney](https://datatalks.club/people/laurencemoroney.html)
(from 12 Apr 2021 to 16 Apr 2021)
* [The Practitioner's Guide to Graph Data](https://datatalks.club/books/20210405-the-practitioners-guide-to-graph-data.html)
by [Denise Gosnell](https://datatalks.club/people/denisegosnell.html)
(from 05 Apr 2021 to 09 Apr 2021)
* [Learning Tensorflow.js](https://datatalks.club/books/20210329-learning-tensorflow-js.html)
by [Gant Laborde](https://datatalks.club/people/gantlaborde.html)
(from 29 Mar 2021 to 02 Apr 2021)
* [Street Coder](https://datatalks.club/books/20210322-street-coder.html)
by [Sedat Kapanoglu](https://datatalks.club/people/sedatkapanoglu.html)
(from 22 Mar 2021 to 26 Mar 2021)
* [Database Internals](https://datatalks.club/books/20210315-database-internals.html)
by [Alex Petrov](https://datatalks.club/people/alexpetrov.html)
(from 15 Mar 2021 to 19 Mar 2021)
* [Designing Data-Intensive Applications](https://datatalks.club/books/20210308-designing-data-intensive-applications.html)
by [Martin Kleppmann](https://datatalks.club/people/martinkleppmann.html)
(from 08 Mar 2021 to 12 Mar 2021)
* [Machine Learning Engineering in Action](https://datatalks.club/books/20210301-ml-engineering.html)
by [Ben Wilson](https://datatalks.club/people/benwilson.html)
(from 01 Mar 2021 to 05 Mar 2021)
* [Machine Learning for Algorithmic Trading](https://datatalks.club/books/20210222-ml-algotrading-2ed.html)
by [Stefan Jansen](https://datatalks.club/people/stefanjansen.html)
(from 22 Feb 2021 to 26 Feb 2021)
* [Math for Programmers](https://datatalks.club/books/20210215-math-for-programmers.html)
by [Paul Orland](https://datatalks.club/people/paulorland.html)
(from 15 Feb 2021 to 19 Feb 2021)
* [Machine Learning Design Patterns](https://datatalks.club/books/20210208-ml-design-patterns.html)
by Valliappa Lakshmanan, Sara Robinson, [Michael Munn](https://datatalks.club/people/michaelmunn.html)
(from 08 Feb 2021 to 12 Feb 2021)
* [Data Teams](https://datatalks.club/books/20210201-data-teams.html)
by [Jesse Anderson](https://datatalks.club/people/jesseanderson.html)
(from 01 Feb 2021 to 05 Feb 2021)
* [Mastering Machine Learning Algorithms - Second Edition](https://datatalks.club/books/20210125-mastering-ml-algorithms-2ed.html)
by [Giuseppe Bonaccorso](https://datatalks.club/people/giuseppebonaccorso.html)
(from 25 Jan 2021 to 29 Jan 2021)
* [Deep Learning with Structured Data](https://datatalks.club/books/20210118-deep-learning-structured-data.html)
by [Mark Ryan](https://datatalks.club/people/markryan.html)
(from 18 Jan 2021 to 22 Jan 2021)
* [Reinforcement Learning](https://datatalks.club/books/20210111-reinforcement-learning.html)
by [Phil Winder](https://datatalks.club/people/philwinder.html)
(from 11 Jan 2021 to 15 Jan 2021)
* [Machine Learning Bookcamp](https://datatalks.club/books/20201214-ml-bookcamp.html)
by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
(from 14 Dec 2020 to 18 Dec 2020)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Johannes Hötter – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Johannes Hötter
I am a Data Scientist/Engineer and co-founder of kern.
I am passionate about Machine Learning and Data Management and the things people build with such. At kern, we help Data Scientists in their most time-consuming tasks such as labeling and managing data efficiently and on large scale.
[](https://twitter.com/johoetter)
[](https://linkedin.com/in/johannesh%C3%B6tter)
[](https://github.com/jhoetter)
### Events
* Building an Open-Source NLP Tool ([watch on youtube](https://www.youtube.com/watch?v=WIpnyiHp4IE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jonas Christensen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jonas Christensen
Jonas Christensen has spent his career leading data science functions across multiple industries. He is an international keynote speaker on data science and analytics leadership, a postgraduate educator and advisor in the field of data science and machine learning and host of the Leaders of Analytics podcast.
Jonas is passionate about what data science and AI can do for the world of business and beyond. He believes data science and AI will be as revolutionary to the way we do business and interact with each other as IT and personal computing has been over the past 40 years.
[](https://linkedin.com/in/jonas-christensen-2235313)
### Books
* [Data-Centric Machine Learning with Python](https://datatalks.club/books/20240408-data-centric-machine-learning-with-python.html)
(the book of the week from 08 Apr 2024 to 12 Apr 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jonathan Rioux – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jonathan Rioux
As a data scientist for an engineering consultancy Jonathan Rioux uses PySpark daily. He teaches the software to data scientists, engineers, and data-savvy business analysts.
[](https://twitter.com/lejonesberg)
[](https://linkedin.com/in/jonathanrx)
[](https://github.com/jonesberg)
### Books
* [Data Analysis with Python and PySpark](https://datatalks.club/books/20211025-data-analysis-with-python-and-pyspark.html)
(the book of the week from 25 Oct 2021 to 29 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# José María Sánchez Salas – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 José María Sánchez Salas
I’m a Computer Scientist focused on Data Engineering, after working in different fields such as software and backend engineering. As part of my focus on data engineering, I also write a data engineering newsletter that leaves no one indifferent.
[](https://linkedin.com/in/jmssalas)
[](https://github.com/jmssalas)
### Events
* Mastering Data Engineering as a Remote Worker ([watch on youtube](https://www.youtube.com/watch?v=UX7UShEioKc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Jon Skeet – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Jon Skeet
Jon Skeet is a staff developer relations engineer at Google, currently working on the Google Cloud Client Libraries for .NET. His contributions to open source include the NodaTime date and time library for .NET, and he’s famous for his contributions to Stack Overflow. Jon is also the author of Manning’s C# in Depth, currently in its fourth edition.
[](https://twitter.com/jonskeet)
[](https://github.com/jskeet)
[](https://jonskeet.uk/)
### Books
* [Software Mistakes and Tradeoffs](https://datatalks.club/books/20210906-software-mistakes-and-tradeoffs.html)
(the book of the week from 06 Sep 2021 to 10 Sep 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Josh Fischer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Josh Fischer
Josh Fischer is an Apache Committer, and part of the project management committee for the Apache Heron distributed stream processing engine. Josh is a software engineer at Scotcro and has worked with moving large datasets in real time for organizations such as 1904labs and Bayer.
[](https://twitter.com/joshfischer1108)
[](https://linkedin.com/in/joshfischer1108)
[](https://github.com/joshfischer1108)
### Books
* [Grokking Streaming Systems](https://datatalks.club/books/20220704-grokking-streaming-systems.html)
(the book of the week from 04 Jul 2022 to 08 Jul 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Josh Tobin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Josh Tobin
Josh Tobin is the founder and CEO of a stealth machine learning startup. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey.
He is also the creator of [Full Stack Deep Learning](http://fullstackdeeplearning.com/)
, the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley advised by Pieter Abbeel.
[](https://twitter.com/josh_tobin_)
[](https://linkedin.com/in/josh-tobin-4b3b10a9)
[](https://github.com/josh-tobin)
[](http://josh-tobin.com/)
### Events
* Evaluation Store: a New Category of ML Engineering Tools ([watch on youtube](https://www.youtube.com/watch?v=KzyKVGIMIU0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Joyce Kay Avila – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Joyce Kay Avila
Joyce Kay Avila is a data and AI strategist who has more than 25 years of experience as a business and technology leader. She has bachelor’s and master’s degrees in the field of computer science and business administration, and completed her PhD coursework in accounting information systems.
Joyce holds numerous Salesforce and Slack certifications, as well as Tableau certification. She is also certified in Snowflake and has been recognized as a Snowflake Data Superhero since that MVP program began in 2019.
For the Salesforce and Snowflake communities, Joyce produces a running series of how-to videos on your YouTube channels. She is also the author of several O’Reilly books, including Snowflake: The Definitive Guide \[1st and 2nd editions\], Hands-on Salesforce Data Cloud, and Salesforce AI.
[](https://linkedin.com/in/joycekayavila)
### Books
* [Snowflake: The Definitive Guide](https://datatalks.club/books/20230123-snowflake-definitive-guide.html)
(the book of the week from 23 Jan 2023 to 27 Jan 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Juan Manuel Perafan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Juan Manuel Perafan
Juan Manuel Perafan is an analytics engineer, educator, and community builder based in Utrecht. He’s the co-author of Fundamentals of Analytics Engineering, host of the SQL Lingua Franca podcast, and a dbt Community Award winner. Juan founded the Analytics Engineering Meetup Netherlands and the Dutch dbt Meetup, and has spoken at events like dbt Coalesce, Linux Foundation OS Summit, and Big Data Expo NL.
[](https://linkedin.com/in/jmperafan)
[](http://juanalytics.com/)
### Events
* Foundations of Analytics Engineer Role: Skills, Scope, and Modern Practices ([watch on youtube](https://datatalks.club/people/juanmanuelperafan.html)
)
* [Analytics Engineering with dbt Workshop](https://luma.com/c8c55og5)
on 27 Jan 2026
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Juan Orduz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Juan Orduz
I am a Berlin based mathematician and data scientist. I am particularly interested in statistical learning, time series analysis, bayesian and geometric methods in data analysis.
[](https://twitter.com/juanitorduz)
[](https://linkedin.com/in/juanitorduz)
[](https://github.com/juanitorduz)
### Events
* Machine Learning in Marketing ([watch on youtube](https://www.youtube.com/watch?v=jsAxUd_bZpw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Juan Pablo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Juan Pablo
Juan Pablo Murillo is an AI and data professional with over a decade of experience spanning AI security, data science, and analytics engineering. He currently works at Google as an AI Security Engineer, where he focuses on developing methods to protect large language models and generative AI systems from emerging threats. His work includes prompt engineering, adversarial testing, and the design of secure AI frameworks that ensure the ethical and safe deployment of advanced AI technologies.
Before joining Google, Juan spent four years at Amazon as a Business Intelligence Engineer, where he designed data models and pipelines on AWS to optimize analytics across large-scale operations. Prior to that, he worked as a Data Scientist at T-Mobile and ProCogia, applying machine learning and customer behavior analysis to improve business performance and decision-making.
Juan holds a strong foundation in mathematics from his academic background at the University of Iowa, where he taught undergraduate courses and conducted research in coding theory at the Mathematical Sciences Research Institute. His career reflects a unique combination of mathematical rigor, data science practice, and AI security innovation—making him a trusted voice in the intersection of AI safety, large-scale data systems, and responsible machine learning.
[](https://twitter.com/thatjuanpablo)
[](https://linkedin.com/in/thatjuanpablo)
### Events
* From Math Teacher to Analytics Engineer ([watch on youtube](https://www.youtube.com/watch?v=qh6-HDhw2xY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Julia Ostheimer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Julia Ostheimer
After starting my work career as a Project Manager, I have been doing a deep dive into the world of artificial intelligence. My thesis focused on human-in-the-loop computing, a machine learning approach to achieving high-accuracy algorithms by combining machine intelligence with human domain expertise. Since then, I have been working & volunteering in the field of Data Science & Machine Learning, always interested to learn and applying new knowledge of AI and ML research.
[](https://linkedin.com/in/julia-ostheimer)
[](https://github.com/JOPloume)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Justin Ryan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Justin Ryan
Justin Ryan, a seasoned expert in cybersecurity and IT, brings over twenty years of rich experience to this book. His journey within the security field began in the U.S. Air Force as a Cybersecurity Operations Manager for the AFCERT (Air Force Computer Emergency Response Team), where he honed his skills in incident response as a certified operator. After nearly ten years in the Air Force, he transitioned to EY as a Manager of Cyber and Privacy Risk Advising. He then advanced to become Vice President of Cyber Risk Management at JPMorgan Chase & Co., focusing on the financial services industry. Justin also served as the Director of Cybersecurity Risk Management at USAA, building the program nearly from the ground up. He still works as a full-time practitioner in the financial services industry, leading the development of a sensitive data management department.
Throughout his career, Justin has held various cybersecurity and privacy roles, including consulting for prestigious global organizations like HSBC, Cisco, Merck & Co., and Rackspace.
Justin holds an Executive Master of Cybersecurity from Brown University and a Master of Science in Technology Commercialization from Northeastern University, among his five degrees. His professional certifications include GIAC Certified Incident Handler (GCIH), Certified Information Systems Security Professional (CISSP), Certified Ethical Hacker (CEH), Certified in Risk and Information Systems Control (CRISC), and Global Industrial Cyber Security Professional (GICSP). Additionally, he completed the 7-month intensive executive leadership program called the Program for Leadership Development at Harvard Business School.
[](https://linkedin.com/in/justincryan)
### Books
* [AI Data Privacy and Protection](https://datatalks.club/books/20240715-ai-data-privacy-and-protection.html)
(the book of the week from 15 Jul 2024 to 19 Jul 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Events – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Events
======
We host multiple types of events to help you learn and connect with the data science community:
Types of Events
---------------
* Webinars – events on Tuesday, with slides, mostly technical
* Live podcasts – events on Friday, a discussion without slides, the recording is published as a podcast
* Workshop – hands-on tutorials about technical topics
* Conference – bigger events with multiple talks, both webinar-type talks and podcast-type talks
> 📅 **Pro tip**: you can also subscribe to [our Google calendar](https://calendar.google.com/calendar/?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
> to get notified about all our events (subscribing works from desktop only).
Upcoming events
---------------
Upcoming Events
---------------
* [Fact-Checking with Wikidata](https://luma.com/7fs5v7os)
on 20 Jan 2026 by [Philippe Saadé](https://datatalks.club/people/philippesaade.html)
* [AI Engineering: Skill Stack, Agents, LLMOps, and How to Ship AI Products](https://luma.com/fx3wplwn)
on 26 Jan 2026 by [Paul Iusztin](https://datatalks.club/people/pauliusztin.html)
* [Analytics Engineering with dbt Workshop](https://luma.com/c8c55og5)
on 27 Jan 2026 by [Juan Manuel Perafan](https://datatalks.club/people/juanmanuelperafan.html)
* [The Future of AI Agents](https://luma.com/0s0dcpdl)
on 10 Feb 2026 by [Aditya Gautam](https://datatalks.club/people/adityagautam.html)
* [From APIs to Warehouses: AI-Assisted Data Ingestion with dlt](https://luma.com/hzis1yzp)
on 17 Feb 2026 by [Aashish Nair](https://datatalks.club/people/aashishnair.html)
Past events
-----------
Past Events
-----------
* Data Engineering Zoomcamp 2026 Course Launch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Foundations of Analytics Engineer Role: Skills, Scope, and Modern Practices by [Juan Manuel Perafan](https://datatalks.club/people/juanmanuelperafan.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Durable Agentic Workflows with Temporal.io by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=N1gaI3Qz6vw)
)
* From Human-in-the-Loop to Agent-in-the-Loop: A Practical Transition Guide by [Ertugrul Mutlu](https://datatalks.club/people/ertugrulmutlu.html)
([watch on youtube](https://www.youtube.com/watch?v=HwCR59VuYn4)
)
* Data Engineering Zoomcamp 2026 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=WB6b1lcguaA)
)
* Automated Prompt Optimization with Evidently AI by [Mikhail Sveshnikov](https://datatalks.club/people/mikhailsveshnikov.html)
([watch on youtube](https://www.youtube.com/watch?v=uMNYVw4jh-8)
)
* Docker for Data Engineering: Postgres, Docker Compose, and Real-World Workflows by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=lP8xXebHmuE)
)
* Building Pet Health Tech: ML, Sensors, and Dog Behavior Data by [Sofya Yulpatova](https://datatalks.club/people/sofyayulpatova.html)
([watch on youtube](https://www.youtube.com/watch?v=4bl2TSHD_Fc)
)
* n8n: From Fundamentals to Building Intelligent Automation Pipeline by [Moein Foroughi](https://datatalks.club/people/moeinforoughi.html)
([watch on youtube](https://www.youtube.com/watch?v=KR9ApZXsV8g)
)
* AI Dev Tools Zoomcamp 2025 Course Launch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=58pn873XO04)
)
* Combining Quantum and AI for Accelerating CFD Simulations - Part 1 by [Aditya Seshaditya](https://datatalks.club/people/adityaseshaditya.html)
([watch on youtube](https://www.youtube.com/watch?v=UHRN7cX_ieE)
)
* Reinventing a Career in Tech by [Xia He-Bleinagel](https://datatalks.club/people/xiahebleinagel.html)
([watch on youtube](https://www.youtube.com/watch?v=D2rw52SOFfM)
)
* AI Dev Tools Zoomcamp 2025 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=sUwrCnP2iGU)
)
* Building with MCP: Tools, Workflows, and Real Examples by [Bhavani Ravi](https://datatalks.club/people/bhavaniravi.html)
([watch on youtube](https://www.youtube.com/watch?v=0IhZdcjddo4)
)
* Practical guide: Fine-tuning Qwen3 with LoRA by [Ivan Potapov](https://datatalks.club/people/ivanpotapov.html)
([watch on youtube](https://www.youtube.com/watch?v=cayFaWkI39A)
)
* Deep Learning with PyTorch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=Ne25VujHRLA)
)
* From Black-Box Systems to Augmented Decision-Making by [Anusha Akkina](https://datatalks.club/people/anushaakkina.html)
([watch on youtube](https://www.youtube.com/watch?v=YZNaLm-_zwA)
)
* How to Build and Evaluate AI systems in the Age of LLMs by [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html)
([watch on youtube](https://www.youtube.com/watch?v=eC3RNuI6ow0)
)
* Advancing Cancer Genomics with Machine Learning by [Rileen Sinha](https://datatalks.club/people/rileensinha.html)
([watch on youtube](https://datatalks.club/events.html)
)
* From Biotechnology to Bioinformatics Software by [Sebastian Ayala Ruano](https://datatalks.club/people/sebastianayalaruano.html)
([watch on youtube](https://www.youtube.com/watch?v=ZFrcrTtnB1Q)
)
* Deploying ML Models with Kubernetes by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=c_CzCsCnWoU)
)
* Lessons from Applied AI: Tesla, Waymo, and Beyond by [Aishwarya Jadhav](https://datatalks.club/people/aishwaryajadhav.html)
([watch on youtube](https://www.youtube.com/watch?v=vK_SxyqIfwk)
)
* From Theme Parks to Tesla: Building Data Products That Work by [Abouzar Abbaspour](https://datatalks.club/people/abouzarabbaspour.html)
([watch on youtube](https://www.youtube.com/watch?v=gXvVMvhfrIY)
)
* Building reliable AI products in the era of Gen AI and Agents by [Ranjitha Kulkarni](https://datatalks.club/people/ranjithakulkarni.html)
([watch on youtube](https://www.youtube.com/watch?v=x2AAjqz2XmM)
)
* From Semiconductors to Machine Learning: A Career in Data and Teaching by [Dashel Ruiz Perez](https://datatalks.club/people/dashelruizperez.html)
([watch on youtube](https://www.youtube.com/watch?v=B2tzuUg5uZs)
)
* Lessons from Two Decades of AI by [Micheal Lanham](https://datatalks.club/people/micheallanham.html)
([watch on youtube](https://www.youtube.com/watch?v=DSxqUlumM3A)
)
* From Algorithms to Agents: Lessons from Building Copilot by [Vadim Smolyakov](https://datatalks.club/people/vadimsmolyakov.html)
([watch on youtube](https://www.youtube.com/watch?v=s8kyzy8V5b8)
)
* ML Zoomcamp 2025 Course Launch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=z064DoidiKg)
)
* From Astronomy to Applied ML by [Daniel Egbo](https://datatalks.club/people/danielegbo.html)
([watch on youtube](https://www.youtube.com/watch?v=b92gwrsVQtg)
)
* ML Zoomcamp 2025 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=ph1PxZIkz1o)
)
* From Medicine to Machine Learning: How Public Learning Turned into a Career by [Pastor Soto](https://datatalks.club/people/pastorsoto.html)
([watch on youtube](https://www.youtube.com/watch?v=5km62e4nDaw)
)
* Deploying ML Models with AWS Lambda (Serverless) by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=sHQaeVm5hT8)
)
* Build an AI Coding Agent by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=-XLgk1O421I)
)
* Deploying ML Models with FastAPI and uv by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=jzGzw98Eikk)
)
* Build Agentic Assistants with OpenAI Function Calling: Part 2 by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=yS_hwnJusDk)
)
* Introduction to Vibe Coding: Build a Game with AI by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=NSMXQk4Axig)
)
* Mindful Data Strategy: From Pipelines to Business Impact by [Lior Barak](https://datatalks.club/people/liorbarak.html)
([watch on youtube](https://www.youtube.com/watch?v=B76J4QkZPWs)
)
* DataTalks.Club Summer 2025 AI Hackathon Launch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=APtJ1SDGdw0)
)
* From RAG to Agents: Making Smart AI Assistants (LLM Zoomcamp bonus module) by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=GH3lrOsU3AU)
)
* From Simulation Algorithms to Production-Grade Data Systems by [Orell Garten](https://datatalks.club/people/orellgarten.html)
([watch on youtube](https://www.youtube.com/watch?v=pkcpH5N-GP8)
)
* From REST to reasoning: ingest, index, and query with dlt and Cognee by [Hiba Jamal](https://datatalks.club/people/hibajamal.html)
([watch on youtube](https://www.youtube.com/watch?v=MNt_KK32gys)
)
* Open-Source LLM Zoomcamp 2025
Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=mmUyeyAPVnU)
)
* MLOps Zoomcamp Competition - Bot or Not? by [Alexander Guschin](https://datatalks.club/people/alexanderguschin.html)
, [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/live/ZxUVBG4z5uE)
)
* LLM Zoomcamp Course Launch by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=FgnelhEJFj0)
)
* AI Math Olympiad: A Technical Debrief of the Competition by [Ilya Boytsov](https://datatalks.club/people/ilyaboytsov.html)
([watch on youtube](https://www.youtube.com/watch?v=GH_IK_HK1HA)
)
* LLM Zoomcamp 2025
Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=8lgiOLMMKcY)
)
* Foundations of Data Engineering Instruments by [Sergei Boitsov](https://datatalks.club/people/sergeiboitsov.html)
([watch on youtube](https://www.youtube.com/watch?v=H3sxOea9qD0)
)
* How to write data pipelines with Apache Airflow® 3.0 by [Timothy Davis](https://datatalks.club/people/timothydavis.html)
([watch on youtube](https://www.youtube.com/watch?v=v_jbmhVWBvM)
)
* MLOps Zoomcamp 2025 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=rv43YJQsZIw)
)
* Stock Market Analytics Zoomcamp 2025 - Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
([watch on youtube](https://www.youtube.com/watch?v=1N1jIwa1uag)
)
* Taking your Freelance Career to the Next Level by [Dimitri Visnadi](https://datatalks.club/people/dimitrivisnadi.html)
([watch on youtube](https://www.youtube.com/watch?v=S93V8RgwBig)
)
* Using GenAI to Create Horror Stories by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=DvhdJWqE47gyoutube)
)
* From Hackathons To Developer Advocacy by [Will Russell](https://datatalks.club/people/willrussell.html)
([watch on youtube](https://www.youtube.com/watch?v=vXbMUfHE1OE)
)
* How to build an Agentic Search Flow by [Florian Hoenicke](https://datatalks.club/people/florianhoenicke.html)
([watch on youtube](https://www.youtube.com/live/Gb-LnXu_Hc0)
)
* Build a Strong Career in Data by [Lavanya Gupta](https://datatalks.club/people/lavanyagupta.html)
([watch on youtube](https://www.youtube.com/watch?v=ekG5zJioyFs)
)
* Predicting Financial Time-Series by [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
([watch on youtube](https://www.youtube.com/watch?v=PxAh08Pcmj4)
)
* From Supply Chain Management to Digital Warehousing and FinOps by [Eddy Zulkifly](https://datatalks.club/people/eddyzulkifly.html)
([watch on youtube](https://www.youtube.com/watch?v=7ePp6wuxM5s)
)
* Data Ingestion From APIs to Warehouses and Data Lakes by [Violetta Mishechkina](https://datatalks.club/people/violettamishechkina.html)
([watch on youtube](https://www.youtube.com/watch?v=pgJWP_xqO1g)
)
* Data Intensive AI by [Bartosz Mikulski](https://datatalks.club/people/bartoszmikulski.html)
([watch on youtube](https://www.youtube.com/watch?v=BP6w_vKySN0)
)
* MLOps in Corporations and Startups by [Nemanja Radojkovic](https://datatalks.club/people/nemanjaradojkovic.html)
([watch on youtube](https://www.youtube.com/watch?v=DX9c__a4jzg)
)
* Trends in Data Engineering by [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
([watch on youtube](https://www.youtube.com/watch?v=AlCFKbFIEM8)
)
* Competitive Machine Learning and Teaching by [Alexander Guschin](https://datatalks.club/people/alexanderguschin.html)
([watch on youtube](https://www.youtube.com/watch?v=NfAJAr7FvyY)
)
* Trends in AI Infrastructure by [Andrey Cheptsov](https://datatalks.club/people/andreycheptsov.html)
([watch on youtube](https://www.youtube.com/watch?v=1aMuynlLM3o)
)
* Linguistics and Fairness by [Tamara Atanasoska](https://datatalks.club/people/tamaraatanasoska.html)
([watch on youtube](https://www.youtube.com/watch?v=sXU9vMDBjmk)
)
* Elevating RAG Systems by [Apurva Misra](https://datatalks.club/people/apurvamisra.html)
([watch on youtube](https://www.youtube.com/watch?v=RexNgpgwmkk)
)
* Career choices, transitions and promotions in and out of tech by [Agita Jaunzeme](https://datatalks.club/people/agitajaunzeme.html)
([watch on youtube](https://www.youtube.com/watch?v=QKWu5-6_6TE)
)
* Economics and Automation Workshop: Building a Data Pipeline for Economic Insights by [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
([watch on youtube](https://www.youtube.com/watch?v=MeDUe75WQaQ)
)
* Career advice, learning, and featuring women in ML and AI by [Isabella Bicalho](https://datatalks.club/people/isabellabicalho.html)
([watch on youtube](https://www.youtube.com/watch?v=GifY8Zn-pnU)
)
* AI in Industry: Trust, Return on Investment and Future by [Maria Sukhareva](https://datatalks.club/people/mariasukhareva.html)
([watch on youtube](https://www.youtube.com/watch?v=bT7-HRNCltk)
)
* Large Hadron Collider and Mentorship by [Anastasia Karavdina](https://datatalks.club/people/anastasiakaravdina.html)
([watch on youtube](https://www.youtube.com/watch?v=kV0ZDy2UtJA)
)
* MLOps as a Team by [Raphaël Hoogvliets](https://datatalks.club/people/raphaelhoogvliets.html)
([watch on youtube](https://www.youtube.com/watch?v=rMq63r3zi4c)
)
* Developing Your Career in ML by Studying by [Ella (Wati) Sahnan](https://datatalks.club/people/ella(wati)sahnan.html)
([watch on youtube](https://datatalks.club/events.html)
)
* DataTalks.Club Anniversary Podcast by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=GHbeXIKnkLQ)
)
* Human-Centered AI for Disordered Speech Recognition by [Katarzyna Foremniak](https://datatalks.club/people/katarzynaforemniak.html)
([watch on youtube](https://www.youtube.com/watch?v=yTZ4cddD7DU)
)
* A Gentle Introduction to LLM Evaluations by [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
([watch on youtube](https://www.youtube.com/watch?v=ac6ZB5QEwGU)
)
* ML Zoomcamp 2024 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=htsvF-QUJDc)
)
* Using Data to Create Liveable Cities by [Rachel Lim](https://datatalks.club/people/rachellim.html)
([watch on youtube](https://www.youtube.com/watch?v=VXQIGHUWeL0)
)
* DataOps, Observability, and The Cure for Data Team Blues by [Christopher Bergh](https://datatalks.club/people/christopherbergh.html)
([watch on youtube](https://www.youtube.com/watch?v=HzGpIxV8HtA)
)
* Open source data ingestion for RAGs with dlt by [Akela Drissner](https://datatalks.club/people/akeladrissner.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Inventory Optimization in E-commerce by [Hagop Dippel](https://datatalks.club/people/hagopdippel.html)
, [Andreas Syrén](https://datatalks.club/people/andreassyren.html)
([watch on youtube](https://www.youtube.com/watch?v=RlXlJK8OXhY)
)
* Working as a Core Developer in the Scikit Learn Universe by [Guillaume Lemaître](https://datatalks.club/people/guillaumelemaitre.html)
([watch on youtube](https://www.youtube.com/watch?v=RR6xaYqKJ3o)
)
* Building a Domestic Risk Assessment Tool by [Ba Linh Le](https://datatalks.club/people/balinhle.html)
, [Sabina Firtala](https://datatalks.club/people/sabinafirtala.html)
([watch on youtube](https://www.youtube.com/watch?v=CpWlBAmD9ok)
)
* LLM Zoomcamp 2024 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Implement a Search Engine by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=nMrGK5QgPVE)
)
* Data Storytelling in Python Altair by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
([watch on youtube](https://www.youtube.com/watch?v=WqcuYXTEeUo)
)
* How to Land Your First Data Engineer Job by [Gonçalo Sequeira](https://datatalks.club/people/goncalosequeira.html)
([watch on youtube](https://www.youtube.com/watch?v=1AchMU9xf0Q)
)
* MLOps Zoomcamp 2024 Pre-Course Live Q&A by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=YmllO3ld5LE)
)
* Chat with Your Own Data: Introduction to the LLM Zoomcamp by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=q-p36Ak6YI8)
)
* Community Building and Teaching in AI & Tech by [Erum Afzal](https://datatalks.club/people/erumafzal.html)
([watch on youtube](https://www.youtube.com/watch?v=7SLd5V7z3xQ)
)
* Working in Open Source - Probabl.ai and sklearn by [Vincent Warmerdam](https://datatalks.club/people/vincentwarmerdam.html)
([watch on youtube](https://www.youtube.com/watch?v=UPlIETGwTg8)
)
* System Design in Data Engineering by [Sergei Shaikin](https://datatalks.club/people/sergeishaikin.html)
([watch on youtube](https://datatalks.club/events.html)
)
* AI for Ecology, Biodiversity, and Conservation by [Tanya Berger-Wolf](https://datatalks.club/people/tanyabergerwolf.html)
([watch on youtube](https://www.youtube.com/watch?v=30tTrozbAkg)
)
* Stock Market Analytics Zoomcamp Pre-Launch Q&A by [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
, [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=oswTLnjkRUg)
)
* Knowledge Graphs and LLMs Across Academia and Industry by [Anahita Pakiman](https://datatalks.club/people/anahitapakiman.html)
([watch on youtube](https://www.youtube.com/watch?v=YncdlUscUOo)
)
* Inclusive Data Leadership Coaching by [Tereza Iofciu](https://datatalks.club/people/terezaiofciu.html)
([watch on youtube](https://www.youtube.com/watch?v=Z4vOTgzLkJQ)
)
* Unlocking the Door to Your Kick-Start Data Engineering Career by [Madiha Khalid](https://datatalks.club/people/madihakhalid.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Building Production Search Systems by [Daniel Svonava](https://datatalks.club/people/danielsvonava.html)
([watch on youtube](https://www.youtube.com/watch?v=gEmSrknGKDE)
)
* Building Machine Learning Products by [Reem Mahmoud](https://datatalks.club/people/reemmahmoud.html)
([watch on youtube](https://www.youtube.com/watch?v=m45tNY-8gY8)
)
* Stream Processing by [Noel Kwan](https://datatalks.club/people/noelkwan.html)
([watch on youtube](https://datatalks.club/events.html)
)
* RAG in Action: Next-Level Retrieval Augmented Generation by [Leonard Püttmann](https://datatalks.club/people/leonardputtmann.html)
([watch on youtube](https://datatalks.club/events.html)
)
* How to Build an LLM-powered QA bot by [Alex Litvinov](https://datatalks.club/people/alexlitvinov.html)
([watch on youtube](https://www.youtube.com/watch?v=QhFLeZV-PVk)
)
* Make an Impact Through Volunteering Open Source Work by [Sara EL-ATEIF](https://datatalks.club/people/saraelateif.html)
([watch on youtube](https://www.youtube.com/watch?v=aHdaIwOEI8Q)
)
* Five Techniques for Improving RAG Chatbots by [Nikita Kozodoi](https://datatalks.club/people/nikitakozodoi.html)
([watch on youtube](https://www.youtube.com/watch?v=xPYmClWk5O8)
)
* Data Ingestion From APIs to Warehouses by [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Accelerating The Job Hunt for The Perfect Job in Tech by [Sarah Mestiri](https://datatalks.club/people/sarahmestiri.html)
([watch on youtube](https://www.youtube.com/watch?v=PchwbIs0tOg)
)
* Applied Causal Inference by [Irina Brudaru](https://datatalks.club/people/irinabrudaru.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Machine Learning Engineering in Finance by [Nemanja Radojkovic](https://datatalks.club/people/nemanjaradojkovic.html)
([watch on youtube](https://www.youtube.com/watch?v=Nl4aibeFwiI)
)
* Introduction to Data Engineering Zoomcamp by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=91b8u9GmqB4)
)
* Bayesian Modeling and Probabilistic Programming by [Rob Zinkov](https://datatalks.club/people/robzinkov.html)
([watch on youtube](https://www.youtube.com/watch?v=kcKvUSInm-M)
)
* Stock Market Analysis with Python and Machine Learning by [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
([watch on youtube](https://www.youtube.com/watch?v=NThHAEIazFk)
)
* How to Boost Your Impact as A Data Professional by [Shachar Meir](https://datatalks.club/people/shacharmeir.html)
([watch on youtube](https://www.youtube.com/watch?v=eNmoRCwFXM8)
)
* Terraform: Reshaping the Data Engineering Experience by [Andrei Tserakhau](https://datatalks.club/people/andreitserakhau.html)
([watch on youtube](https://www.youtube.com/watch?v=YFr_5NTjv0Q)
)
* Searching Beyond the Surface: Navigating Challenges and Innovations in Search Technologies by [Atita Arora](https://datatalks.club/people/atitaarora.html)
([watch on youtube](https://www.youtube.com/watch?v=_fbe1QyJ1PY)
)
* Prompt Engineering 101 by [Krzysztof Ograbek](https://datatalks.club/people/krzysztofograbek.html)
([watch on youtube](https://www.youtube.com/watch?v=qLGnW2vxors)
)
* The Entrepreneurship Journey: From Freelancing to Starting a Company by [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
([watch on youtube](https://www.youtube.com/watch?v=vOpEQiCsaLw)
)
* Large Language Models Agents by [Anish Shah](https://datatalks.club/people/anishshah.html)
([watch on youtube](https://www.youtube.com/watch?v=m5CZzhXPgd0)
)
* Become a Data Freelancer by [Dimitri Visnadi](https://datatalks.club/people/dimitrivisnadi.html)
([watch on youtube](https://www.youtube.com/watch?v=R_EnSa9aZtE)
)
* AI for Digital Health by [Maria Bruckert](https://datatalks.club/people/mariabruckert.html)
([watch on youtube](https://www.youtube.com/watch?v=whpkDmVVGUE)
)
* Cracking the Code: Machine Learning Made Understandable by [Christoph Molnar](https://datatalks.club/people/christophmolnar.html)
([watch on youtube](https://www.youtube.com/watch?v=LBuGzyOkx7c)
)
* The Unwritten Rules for Success in Machine Learning by [Jack Blandin](https://datatalks.club/people/jackblandin.html)
([watch on youtube](https://www.youtube.com/watch?v=su2M058m3Lw)
)
* The Trade-off Between Simplicity And Optimality In Problem-solving by [Arman Jabbari](https://datatalks.club/people/armanjabbari.html)
([watch on youtube](https://www.youtube.com/watch?v=iSCd31T8S3M)
)
* From a Research Scientist at Amazon to a Machine learning/AI Consultant by [Verena Weber](https://datatalks.club/people/verenaweber.html)
([watch on youtube](https://www.youtube.com/watch?v=4RargY8iOaE)
)
* Machine Learning with Ray: Supercharge Your GPU Clusters by [Sadik Bakiu](https://datatalks.club/people/sadikbakiu.html)
([watch on youtube](https://datatalks.club/events.html)
)
* From Marketing to Product Owner in Search by [Lera Kaimashnіkova](https://datatalks.club/people/lerakaimashnikova.html)
([watch on youtube](https://www.youtube.com/watch?v=-HbQQ_bVdfE)
)
* Collaborative Data Science in Business by [Ioannis Mesionis](https://datatalks.club/people/ioannismesionis.html)
([watch on youtube](https://www.youtube.com/watch?v=1pExOVuCF8Q)
)
* Bridging Data Science and Healthcare by [Eleni Stamatelou](https://datatalks.club/people/elenistamatelou.html)
([watch on youtube](https://www.youtube.com/watch?v=pDOwlulDh0c)
)
* Make Data Magical with Mage by [Matt Palmer](https://datatalks.club/people/mattpalmer.html)
([watch on youtube](https://www.youtube.com/watch?v=JKALtxziBG0)
)
* DataTalks.Club Anniversary Interview by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, [Johanna Bayer](https://datatalks.club/people/johannabayer.html)
([watch on youtube](https://www.youtube.com/watch?v=nCqwZT9zA0M)
)
* Data Engineering for Fraud Prevention by [Angela Ramirez](https://datatalks.club/people/angelaramirez.html)
([watch on youtube](https://www.youtube.com/watch?v=ZXNKjrrKU_I)
)
* From Data Manager to Data Architect by [Loïc Magnien](https://datatalks.club/people/loicmagnien.html)
([watch on youtube](https://www.youtube.com/watch?v=qWG--iYO2uc)
)
* Pragmatic and Standardized MLOps by [Maria Vechtomova](https://datatalks.club/people/mariavechtomova.html)
([watch on youtube](https://www.youtube.com/watch?v=q3DTR3Od1MA)
)
* Introduction to ML Engineering and ML Zoomcamp by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=a7phcSmuNY0)
)
* Democratizing Causality by [Aleksander Molak](https://datatalks.club/people/aleksandermolak.html)
([watch on youtube](https://www.youtube.com/watch?v=0I2FHH95Ofs)
)
* Mastering Data Engineering as a Remote Worker by [José María Sánchez Salas](https://datatalks.club/people/josemaria.html)
([watch on youtube](https://www.youtube.com/watch?v=UX7UShEioKc)
)
* The Good, the Bad and the Ugly of GPT by [Sandra Kublik](https://datatalks.club/people/sandrakublik.html)
([watch on youtube](https://www.youtube.com/watch?v=bM6AR4A-f98)
)
* LLMs for Everyone by [Meryem Arik](https://datatalks.club/people/meryemarik.html)
([watch on youtube](https://www.youtube.com/watch?v=6dn6uZFkk04)
)
* Investing in Open-Source Data Tools by [Bela Wiertz](https://datatalks.club/people/belawiertz.html)
([watch on youtube](https://www.youtube.com/watch?v=7Bg1JQLnCao)
)
* Building a Group Wide Data Lakehouse for Data Science & BI by [Loïc Magnien](https://datatalks.club/people/loicmagnien.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Why Machine Learning Design is Broken by [Valerii Babushkin](https://datatalks.club/people/valeriybabushkin.html)
([watch on youtube](https://www.youtube.com/watch?v=6YBMU6475KQ)
)
* Interpretable AI and ML by [Polina Mosolova](https://datatalks.club/people/polinamosolova.html)
([watch on youtube](https://www.youtube.com/watch?v=EQcY83VA0Us)
)
* From Scratch to Success: Building an MLOps Team and ML Platform by [Simon Stiebellehner](https://datatalks.club/people/simonstiebellehner.html)
([watch on youtube](https://www.youtube.com/watch?v=CB1YIsxQRtc)
)
* Data Plumbing without the poop by [Tommy Dang](https://datatalks.club/people/tommydang.html)
([watch on youtube](https://www.youtube.com/watch?v=nUfAqM2Sguc)
)
* Identity Resolution Essentials from a Data Scientist by [Nathan Wang](https://datatalks.club/people/nathanwang.html)
([watch on youtube](https://www.youtube.com/watch?v=hHvvXagqY9k)
)
* From MLOps to DataOps by [Santona Tuli](https://datatalks.club/people/santonatuli.html)
([watch on youtube](https://www.youtube.com/watch?v=kSTfhQ_SZgc)
)
* Data Developer Relations by [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html)
([watch on youtube](https://www.youtube.com/watch?v=z7BvslwVRbQ)
)
* MLOps Zoomcamp - Experiment Tracking with Weights and Biases by [Soumik Rakshit](https://datatalks.club/people/soumikrakshit.html)
([watch on youtube](https://www.youtube.com/watch?v=yNyqFMwEyL4)
)
* Building and Scaling a Machine Learning Platform by [Magdalena Kuhn](https://datatalks.club/people/magdalenakuhn.html)
([watch on youtube](https://www.youtube.com/watch?v=bNrBJwiLBWU)
)
* Lessons Learned from Freelancing and Working in a Start-up by [Antonis Stellas](https://datatalks.club/people/antonisstellas.html)
([watch on youtube](https://www.youtube.com/watch?v=-Gj7SaI-QW4)
)
* Data Access Management by [Bart Vandekerckhove](https://datatalks.club/people/bartvandekerckhove.html)
([watch on youtube](https://www.youtube.com/watch?v=IiPOIiUy5b4)
)
* Data Strategy: Key Principles and Best Practices by [Boyan Angelov](https://datatalks.club/people/boyanangelov.html)
([watch on youtube](https://www.youtube.com/watch?v=jGbfeYdlCiQ)
)
* From Data to Deployment by [Agostino Calamia](https://datatalks.club/people/agostinocalamia.html)
([watch on youtube](https://www.youtube.com/watch?v=DaW7I5ag9CI)
)
* Practical Data Privacy by [Katharine Jarmul](https://datatalks.club/people/katharinejarmul.html)
([watch on youtube](https://www.youtube.com/watch?v=gbjoFfrm4iw)
)
* Introduction to MLOps and MLOps Zoomcamp by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=o34Q_61iA4Y)
)
* Building Scalable and Reliable Machine Learning Systems by [Arseny Kravchenko](https://datatalks.club/people/arsenykravchenko.html)
([watch on youtube](https://www.youtube.com/watch?v=i-pIdekjUow)
)
* Open Source MLOps on Public Cloud by [Andreea Munteanu](https://datatalks.club/people/andreeamunteanu.html)
([watch on youtube](https://www.youtube.com/watch?v=kfCTwyfwV_s)
)
* Machine Learning in Digital Identity by [Geo Jolly](https://datatalks.club/people/geojolly.html)
([watch on youtube](https://www.youtube.com/watch?v=YoOWHfPuKhk)
)
* Effective Domain-Driven Design for Machine Learning Products by [Larysa Visengeriyeva](https://datatalks.club/people/larysavisengeriyeva.html)
([watch on youtube](https://www.youtube.com/watch?v=LPW7DDCUXHY)
)
* Building an Open-Source NLP Tool by [Johannes Hötter](https://datatalks.club/people/johanneshotter.html)
([watch on youtube](https://www.youtube.com/watch?v=WIpnyiHp4IE)
)
* Navigating Industrial Data Challenges by [Rosona Eldred](https://datatalks.club/people/rosonaeldred.html)
([watch on youtube](https://www.youtube.com/watch?v=rwuud5wr3J4)
)
* Fraud Detection with Graph Features and GNN by [Nikita Iserson](https://datatalks.club/people/nikitaiserson.html)
([watch on youtube](https://www.youtube.com/watch?v=ykVq8PemUsU)
)
* Mastering Self-Learning in Machine Learning by [Aaisha Muhammad](https://datatalks.club/people/aaishamuhammad.html)
([watch on youtube](https://www.youtube.com/watch?v=Kc3Puh3UCRQ)
)
* The Secret Sauce of Data Science Management by [Shir Meir Lador](https://datatalks.club/people/shirmeirlador.html)
([watch on youtube](https://www.youtube.com/watch?v=gcxP0qRO-MY)
)
* SE4ML - Software Engineering for Machine Learning by [Nadia Nahar](https://datatalks.club/people/nadianahar.html)
([watch on youtube](https://www.youtube.com/watch?v=35Ch8xL2SA8)
)
* Starting a Consultancy in the Data Space by [Aleksander Kruszelnicki](https://datatalks.club/people/aleksanderkruszelnicki.html)
([watch on youtube](https://www.youtube.com/watch?v=rh_pE35m3vE)
)
* Biohacking for Data Scientists and ML Engineers by [Ruslan Shchuchkin](https://datatalks.club/people/ruslanshchuchkin.html)
([watch on youtube](https://www.youtube.com/watch?v=uyxUBADZYpU)
)
* Maximizing Confidence in Your Data Model Changes with dbt and PipeRider by [Dave Flynn](https://datatalks.club/people/daveflynn.html)
([watch on youtube](https://www.youtube.com/watch?v=O-tyUOQccSs)
)
* GitOps for ML: Converting Notebooks to Reproducible Pipelines by [Rob De Wit](https://datatalks.club/people/robdewit.html)
([watch on youtube](https://www.youtube.com/watch?v=t92ISBh4y_E)
)
* Analytics for a Better World by [Parvathy Krishnan](https://datatalks.club/people/parvathykrishnan.html)
([watch on youtube](https://www.youtube.com/watch?v=b6x5zZ3C6sQ)
)
* Accelerating the Adoption of AI through Diversity by [Dânia Meira](https://datatalks.club/people/daniameira.html)
([watch on youtube](https://www.youtube.com/watch?v=SRUwwvk_YCk)
)
* Staff AI Engineer by [Tatiana Gabruseva](https://datatalks.club/people/tatianagabruseva.html)
([watch on youtube](https://www.youtube.com/watch?v=_xr1_xb736E)
)
* Navigating Career Changes in Machine Learning by [Krzysztof Szafanek](https://datatalks.club/people/krzysztofszafanek.html)
([watch on youtube](https://www.youtube.com/watch?v=cUxZBXQgZaU)
)
* The Journey of a Data Generalist: From Bioinformatics to Freelancing by [Jekaterina Kokatjuhha](https://datatalks.club/people/jekaterinakokatjuhha.html)
([watch on youtube](https://www.youtube.com/watch?v=FRi0SUtxdMw)
)
* Practical Learning-to-Rank: Deep, Fast, Precise by [Roman Grebennikov](https://datatalks.club/people/romangrebennikov.html)
([watch on youtube](https://www.youtube.com/watch?v=oXfFqAKf4Ac)
)
* Preparing for a Data Science Interview by [Luke Whipps](https://datatalks.club/people/lukewhipps.html)
([watch on youtube](https://www.youtube.com/watch?v=NnZjlMowkWA)
)
* Indie Hacking by [Pauline Clavelloux](https://datatalks.club/people/paulineclavelloux.html)
([watch on youtube](https://www.youtube.com/watch?v=KsV_SVXlTo8)
)
* Doing Software Engineering in Academia by [Johanna Bayer](https://datatalks.club/people/johannabayer.html)
([watch on youtube](https://www.youtube.com/watch?v=K0PdQITQzVQ)
)
* Bring Your Own Data by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=POGiLFWxQWQ)
)
* Modern Data Pipelines in AdTech - Life in the Trenches by [Roksolana Diachuk](https://datatalks.club/people/roksolanadiachuk.html)
([watch on youtube](https://www.youtube.com/watch?v=DCcRE1m6960)
)
* Observing Python Applications Using Prometheus by [Jessica Greene](https://datatalks.club/people/jessicagreene.html)
, [Vanessa Aguilar](https://datatalks.club/people/vanessaaguilar.html)
([watch on youtube](https://www.youtube.com/watch?v=Jb5ikhhttxc)
)
* Data-Centric AI by [Marysia Winkels](https://datatalks.club/people/marysiawinkels.html)
([watch on youtube](https://www.youtube.com/watch?v=t3HDdVWQzNM)
)
* Keeping Your Model In Production by [Amber Roberts](https://datatalks.club/people/amberroberts.html)
([watch on youtube](https://www.youtube.com/watch?v=UG_z9ekrJt0)
)
* Business Skills for Data Professionals by [Loris Marini](https://datatalks.club/people/lorismarini.html)
([watch on youtube](https://www.youtube.com/watch?v=xMYRUiTu960)
)
* Introduction to Bayesian Inference for Parameter Estimation by [Prasoon Shukla](https://datatalks.club/people/prasoonshukla.html)
([watch on youtube](https://www.youtube.com/watch?v=q1B6YwINFvc)
)
* From Software Engineer to Data Science Manager by [Sadat Anwar](https://datatalks.club/people/sadatanwar.html)
([watch on youtube](https://www.youtube.com/watch?v=xyTfqIWeKf8)
)
* Teaching and Mentoring in Data Analytics by [Irina Brudaru](https://datatalks.club/people/irinabrudaru.html)
([watch on youtube](https://www.youtube.com/watch?v=saaRRzgHsmE)
)
* Technical Writing and Data Journalism by [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
([watch on youtube](https://www.youtube.com/watch?v=uO_lk12q02A)
)
* From Digital Marketing to Analytics Engineering by [Nikola Maksimovic](https://datatalks.club/people/nikolamaksimovic.html)
([watch on youtube](https://www.youtube.com/watch?v=GawJ7mG5ElQ)
)
* Building an ML Service Platform from the Ground Up by [Sean Sheng](https://datatalks.club/people/seansheng.html)
([watch on youtube](https://www.youtube.com/watch?v=8h7vIN2WzT4)
)
* Product Owners in Data Science by [Anna Hannemann](https://datatalks.club/people/annahannemann.html)
([watch on youtube](https://www.youtube.com/watch?v=rTRTjB6cGng)
)
* Machine Learning Workflows in Production by [Krzysztof Szafanek](https://datatalks.club/people/krzysztofszafanek.html)
([watch on youtube](https://www.youtube.com/watch?v=CO4Gqd95j6k)
)
* Building Data Science Practice by [Andrey Shtylenko](https://datatalks.club/people/andreyshtylenko.html)
([watch on youtube](https://www.youtube.com/watch?v=XbDQv8FTA4U)
)
* Large-Scale Entity Resolution by [Sonal Goyal](https://datatalks.club/people/sonalgoyal.html)
([watch on youtube](https://www.youtube.com/watch?v=lpjffCOPxlY)
)
* Data Monetization with Machine Learning by [Tomek Jamiński](https://datatalks.club/people/tomekjaminski.html)
([watch on youtube](https://www.youtube.com/watch?v=CoO8PGcOgtE)
)
* From Data Science to DataOps by [Tomasz Hinc](https://datatalks.club/people/tomaszhinc.html)
([watch on youtube](https://www.youtube.com/watch?v=lem7knxqNzg)
)
* Data Science Career Development by [Katie Bauer](https://datatalks.club/people/katiebauer.html)
([watch on youtube](https://www.youtube.com/watch?v=i1NHRroQClQ)
)
* A Practical Guide to A/B Testing in MLOps by [Sadik Bakiu](https://datatalks.club/people/sadikbakiu.html)
([watch on youtube](https://www.youtube.com/watch?v=k9CqduJ7ha4)
)
* From Testing Phones to Managing NLP Projects by [Alvaro Navas Peire](https://datatalks.club/people/alvaronavaspeire.html)
([watch on youtube](https://www.youtube.com/watch?v=-xumbiXOlA8)
)
* Effective Machine Learning Inside Your Database by [Patricio Cerda Mardini](https://datatalks.club/people/patriciocerdamardini.html)
([watch on youtube](https://www.youtube.com/watch?v=hfIOh_r8NEE)
)
* Responsible and Explainable AI by [Supreet Kaur](https://datatalks.club/people/supreetkaur.html)
([watch on youtube](https://www.youtube.com/watch?v=8Eb5mG-pC3o)
)
* Building the Modern Geospatial Data Stack by [Ramiro Aznar](https://datatalks.club/people/ramiroaznar.html)
([watch on youtube](https://www.youtube.com/watch?v=VntcKnMGdhM)
)
* Data Incident Management with Soda and dbt by [Bastien Boutonnet](https://datatalks.club/people/bastienboutonnet.html)
([watch on youtube](https://www.youtube.com/watch?v=I3Fwhx1fw6M)
)
* Probabilistic Demand Forecasting at Scale by [Hagop Dippel](https://datatalks.club/people/hagopdippel.html)
([watch on youtube](https://www.youtube.com/watch?v=it7QyLPi4gU)
)
* Leading Data Research by [David Bader](https://datatalks.club/people/davidbader.html)
([watch on youtube](https://www.youtube.com/watch?v=vZLlpsUlchQ)
)
* Feature Selection in Machine Learning with Python by [Soledad Galli](https://datatalks.club/people/soledadgalli.html)
([watch on youtube](https://www.youtube.com/watch?v=blvmNWbcPDo)
)
* Dataset Creation and Curation by [Christiaan Swart](https://datatalks.club/people/christiannswart.html)
([watch on youtube](https://www.youtube.com/watch?v=QggWydGrWoo)
)
* Data Mesh 101 by [Zhamak Dehghani](https://datatalks.club/people/zhamakdehghani.html)
([watch on youtube](https://www.youtube.com/watch?v=346N_pCtYZU)
)
* Three Stages of Real-Time Data Monitoring by [Sage Elliott](https://datatalks.club/people/sageelliott.html)
([watch on youtube](https://www.youtube.com/watch?v=gZZjirywiUI)
)
* Growing Data Engineering Team in a Scale-Up by [Mehdi OUAZZA](https://datatalks.club/people/mehdiouazza.html)
([watch on youtube](https://www.youtube.com/watch?v=acJ6sVqKOUk)
)
* Lessons Learned About Data & AI at Enterprises by [Alexander Hendorf](https://datatalks.club/people/alexanderhendorf.html)
([watch on youtube](https://www.youtube.com/watch?v=Vms29u9xC3k)
)
* Feature Engineering for Time Series Forecasting by [Kishan Manani](https://datatalks.club/people/kishanmanani.html)
([watch on youtube](https://www.youtube.com/watch?v=2vMNiSeNUjI)
)
* MLOps Architect by [Danny Leybzon](https://datatalks.club/people/dannyleybzon.html)
([watch on youtube](https://www.youtube.com/watch?v=p1gVaS4Zx5M)
)
* Decoding Data Science Job Descriptions by [Tereza Iofciu](https://datatalks.club/people/terezaiofciu.html)
([watch on youtube](https://www.youtube.com/watch?v=bqxBiIwtmX4)
)
* Speeding up Python Code 500x by [Jamie Broomall](https://datatalks.club/people/jamiebroomall.html)
([watch on youtube](https://www.youtube.com/watch?v=LmHdEghy204)
)
* Data Science for Social Impact by [Christine Cepelak](https://datatalks.club/people/christinecepelak.html)
([watch on youtube](https://www.youtube.com/watch?v=xWC1HAfekRk)
)
* Hiring Data Science Talent by [Olga Ivina](https://datatalks.club/people/olgaivina.html)
([watch on youtube](https://www.youtube.com/watch?v=Af9t9r2b0z0)
)
* From Open-Source Maintainer to Founder by [Will McGugan](https://datatalks.club/people/willmcgugan.html)
([watch on youtube](https://www.youtube.com/watch?v=bwfR9dyxf1M)
)
* Best MLOps Practices for Building End-to-End ML Projects by [Alex Kim](https://datatalks.club/people/alexkim.html)
([watch on youtube](https://www.youtube.com/watch?v=2vDwAX9Bf8c)
)
* Designing a Data Science Organization by [Lisa Cohen](https://datatalks.club/people/lisacohen.html)
([watch on youtube](https://www.youtube.com/watch?v=F_rJ4fg5ZEA)
)
* Building the End to end Roadmap for your Data by [Mary Jane Dykeman](https://datatalks.club/people/maryjanedykeman.html)
([watch on youtube](https://www.youtube.com/watch?v=xiKcS1c0sBk)
)
* Developer Advocacy Engineer for Open-Source by [Merve Noyan](https://datatalks.club/people/mervenoyan.html)
([watch on youtube](https://www.youtube.com/watch?v=SnEYvF-Ztb8)
)
* Data Scientists at Work by [Mısra Turp](https://datatalks.club/people/misraturp.html)
([watch on youtube](https://www.youtube.com/watch?v=oUycqtMoYr8)
)
* Freelancing and Consulting with Data Engineering by [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
([watch on youtube](https://www.youtube.com/watch?v=9DTTrN-khCk)
)
* Getting a Data Engineering Job by [Jeff Katz](https://datatalks.club/people/jeffkatz.html)
([watch on youtube](https://www.youtube.com/watch?v=yvEWG-S1F_M)
)
* Using Data for Asteroid Mining by [Daynan Crull](https://datatalks.club/people/daynancrull.html)
([watch on youtube](https://www.youtube.com/watch?v=YxijEUoDCfw)
)
* Machine Learning in Marketing by [Juan Orduz](https://datatalks.club/people/juanorduz.html)
([watch on youtube](https://www.youtube.com/watch?v=jsAxUd_bZpw)
)
* Recommender Systems for Digital Advertising by [Nataliya Portman](https://datatalks.club/people/nataliyaportman.html)
([watch on youtube](https://www.youtube.com/watch?v=96HudrBW3rU)
)
* From Academia to Data Analytics and Engineering by [Gloria Quiceno](https://datatalks.club/people/gloriaquiceno.html)
([watch on youtube](https://www.youtube.com/watch?v=0wANfIvum4U)
)
* Mitigating Bias with the XAI Toolbox by [Serg Masis](https://datatalks.club/people/sergmasis.html)
([watch on youtube](https://www.youtube.com/watch?v=mP_9zEFwrho)
)
* Teaching Data Engineers by [Jeff Katz](https://datatalks.club/people/jeffkatz.html)
([watch on youtube](https://www.youtube.com/watch?v=dFo10l8B6Go)
)
* Data Quality and Reliability with Soda Core by [Vijay Kiran](https://datatalks.club/people/vijaykiran.html)
([watch on youtube](https://www.youtube.com/watch?v=CSqHZ1eJ5is)
)
* From Roasting Coffee to Backend Development by [Jessica Greene](https://datatalks.club/people/jessicagreene.html)
([watch on youtube](https://www.youtube.com/watch?v=BKqmNdxsBko)
)
* Building Machine Learning Pipelines with Kedro by [Merel Theisen](https://datatalks.club/people/mereltheisen.html)
([watch on youtube](https://www.youtube.com/watch?v=AUmDliHzWp0)
)
* Recruiting Data Engineers by [Nicolas Rassam](https://datatalks.club/people/nicolasrassam.html)
([watch on youtube](https://www.youtube.com/watch?v=hylxiu4VGTo)
)
* Federated Learning to the Rescue by [Orlando Hohmeier](https://datatalks.club/people/orlandohohmeier.html)
, [Jan Schlicht](https://datatalks.club/people/janschlicht.html)
([watch on youtube](https://www.youtube.com/watch?v=GtJsn3es2kA)
)
* Storytime for DataOps by [Christopher Bergh](https://datatalks.club/people/christopherbergh.html)
([watch on youtube](https://www.youtube.com/watch?v=0Fx5PCoLkf4)
)
* Analytics Use Cases Across Non-Automated Operations by [Delina Ivanova](https://datatalks.club/people/delinaivanova.html)
([watch on youtube](https://www.youtube.com/watch?v=TolOCxHdVx4)
)
* Machine Learning and Personalization in Healthcare by [Stefan Gudmundsson](https://datatalks.club/people/stefangudmundsson.html)
([watch on youtube](https://www.youtube.com/watch?v=IDzhmmKeNG4)
)
* Hands-On Data Monitoring with whylogs by [Danny Leybzon](https://datatalks.club/people/dannyleybzon.html)
([watch on youtube](https://www.youtube.com/watch?v=b6yk9b7A4CQ)
)
* Innovation and Design for Machine Learning by [Liesbeth Dingemans](https://datatalks.club/people/liesbethdingemans.html)
([watch on youtube](https://www.youtube.com/watch?v=tcqBfZw41FM)
)
* Modern Data Warehouse by [Rahul Jain](https://datatalks.club/people/16rahuljain.html)
([watch on youtube](https://www.youtube.com/watch?v=x2yNK3LlWUc)
)
* Hacking Your Data Career by [Marijn Markus](https://datatalks.club/people/marijnmarkus.html)
([watch on youtube](https://www.youtube.com/watch?v=RhSg8ill1So)
)
* Productionizing a Feature Store for Fraud Detection at Ubisoft by [Jeanine Harb](https://datatalks.club/people/jeanineharb.html)
([watch on youtube](https://www.youtube.com/watch?v=EmZIYcHGX9U)
)
* Visualising Machine Learning by [Meor Amer](https://datatalks.club/people/meoramer.html)
([watch on youtube](https://www.youtube.com/watch?v=OuCuk-7RHjM)
)
* From Math Teacher to Analytics Engineer by [Juan Pablo](https://datatalks.club/people/juanpablo.html)
([watch on youtube](https://www.youtube.com/watch?v=qh6-HDhw2xY)
)
* From Data Science to Data Engineering by [Ellen König](https://datatalks.club/people/ellenkonig.html)
([watch on youtube](https://www.youtube.com/watch?v=3TTu-hYzxeg)
)
* The Unspoken Relationship - Product Managers and Data Scientists by [Sivan Biham](https://datatalks.club/people/sivanbiham.html)
([watch on youtube](https://www.youtube.com/watch?v=4FmApxkyoXQ)
)
* Helping People Get Jobs with Data by [Padma Chitturi](https://datatalks.club/people/padmachitturi.html)
([watch on youtube](https://www.youtube.com/watch?v=wVwTMkRS1VM)
)
* Becoming a Data Engineering Manager by [Rahul Jain](https://datatalks.club/people/16rahuljain.html)
([watch on youtube](https://www.youtube.com/watch?v=FljnbUQ796w)
)
* Monitoring Model Performance with Crowdsourcing by [Magdalena Konkiewicz](https://datatalks.club/people/magdalenakonkiewicz.html)
([watch on youtube](https://www.youtube.com/watch?v=sFh7F7pJ6JI)
)
* A/B Testing by [Jakob Graff](https://datatalks.club/people/jakobgraff.html)
([watch on youtube](https://www.youtube.com/watch?v=0Gqx1LtqRZU)
)
* Load Testing ML Microservices for Robustness and Scalability by [Emmanuel Raj](https://datatalks.club/people/emmanuelraj.html)
([watch on youtube](https://www.youtube.com/watch?v=NE9vl4ma3Tk)
)
* Machine Learning System Design Interview by [Valerii Babushkin](https://datatalks.club/people/valeriybabushkin.html)
([watch on youtube](https://www.youtube.com/watch?v=0RsmRjar66E)
)
* Career Coaching by [Lindsay McQuade](https://datatalks.club/people/lindsaymcquade.html)
([watch on youtube](https://www.youtube.com/watch?v=_U8GrYJvmJM)
)
* Product Management Essentials for Data Professionals by [Greg Coquillo](https://datatalks.club/people/gregcoquillo.html)
([watch on youtube](https://www.youtube.com/watch?v=p4wg0Vd2uD4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Product-Management-Essentials-for-Data-Professionals---Greg-Coquillo-e1dr8g5)
)
* Software Testing for Machine Learning Pipelines by [Arseny Kravchenko](https://datatalks.club/people/arsenykravchenko.html)
([watch on youtube](https://www.youtube.com/watch?v=zrz6uj6Lr74)
)
* Recruiting Data Professionals by [Alicja Notowska](https://datatalks.club/people/alicjanotowska.html)
([watch on youtube](https://www.youtube.com/watch?v=WSMDXsjKYx4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Recruiting-Data-Professionals---Alicja-Notowska-e1dj2qn)
)
* DataTalks.Club Behind the Scenes by [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
, [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=IxTyq96juVE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/DataTalks-Club-Behind-the-Scenes---Eugene-Yan--Alexey-Grigorev-e1d4567)
)
* Fraud Detection in Financial Industry by [Kate Ogochukwu Nwankwo](https://datatalks.club/people/kateogochukwunwankwo.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Becoming a Data Science Manager by [Mariano Semelman](https://datatalks.club/people/marianosemelman.html)
([watch on youtube](https://www.youtube.com/watch?v=qOLR84-KHoY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-Science-Manager---Mariano-Semelman-e1cbrf7)
)
* Leading NLP Teams by [Ivan Bilan](https://datatalks.club/people/ivanbilan.html)
([watch on youtube](https://www.youtube.com/watch?v=RJEf6mzxh1w)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Leading-NLP-Teams---Ivan-Bilan-e1c4929)
)
* AI in Fashion - Size & Fit by [Nour Karessli](https://datatalks.club/people/nourkaressli.html)
([watch on youtube](https://datatalks.club/events.html)
)
* AI-Powered Computer Vision Applications in Media Industry by [Yulia Pavlova](https://datatalks.club/people/yuliapavlova.html)
([watch on youtube](https://www.youtube.com/watch?v=bINeBnWqFwo)
)
* Product Management for Machine Learning by [Geo Jolly](https://datatalks.club/people/geojolly.html)
([watch on youtube](https://www.youtube.com/watch?v=PjqjPvHliqg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Product-Management-for-Machine-Learning---Geo-Jolly-e1brpvm)
)
* Hypothesis Testing: Bayesian or Frequentist? Two ways of looking at the same coin by [Andre Schumacher](https://datatalks.club/people/andreschumacher.html)
([watch on youtube](https://www.youtube.com/watch?v=AkWmyE3kQ9k)
)
* Moving from Academia to Industry by [CJ Jenkins](https://datatalks.club/people/cjjenkins.html)
([watch on youtube](https://www.youtube.com/watch?v=m4F651BpUFk)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Moving-from-Academia-to-Industry---CJ-Jenkins-e1bh84o)
)
* Paradoxes in Data Science by [Pier Paolo Ippolito](https://datatalks.club/people/pierpaoloippolito.html)
([watch on youtube](https://www.youtube.com/watch?v=bIgGMfuctXQ)
)
* Advancing Big Data Analytics: Post-Doctoral Research by [Eleni Tzirita Zacharatou](https://datatalks.club/people/elenitziritazacharatou.html)
([watch on youtube](https://www.youtube.com/watch?v=7jgmIQGMhGE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Advancing-Big-Data-Analytics-Post-Doctoral-Research---Eleni-Tzirita-Zacharatou-e1b6f41)
)
* Notebooks in Production - Fun or not Fun? by [Nick Bilozerov](https://datatalks.club/people/nickbilozerov.html)
([watch on youtube](https://www.youtube.com/watch?v=yy88MqWQ-eM)
)
* Becoming a Data Product Manager by [Sara Menefee](https://datatalks.club/people/saramenefee.html)
([watch on youtube](https://www.youtube.com/watch?v=nt__pVuuC-k)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-Product-Manager---Sara-Menefee-e1arc4a)
)
* Back-of-the-Envelope Calculation For Machine Learning Projects by [Larysa Visengeriyeva](https://datatalks.club/people/larysavisengeriyeva.html)
([watch on youtube](https://www.youtube.com/watch?v=LFFdahY0w7A)
)
* Introduction to NLP for Industry Use by [Ivan Bilan](https://datatalks.club/people/ivanbilan.html)
([watch on youtube](https://www.youtube.com/watch?v=VRur3xey31s)
)
* Data Science Manager vs Data Science Expert by [Barbara Sobkowiak](https://datatalks.club/people/barbarasobkowiak.html)
([watch on youtube](https://www.youtube.com/watch?v=hFmIgaN-F8Y)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Science-Manager-vs-Data-Science-Expert---Barbara-Sobkowiak-e1ah3od)
)
* Algorithmic Fairness by [Gráinne McKnight](https://datatalks.club/people/grainnemcknight.html)
([watch on youtube](https://www.youtube.com/watch?v=CMojXBatk2c)
)
* Ace Non-Technical Data Science Interviews by [Nick Singh](https://datatalks.club/people/nicksingh.html)
([watch on youtube](https://www.youtube.com/watch?v=tRdLVUKU7Bo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Ace-Non-Technical-Data-Science-Interviews---Nick-Singh-e1a5qtd)
)
* The War of Gradient Boosted Trees by [Naomi Nguyen](https://datatalks.club/people/naominguyen.html)
([watch on youtube](https://www.youtube.com/watch?v=croe7mMze6s)
)
* Becoming a Solopreneur in Data by [Noah Gift](https://datatalks.club/people/noahgift.html)
([watch on youtube](https://www.youtube.com/watch?v=gCLUY37HGtw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Solopreneur-in-Data---Noah-Gift-e19gqbr)
)
* Building Business Acumen for Data Professionals by [Thom Ives](https://datatalks.club/people/thomives.html)
([watch on youtube](https://www.youtube.com/watch?v=pImYf9ML95Q)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-Business-Acumen-for-Data-Professionals---Thom-Ives-e19gq91)
)
* PyTorch Contributor's Guide: How and Why? by [Emil Bogomolov](https://datatalks.club/people/emilbogomolov.html)
([watch on youtube](https://www.youtube.com/watch?v=RCfnWe9VVGM)
)
* Conquering the Last Mile in Data by [Caitlin Moorman](https://datatalks.club/people/caitlinmoorman.html)
([watch on youtube](https://www.youtube.com/watch?v=HfMpG2zpa2I)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Conquering-the-Last-Mile-in-Data---Caitlin-Moorman-e1958c1)
)
* Similarities and Differences between ML and Analytics by [Rishabh Bhargava](https://datatalks.club/people/rishabhbhargava.html)
([watch on youtube](https://www.youtube.com/watch?v=rMRUa8WxDz4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Similarities-and-Differences-between-ML-and-Analytics---Rishabh-Bhargava-e18rcam)
)
* Building an Open-Source Feature Store with Apache Flink by [Roman Grebennikov](https://datatalks.club/people/romangrebennikov.html)
([watch on youtube](https://www.youtube.com/watch?v=BskjQPkrYec)
)
* Building and Leading Data Teams by [Tammy Liang](https://datatalks.club/people/tammyliang.html)
([watch on youtube](https://www.youtube.com/watch?v=kI4V2iBbaH0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-and-Leading-Data-Teams---Tammy-Liang-e18efdl)
)
* Introduction to Transformers for NLP by [Olga Petrova](https://datatalks.club/people/olgapetrova.html)
([watch on youtube](https://www.youtube.com/watch?v=hGNycrko5kc)
)
* What Researchers and Engineers Can Learn from Each Other by [Mihail Eric](https://datatalks.club/people/mihaileric.html)
([watch on youtube](https://www.youtube.com/watch?v=d9xVXqKq3sU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-Researchers-and-Engineers-Can-Learn-from-Each-Other---Mihail-Eric-e1854bj)
)
* Doing Good with Data by [Marijn Markus](https://datatalks.club/people/marijnmarkus.html)
([watch on youtube](https://www.youtube.com/watch?v=esv5-4HRXvI)
)
* Modern Data Stack for Analytics Engineering by [Kyle Shannon](https://datatalks.club/people/kyleshannon.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Introducing Data Science in Startups by [Marianna Diachuk](https://datatalks.club/people/mariannadiachuk.html)
([watch on youtube](https://youtube.com/watch?v=KMSE9GkU2mE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Introducing-Data-Science-in-Startups---Marianna-Diachuk-e17rc4i)
)
* Machine Learning Observability by [Aparna Dhinakaran](https://datatalks.club/people/aparnadhinakaran.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Defining Success: Metrics and KPIs by [Adam Sroka](https://datatalks.club/people/adamsroka.html)
([watch on youtube](https://www.youtube.com/watch?v=H4P2RfKvXGs)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Defining-Success-Metrics-and-KPIs---Adam-Sroka-e17gfp0)
)
* Getting Started with Network Analytics in Python by [Eric Sims](https://datatalks.club/people/ericsims.html)
([watch on youtube](https://www.youtube.com/watch?v=LwSeYUlvvtE)
)
* Making Sense of Data Engineering Acronyms and Buzzwords by [Natalie Kwong](https://datatalks.club/people/nataliekwong.html)
([watch on youtube](https://www.youtube.com/watch?v=t9Z1S3OYnJU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Making-Sense-of-Data-Engineering-Acronyms-and-Buzzwords---Natalie-Kwong-e177303)
)
* Make Your First ML Chatbot by [Rachael Tatman](https://datatalks.club/people/rachaeltatman.html)
([watch on youtube](https://www.youtube.com/watch?v=YwjauyAflmc)
)
* Mastering Algorithms and Data Structures by [Marcello La Rocca](https://datatalks.club/people/marcellolarocca.html)
([watch on youtube](https://www.youtube.com/watch?v=RiQa-9LguW8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Mastering-Algorithms-and-Data-Structures---Marcello-La-Rocca-e16s7lf)
)
* Building Streaming Analytics: The Journey and Learnings by [Maxim Lukichev](https://datatalks.club/people/maximlukichev.html)
([watch on youtube](https://www.youtube.com/watch?v=DN_p_tFSw6c)
)
* Chief Data Officer by [Marco De Sa](https://datatalks.club/people/marcodesa.html)
([watch on youtube](https://www.youtube.com/watch?v=IdaZOD46FEw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Chief-Data-Officer---Marco-De-Sa-e16hm4t)
)
* Freelancing in Machine Learning by [Mikio Braun](https://datatalks.club/people/mikiobraun.html)
([watch on youtube](https://www.youtube.com/watch?v=HfF791e0HR8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Freelancing-in-Machine-Learning---Mikio-Braun-e166n7r)
)
* Orchestrating Enterprise ML Workload Jobs Across Clouds by [Srivathsan Canchi](https://datatalks.club/people/srivathsancanchi.html)
, [Alex Chung](https://datatalks.club/people/alexchung.html)
([watch on youtube](https://www.youtube.com/watch?v=ocDP-94YFjI)
)
* Launching a Startup: From Idea to First Hire by [Carmine Paolino](https://datatalks.club/people/carminepaolino.html)
([watch on youtube](https://www.youtube.com/watch?v=s-w8_GDgIlU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Launching-a-Startup-From-Idea-to-First-Hire---Carmine-Paolino-e15sk4i)
)
* Unboxing Design Docs for Data Scientists by [Vincent Tatan](https://datatalks.club/people/vincenttatan.html)
([watch on youtube](https://www.youtube.com/watch?v=vO3Rr50hWSU)
)
* Modeling the Human Brain by [Jessie Yaros](https://datatalks.club/people/jessieyaros.html)
([watch on youtube](https://www.youtube.com/watch?v=6X7P-zwSi7E)
)
* The Importance of Data Quality by [Fabiana Clemente](https://datatalks.club/people/fabianaclemente.html)
([watch on youtube](https://www.youtube.com/watch?v=SQULlY5ZOcw)
)
* Why Your Search Relevance Project Will Fail by [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
([watch on youtube](https://www.youtube.com/watch?v=Ms9QBBB8MxE)
)
* Humans in the Loop by [Lina Weichbrodt](https://datatalks.club/people/linaweichbrodt.html)
([watch on youtube](https://www.youtube.com/watch?v=o50j_Ndx2Hg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Humans-in-the-Loop---Lina-Weichbrodt-e14npgp)
)
* Running from Complexity by [Ben Wilson](https://datatalks.club/people/benwilson.html)
([watch on youtube](https://www.youtube.com/watch?v=sMy8NYZnsy8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Running-from-Complexity---Ben-Wilson-e14np51)
)
* Setting Up AI Projects for Success by [Jan Zawadzki](https://datatalks.club/people/janzawadzki.html)
([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* DataTalks.Club Summer Marathon: Machine Learning in Production by [Jan Zawadzki](https://datatalks.club/people/janzawadzki.html)
, [Ben Wilson](https://datatalks.club/people/benwilson.html)
, [Lina Weichbrodt](https://datatalks.club/people/linaweichbrodt.html)
, [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
, [Fabiana Clemente](https://datatalks.club/people/fabianaclemente.html)
([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* I Want to Build a Machine Learning Startup! by [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
([watch on youtube](https://www.youtube.com/watch?v=DiDs5aMjEWg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/I-Want-to-Build-a-Machine-Learning-Startup----Elena-Samuylova-e139ste)
)
* Big Data Engineer vs Data Scientist by [Roksolana Diachuk](https://datatalks.club/people/roksolanadiachuk.html)
([watch on youtube](https://www.youtube.com/watch?v=yg3d1lFd7Uo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Big-Data-Engineer-vs-Data-Scientist---Roksolana-Diachuk-e139sl8)
)
* Build Your Own Data Pipeline by [Andreas Kretz](https://datatalks.club/people/andreaskretz.html)
([watch on youtube](https://www.youtube.com/watch?v=IrZPAG6OBqo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Build-Your-Own-Data-Pipeline---Andreas-Kretz-e139se1)
)
* From Software Engineering to Machine Learning by [Santiago Valdarrama](https://datatalks.club/people/svpino.html)
([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/From-Software-Engineering-to-Machine-Learning---Santiago-Valdarrama-e139s63)
)
* DataTalks.Club Summer Marathon: Career in Data by [Santiago Valdarrama](https://datatalks.club/people/svpino.html)
, [Daliana Liu](https://datatalks.club/people/dalianaliu.html)
, [Andreas Kretz](https://datatalks.club/people/andreaskretz.html)
, [Roksolana Diachuk](https://datatalks.club/people/roksolanadiachuk.html)
, [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
* Analytics Engineer: New Role in a Data Team by [Victoria Perez Mola](https://datatalks.club/people/victoriaperezmola.html)
([watch on youtube](https://www.youtube.com/watch?v=C5UcxBwdCEg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Analytics-Engineer-New-Role-in-a-Data-Team---Victoria-Perez-Mola-e131e3n)
)
* Ingestion and Historization in the Data Lake by [Illia Todor](https://datatalks.club/people/illiatodor.html)
([watch on youtube](https://datatalks.club/events.html)
)
* Data Governance by [Jessi Ashdown](https://datatalks.club/people/jessiashdown.html)
, [Uri Gilad](https://datatalks.club/people/urigilad.html)
([watch on youtube](https://www.youtube.com/watch?v=tJ3v8h7A7RY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Governance---Jessi-Ashdown--Uri-Gilad-e12jmoo)
)
* Conversational AI by [Merve Noyan](https://datatalks.club/people/mervenoyan.html)
([watch on youtube](https://www.youtube.com/watch?v=Jrigg7n-bt8)
)
* What Data Scientists Don’t Mention in Their LinkedIn Profiles by [Yury Kashnitsky](https://datatalks.club/people/yurykashnitsky.html)
([watch on youtube](https://www.youtube.com/watch?v=c6dK1LWpv4g)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-Data-Scientists-Dont-Mention-in-Their-LinkedIn-Profiles---Yury-Kashnitsky-e125jjl)
)
* Productionizing ML Systems without Fear nor Heroism by [Nastasia Saby](https://datatalks.club/people/nastasiasaby.html)
([watch on youtube](https://www.youtube.com/watch?v=KkOGMaz4Xws)
)
* Becoming a Data-led Professional by [Arpit Choudhury](https://datatalks.club/people/arpitchoudhury.html)
([watch on youtube](https://www.youtube.com/watch?v=8v5KpHWgyYw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-led-Professional---Arpit-Choudhury-e11mkgq)
)
* A Framework for Feature Engineering and Machine Learning Pipelines by [Jacques Peeters](https://datatalks.club/people/jacquespeeters.html)
([watch on youtube](https://www.youtube.com/watch?v=HK-kBkERTBY)
)
* How to Market Yourself (without Being a Celebrity) by [Shawn Swyx Wang](https://datatalks.club/people/swyx.html)
([watch on youtube](https://www.youtube.com/watch?v=tkBCPqWKCL8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/How-to-Market-Yourself-without-Being-a-Celebrity---Shawn-Swyx-Wang-e11ai8t)
)
* Hands-on AI Observability by [Andy Petrella](https://datatalks.club/people/andypetrella.html)
([watch on youtube](https://www.youtube.com/watch?v=ARfrqa-t-Xk)
)
* From Physics to Machine Learning by [Tatiana Gabruseva](https://datatalks.club/people/tatianagabruseva.html)
([watch on youtube](https://www.youtube.com/watch?v=wJPi6Ip9PX0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/From-Physics-to-Machine-Learning---Tatiana-Gabruseva-e10r4pl)
)
* Setting up an A/B Testing Framework by [Agnes van Belle](https://datatalks.club/people/agnesvanbelle.html)
([watch on youtube](https://www.youtube.com/watch?v=u3afDiIrKo4)
)
* What I Learned After Interviewing 300 Data Scientists by [Oleg Novikov](https://datatalks.club/people/olegnovikov.html)
([watch on youtube](https://www.youtube.com/watch?v=AYi7b-8GPm4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-I-Learned-After-Interviewing-300-Data-Scientists---Oleg-Novikov-e10ctbs)
)
* Reinforcement Learning for Search by [Roman Grebennikov](https://datatalks.club/people/romangrebennikov.html)
([watch on youtube](https://www.youtube.com/watch?v=oVkD-2xbqKM)
)
* Effective Communication with Business for Data Professionals by [Lior Barak](https://datatalks.club/people/liorbarak.html)
([watch on youtube](https://www.youtube.com/watch?v=gqroEsTyLD0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Effective-Communication-with-Business-for-Data-Professionals---Lior-Barak-e1002rm)
)
* Evaluation Store: a New Category of ML Engineering Tools by [Josh Tobin](https://datatalks.club/people/joshtobin.html)
([watch on youtube](https://www.youtube.com/watch?v=KzyKVGIMIU0)
)
* Data Observability: The Next Frontier of Data Engineering by [Barr Moses](https://datatalks.club/people/barrmoses.html)
([watch on youtube](https://www.youtube.com/watch?v=TrMG1SOqZkQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Observability---Barr-Moses-evghmh)
)
* Data Mesh in Practice by [Max Schultze](https://datatalks.club/people/maxschultze.html)
([watch on youtube](https://www.youtube.com/watch?v=ekEc8D_D3zY)
)
* Shifting Career from Analytics to Data Science by [Andrada Olteanu](https://datatalks.club/people/andradaolteanu.html)
([watch on youtube](https://www.youtube.com/watch?v=ixmTewD5Waw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Shifting-Career-from-Analytics-to-Data-Science---Andrada-Olteanu-ev19ma)
)
* Data Science for Social Good by [Filipa Castro](https://datatalks.club/people/filipacastro.html)
, [Agnieszka Mikołajczyk](https://datatalks.club/people/agnieszkamikolajczyk.html)
([watch on youtube](https://www.youtube.com/watch?v=arwHfVX8_cc)
)
* Transitioning from Project Management to Data Science by [Ksenia Legostay](https://datatalks.club/people/ksenialegostay.html)
([watch on youtube](https://www.youtube.com/watch?v=rBKezdb9jEc)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Transitioning-from-Project-Management-to-Data-Science---Ksenia-Legostay-euig2a)
)
* Deep Learning Recommender Systems by [Cristian Martinez](https://datatalks.club/people/cristianmartinez.html)
, [Ilia Ivanov](https://datatalks.club/people/iliaivanov.html)
([watch on youtube](https://www.youtube.com/watch?v=LWAQUgJOYm0)
)
* Building and Growing Online Communities by [Demetrios Brinkmann](https://datatalks.club/people/demetriosbrinkmann.html)
([watch on youtube](https://www.youtube.com/watch?v=ByCE1vSrIr8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-Online-Tech-Communities---Demetrios-Brinkmann-eu35fo)
)
* Bodywork: GitOps for Machine Learning by [Alex Ioannides](https://datatalks.club/people/alexioannides.html)
([watch on youtube](https://www.youtube.com/watch?v=m4cn7HJUxng)
)
* DataOps 101 by [Lars Albertsson](https://datatalks.club/people/larsalbertsson.html)
([watch on youtube](https://www.youtube.com/watch?v=vyF3yGsF6UY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/DataOps-101---Lars-Albertsson-ethsp1)
)
* Active and Self-Supervised Learning for Data Selection by [Igor Susmelj](https://datatalks.club/people/igorsusmelj.html)
([watch on youtube](https://www.youtube.com/watch?v=US1TTdRRUQQ)
)
* The Essentials of Public Speaking for Career in Data Science by [Ben Taylor](https://datatalks.club/people/bentaylor.html)
([watch on youtube](https://www.youtube.com/watch?v=wOFvlR9UBxI)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Essentials-of-Public-Speaking-for-Career-in-Data-Science---Ben-Taylor-et0m4p)
)
* Building Scalable End-to-End Deep Learning Pipelines in the Cloud by [Rustem Feyzkhanov](https://datatalks.club/people/rustemfeyzkhanov.html)
([watch on youtube](https://www.youtube.com/watch?v=Nq8-VdBEY98)
)
* Continuous Integration for Machine Learning by [Elle O'Brien](https://datatalks.club/people/elleobrien.html)
([watch on youtube](https://www.youtube.com/watch?v=A3OEaaiGPhk)
)
* Machine Learning Design Patterns by [Sara Robinson](https://datatalks.club/people/sararobinson.html)
([watch on youtube](https://www.youtube.com/watch?v=aL_zQfi7lDI)
)
* Putting Data Science in Production by [Mikio Braun](https://datatalks.club/people/mikiobraun.html)
([watch on youtube](https://www.youtube.com/watch?v=gFuEgIeZzIo)
)
* 10 Foundational Practices of Machine Learning Engineering by [Larysa Visengeriyeva](https://datatalks.club/people/larysavisengeriyeva.html)
([watch on youtube](https://www.youtube.com/watch?v=4gHSUjNUopc)
)
* DataTalks.Club Conference: ML in Production by [Larysa Visengeriyeva](https://datatalks.club/people/larysavisengeriyeva.html)
, [Mikio Braun](https://datatalks.club/people/mikiobraun.html)
, [Sara Robinson](https://datatalks.club/people/sararobinson.html)
, [Elle O'Brien](https://datatalks.club/people/elleobrien.html)
([watch on youtube](https://www.youtube.com/watch?v=og1DG1KZ71c)
)
* New Roles and Key Skills to Monetize Machine Learning by [Vin Vashishta](https://datatalks.club/people/vinvashishta.html)
([watch on youtube](https://www.youtube.com/watch?v=xCjzA_8S4kI)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/New-Roles-and-Key-Skills-to-Monetize-Machine-Learning---Vin-Vashishta-escer6)
)
* Career Transitioning into Data Science by [Parul Pandey](https://datatalks.club/people/parulpandey.html)
([watch on youtube](https://www.youtube.com/watch?v=slczbYNn_Kg)
)
* Personal Branding by [Admond Lee Kin Lim](https://datatalks.club/people/admondleekinlim.html)
([watch on youtube](https://www.youtube.com/watch?v=tQRQnz_aHYQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Personal-Branding---Admond-Lee-Kin-Lim-ern77e)
)
* The ABC’s of Data Science by [Danny Ma](https://datatalks.club/people/dannyma.html)
([watch on youtube](https://www.youtube.com/watch?v=HVQ0DZOQcts)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-ABCs-of-Data-Science---Danny-Ma-er33oa)
)
* DataTalks.Club Conference: Career in Data by [Danny Ma](https://datatalks.club/people/dannyma.html)
, [Admond Lee Kin Lim](https://datatalks.club/people/admondleekinlim.html)
, [Parul Pandey](https://datatalks.club/people/parulpandey.html)
, [Vin Vashishta](https://datatalks.club/people/vinvashishta.html)
([watch on youtube](https://www.youtube.com/watch?v=ltFkvoiA57M)
)
* Translating ML Predictions Into Better Real-World Results with Decision Optimization by [Dan Becker](https://datatalks.club/people/danbecker.html)
([watch on youtube](https://www.youtube.com/watch?v=SJuzQ4bcU2c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Translating-ML-Predictions-Into-Better-Real-World-Results-with-Decision-Optimization---Dan-Becker-eqk0b1)
)
* How Your Machine Learning Project Will Fail by [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
([watch on youtube](https://www.youtube.com/watch?v=bVxHQH2-PMo)
)
* Dangers of Dirty Data by [Susan Walsh](https://datatalks.club/people/susanwalsh.html)
([watch on youtube](https://www.youtube.com/watch?v=N6994_LkdaI)
)
* Building Data-Intensive Teams by [Elias Nema](https://datatalks.club/people/eliasnema.html)
([watch on youtube](https://www.youtube.com/watch?v=5rBE6MO4lac)
)
* DataTalks.Club Conference: Product and Process by [Elias Nema](https://datatalks.club/people/eliasnema.html)
, [Susan Walsh](https://datatalks.club/people/susanwalsh.html)
, [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
, [Dan Becker](https://datatalks.club/people/danbecker.html)
([watch on youtube](https://www.youtube.com/watch?v=dvzPU43tqFM)
)
* Building an ML System for Southeast Asia’s Largest Hospital Group by [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
([watch on youtube](https://www.youtube.com/watch?v=G5F-L7hdqSQ)
)
* Build Your AI Machine Vision System by Yourself by [Mahmoud AbdelAziz](https://datatalks.club/people/mahmoudaziz.html)
([watch on youtube](https://www.youtube.com/watch?v=GPeJKcBvKrw)
)
* How to use AI in Consumer Food Product Innovation by [Himanshu Upreti](https://datatalks.club/people/himanshuupreti.html)
([watch on youtube](https://www.youtube.com/watch?v=7DuTkKrOZ1Y)
)
* Industrial Applications of Reinforcement Learning by [Phil Winder](https://datatalks.club/people/philwinder.html)
([watch on youtube](https://www.youtube.com/watch?v=ih6IEv89IV4)
)
* DataTalks.Club Conference: ML Use Cases by [Phil Winder](https://datatalks.club/people/philwinder.html)
, [Himanshu Upreti](https://datatalks.club/people/himanshuupreti.html)
, [Mahmoud AbdelAziz](https://datatalks.club/people/mahmoudaziz.html)
, [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
([watch on youtube](https://www.youtube.com/watch?v=jvqS1_GnLsk)
)
* Feature Stores: Cutting through the Hype by [Willem Pienaar](https://datatalks.club/people/willempienaar.html)
([watch on youtube](https://www.youtube.com/watch?v=FQYTb4uWljQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Feature-Stores-Cutting-through-the-Hype---Willem-Pienaar-ept6m8)
)
* The Rise of MLOps by [Theofilos Papapanagiotou](https://datatalks.club/people/theofilospapapanagiotou.html)
([watch on youtube](https://www.youtube.com/watch?v=-i0fVp0ntYA)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Rise-of-MLOps---Theofilos-Papapanagiotou-ept67o)
)
* Getting Started with Open Source by [Vincent Warmerdam](https://datatalks.club/people/vincentwarmerdam.html)
([watch on youtube](https://www.youtube.com/watch?v=IxV9EH-tphQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Getting-Started-with-Open-Source---Vincent-Warmerdam-epk60j)
)
* Machine Learning for Customer Service by [Neal Lathia](https://datatalks.club/people/neallathia.html)
([watch on youtube](https://www.youtube.com/watch?v=ohjSvKdUumY)
)
* Developer Advocacy for Data Science by [Elle O'Brien](https://datatalks.club/people/elleobrien.html)
([watch on youtube](https://www.youtube.com/watch?v=jv5W4jXk4P4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Developer-Advocacy-for-Data-Science---Elle-OBrien-epcbak)
)
* Machine Learning Performance Monitoring by [Emeli Dral](https://datatalks.club/people/emelidral.html)
([watch on youtube](https://www.youtube.com/watch?v=iiLadbM_It8)
)
* The Importance of Writing in a Tech Career by [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
([watch on youtube](https://www.youtube.com/watch?v=vXWGd7olv3c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Importance-of-Writing-in-a-Tech-Career---Eugene-Yan-ep17du)
)
* Data Versioning Explained by [Itai Admi](https://datatalks.club/people/itaiadmi.html)
([watch on youtube](https://www.youtube.com/watch?v=FU2tQSobPNk)
)
* Mentoring by [Rahul Jain](https://datatalks.club/people/rahuljain.html)
([watch on youtube](https://www.youtube.com/watch?v=LQvwTNQbPg4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Mentoring---Rahul-Jain-eo7cmu)
)
* AWS Glue DataBrew by [Dmitry Muzalevskiy](https://datatalks.club/people/dmitrymuzalevskiy.html)
([watch on youtube](https://www.youtube.com/watch?v=HQTKAcuCbTc)
)
* Standing out as a Data Scientist by [Luke Whipps](https://datatalks.club/people/lukewhipps.html)
([watch on youtube](https://www.youtube.com/watch?v=Sb4CJlonB3c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Standing-out-as-a-Data-Scientist---Luke-Whipps-envr7e)
)
* Deploying Serverless Machine Learning with AWS by [Sejal Vaidya](https://datatalks.club/people/sejalvaidya.html)
([watch on youtube](https://www.youtube.com/watch?v=79B8AOKkpho)
)
* Building a Data Science Team by [Dat Tran](https://datatalks.club/people/dattran.html)
([watch on youtube](https://www.youtube.com/watch?v=ScDIB-3O77A)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-a-Data-Science-Team---Dat-Tran-enlmef)
)
* Processes in a Data Science Project by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=SesVTDklFYQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Processes-in-a-Data-Science-Project---Alexey-Grigorev-encdlg)
)
* Essential Communication Skills for Data Professionals by [David Gates](https://datatalks.club/people/davidgates.html)
([watch on youtube](https://www.youtube.com/watch?v=ZRtVBflBkuc)
)
* Fighting Fraud with Triplet Loss by [Artemii Frolov](https://datatalks.club/people/artemiifrolov.html)
([watch on youtube](https://www.youtube.com/watch?v=1Jabakbryyk)
)
* Roles in a Data Team by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
([watch on youtube](https://www.youtube.com/watch?v=2ZOnA19sDpM)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Roles-in-a-data-team---Alexey-Grigorev-emqcft)
)
* Customer Segmentation 2.0 by [Nishant Mohan](https://datatalks.club/people/nishantmohan.html)
([watch on youtube](https://www.youtube.com/watch?v=pWqD7SGuihs)
)
* Deploying models with AWS Sagemaker by [Dmitry Muzalevskiy](https://datatalks.club/people/dmitrymuzalevskiy.html)
([watch on youtube](https://www.youtube.com/watch?v=2ZOnA19sDpM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Katarzyna Foremniak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Katarzyna Foremniak
Katarzyna is a computational linguist with over 10 years of experience in NLP and speech recognition. She has developed language models for automotive brands like Audi and Porsche and specializes in phonetics, morpho-syntax, and sentiment analysis. Kasia also teaches at the University of Warsaw and is passionate about human-centered AI and multilingual NLP.
[](https://linkedin.com/in/katarzyna-foremniak)
### Events
* Human-Centered AI for Disordered Speech Recognition ([watch on youtube](https://www.youtube.com/watch?v=yTZ4cddD7DU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kate Ogochukwu Nwankwo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kate Ogochukwu Nwankwo
Kate is co-founder of Datatera and Data Science and Al Engineering fellow at Women Techsters. Passionate about using AI-powered technology and data to drive business growth.
[](https://linkedin.com/in/nwankwo-kate-ogochukwu-1a1a1a170)
### Events
* Fraud Detection in Financial Industry ([watch on youtube](https://datatalks.club/people/kateogochukwunwankwo.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Katharine Jarmul – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Katharine Jarmul
Katharine Jarmul is a privacy activist, machine learning engineer, and Principal Data Scientist at Thoughtworks Germany. Previously, Katharine held numerous roles at large companies and startups in the US and Germany, implementing data processing and machine learning systems with a focus on reliability, testability, privacy and security.
[](https://twitter.com/kjam)
[](https://linkedin.com/in/katharinejarmul)
### Events
* Practical Data Privacy ([watch on youtube](https://www.youtube.com/watch?v=gbjoFfrm4iw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Katie Bauer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Katie Bauer
Katie Bauer is a data science leader also known for her writing and thinking about data careers and managing effective data teams. She is currently the Head of Data at GlossGenius, a leading software platform for beauty professionals in the US salon and studio space. Previously she led Twitter’s infrastructure data science and analytics organization, ran the consumer data science team at Reddit, and has worked as an analyst and ML engineer in Adtech and natural language search.
[](https://twitter.com/imightbemary)
[](https://linkedin.com/in/mkatiebauer)
[](https://github.com/imightbemary)
[](http://katiebauer.net/about)
### Events
* Data Science Career Development ([watch on youtube](https://www.youtube.com/watch?v=i1NHRroQClQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ken Youens-Clark – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ken Youens-Clark
Ken Youens-Clark is a Senior Scientific Programmer at the University of Arizona. He has an MS in Biosystems Engineering and has been programming for over 20 years.
[](https://twitter.com/kycl4rk)
[](https://linkedin.com/in/kycl4rk)
[](https://github.com/kyclark)
[](http://www.kyclark.us/)
### Books
* [Tiny Python Projects](https://datatalks.club/books/20210426-tiny-python-projects.html)
(the book of the week from 26 Apr 2021 to 30 Apr 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kevin Huo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kevin Huo
Kevin Huo is currently a Data Scientist at a Hedge Fund, and previously was a Data Scientist at Facebook working on Facebook Groups. He holds a degree in Computer Science from the University of Pennsylvania and a degree in business from Wharton. In college he interned on Wall Street, at Facebook and Bloomberg.
[](https://linkedin.com/in/kevin-huo)
[](https://www.acethedatascienceinterview.com/)
### Books
* [Ace The Data Science Interview](https://datatalks.club/books/20211115-ace-the-data-science-interview.html)
(the book of the week from 15 Nov 2021 to 19 Nov 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Khuyen Tran – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Khuyen Tran
Khuyen Tran transforms how data scientists learn and work. She has written over 180 articles as a top writer on Towards Data Science, helping data professionals bridge the gap between prototyping and production. As founder of CodeCut, she publishes daily Python tips in her newsletter that reach over 10,000 views per month and has built a community of 110,000 LinkedIn followers. Previously an MLOps Engineer and Senior Data Engineer at Accenture, she built enterprise data solutions for clients worldwide.
[](https://twitter.com/KhuyenTran16)
[](https://linkedin.com/in/khuyen-tran-1401)
[](https://github.com/khuyentran1401)
[](https://khuyentran1476.medium.com/)
### Books
* [Production‑Ready Data Science](https://datatalks.club/books/20250728-production-ready-data-science.html)
(the book of the week from 28 Jul 2025 to 01 Aug 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kim Falk – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kim Falk
Passionate data scientist with experience in machine learning specialized in Recommender systems. Worked on recommenders for customers like BT and Manning Publishing. Added user segmentation in Sitecore CMS and worked on Danish NLP Models for named entity extraction as well as Deep Learning classifier to predict verdicts of legal cases. Experience in leading small teams. Keen on research and keeping up to date, with a focus on results.
Author of Practical Recommender Systems.
[](https://twitter.com/kimfalk)
[](https://linkedin.com/in/kimfalken)
[](https://github.com/kimfalk)
[](https://kimfalk.org/)
### Books
* [Practical Recommender Systems](https://datatalks.club/books/20210802-practical-recommender-systems.html)
(the book of the week from 02 Aug 2021 to 06 Aug 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kishan Manani – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kishan Manani
Kishan is a machine learning and data science lead, online course instructor, and open source software contributor. He leads data science teams to deliver data and machine learning products end-to-end. He has 10+ years of experience in applying machine learning and statistics in finance, e-commerce, and healthcare research. He contributes to well known Python packages including Statsmodels, Feature-engine, and Prophet. Kishan Attained a PhD in Physics from Imperial College London in applied large scale time-series analysis and modelling of cardiac arrhythmias.
[](https://twitter.com/KishManani)
[](https://linkedin.com/in/kishanmanani)
### Events
* Feature Engineering for Time Series Forecasting ([watch on youtube](https://www.youtube.com/watch?v=2vMNiSeNUjI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Konrad Banachewicz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Konrad Banachewicz
Konrad Banachewicz holds a PhD in statistics from Vrije Universiteit Amsterdam. He is a lead data scientist at eBay and a Kaggle Grandmaster. He worked in a variety of financial institutions on a wide array of quantitative data analysis problems. In the process, he became an expert on the entire lifetime of a data product cycle.
[](https://twitter.com/tng_konrad)
[](https://linkedin.com/in/konrad-banachewicz-8879572)
[](https://github.com/tng-konrad)
### Books
* [The Kaggle Book](https://datatalks.club/books/20220919-kaggle-book.html)
(the book of the week from 19 Sep 2022 to 23 Sep 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kranti K. Parisa – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kranti K. Parisa
Kranti K. Parisa is currently the Vice President & Head of Product Engineering at Dialpad. His teams build large scale, cloud native real-time business communications & collaboration software with industry leading in-house AI/ML & Telephony technology. Before Dialpad, he has led teams that are responsible for search and personalization platforms, products and services at Apple. Kranti was a cofounder, CTO and technical advisor of multiple start-ups focusing on cloud computing, SaaS, and enterprise search. He has contributed to the Apache Lucene/Solr community and co-authored the book Apache Solr Enterprise Search Server. For his outstanding contributions to Search & Discovery, U.S. Government has recognized him as a Person of Extraordinary Ability (EB1A).
[](https://twitter.com/krantiparisa)
[](https://linkedin.com/in/krantiparisa)
### Books
* [Reliable Machine Learning](https://datatalks.club/books/20221121-reliable-machine-learning.html)
(the book of the week from 05 Dec 2022 to 09 Dec 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Krzysztof Ograbek – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Krzysztof Ograbek
Krzysztof (Kris) Ograbek is a content creator, lifelong learner, and a member of the “5 A.M. Club”. In his 3-year-long search for a perfect niche, he created content about various topics, such as Blockchain, fatherhood, learning how to learn, and many more. He creates practical projects based on Large Language Models, his current (and hopefully final) passion. As a believer in the “teach everything you learn” motto, he documents his journey on 3 platforms: YouTube, Medium, and LinkedIn. After work, he turns into a world-class dad and husband.
[](https://linkedin.com/in/kris-ograbek)
### Events
* Prompt Engineering 101 ([watch on youtube](https://www.youtube.com/watch?v=qLGnW2vxors)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# DataTalks.Club Podcast – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Podcast
=======
Listen to conversations with data science experts, machine learning practitioners, and AI researchers. Subscribe to stay updated with the latest episodes.
Listen to or watch on your favorite platform
--------------------------------------------
[\
\
Apple Podcasts](https://podcasts.apple.com/us/podcast/id1541710331)
[\
\
Spotify](https://open.spotify.com/show/0pck8zuiXdI0OrCg86DAPy)
[\
\
YouTube](https://www.youtube.com/c/DataTalksClub)
[\
\
Anchor](https://anchor.fm/datatalksclub)
Register for upcoming podcast events in [events](https://datatalks.club/events.html)
.
All Podcast Episodes
--------------------
### Season #22
* [Building Pet Health Tech: ML, Sensors, and Dog Behavior Data](https://datatalks.club/podcast/s22e08-building-pet-health-tech-ml-sensors-and-dog-behavior-data.html "Listen to: Building Pet Health Tech: ML, Sensors, and Dog Behavior Data")
with [Sofya Yulpatova](https://datatalks.club/people/sofyayulpatova.html)
* [Reinventing a Career in Tech](https://datatalks.club/podcast/s22e07-reinventing-career-in-tech.html "Listen to: Reinventing a Career in Tech")
with [Xia He-Bleinagel](https://datatalks.club/people/xiahebleinagel.html)
* [From Black-Box Systems to Augmented Decision-Making](https://datatalks.club/podcast/s22e06-from-black-box-systems-to-augmented-decision-making.html "Listen to: From Black-Box Systems to Augmented Decision-Making")
with [Anusha Akkina](https://datatalks.club/people/anushaakkina.html)
* [Practical LLM Engineering and RAG: Prompting, Evaluation and Real-World Workflows](https://datatalks.club/podcast/practical-llm-engineering-and-rag.html "Listen to: Practical LLM Engineering and RAG: Prompting, Evaluation and Real-World Workflows")
with [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html)
* [Bioinformatics Workflows in Practice: Sequencing, Metagenomics, and Open-Source Tools](https://datatalks.club/podcast/bioinformatics-worflows-tools-and-data-science.html "Listen to: Bioinformatics Workflows in Practice: Sequencing, Metagenomics, and Open-Source Tools")
with [Sebastian Ayala Ruano](https://datatalks.club/people/sebastianayalaruano.html)
* [Applying Computer Vision Research to Building Production-Ready AI Systems for Real-World Deployment](https://datatalks.club/podcast/from-computer-vision-research-to-autonomous-driving-ai.html "Listen to: Applying Computer Vision Research to Building Production-Ready AI Systems for Real-World Deployment")
with [Aishwarya Jadhav](https://datatalks.club/people/aishwaryajadhav.html)
* [Building Agentic AI Systems: Pragmatic Agent Engineering, Tooling, Retrieval & Evaluation](https://datatalks.club/podcast/building-agentic-ai-engineering-tooling-retrieval-evaluation.html "Listen to: Building Agentic AI Systems: Pragmatic Agent Engineering, Tooling, Retrieval & Evaluation")
with [Ranjitha Kulkarni](https://datatalks.club/people/ranjithakulkarni.html)
### Season #21
* [From Theme Parks to Tesla: Building Data Products Through Applied ML and Full-Stack Engineering](https://datatalks.club/podcast/theme-park-crowd-modeling-to-tesla-full-stack-data-engineering.html "Listen to: From Theme Parks to Tesla: Building Data Products Through Applied ML and Full-Stack Engineering")
with [Abouzar Abbaspour](https://datatalks.club/people/abouzarabbaspour.html)
* [From Classical Guitar to Production ML: Nonlinear Career Path Through Semiconductors, Yield Analytics & Community-Driven Learning](https://datatalks.club/podcast/from-semiconductor-data-to-applied-machine-learning.html "Listen to: From Classical Guitar to Production ML: Nonlinear Career Path Through Semiconductors, Yield Analytics & Community-Driven Learning")
with [Dashel Ruiz Perez](https://datatalks.club/people/dashelruizperez.html)
* [From Game AI to LLM Agents: 20-Year Evolution of Multi-Agent Systems, Evolutionary Algorithms & Modern AI Tooling](https://datatalks.club/podcast/from-game-ai-to-modern-ai-agents.html "Listen to: From Game AI to LLM Agents: 20-Year Evolution of Multi-Agent Systems, Evolutionary Algorithms & Modern AI Tooling")
with [Micheal Lanham](https://datatalks.club/people/micheallanham.html)
* [From Radio Astronomy to Applied ML: MEERKAT Data Pipelines, Multi-Wavelength Cross-Matching & Production-Grade ML Systems](https://datatalks.club/podcast/from-radio-astronomy-to-machine-learning-and-data-engineering.html "Listen to: From Radio Astronomy to Applied ML: MEERKAT Data Pipelines, Multi-Wavelength Cross-Matching & Production-Grade ML Systems")
with [Daniel Egbo](https://datatalks.club/people/danielegbo.html)
* [From Medicine to Machine Learning: Skill Stacking, Public Learning & Freelance-Driven Career Building](https://datatalks.club/podcast/nonlinear-path-to-machine-learning-freelancing-and-public-learning.html "Listen to: From Medicine to Machine Learning: Skill Stacking, Public Learning & Freelance-Driven Career Building")
with [Pastor Soto](https://datatalks.club/people/pastorsoto.html)
* [Mindful Data Strategy for Business Impact: Wabi-Sabi Approach, Data Trust & Maintenance-Innovation Balance](https://datatalks.club/podcast/mindful-data-strategy-for-business-impact.html "Listen to: Mindful Data Strategy for Business Impact: Wabi-Sabi Approach, Data Trust & Maintenance-Innovation Balance")
with [Lior Barak](https://datatalks.club/people/liorbarak.html)
* [From Academic Research to Lean Data Consulting: MVP Strategy, Problem-First Thinking & Freelance Practice Building](https://datatalks.club/podcast/from-academic-research-to-data-engineering-freelancing.html "Listen to: From Academic Research to Lean Data Consulting: MVP Strategy, Problem-First Thinking & Freelance Practice Building")
with [Orell Garten](https://datatalks.club/people/orellgarten.html)
### Season #20
* [Building a Sustainable Data Freelancing Career: Market Validation, Client Acquisition & Strategic Positioning](https://datatalks.club/podcast/data-freelancing-career-strategy-market-demand-and-client-acquisition.html "Listen to: Building a Sustainable Data Freelancing Career: Market Validation, Client Acquisition & Strategic Positioning")
with [Dimitri Visnadi](https://datatalks.club/people/dimitrivisnadi.html)
* [Developer Advocacy Through Community Impact: Technical Leadership, Open Source Mentorship & Demo-Driven Communication](https://datatalks.club/podcast/practical-devrel-demofirst-education-and-open-source.html "Listen to: Developer Advocacy Through Community Impact: Technical Leadership, Open Source Mentorship & Demo-Driven Communication")
with [Will Russell](https://datatalks.club/people/willrussell.html)
* [Applied LLM Research & Career Growth: Long-Context Evaluation, Prototyping & Industry Publishing](https://datatalks.club/podcast/applied-llm-research-and-career-growth-in-practice.html "Listen to: Applied LLM Research & Career Growth: Long-Context Evaluation, Prototyping & Industry Publishing")
with [Lavanya Gupta](https://datatalks.club/people/lavanyagupta.html)
* [FinOps for Data Engineers: Optimize Cloud Costs, BigQuery & Modern Data Stack](https://datatalks.club/podcast/finops-for-data-engineers.html "Listen to: FinOps for Data Engineers: Optimize Cloud Costs, BigQuery & Modern Data Stack")
with [Eddy Zulkifly](https://datatalks.club/people/eddyzulkifly.html)
* [Production AI Engineering: Data Pipelines, Prompt Optimization and Caching](https://datatalks.club/podcast/production-ready-ai-engineering.html "Listen to: Production AI Engineering: Data Pipelines, Prompt Optimization and Caching")
with [Bartosz Mikulski](https://datatalks.club/people/bartoszmikulski.html)
* [Lean MLOps for Startups: SaaS-First MVP Stack, Avoid Vendor Lock-In & Manage Tech Debt](https://datatalks.club/podcast/lean-mlops-for-startups.html "Listen to: Lean MLOps for Startups: SaaS-First MVP Stack, Avoid Vendor Lock-In & Manage Tech Debt")
with [Nemanja Radojkovic](https://datatalks.club/people/nemanjaradojkovic.html)
* [Modern Data Engineering: Iceberg, Delta Lake & AI-Powered Pipelines](https://datatalks.club/podcast/trends-in-modern-data-engineering.html "Listen to: Modern Data Engineering: Iceberg, Delta Lake & AI-Powered Pipelines")
with [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
* [From Kaggle Grandmaster to Production ML: Competition Rigor, System Design & Large-Scale Education](https://datatalks.club/podcast/kaggle-grandmaster-to-production-ml-and-education.html "Listen to: From Kaggle Grandmaster to Production ML: Competition Rigor, System Design & Large-Scale Education")
with [Alexander Guschin](https://datatalks.club/people/alexanderguschin.html)
* [Post-ChatGPT AI Infrastructure: Open Source Orchestration, On-Prem Economics & Distributed Training at Scale](https://datatalks.club/podcast/ai-infrastructure-hybrid-cloud-on-prem-distributed-training.html "Listen to: Post-ChatGPT AI Infrastructure: Open Source Orchestration, On-Prem Economics & Distributed Training at Scale")
with [Andrey Cheptsov](https://datatalks.club/people/andreycheptsov.html)
### Season #19
* [Fairness in AI/ML Engineering: Interpretability, Metrics and Sociotechnical Design](https://datatalks.club/podcast/fairness-in-ai-ml-engineering.html "Listen to: Fairness in AI/ML Engineering: Interpretability, Metrics and Sociotechnical Design")
with [Tamara Atanasoska](https://datatalks.club/people/tamaraatanasoska.html)
* [From DevOps to Data Engineering: Automation, Open Source & Volunteering for Career Transitions](https://datatalks.club/podcast/from-devops-to-data-engineering-automation-open-source-volunteering.html "Listen to: From DevOps to Data Engineering: Automation, Open Source & Volunteering for Career Transitions")
with [Agita Jaunzeme](https://datatalks.club/people/agitajaunzeme.html)
* [From Biology to ML: Build a Data Science Portfolio with Open-Source, Computer Vision & Transformers](https://datatalks.club/podcast/from-biology-to-machine-learning-data-science-portfolio-open-source-computer-vision-transformers.html "Listen to: From Biology to ML: Build a Data Science Portfolio with Open-Source, Computer Vision & Transformers")
with [Isabella Bicalho](https://datatalks.club/people/isabellabicalho.html)
* [Hardening Generative AI Chatbots: Prevent Prompt Injection, Data Exfiltration & Hallucinations](https://datatalks.club/podcast/generative-ai-chatbots-in-production-security.html "Listen to: Hardening Generative AI Chatbots: Prevent Prompt Injection, Data Exfiltration & Hallucinations")
with [Maria Sukhareva](https://datatalks.club/people/mariasukhareva.html)
* [From Collider Physics to Data Science: Research Software Engineering, Interview Prep & Mentorship](https://datatalks.club/podcast/from-large-hadron-collider-to-data-science-research-software-engineering.html "Listen to: From Collider Physics to Data Science: Research Software Engineering, Interview Prep & Mentorship")
with [Anastasia Karavdina](https://datatalks.club/people/anastasiakaravdina.html)
* [MLOps at Scale: CI/CD, Reproducibility, Model Monitoring & Adoption Strategies](https://datatalks.club/podcast/mlops-at-scale-reproducibility-adoption.html "Listen to: MLOps at Scale: CI/CD, Reproducibility, Model Monitoring & Adoption Strategies")
with [Raphaël Hoogvliets](https://datatalks.club/people/raphaelhoogvliets.html)
* [Inside Scaling DataTalks.Club: How We Built Free Data Engineering, MLOps & LLM Courses](https://datatalks.club/podcast/datatalksclub-scaling-and-free-courses.html "Listen to: Inside Scaling DataTalks.Club: How We Built Free Data Engineering, MLOps & LLM Courses")
with [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [Human-Centered Speech Recognition: ASR for Disordered Speech and Accents](https://datatalks.club/podcast/human-centered-ai-automatic-speech-recognition.html "Listen to: Human-Centered Speech Recognition: ASR for Disordered Speech and Accents")
with [Katarzyna Foremniak](https://datatalks.club/people/katarzynaforemniak.html)
* [Urban Data Science: Transport Analytics, Sensors and Liveable Cities](https://datatalks.club/podcast/urban-data-science.html "Listen to: Urban Data Science: Transport Analytics, Sensors and Liveable Cities")
with [Rachel Lim](https://datatalks.club/people/rachellim.html)
### Season #18
* [DataOps for Data Engineering: Automation, Observability, CI/CD & Reliable ML Deployments](https://datatalks.club/podcast/dataops-for-data-engineering.html "Listen to: DataOps for Data Engineering: Automation, Observability, CI/CD & Reliable ML Deployments")
with [Christopher Bergh](https://datatalks.club/people/christopherbergh.html)
* [Building a Domestic Risk Assessment Tool: Data Cleaning, Risk Scoring Models and Privacy Compliance](https://datatalks.club/podcast/building-domestic-risk-assessment-tool.html "Listen to: Building a Domestic Risk Assessment Tool: Data Cleaning, Risk Scoring Models and Privacy Compliance")
with [Sabina Firtala](https://datatalks.club/people/sabinafirtala.html)
* [Community Building and Teaching in AI & Tech: Project-to-Course Model for AI Education](https://datatalks.club/podcast/community-building-and-teaching-in-ai-tech.html "Listen to: Community Building and Teaching in AI & Tech: Project-to-Course Model for AI Education")
with [Erum Afzal](https://datatalks.club/people/erumafzal.html)
* [Open Source ML Tools: Scikit-Learn Governance, Sustainability and Business Models](https://datatalks.club/podcast/open-source-ml-tools-strategy-and-business-models.html "Listen to: Open Source ML Tools: Scikit-Learn Governance, Sustainability and Business Models")
with [Vincent Warmerdam](https://datatalks.club/people/vincentwarmerdam.html)
* [AI for Ecology, Biodiversity, and Conservation: Computer Vision, Remote Sensing and Citizen Science](https://datatalks.club/podcast/ai-for-ecology-biodiversity-and-conservation.html "Listen to: AI for Ecology, Biodiversity, and Conservation: Computer Vision, Remote Sensing and Citizen Science")
with [Tanya Berger-Wolf](https://datatalks.club/people/tanyabergerwolf.html)
* [Using Knowledge Graphs & LLMs for Automotive R&D: RAG, Graph ML & Crash Simulation](https://datatalks.club/podcast/knowledge-graphs-and-llms-for-automotive-rnd.html "Listen to: Using Knowledge Graphs & LLMs for Automotive R&D: RAG, Graph ML & Crash Simulation")
with [Anahita Pakiman](https://datatalks.club/people/anahitapakiman.html)
* [Data Leadership Coaching: Transition to Manager, Stakeholder Skills and Team Impact](https://datatalks.club/podcast/data-leadership-coaching.html "Listen to: Data Leadership Coaching: Transition to Manager, Stakeholder Skills and Team Impact")
with [Tereza Iofciu](https://datatalks.club/people/terezaiofciu.html)
### Season #17
* [Building Search Systems: Dense Embeddings, MLOps and Evaluation Metrics](https://datatalks.club/podcast/building-production-search-systems.html "Listen to: Building Search Systems: Dense Embeddings, MLOps and Evaluation Metrics")
with [Daniel Svonava](https://datatalks.club/people/danielsvonava.html)
* [Production ML Search: Embeddings, Hybrid Architectures and Scalable Indexing](https://datatalks.club/podcast/production-ml-search-vector-search-embeddings-hybrid%20search.html "Listen to: Production ML Search: Embeddings, Hybrid Architectures and Scalable Indexing")
with [Reem Mahmoud](https://datatalks.club/people/reemmahmoud.html)
* [Open Source and Volunteering: Building AI Projects and Career Momentum](https://datatalks.club/podcast/open-source-and-volunteering-in-ai-for-data-ml-career-growth.html "Listen to: Open Source and Volunteering: Building AI Projects and Career Momentum")
with [Sara EL-ATEIF](https://datatalks.club/people/saraelateif.html)
* [Tech Job Search Strategy: Portfolio Projects, Resume Tips and Networking](https://datatalks.club/podcast/job-search-strategy-in-tech-projects-skills-cv-networking.html "Listen to: Tech Job Search Strategy: Portfolio Projects, Resume Tips and Networking")
with [Sarah Mestiri](https://datatalks.club/people/sarahmestiri.html)
* [MLOps in Finance: Regulated Deployment, CI/CD and Model Governance](https://datatalks.club/podcast/mlops-and-ml-engineering-in-finance.html "Listen to: MLOps in Finance: Regulated Deployment, CI/CD and Model Governance")
with [Nemanja Radojkovic](https://datatalks.club/people/nemanjaradojkovic.html)
* [Bayesian Modeling: PyMC, Stan and Probabilistic Programming Workflows](https://datatalks.club/podcast/bayesian-modeling-workflows-and-tools.html "Listen to: Bayesian Modeling: PyMC, Stan and Probabilistic Programming Workflows")
with [Rob Zinkov](https://datatalks.club/people/robzinkov.html)
* [Algorithmic Trading with Python: Backtesting, Risk Management and Deployment](https://datatalks.club/podcast/algorithmic-trading-with-python-and-machine-learning.html "Listen to: Algorithmic Trading with Python: Backtesting, Risk Management and Deployment")
with [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html)
* [Modern Search Systems: Vector Databases, LLMs and Semantic Retrieval](https://datatalks.club/podcast/modern-search-systems-vector-databases-llms-semantic-retrieval.html "Listen to: Modern Search Systems: Vector Databases, LLMs and Semantic Retrieval")
with [Atita Arora](https://datatalks.club/people/atitaarora.html)
* [From Data Freelancer to Startup: Open-Source Products and Bottom-Up Adoption](https://datatalks.club/podcast/from-data-freelancer-to-startup-open-source-products.html "Listen to: From Data Freelancer to Startup: Open-Source Products and Bottom-Up Adoption")
with [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
### Season #16
* [Becoming a Data Freelancer: Pricing, Client Acquisition and Contract Strategy](https://datatalks.club/podcast/becoming-data-freelancer.html "Listen to: Becoming a Data Freelancer: Pricing, Client Acquisition and Contract Strategy")
with [Dimitri Visnadi](https://datatalks.club/people/dimitrivisnadi.html)
* [Building Digital Health Startups: MVP Strategy, AI Diagnosis and Telemedicine](https://datatalks.club/podcast/building-ai-digital-health-startups.html "Listen to: Building Digital Health Startups: MVP Strategy, AI Diagnosis and Telemedicine")
with [Maria Bruckert](https://datatalks.club/people/mariabruckert.html)
* [Interpretable Machine Learning: SHAP, Conformal Prediction and Model Trust](https://datatalks.club/podcast/interpretable-machine-learning.html "Listen to: Interpretable Machine Learning: SHAP, Conformal Prediction and Model Trust")
with [Christoph Molnar](https://datatalks.club/people/christophmolnar.html)
* [From Software Engineer to VP of Machine Learning: Stakeholder Buy-In, Rapid POCs and Full-Stack Skills](https://datatalks.club/podcast/from-software-engineering-to-vp-of-machine-learning-applied-ml-leadership.html "Listen to: From Software Engineer to VP of Machine Learning: Stakeholder Buy-In, Rapid POCs and Full-Stack Skills")
with [Jack Blandin](https://datatalks.club/people/jackblandin.html)
* [Launching a Freelance Generative AI Business: NLP Services and Client Acquisition](https://datatalks.club/podcast/practical-generative-ai-consulting-from-expertise-to-impact.html "Listen to: Launching a Freelance Generative AI Business: NLP Services and Client Acquisition")
with [Verena Weber](https://datatalks.club/people/verenaweber.html)
* [Building Data Products at Scale: Intake, A/B Testing, and MLOps in a Marketing Organization](https://datatalks.club/podcast/building-data-products-lead-data-scientist.html "Listen to: Building Data Products at Scale: Intake, A/B Testing, and MLOps in a Marketing Organization")
with [Ioannis Mesionis](https://datatalks.club/people/ioannismesionis.html)
* [Building Healthcare ML Systems: From Sepsis Prediction to Low-Resource Clinical Deployment](https://datatalks.club/podcast/building-healthcare-machine-learning-systems.html "Listen to: Building Healthcare ML Systems: From Sepsis Prediction to Low-Resource Clinical Deployment")
with [Eleni Stamatelou](https://datatalks.club/people/elenistamatelou.html)
* [Building a Sustainable Data Community: 3 Years of DataTalks.Club Growth and Evolution](https://datatalks.club/podcast/datatalksclub-building-sustainable-data-community-3-years-anniversary.html "Listen to: Building a Sustainable Data Community: 3 Years of DataTalks.Club Growth and Evolution")
with [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
, [Johanna Bayer](https://datatalks.club/people/johannabayer.html)
### Season #15
* [Build and Scale Data Engineering Systems for Fraud Detection: Feature Pipelines, Real-Time Inference, Graph Databases & Production Debugging](https://datatalks.club/podcast/building-and-scaling-data-engineering-systems-for-fraud-detection.html "Listen to: Build and Scale Data Engineering Systems for Fraud Detection: Feature Pipelines, Real-Time Inference, Graph Databases & Production Debugging")
with [Angela Ramirez](https://datatalks.club/people/angelaramirez.html)
* [From Hands-On IoT Data Engineering to Leading Data Architecture: Pipelines, Cloud Adaptation & Analytics Modeling](https://datatalks.club/podcast/from-iot-data-engineering-to-leading-data-architect.html "Listen to: From Hands-On IoT Data Engineering to Leading Data Architecture: Pipelines, Cloud Adaptation & Analytics Modeling")
with [Loïc Magnien](https://datatalks.club/people/loicmagnien.html)
* [Pragmatic MLOps: Build Standardized CI/CD, Model Registries, Monitoring & Org Best Practices](https://datatalks.club/podcast/pragmatic-and-standardized-mlops.html "Listen to: Pragmatic MLOps: Build Standardized CI/CD, Model Registries, Monitoring & Org Best Practices")
with [Maria Vechtomova](https://datatalks.club/people/mariavechtomova.html)
* [Causal Inference for Real-World ML: Uplift Modeling, Counterfactuals, Treatment Effects & LLM Integration](https://datatalks.club/podcast/causal-inference-for-machine-learning.html "Listen to: Causal Inference for Real-World ML: Uplift Modeling, Counterfactuals, Treatment Effects & LLM Integration")
with [Aleksander Molak](https://datatalks.club/people/aleksandermolak.html)
* [Remote Data Engineering Life: Building IoT Platforms, Career Transitions & Newsletter-Driven Personal Growth](https://datatalks.club/podcast/remote-data-engineering-work-and-building-iot-platforms.html "Listen to: Remote Data Engineering Life: Building IoT Platforms, Career Transitions & Newsletter-Driven Personal Growth")
with [José María Sánchez Salas](https://datatalks.club/people/josemaria.html)
* [LLM Value Creation: GPT Communities, Business Use Cases & Human-in-the-Loop AI Applications](https://datatalks.club/podcast/practical-llm-use-cases-and-product-patterns.html "Listen to: LLM Value Creation: GPT Communities, Business Use Cases & Human-in-the-Loop AI Applications")
with [Sandra Kublik](https://datatalks.club/people/sandrakublik.html)
* [Deploying LLMs in Production: Fine-Tuning, Retrieval & Open-Source vs API Tradeoffs](https://datatalks.club/podcast/deploying-llms-in-production-fine-tuning-retrieval-open-source-api.html "Listen to: Deploying LLMs in Production: Fine-Tuning, Retrieval & Open-Source vs API Tradeoffs")
with [Meryem Arik](https://datatalks.club/people/meryemarik.html)
* [Early-Stage Investing in Open Source Developer Tools: Deal Sourcing, Due Diligence & Commercialization Models](https://datatalks.club/podcast/investing-in-open-source-developer-tools.html "Listen to: Early-Stage Investing in Open Source Developer Tools: Deal Sourcing, Due Diligence & Commercialization Models")
with [Bela Wiertz](https://datatalks.club/people/belawiertz.html)
* [ML System Design Playbook: Fail-Fast Design Docs, Modular Architecture & Data Drift Monitoring](https://datatalks.club/podcast/ml-system-design.html "Listen to: ML System Design Playbook: Fail-Fast Design Docs, Modular Architecture & Data Drift Monitoring")
with [Valerii Babushkin](https://datatalks.club/people/valeriybabushkin.html)
### Season #14
* [Build Explainable and Actionable AI/ML Systems: Industrial PhD, Trust Theory & Production Deployment](https://datatalks.club/podcast/building-explainable-and-actionable-ai-ml-systems.html "Listen to: Build Explainable and Actionable AI/ML Systems: Industrial PhD, Trust Theory & Production Deployment")
with [Polina Mosolova](https://datatalks.club/people/polinamosolova.html)
* [Building Production ML Platforms: Infrastructure, Workflows, Teams & Governance That Scale](https://datatalks.club/podcast/building-production-ml-platform-and-mlops-team.html "Listen to: Building Production ML Platforms: Infrastructure, Workflows, Teams & Governance That Scale")
with [Simon Stiebellehner](https://datatalks.club/people/simonstiebellehner.html)
* [Modern Data Pipeline Architecture: Ingestion, Orchestration, Transformation & MLOps Systems](https://datatalks.club/podcast/modern-data-pipelines-orchestration-ingestion-modeling.html "Listen to: Modern Data Pipeline Architecture: Ingestion, Orchestration, Transformation & MLOps Systems")
with [Santona Tuli](https://datatalks.club/people/santonatuli.html)
* [DevRel Role for Machine Learning: ML Ecosystems, Open-Source Governance & Developer Experience with Metaflow](https://datatalks.club/podcast/devrel-open-source-machine-learning.html "Listen to: DevRel Role for Machine Learning: ML Ecosystems, Open-Source Governance & Developer Experience with Metaflow")
with [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html)
* [Freelance Data Scientist Playbook: MLOps, Model Monitoring, Upwork & Startup Skills](https://datatalks.club/podcast/from-startup-engineering-to-freelance-data-science.html "Listen to: Freelance Data Scientist Playbook: MLOps, Model Monitoring, Upwork & Startup Skills")
with [Antonis Stellas](https://datatalks.club/people/antonisstellas.html)
* [Data Governance & Data Access Management: Access Controls, Data Catalogs & Access-as-Code](https://datatalks.club/podcast/data-governance-data-access-management.html "Listen to: Data Governance & Data Access Management: Access Controls, Data Catalogs & Access-as-Code")
with [Bart Vandekerckhove](https://datatalks.club/people/bartvandekerckhove.html)
* [Actionable Data Strategy & DataOps for AI-Powered Products: Pitch, Measure, Use GPT](https://datatalks.club/podcast/data-strategy-and-dataops-for-ai-powered-products.html "Listen to: Actionable Data Strategy & DataOps for AI-Powered Products: Pitch, Measure, Use GPT")
with [Boyan Angelov](https://datatalks.club/people/boyanangelov.html)
* [Data Privacy Playbook: Differential Privacy, Federated Learning, PETs & Consent UX](https://datatalks.club/podcast/data-privacy-engineering-gdpr-machine-learning.html "Listen to: Data Privacy Playbook: Differential Privacy, Federated Learning, PETs & Consent UX")
with [Katharine Jarmul](https://datatalks.club/people/katharinejarmul.html)
* [Build Scalable, Reliable ML Systems (MLOps): Design Docs, Data Strategy & Edge Constraints](https://datatalks.club/podcast/building-scalable-and-reliable-machine-learning-systems.html "Listen to: Build Scalable, Reliable ML Systems (MLOps): Design Docs, Data Strategy & Edge Constraints")
with [Arseny Kravchenko](https://datatalks.club/people/arsenykravchenko.html)
### Season #13
* [Build Open-Source NLP Tools: Weak Supervision, LLM Heuristics & Enterprise ML Product Strategy](https://datatalks.club/podcast/building-open-source-nlp-tool.html "Listen to: Build Open-Source NLP Tools: Weak Supervision, LLM Heuristics & Enterprise ML Product Strategy")
with [Johannes Hötter](https://datatalks.club/people/johanneshotter.html)
* [Master Industrial Data: Synthetic Tabular Data, Small-Data Modeling, Sensors & MLOps](https://datatalks.club/podcast/industrial-data-small-data-production-machine-learning.html "Listen to: Master Industrial Data: Synthetic Tabular Data, Small-Data Modeling, Sensors & MLOps")
with [Rosona Eldred](https://datatalks.club/people/rosonaeldred.html)
* [How to Teach Yourself Bioinformatics & ML: Project-First Learning, Resources, and MLOps](https://datatalks.club/podcast/learning-machine-learning-self-taught-bioinformatics.html "Listen to: How to Teach Yourself Bioinformatics & ML: Project-First Learning, Resources, and MLOps")
with [Aaisha Muhammad](https://datatalks.club/people/aaishamuhammad.html)
* [Master Data Science Management: Agile ML, Debrief Culture, Metrics & Scale to Production](https://datatalks.club/podcast/data-science-management-and-agile-machine-learning.html "Listen to: Master Data Science Management: Agile ML, Debrief Culture, Metrics & Scale to Production")
with [Shir Meir Lador](https://datatalks.club/people/shirmeirlador.html)
* [Software Engineering for ML: Prevent Hidden Technical Debt with MLOps, Documentation & Team Alignment](https://datatalks.club/podcast/software-engineering-for-machine-learning.html "Listen to: Software Engineering for ML: Prevent Hidden Technical Debt with MLOps, Documentation & Team Alignment")
with [Nadia Nahar](https://datatalks.club/people/nadianahar.html)
* [Build a Data Consulting Business: Customer Validation, User Interviews & Pricing Strategy](https://datatalks.club/podcast/data-consulting-business-pricing-and-client-acquisition.html "Listen to: Build a Data Consulting Business: Customer Validation, User Interviews & Pricing Strategy")
with [Aleksander Kruszelnicki](https://datatalks.club/people/aleksanderkruszelnicki.html)
* [Actionable Biohacks to Boost Productivity: Sleep, Circadian Light, Dopamine & Habits](https://datatalks.club/podcast/biohacking-productivity-for-data-scientists-and-ml-engineers.html "Listen to: Actionable Biohacks to Boost Productivity: Sleep, Circadian Light, Dopamine & Habits")
with [Ruslan Shchuchkin](https://datatalks.club/people/ruslanshchuchkin.html)
* [Analytics for Nonprofits: Build Data Maturity, Teams, Tools & Optimization Strategies](https://datatalks.club/podcast/data-science-and-analytics-for-nonprofits-tech-for-good.html "Listen to: Analytics for Nonprofits: Build Data Maturity, Teams, Tools & Optimization Strategies")
with [Parvathy Krishnan](https://datatalks.club/people/parvathykrishnan.html)
* [How to Build & Scale a Data Science Community: Diversity, ML Deployment & Career Growth](https://datatalks.club/podcast/building-ml-communities-diversity-and-career-growth.html "Listen to: How to Build & Scale a Data Science Community: Diversity, ML Deployment & Career Growth")
with [Dânia Meira](https://datatalks.club/people/daniameira.html)
### Season #12
* [Transitioning from Academia to Industry as a Staff AI Engineer: Interview Prep, MLOps & Onboarding](https://datatalks.club/podcast/from-academia-to-staff-ai-engineer-interviews-and-career-growth.html "Listen to: Transitioning from Academia to Industry as a Staff AI Engineer: Interview Prep, MLOps & Onboarding")
with [Tatiana Gabruseva](https://datatalks.club/people/tatianagabruseva.html)
* [How to Grow Your ML Engineering Career: Platform Work, LLM Workflows & Debugging Skills](https://datatalks.club/podcast/how-to-grow-your-ml-engineering-career.html "Listen to: How to Grow Your ML Engineering Career: Platform Work, LLM Workflows & Debugging Skills")
with [Krzysztof Szafanek](https://datatalks.club/people/krzysztofszafanek.html)
* [Master Machine Learning & Data Science Interviews: Recruiter-Proven Stages, Prep & Resources](https://datatalks.club/podcast/machine-learning-data-science-interview-prep.html "Listen to: Master Machine Learning & Data Science Interviews: Recruiter-Proven Stages, Prep & Resources")
with [Luke Whipps](https://datatalks.club/people/lukewhipps.html)
* [Indie Hacking and Bootstrapping Side Projects for Data Scientists: Build, Launch & Monetize Indie Hacker Products](https://datatalks.club/podcast/data-scientist-and-indie-hacker-bootstrapping-side-projects.html "Listen to: Indie Hacking and Bootstrapping Side Projects for Data Scientists: Build, Launch & Monetize Indie Hacker Products")
with [Pauline Clavelloux](https://datatalks.club/people/paulineclavelloux.html)
* [Teaching Open Science & Reproducible Research: Research Software Engineering Practices for Academia](https://datatalks.club/podcast/teaching-reproducible-research-and-open-science-coding-practices-for-academia.html "Listen to: Teaching Open Science & Reproducible Research: Research Software Engineering Practices for Academia")
with [Johanna Bayer](https://datatalks.club/people/johannabayer.html)
* [Data-Centric AI: Improve Label Quality & Edit Datasets to Boost Model Performance](https://datatalks.club/podcast/data-centric-ai.html "Listen to: Data-Centric AI: Improve Label Quality & Edit Datasets to Boost Model Performance")
with [Marysia Winkels](https://datatalks.club/people/marysiawinkels.html)
* [Practical Skills for Data Professionals in SaaS: Bridging the Gap between Data and Business](https://datatalks.club/podcast/data-professionals-business-skills-in-saas.html "Listen to: Practical Skills for Data Professionals in SaaS: Bridging the Gap between Data and Business")
with [Loris Marini](https://datatalks.club/people/lorismarini.html)
* [Transitioning from Software Engineer to Data Science Manager: Search, ML & Leadership](https://datatalks.club/podcast/from-software-engineering-to-leading-data-science-teams.html "Listen to: Transitioning from Software Engineer to Data Science Manager: Search, ML & Leadership")
with [Sadat Anwar](https://datatalks.club/people/sadatanwar.html)
### Season #11
* [Designing FinTech Data Analytics Curriculum: Fraud Detection, BigQuery Labs & Mentoring](https://datatalks.club/podcast/teaching-mentoring-data-analytics-fintech.html "Listen to: Designing FinTech Data Analytics Curriculum: Fraud Detection, BigQuery Labs & Mentoring")
with [Irina Brudaru](https://datatalks.club/people/irinabrudaru.html)
* [Practical Data Journalism: Sourcing, Storytelling, Visualization & Tools (Python, Tableau)](https://datatalks.club/podcast/data-journalism-python-visualization-storytelling.html "Listen to: Practical Data Journalism: Sourcing, Storytelling, Visualization & Tools (Python, Tableau)")
with [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html)
* [Marketing to Analytics Engineering: DBT, SQL, Data Modeling & Career Playbook](https://datatalks.club/podcast/from-marketing-to-analytics-engineering-sql-dbt-career-switch.html "Listen to: Marketing to Analytics Engineering: DBT, SQL, Data Modeling & Career Playbook")
with [Nikola Maksimovic](https://datatalks.club/people/nikolamaksimovic.html)
* [Building Data Products at Scale: Recommenders, Domain Ownership, and Hiring for Production ML](https://datatalks.club/podcast/building-data-products-product-owner-vs-product-manager.html "Listen to: Building Data Products at Scale: Recommenders, Domain Ownership, and Hiring for Production ML")
with [Anna Hannemann](https://datatalks.club/people/annahannemann.html)
* [Building and Scaling Data Science Practice in Industrial Enterprises: AI Adoption, MLOps Maturity & Career Growth](https://datatalks.club/podcast/building-and-scaling-data-science-practice-industrial-ai-mlops.html "Listen to: Building and Scaling Data Science Practice in Industrial Enterprises: AI Adoption, MLOps Maturity & Career Growth")
with [Andrey Shtylenko](https://datatalks.club/people/andreyshtylenko.html)
* [Building an Open-Source ML-Powered Identity Resolution Tool in the Modern Data Stack](https://datatalks.club/podcast/building-open-source-data-product-for-identity-resolution.html "Listen to: Building an Open-Source ML-Powered Identity Resolution Tool in the Modern Data Stack")
with [Sonal Goyal](https://datatalks.club/people/sonalgoyal.html)
* [DataOps & GitOps for Data Teams: Onboarding, IaC, Reproducibility & Production Best Practices](https://datatalks.club/podcast/dataops-and-gitops-best-practices-for-data-teams.html "Listen to: DataOps & GitOps for Data Teams: Onboarding, IaC, Reproducibility & Production Best Practices")
with [Tomasz Hinc](https://datatalks.club/people/tomaszhinc.html)
* [How to Hire, Manage, and Grow a Data Science Team in B2B SaaS](https://datatalks.club/podcast/hiring-and-managing-data-science-teams-in-b2b-saas.html "Listen to: How to Hire, Manage, and Grow a Data Science Team in B2B SaaS")
with [Katie Bauer](https://datatalks.club/people/katiebauer.html)
* [Transition from QA to Machine Learning & Data Engineering: Projects, Cloud & Interview Prep](https://datatalks.club/podcast/how-to-transition-into-ml-and-data-engineering-from-qa.html "Listen to: Transition from QA to Machine Learning & Data Engineering: Projects, Cloud & Interview Prep")
with [Alvaro Navas Peire](https://datatalks.club/people/alvaronavaspeire.html)
### Season #10
* [Responsible & Explainable AI: Practical Guide to Bias Detection, Fairness & Governance](https://datatalks.club/podcast/responsible-explainable-ai-bias-detection.html "Listen to: Responsible & Explainable AI: Practical Guide to Bias Detection, Fairness & Governance")
with [Supreet Kaur](https://datatalks.club/people/supreetkaur.html)
* [Build Data Science Programs, Democratize HPC & Scale Graph Analytics with Arkouda](https://datatalks.club/podcast/building-data-science-programs-and-democratizing-high-performance-computing.html "Listen to: Build Data Science Programs, Democratize HPC & Scale Graph Analytics with Arkouda")
with [David Bader](https://datatalks.club/people/davidbader.html)
* [Practical Guide to Dataset Creation & Annotation for NLP: Active Learning, Weak Supervision, Tools](https://datatalks.club/podcast/nlp-dataset-creation-annotation-tools-workflows.html "Listen to: Practical Guide to Dataset Creation & Annotation for NLP: Active Learning, Weak Supervision, Tools")
with [Christiaan Swart](https://datatalks.club/people/christiannswart.html)
* [Data Mesh Implementation: Build Decentralized Data Products, Contracts & Federated Governance](https://datatalks.club/podcast/data-mesh-architecture-decentralized-data-products.html "Listen to: Data Mesh Implementation: Build Decentralized Data Products, Contracts & Federated Governance")
with [Zhamak Dehghani](https://datatalks.club/people/zhamakdehghani.html)
* [Scale Data Engineering Teams: Build Self-Service Data Platforms, Hire Senior Engineers & Use Kafka](https://datatalks.club/podcast/scaling-data-engineering-teams-self-service-platforms.html "Listen to: Scale Data Engineering Teams: Build Self-Service Data Platforms, Hire Senior Engineers & Use Kafka")
with [Mehdi OUAZZA](https://datatalks.club/people/mehdiouazza.html)
* [Scale Enterprise AI: Data-First Strategies, MLOps Best Practices & Realistic Experiments](https://datatalks.club/podcast/scaling-enterprise-ai-mlops-data-first-strategy.html "Listen to: Scale Enterprise AI: Data-First Strategies, MLOps Best Practices & Realistic Experiments")
with [Alexander Hendorf](https://datatalks.club/people/alexanderhendorf.html)
* [MLOps Architect Guide: Production Model Monitoring, Data Observability & Tooling](https://datatalks.club/podcast/mlops-model-monitoring-data-observability.html "Listen to: MLOps Architect Guide: Production Model Monitoring, Data Observability & Tooling")
with [Danny Leybzon](https://datatalks.club/people/dannyleybzon.html)
* [Data Science Jobs: How to Spot Misleading Job Titles, Hiring Red Flags & Build Better Data Teams](https://datatalks.club/podcast/data-science-job-red-flags-and-mismatched-roles.html "Listen to: Data Science Jobs: How to Spot Misleading Job Titles, Hiring Red Flags & Build Better Data Teams")
with [Tereza Iofciu](https://datatalks.club/people/terezaiofciu.html)
* [Data Science for Public Policy — Ethical AI, Climate Justice & Impact Projects](https://datatalks.club/podcast/data-science-for-public-policy-ethical-ai-social-impact.html "Listen to: Data Science for Public Policy — Ethical AI, Climate Justice & Impact Projects")
with [Christine Cepelak](https://datatalks.club/people/christinecepelak.html)
### Season #9
* [How to Hire Data Scientists: Interview Questions, MLOps, AutoML Limits & Inclusive Hiring](https://datatalks.club/podcast/hiring-for-data-science-jobs-interview-questions-skills.md.html "Listen to: How to Hire Data Scientists: Interview Questions, MLOps, AutoML Limits & Inclusive Hiring")
with [Olga Ivina](https://datatalks.club/people/olgaivina.html)
* [From Developer to Startup Founder: Building a Career Through Open Source](https://datatalks.club/podcast/open-source-turned-into-career-and-startup-creation.html "Listen to: From Developer to Startup Founder: Building a Career Through Open Source")
with [Will McGugan](https://datatalks.club/people/willmcgugan.html)
* [Designing High-Impact Data Science Teams: Centralized vs Embedded Models, Experimentation & Staffing](https://datatalks.club/podcast/data-science-team-structure-and-org-design.html "Listen to: Designing High-Impact Data Science Teams: Centralized vs Embedded Models, Experimentation & Staffing")
with [Lisa Cohen](https://datatalks.club/people/lisacohen.html)
* [Contribute to Hugging Face & Build an NLP Portfolio: Open Source, Datasets, Spaces](https://datatalks.club/podcast/hugging-face-contributions-and-nlp-portfolio.html "Listen to: Contribute to Hugging Face & Build an NLP Portfolio: Open Source, Datasets, Spaces")
with [Merve Noyan](https://datatalks.club/people/mervenoyan.html)
* [Data Science Career Playbook: Job Hunt, Portfolios, DALL·E 2 & Overcoming FOMO](https://datatalks.club/podcast/how-to-break-into-data-science.html "Listen to: Data Science Career Playbook: Job Hunt, Portfolios, DALL·E 2 & Overcoming FOMO")
with [Mısra Turp](https://datatalks.club/people/misraturp.html)
* [Freelance Data Engineering Playbook: Pricing, Client Acquisition & Tools](https://datatalks.club/podcast/freelance-data-engineering-pricing-and-clients.html "Listen to: Freelance Data Engineering Playbook: Pricing, Client Acquisition & Tools")
with [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html)
* [Data Engineering Job Prep & Interview Guide: Python, SQL, Portfolio & Job Search Tips](https://datatalks.club/podcast/get-data-engineering-job-prep-and-interview.html "Listen to: Data Engineering Job Prep & Interview Guide: Python, SQL, Portfolio & Job Search Tips")
with [Jeff Katz](https://datatalks.club/people/jeffkatz.html)
* [Asteroid Mining: Using ML & Hyperspectral Spectroscopy to Detect Water for ISRU](https://datatalks.club/podcast/machine-learning-for-asteroid-mining-and-water-detection.html "Listen to: Asteroid Mining: Using ML & Hyperspectral Spectroscopy to Detect Water for ISRU")
with [Daynan Crull](https://datatalks.club/people/daynancrull.html)
* [Marketing Data Science: Attribution, Media Mix Modeling, Uplift & Cookieless Tracking](https://datatalks.club/podcast/machine-learning-in-marketing-attribution-marketing-mix-modeling.html "Listen to: Marketing Data Science: Attribution, Media Mix Modeling, Uplift & Cookieless Tracking")
with [Juan Orduz](https://datatalks.club/people/juanorduz.html)
### Season #8
* [How I Landed a Data Engineering Job: Bootcamp, Docker, Airflow, AWS & Interview Tips](https://datatalks.club/podcast/get-data-analytics-and-data-engineering-job.html "Listen to: How I Landed a Data Engineering Job: Bootcamp, Docker, Airflow, AWS & Interview Tips")
with [Gloria Quiceno](https://datatalks.club/people/gloriaquiceno.html)
* [Build a Data Engineering Career: Bootcamp Curriculum, SQL Mastery & Interview Prep](https://datatalks.club/podcast/data-engineering-career-path-and-skills.html "Listen to: Build a Data Engineering Career: Bootcamp Curriculum, SQL Mastery & Interview Prep")
with [Jeff Katz](https://datatalks.club/people/jeffkatz.html)
* [How to Switch to Tech: Community Meetups, Open Source Fellowships & Landing an Ecosia Internship](https://datatalks.club/podcast/how-to-switch-to-ml-tech-without-experience.html "Listen to: How to Switch to Tech: Community Meetups, Open Source Fellowships & Landing an Ecosia Internship")
with [Jessica Greene](https://datatalks.club/people/jessicagreene.html)
* [Hiring Data Engineers in Europe: Nicolas Rassam on Interviews, Skills & Career Switches](https://datatalks.club/podcast/hiring-for-data-engineering-jobs-in-europe.html "Listen to: Hiring Data Engineers in Europe: Nicolas Rassam on Interviews, Skills & Career Switches")
with [Nicolas Rassam](https://datatalks.club/people/nicolasrassam.html)
* [Mastering DataOps: Automation, Observability & CI/CD for Reliable Data Pipelines](https://datatalks.club/podcast/dataops-automation-and-reliable-data-pipelines.html "Listen to: Mastering DataOps: Automation, Observability & CI/CD for Reliable Data Pipelines")
with [Christopher Bergh](https://datatalks.club/people/christopherbergh.html)
* [AI in Healthcare & Digital Therapeutics: Building Data Teams, Personalization, A/B Testing & Ethics](https://datatalks.club/podcast/ai-in-healthcare-and-digital-therapeutics.html "Listen to: AI in Healthcare & Digital Therapeutics: Building Data Teams, Personalization, A/B Testing & Ethics")
with [Stefan Gudmundsson](https://datatalks.club/people/stefangudmundsson.html)
* [AI Product Design: Algorithm-Ready UX, Rapid Experiments & Data-Driven Roadmaps](https://datatalks.club/podcast/ai-ml-product-design-and-experimentation.html "Listen to: AI Product Design: Algorithm-Ready UX, Rapid Experiments & Data-Driven Roadmaps")
with [Liesbeth Dingemans](https://datatalks.club/people/liesbethdingemans.html)
* [Data Science Career Playbook: Build Unique IoT Portfolios, Explainable AI, OSINT & LinkedIn Growth](https://datatalks.club/podcast/how-to-stand-out-in-data-science.html "Listen to: Data Science Career Playbook: Build Unique IoT Portfolios, Explainable AI, OSINT & LinkedIn Growth")
with [Marijn Markus](https://datatalks.club/people/marijnmarkus.html)
* [Using Visualizations to Explain Machine Learning: Build Intuition with kDimensions, Figma & Templates](https://datatalks.club/podcast/visualizing-machine-learning-concepts-to-explain-ml.html "Listen to: Using Visualizations to Explain Machine Learning: Build Intuition with kDimensions, Figma & Templates")
with [Meor Amer](https://datatalks.club/people/meoramer.html)
### Season #7
* [How to Break into Data Analytics: Networking, Portfolio, SQL & Interview Prep](https://datatalks.club/podcast/from-math-graduate-to-data-analytics.html "Listen to: How to Break into Data Analytics: Networking, Portfolio, SQL & Interview Prep")
with [Juan Pablo](https://datatalks.club/people/juanpablo.html)
* [How to Become a Data Engineer: Skills, MLOps, Pipelines, SQL, CI/CD & Cloud](https://datatalks.club/podcast/from-software-engineering-data-science-to-data-engineering-leadership.html "Listen to: How to Become a Data Engineer: Skills, MLOps, Pipelines, SQL, CI/CD & Cloud")
with [Ellen König](https://datatalks.club/people/ellenkonig.html)
* [Data Engineering Leadership: Scale ETL to ELT, Build Robust Data Platforms & Teams](https://datatalks.club/podcast/data-engineering-leadership-and-modern-data-platforms.html "Listen to: Data Engineering Leadership: Scale ETL to ELT, Build Robust Data Platforms & Teams")
with [Rahul Jain](https://datatalks.club/people/16rahuljain.html)
* [Product Analytics & A/B Testing: Causality, Metrics, Power Analysis, A/A Tests](https://datatalks.club/podcast/ab-testing-and-product-experimentation.html "Listen to: Product Analytics & A/B Testing: Causality, Metrics, Power Analysis, A/A Tests")
with [Jakob Graff](https://datatalks.club/people/jakobgraff.html)
* [ML System Design Interviews: Production ML, Fraud Detection, Features, A/B Testing & MLOps](https://datatalks.club/podcast/machine-learning-system-design-interview.html "Listen to: ML System Design Interviews: Production ML, Fraud Detection, Features, A/B Testing & MLOps")
with [Valerii Babushkin](https://datatalks.club/people/valeriybabushkin.html)
* [Land Junior Data Jobs: CVs, Interviews, Transferable Skills & Overcome Imposter Syndrome](https://datatalks.club/podcast/get-junior-data-job-and-transferable-skills.html "Listen to: Land Junior Data Jobs: CVs, Interviews, Transferable Skills & Overcome Imposter Syndrome")
with [Lindsay McQuade](https://datatalks.club/people/lindsaymcquade.html)
* [Build & Scale Data Products for AI: Roadmaps, MLOps, Customer Research & Metrics](https://datatalks.club/podcast/building-and-scaling-ai-data-products-with-mlops.html "Listen to: Build & Scale Data Products for AI: Roadmaps, MLOps, Customer Research & Metrics")
with [Greg Coquillo](https://datatalks.club/people/gregcoquillo.html)
* [Hiring Data Scientists & Analysts: Talent Pipelines, Job Specs, CV Screening & Salary Tips](https://datatalks.club/podcast/hiring-data-scientists-and-analysts.html "Listen to: Hiring Data Scientists & Analysts: Talent Pipelines, Job Specs, CV Screening & Salary Tips")
with [Alicja Notowska](https://datatalks.club/people/alicjanotowska.html)
* [DataTalks.Club Behind the Scenes: Alexey Grigorev on Scaling and Growing the Community](https://datatalks.club/podcast/datatalksclub-building-scaling-data-community.html "Listen to: DataTalks.Club Behind the Scenes: Alexey Grigorev on Scaling and Growing the Community")
with [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
, [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
### Season #6
* [Data Science Leadership: Product-First ML, Recommenders & RTB, MLOps, Hiring & Mentoring](https://datatalks.club/podcast/data-science-leadership-hiring-mlops.html "Listen to: Data Science Leadership: Product-First ML, Recommenders & RTB, MLOps, Hiring & Mentoring")
with [Mariano Semelman](https://datatalks.club/people/marianosemelman.html)
* [Lead NLP Teams: Hiring, Production Pipelines, MLOps & LLM Tradeoffs (GPT-3, spaCy)](https://datatalks.club/podcast/nlp-team-hiring-and-production-mlops.html "Listen to: Lead NLP Teams: Hiring, Production Pipelines, MLOps & LLM Tradeoffs (GPT-3, spaCy)")
with [Ivan Bilan](https://datatalks.club/people/ivanbilan.html)
* [Become an ML Product Manager: MLOps Platforms, Observability & Adoption](https://datatalks.club/podcast/ml-product-manager-and-mlops-platform-strategy.html "Listen to: Become an ML Product Manager: MLOps Platforms, Observability & Adoption")
with [Geo Jolly](https://datatalks.club/people/geojolly.html)
* [From Postdoc to Data Science Lead: ML Foundations, Docker Deployment & Hiring Tips](https://datatalks.club/podcast/postdoc-to-data-science-lead-career-transition.html "Listen to: From Postdoc to Data Science Lead: ML Foundations, Docker Deployment & Hiring Tips")
with [CJ Jenkins](https://datatalks.club/people/cjjenkins.html)
* [Master Spatial Big Data Analytics: Nebula Stream Systems, Postdoc Mentoring & PhD Tips](https://datatalks.club/podcast/big-data-analytics-and-postdoc-research.html "Listen to: Master Spatial Big Data Analytics: Nebula Stream Systems, Postdoc Mentoring & PhD Tips")
with [Eleni Tzirita Zacharatou](https://datatalks.club/people/elenitziritazacharatou.html)
* [How to Transition from Design to Data Product Manager: SQL, Customer Discovery & Data Quality](https://datatalks.club/podcast/product-designer-to-data-product-manager.html "Listen to: How to Transition from Design to Data Product Manager: SQL, Customer Discovery & Data Quality")
with [Sara Menefee](https://datatalks.club/people/saramenefee.html)
* [Data Science Manager vs Expert: Hiring Strategy, Skills, Team Building & When to Use ML](https://datatalks.club/podcast/data-science-manager-vs-expert-hiring-guide.html "Listen to: Data Science Manager vs Expert: Hiring Strategy, Skills, Team Building & When to Use ML")
with [Barbara Sobkowiak](https://datatalks.club/people/barbarasobkowiak.html)
* [Ace Data Interviews: Behavioral STARs, Case Strategy, Portfolios & Cold Emails](https://datatalks.club/podcast/data-interview-behavioral-and-portfolio-prep-guide.html "Listen to: Ace Data Interviews: Behavioral STARs, Case Strategy, Portfolios & Cold Emails")
with [Nick Singh](https://datatalks.club/people/nicksingh.html)
* [Solopreneur Guide: Diversify Income with Courses, Consulting, Books & Side-Gigs](https://datatalks.club/podcast/solopreneur-developer-and-data-professional.html "Listen to: Solopreneur Guide: Diversify Income with Courses, Consulting, Books & Side-Gigs")
with [Noah Gift](https://datatalks.club/people/noahgift.html)
### Season #5
* [Practical Data Science & ML: Feature Engineering, Model Monitoring, Data Governance & Storytelling](https://datatalks.club/podcast/feature-engineering-model-monitoring-and-data-governance.html "Listen to: Practical Data Science & ML: Feature Engineering, Model Monitoring, Data Governance & Storytelling")
with [Thom Ives](https://datatalks.club/people/thomives.html)
* [Last-Mile Data Delivery for the Modern Data Stack: Build Data Products to Boost Adoption](https://datatalks.club/podcast/last-mile-data-delivery-and-data-product-adoption-modern-data-stack.html "Listen to: Last-Mile Data Delivery for the Modern Data Stack: Build Data Products to Boost Adoption")
with [Caitlin Moorman](https://datatalks.club/people/caitlinmoorman.html)
* [From Analytics to Production ML: Team Building, Experiments, MLOps & Fraud Detection](https://datatalks.club/podcast/production-ml-mlops-and-data-team-building.html "Listen to: From Analytics to Production ML: Team Building, Experiments, MLOps & Fraud Detection")
with [Rishabh Bhargava](https://datatalks.club/people/rishabhbhargava.html)
* [How to Build & Scale a Data Team: Hiring, Production ML, Forecasting & Driving Adoption](https://datatalks.club/podcast/building-and-scaling-data-team.html "Listen to: How to Build & Scale a Data Team: Hiring, Production ML, Forecasting & Driving Adoption")
with [Tammy Liang](https://datatalks.club/people/tammyliang.html)
* [From Research to Production: Build Reproducible, Deployable Full-Stack ML Systems](https://datatalks.club/podcast/research-to-production-ml-systems-roadmap.html "Listen to: From Research to Production: Build Reproducible, Deployable Full-Stack ML Systems")
with [Mihail Eric](https://datatalks.club/people/mihaileric.html)
* [Solo Data Scientist Playbook: 90-Day Roadmap, Pipelines, A/B Tests & Prioritization](https://datatalks.club/podcast/solopreneur-data-scientist.html "Listen to: Solo Data Scientist Playbook: 90-Day Roadmap, Pipelines, A/B Tests & Prioritization")
with [Marianna Diachuk](https://datatalks.club/people/mariannadiachuk.html)
* [KPI Design & Metrics Strategy: Prioritize Impact, Avoid Vanity Metrics, and Prove ROI](https://datatalks.club/podcast/ml-engineering-kpis-and-metrics-strategy.html "Listen to: KPI Design & Metrics Strategy: Prioritize Impact, Avoid Vanity Metrics, and Prove ROI")
with [Adam Sroka](https://datatalks.club/people/adamsroka.html)
* [ETL vs ELT & Data Lake vs Warehouse: Airbyte, dbt, CDC for Modern Data Engineering](https://datatalks.club/podcast/data-engineering-tools-modern-data-stack.html "Listen to: ETL vs ELT & Data Lake vs Warehouse: Airbyte, dbt, CDC for Modern Data Engineering")
with [Natalie Kwong](https://datatalks.club/people/nataliekwong.html)
* [Practical Algorithms for Engineers: Bloom Filters, Approximate Nearest-Neighbor & Performance](https://datatalks.club/podcast/algorithms-data-structures-for-engineers.html "Listen to: Practical Algorithms for Engineers: Bloom Filters, Approximate Nearest-Neighbor & Performance")
with [Marcello La Rocca](https://datatalks.club/people/marcellolarocca.html)
### Season #4
* [Mastering the Chief Data Officer Role: Build Data Strategy, Org Design & AI](https://datatalks.club/podcast/chief-data-officer-data-strategy-and-org-design.html "Listen to: Mastering the Chief Data Officer Role: Build Data Strategy, Org Design & AI")
with [Marco De Sa](https://datatalks.club/people/marcodesa.html)
* [Freelancing in Machine Learning: Pricing, Client Acquisition & Proposals](https://datatalks.club/podcast/freelancing-in-machine-learning.html "Listen to: Freelancing in Machine Learning: Pricing, Client Acquisition & Proposals")
with [Mikio Braun](https://datatalks.club/people/mikiobraun.html)
* [Build a Grocery Retail OS to Cut Supermarket Food Waste & Scale Your Startup](https://datatalks.club/podcast/launch-and-build-retail-startup.html "Listen to: Build a Grocery Retail OS to Cut Supermarket Food Waste & Scale Your Startup")
with [Carmine Paolino](https://datatalks.club/people/carminepaolino.html)
* [Master Human-Centered MLOps: Stakeholder Buy-In, Monitoring, Debugging & Incident Response](https://datatalks.club/podcast/human-centered-mlops-and-model-monitoring.html "Listen to: Master Human-Centered MLOps: Stakeholder Buy-In, Monitoring, Debugging & Incident Response")
with [Lina Weichbrodt](https://datatalks.club/people/linaweichbrodt.html)
* [Practical Machine Learning Engineering for Production: Ship Maintainable Models, Avoid Complexity](https://datatalks.club/podcast/machine-learning-engineering-production-best-practices.html "Listen to: Practical Machine Learning Engineering for Production: Ship Maintainable Models, Avoid Complexity")
with [Ben Wilson](https://datatalks.club/people/benwilson.html)
* [How to Build a Successful ML Startup: MLOps, Model Monitoring, Open Source & Founder Fit](https://datatalks.club/podcast/building-mlops-startup.html "Listen to: How to Build a Successful ML Startup: MLOps, Model Monitoring, Open Source & Founder Fit")
with [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html)
* [Big Data Engineer vs Data Scientist: Skills, Tools, and Career Paths](https://datatalks.club/podcast/big-data-engineer-vs-data-scientist.html "Listen to: Big Data Engineer vs Data Scientist: Skills, Tools, and Career Paths")
with [Roksolana Diachuk](https://datatalks.club/people/roksolanadiachuk.html)
* [From Notebooks to Production: Build Data Pipelines & Deploy ML (AWS, Kafka, Streaming)](https://datatalks.club/podcast/production-ml-pipelines-with-aws-and-kafka.html "Listen to: From Notebooks to Production: Build Data Pipelines & Deploy ML (AWS, Kafka, Streaming)")
with [Andreas Kretz](https://datatalks.club/people/andreaskretz.html)
* [From Software Engineering to Machine Learning: 7 Lessons, Tools, MLOps & Project Roadmap](https://datatalks.club/podcast/from-software-engineer-to-machine-learning.html "Listen to: From Software Engineering to Machine Learning: 7 Lessons, Tools, MLOps & Project Roadmap")
with [Santiago Valdarrama](https://datatalks.club/people/svpino.html)
### Season #3
* [Master Analytics Engineering: Skills, Toolstack, Career Roadmap](https://datatalks.club/podcast/analytics-engineer-skills-tools.html "Listen to: Master Analytics Engineering: Skills, Toolstack, Career Roadmap")
with [Victoria Perez Mola](https://datatalks.club/people/victoriaperezmola.html)
* [How to Build Data Governance in the Cloud: Classification, Catalogs, Policies & ROI](https://datatalks.club/podcast/cloud-data-governance.html "Listen to: How to Build Data Governance in the Cloud: Classification, Catalogs, Policies & ROI")
with [Jessi Ashdown](https://datatalks.club/people/jessiashdown.html)
, [Uri Gilad](https://datatalks.club/people/urigilad.html)
* [Turn Data Science Project Failures into Career Wins: Production Lessons, MLOps Fixes & Framing Failures on LinkedIn](https://datatalks.club/podcast/data-science-failures-and-mlops-lessons.html "Listen to: Turn Data Science Project Failures into Career Wins: Production Lessons, MLOps Fixes & Framing Failures on LinkedIn")
with [Yury Kashnitsky](https://datatalks.club/people/yurykashnitsky.html)
* [How to Build a Data-Led Growth Stack: Event Tracking, Tracking Plans & Reverse ETL](https://datatalks.club/podcast/data-led-growth-event-tracking-and-reverse-etl.html "Listen to: How to Build a Data-Led Growth Stack: Event Tracking, Tracking Plans & Reverse ETL")
with [Arpit Choudhury](https://datatalks.club/people/arpitchoudhury.html)
* [Learn in Public: Personal Branding & Career Marketing for Developers](https://datatalks.club/podcast/developer-personal-brand-learn-in-public.html "Listen to: Learn in Public: Personal Branding & Career Marketing for Developers")
with [Shawn Swyx Wang](https://datatalks.club/people/swyx.html)
* [Switch to Computer Vision & Deep Learning: Roadmap, Kaggle Projects, Mentors & Interview Prep](https://datatalks.club/podcast/from-physics-to-computer-vision-career-transition.html "Listen to: Switch to Computer Vision & Deep Learning: Roadmap, Kaggle Projects, Mentors & Interview Prep")
with [Tatiana Gabruseva](https://datatalks.club/people/tatianagabruseva.html)
* [Data Strategist Guide: Effective Communication to Bridge Data Teams & Management for Data-Driven Growth](https://datatalks.club/podcast/data-translator-role-and-data-strategy.html "Listen to: Data Strategist Guide: Effective Communication to Bridge Data Teams & Management for Data-Driven Growth")
with [Lior Barak](https://datatalks.club/people/liorbarak.html)
* [Data Science Interview Guide: CV Optimization, Take-Home Projects, Mock Interviews & Negotiation](https://datatalks.club/podcast/data-science-interview-and-cv-guide.html "Listen to: Data Science Interview Guide: CV Optimization, Take-Home Projects, Mock Interviews & Negotiation")
with [Oleg Novikov](https://datatalks.club/people/olegnovikov.html)
* [Data Observability Explained: 5 Pillars to Prevent Downtime, Drift & False Positives](https://datatalks.club/podcast/data-quality-data-observability-data-reliability.html "Listen to: Data Observability Explained: 5 Pillars to Prevent Downtime, Drift & False Positives")
with [Barr Moses](https://datatalks.club/people/barrmoses.html)
* [Career Transition from Analytics to Data Science: Build a Kaggle Notebook Portfolio, Learn Python & Get Hired](https://datatalks.club/podcast/analytics-to-data-science-with-kaggle-portfolio.html "Listen to: Career Transition from Analytics to Data Science: Build a Kaggle Notebook Portfolio, Learn Python & Get Hired")
with [Andrada Olteanu](https://datatalks.club/people/andradaolteanu.html)
* [From Project Manager to Data Scientist: Skills, Tools, ML Courses & Job Search](https://datatalks.club/podcast/project-manager-to-data-scientist.html "Listen to: From Project Manager to Data Scientist: Skills, Tools, ML Courses & Job Search")
with [Ksenia Legostay](https://datatalks.club/people/ksenialegostay.html)
### Season #2
* [MLOps Community Playbook: Launch, Grow & Retain Meetups, Members, and Contributors](https://datatalks.club/podcast/mlops-community-building-and-meetups.html "Listen to: MLOps Community Playbook: Launch, Grow & Retain Meetups, Members, and Contributors")
with [Demetrios Brinkmann](https://datatalks.club/people/demetriosbrinkmann.html)
* [DataOps 101 for Scaling Data Platforms: Immutable Pipelines, Self-Service Lakehouse & Reproducibility](https://datatalks.club/podcast/dataops-principles-and-scalable-data-platforms.html "Listen to: DataOps 101 for Scaling Data Platforms: Immutable Pipelines, Self-Service Lakehouse & Reproducibility")
with [Lars Albertsson](https://datatalks.club/people/larsalbertsson.html)
* [Public Speaking for Data Scientists: Master AI Evangelism, Storytelling & Keynotes](https://datatalks.club/podcast/public-speaking-for-data-scientists.html "Listen to: Public Speaking for Data Scientists: Master AI Evangelism, Storytelling & Keynotes")
with [Ben Taylor](https://datatalks.club/people/bentaylor.html)
* [Monetize Machine Learning: Convert Models to ARR/MRR with ML Product & MLOps Strategy](https://datatalks.club/podcast/make-money-with-machine-learning-roles-skills.html "Listen to: Monetize Machine Learning: Convert Models to ARR/MRR with ML Product & MLOps Strategy")
with [Vin Vashishta](https://datatalks.club/people/vinvashishta.html)
* [Build a Personal Brand: Publish on LinkedIn/Medium, Grow Audience, Monetize with Online Courses](https://datatalks.club/podcast/personal-brand-for-data-professionals.html "Listen to: Build a Personal Brand: Publish on LinkedIn/Medium, Grow Audience, Monetize with Online Courses")
with [Admond Lee Kin Lim](https://datatalks.club/people/admondleekinlim.html)
* [Data Science Career Guide: ABC Framework (Analyst, Builder, Consultant) & Transition Tips](https://datatalks.club/podcast/data-science-career-abc-framework.html "Listen to: Data Science Career Guide: ABC Framework (Analyst, Builder, Consultant) & Transition Tips")
with [Danny Ma](https://datatalks.club/people/dannyma.html)
* [Optimize Decisions with ML: Prescriptive & Robust Optimization for Supply Chain and Pricing](https://datatalks.club/podcast/machine-learning-decision-optimization.html "Listen to: Optimize Decisions with ML: Prescriptive & Robust Optimization for Supply Chain and Pricing")
with [Dan Becker](https://datatalks.club/people/danbecker.html)
* [Feature Stores for MLOps: Real-Time Feature Engineering, Feast & Tecton Guide](https://datatalks.club/podcast/mlops-feature-stores-feature-stores-feast-tecton.html "Listen to: Feature Stores for MLOps: Real-Time Feature Engineering, Feast & Tecton Guide")
with [Willem Pienaar](https://datatalks.club/people/willempienaar.html)
* [Mastering MLOps: Kubeflow Pipelines, Model Monitoring & Automated Retraining](https://datatalks.club/podcast/mlops-kubeflow-model-monitoring.html "Listen to: Mastering MLOps: Kubeflow Pipelines, Model Monitoring & Automated Retraining")
with [Theofilos Papapanagiotou](https://datatalks.club/people/theofilospapapanagiotou.html)
* [Contribute to Open Source ML: scikit-learn Pipelines, PRs, Docs & Rasa Conversational AI](https://datatalks.club/podcast/open-source-ml-contributions.html "Listen to: Contribute to Open Source ML: scikit-learn Pipelines, PRs, Docs & Rasa Conversational AI")
with [Vincent Warmerdam](https://datatalks.club/people/vincentwarmerdam.html)
* [DevRel for Data Science: Build Community, Create Content, and Grow Your Career](https://datatalks.club/podcast/devrel-data-science-open-source-tools.html "Listen to: DevRel for Data Science: Build Community, Create Content, and Grow Your Career")
with [Elle O'Brien](https://datatalks.club/people/elleobrien.html)
* [Master Technical Writing: 7-Day Workflow to Accelerate Your Data Science Career](https://datatalks.club/podcast/technical-writing-for-data-scientists.html "Listen to: Master Technical Writing: 7-Day Workflow to Accelerate Your Data Science Career")
with [Eugene Yan](https://datatalks.club/people/eugeneyan.html)
### Season #1
* [How to Find a Mentor and Become One: Mentoring Strategies for Tech Careers](https://datatalks.club/podcast/mentoring-in-tech-how-to-find-and-become-a-mentor.md.html "Listen to: How to Find a Mentor and Become One: Mentoring Strategies for Tech Careers")
with [Rahul Jain](https://datatalks.club/people/rahuljain.html)
* [Land Data Scientist Roles: Resumes, Portfolios, Interviews & Recruiter Workflow](https://datatalks.club/podcast/get-data-scientist-job.html "Listen to: Land Data Scientist Roles: Resumes, Portfolios, Interviews & Recruiter Workflow")
with [Luke Whipps](https://datatalks.club/people/lukewhipps.html)
* [How to Build and Scale ML Teams: Hiring, MLOps & Product-Driven AI for Startups](https://datatalks.club/podcast/building-data-team.html "Listen to: How to Build and Scale ML Teams: Hiring, MLOps & Product-Driven AI for Startups")
with [Dat Tran](https://datatalks.club/people/dattran.html)
* [CRISP-DM Methodology for Data Science Projects: Business Understanding, Data Preparation, Modeling, Evaluation & Deployment](https://datatalks.club/podcast/crisp-dm.html "Listen to: CRISP-DM Methodology for Data Science Projects: Business Understanding, Data Preparation, Modeling, Evaluation & Deployment")
with [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
* [Data Team Roles Explained: Skills, Responsibilities, and How Teams Ship ML Products](https://datatalks.club/podcast/data-team-roles.html "Listen to: Data Team Roles Explained: Skills, Responsibilities, and How Teams Ship ML Products")
with [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Krzysztof Szafanek – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Krzysztof Szafanek
Krzysztof Szafanek is a seasoned engineer with 17 years of professional experience in building software in industries such as pharma, geo services, gaming and online retail. At Zalando, Krzysztof has been supporting machine learning practitioners as ML Platform engineer and internal consultant.
[](https://twitter.com/szafranek)
[](https://linkedin.com/in/szafranek)
[](https://szafranek.net/)
### Events
* Machine Learning Workflows in Production ([watch on youtube](https://www.youtube.com/watch?v=CO4Gqd95j6k)
)
* Navigating Career Changes in Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=cUxZBXQgZaU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# People of DataTalks.Club – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
People
======
Discover **422** data science professionals, authors, and experts from our global community.
All Community Members
---------------------
* [Aashish Nair](https://datatalks.club/people/%20aashishnair.html "Learn more about Aashish Nair")
* [Rahul Jain](https://datatalks.club/people/16rahuljain.html "Learn more about Rahul Jain")
* [Aaisha Muhammad](https://datatalks.club/people/aaishamuhammad.html "Learn more about Aaisha Muhammad")
* [Aaron Wishnick](https://datatalks.club/people/aaronwishnick.html "Learn more about Aaron Wishnick")
* [Abouzar Abbaspour](https://datatalks.club/people/abouzarabbaspour.html "Learn more about Abouzar Abbaspour")
* [Adam Sroka](https://datatalks.club/people/adamsroka.html "Learn more about Adam Sroka")
* [Aditya Gautam](https://datatalks.club/people/adityagautam.html "Learn more about Aditya Gautam")
* [Aditya Seshaditya](https://datatalks.club/people/adityaseshaditya.html "Learn more about Aditya Seshaditya")
* [Admond Lee Kin Lim](https://datatalks.club/people/admondleekinlim.html "Learn more about Admond Lee Kin Lim")
* [Adrian Brudaru](https://datatalks.club/people/adrianbrudaru.html "Learn more about Adrian Brudaru")
* [Agita Jaunzeme](https://datatalks.club/people/agitajaunzeme.html "Learn more about Agita Jaunzeme")
* [Agnes van Belle](https://datatalks.club/people/agnesvanbelle.html "Learn more about Agnes van Belle")
* [Agnieszka Mikołajczyk](https://datatalks.club/people/agnieszkamikolajczyk.html "Learn more about Agnieszka Mikołajczyk")
* [Agostino Calamia](https://datatalks.club/people/agostinocalamia.html "Learn more about Agostino Calamia")
* [Aishwarya Jadhav](https://datatalks.club/people/aishwaryajadhav.html "Learn more about Aishwarya Jadhav")
* [Akela Drissner](https://datatalks.club/people/akeladrissner.html "Learn more about Akela Drissner")
* [Aleksander Kruszelnicki](https://datatalks.club/people/aleksanderkruszelnicki.html "Learn more about Aleksander Kruszelnicki")
* [Aleksander Molak](https://datatalks.club/people/aleksandermolak.html "Learn more about Aleksander Molak")
* [Alena Astrakhantseva](https://datatalks.club/people/alenaastrakhantseva.html "Learn more about Alena Astrakhantseva")
* [Alexander Daniel Rios](https://datatalks.club/people/alexanderdanielrios.html "Learn more about Alexander Daniel Rios")
* [Alexander Guschin](https://datatalks.club/people/alexanderguschin.html "Learn more about Alexander Guschin")
* [Alexander Hendorf](https://datatalks.club/people/alexanderhendorf.html "Learn more about Alexander Hendorf")
* [Alex Chung](https://datatalks.club/people/alexchung.html "Learn more about Alex Chung")
* [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html "Learn more about Alexey Grigorev")
* [Alexia Audevart](https://datatalks.club/people/alexiaaudevart.html "Learn more about Alexia Audevart")
* [Alex Ioannides](https://datatalks.club/people/alexioannides.html "Learn more about Alex Ioannides")
* [Alex Kim](https://datatalks.club/people/alexkim.html "Learn more about Alex Kim")
* [Alex Litvinov](https://datatalks.club/people/alexlitvinov.html "Learn more about Alex Litvinov")
* [Alex Petrov](https://datatalks.club/people/alexpetrov.html "Learn more about Alex Petrov")
* [Alicja Notowska](https://datatalks.club/people/alicjanotowska.html "Learn more about Alicja Notowska")
* [Alvaro Navas Peire](https://datatalks.club/people/alvaronavaspeire.html "Learn more about Alvaro Navas Peire")
* [Amber Roberts](https://datatalks.club/people/amberroberts.html "Learn more about Amber Roberts")
* [Anahita Pakiman](https://datatalks.club/people/anahitapakiman.html "Learn more about Anahita Pakiman")
* [Anastasia Karavdina](https://datatalks.club/people/anastasiakaravdina.html "Learn more about Anastasia Karavdina")
* [Andrada Olteanu](https://datatalks.club/people/andradaolteanu.html "Learn more about Andrada Olteanu")
* [Andreas Kretz](https://datatalks.club/people/andreaskretz.html "Learn more about Andreas Kretz")
* [Andreas Syrén](https://datatalks.club/people/andreassyren.html "Learn more about Andreas Syrén")
* [Andreea Munteanu](https://datatalks.club/people/andreeamunteanu.html "Learn more about Andreea Munteanu")
* [Andrei Tserakhau](https://datatalks.club/people/andreitserakhau.html "Learn more about Andrei Tserakhau")
* [Andre Schumacher](https://datatalks.club/people/andreschumacher.html "Learn more about Andre Schumacher")
* [Andrew Jones](https://datatalks.club/people/andrewjones.html "Learn more about Andrew Jones")
* [Andrew McMahon](https://datatalks.club/people/andrewmcmahon.html "Learn more about Andrew McMahon")
* [Andrey Cheptsov](https://datatalks.club/people/andreycheptsov.html "Learn more about Andrey Cheptsov")
* [Andrey Shtylenko](https://datatalks.club/people/andreyshtylenko.html "Learn more about Andrey Shtylenko")
* [Andy Petrella](https://datatalks.club/people/andypetrella.html "Learn more about Andy Petrella")
* [Angela Ramirez](https://datatalks.club/people/angelaramirez.html "Learn more about Angela Ramirez")
* [Angelica Lo Duca](https://datatalks.club/people/angelicaloduca.html "Learn more about Angelica Lo Duca")
* [Anish Shah](https://datatalks.club/people/anishshah.html "Learn more about Anish Shah")
* [Ankur A. Patel](https://datatalks.club/people/ankurapatel.html "Learn more about Ankur A. Patel")
* [Anna Hannemann](https://datatalks.club/people/annahannemann.html "Learn more about Anna Hannemann")
* [Anthony Virtuoso](https://datatalks.club/people/anthonyvirtuoso.html "Learn more about Anthony Virtuoso")
* [Antje Barth](https://datatalks.club/people/antjebarth.html "Learn more about Antje Barth")
* [Antonis Stellas](https://datatalks.club/people/antonisstellas.html "Learn more about Antonis Stellas")
* [Anusha Akkina](https://datatalks.club/people/anushaakkina.html "Learn more about Anusha Akkina")
* [Aparna Dhinakaran](https://datatalks.club/people/aparnadhinakaran.html "Learn more about Aparna Dhinakaran")
* [Apurva Misra](https://datatalks.club/people/apurvamisra.html "Learn more about Apurva Misra")
* [Arman Jabbari](https://datatalks.club/people/armanjabbari.html "Learn more about Arman Jabbari")
* [Arpit Choudhury](https://datatalks.club/people/arpitchoudhury.html "Learn more about Arpit Choudhury")
* [Arseny Kravchenko](https://datatalks.club/people/arsenykravchenko.html "Learn more about Arseny Kravchenko")
* [Artemii Frolov](https://datatalks.club/people/artemiifrolov.html "Learn more about Artemii Frolov")
* [Ashish Patel](https://datatalks.club/people/ashishpatel.html "Learn more about Ashish Patel")
* [Atita Arora](https://datatalks.club/people/atitaarora.html "Learn more about Atita Arora")
* [Ba Linh Le](https://datatalks.club/people/balinhle.html "Learn more about Ba Linh Le")
* [Barbara Sobkowiak](https://datatalks.club/people/barbarasobkowiak.html "Learn more about Barbara Sobkowiak")
* [Barr Moses](https://datatalks.club/people/barrmoses.html "Learn more about Barr Moses")
* [Bartosz Mikulski](https://datatalks.club/people/bartoszmikulski.html "Learn more about Bartosz Mikulski")
* [Bart Vandekerckhove](https://datatalks.club/people/bartvandekerckhove.html "Learn more about Bart Vandekerckhove")
* [Bastien Boutonnet](https://datatalks.club/people/bastienboutonnet.html "Learn more about Bastien Boutonnet")
* [Bela Wiertz](https://datatalks.club/people/belawiertz.html "Learn more about Bela Wiertz")
* [Ben Taylor](https://datatalks.club/people/bentaylor.html "Learn more about Ben Taylor")
* [Ben Wilson](https://datatalks.club/people/benwilson.html "Learn more about Ben Wilson")
* [Bhavani Ravi](https://datatalks.club/people/bhavaniravi.html "Learn more about Bhavani Ravi")
* [Boyan Angelov](https://datatalks.club/people/boyanangelov.html "Learn more about Boyan Angelov")
* [Caitlin Moorman](https://datatalks.club/people/caitlinmoorman.html "Learn more about Caitlin Moorman")
* [Carmine Paolino](https://datatalks.club/people/carminepaolino.html "Learn more about Carmine Paolino")
* [Cathy Chen](https://datatalks.club/people/cathychen.html "Learn more about Cathy Chen")
* [Chip Huyen](https://datatalks.club/people/chiphuyen.html "Learn more about Chip Huyen")
* [Chris Fregly](https://datatalks.club/people/chrisfregly.html "Learn more about Chris Fregly")
* [Christiaan Swart](https://datatalks.club/people/christiannswart.html "Learn more about Christiaan Swart")
* [Christian Winkler](https://datatalks.club/people/christianwinkler.html "Learn more about Christian Winkler")
* [Christine Cepelak](https://datatalks.club/people/christinecepelak.html "Learn more about Christine Cepelak")
* [Christopher Bergh](https://datatalks.club/people/christopherbergh.html "Learn more about Christopher Bergh")
* [Christoph Molnar](https://datatalks.club/people/christophmolnar.html "Learn more about Christoph Molnar")
* [CJ Jenkins](https://datatalks.club/people/cjjenkins.html "Learn more about CJ Jenkins")
* [Cristian Martinez](https://datatalks.club/people/cristianmartinez.html "Learn more about Cristian Martinez")
* [Daliana Liu](https://datatalks.club/people/dalianaliu.html "Learn more about Daliana Liu")
* [Dan Becker](https://datatalks.club/people/danbecker.html "Learn more about Dan Becker")
* [Dânia Meira](https://datatalks.club/people/daniameira.html "Learn more about Dânia Meira")
* [Daniel Egbo](https://datatalks.club/people/danielegbo.html "Learn more about Daniel Egbo")
* [Daniel Svonava](https://datatalks.club/people/danielsvonava.html "Learn more about Daniel Svonava")
* [Danny Leybzon](https://datatalks.club/people/dannyleybzon.html "Learn more about Danny Leybzon")
* [Danny Ma](https://datatalks.club/people/dannyma.html "Learn more about Danny Ma")
* [Dashel Ruiz Perez](https://datatalks.club/people/dashelruizperez.html "Learn more about Dashel Ruiz Perez")
* [Dat Tran](https://datatalks.club/people/dattran.html "Learn more about Dat Tran")
* [Dave Bechberger](https://datatalks.club/people/davebechberger.html "Learn more about Dave Bechberger")
* [Dave Flynn](https://datatalks.club/people/daveflynn.html "Learn more about Dave Flynn")
* [David Bader](https://datatalks.club/people/davidbader.html "Learn more about David Bader")
* [David Bednar](https://datatalks.club/people/davidbednar.html "Learn more about David Bednar")
* [David Gates](https://datatalks.club/people/davidgates.html "Learn more about David Gates")
* [David Mertz](https://datatalks.club/people/davidmertz.html "Learn more about David Mertz")
* [David Stephenson](https://datatalks.club/people/davidstephenson.html "Learn more about David Stephenson")
* [David Sweet](https://datatalks.club/people/davidsweet.html "Learn more about David Sweet")
* [Daynan Crull](https://datatalks.club/people/daynancrull.html "Learn more about Daynan Crull")
* [Delina Ivanova](https://datatalks.club/people/delinaivanova.html "Learn more about Delina Ivanova")
* [Demetrios Brinkmann](https://datatalks.club/people/demetriosbrinkmann.html "Learn more about Demetrios Brinkmann")
* [Denise Gosnell](https://datatalks.club/people/denisegosnell.html "Learn more about Denise Gosnell")
* [Denis Rothman](https://datatalks.club/people/denisrothman.html "Learn more about Denis Rothman")
* [Dimitri Visnadi](https://datatalks.club/people/dimitrivisnadi.html "Learn more about Dimitri Visnadi")
* [Dmitry Muzalevskiy](https://datatalks.club/people/dmitrymuzalevskiy.html "Learn more about Dmitry Muzalevskiy")
* [Don Jones](https://datatalks.club/people/donjones.html "Learn more about Don Jones")
* [Douglas Gray](https://datatalks.club/people/douglasgray.html "Learn more about Douglas Gray")
* [Doug Turnbull](https://datatalks.club/people/dougturnbull.html "Learn more about Doug Turnbull")
* [Duygu Altinok](https://datatalks.club/people/duygualtinok.html "Learn more about Duygu Altinok")
* [Eddy Zulkifly](https://datatalks.club/people/eddyzulkifly.html "Learn more about Eddy Zulkifly")
* [Elena Samuylova](https://datatalks.club/people/elenasamuylova.html "Learn more about Elena Samuylova")
* [Eleni Stamatelou](https://datatalks.club/people/elenistamatelou.html "Learn more about Eleni Stamatelou")
* [Eleni Tzirita Zacharatou](https://datatalks.club/people/elenitziritazacharatou.html "Learn more about Eleni Tzirita Zacharatou")
* [Elias Nema](https://datatalks.club/people/eliasnema.html "Learn more about Elias Nema")
* [Ella (Wati) Sahnan](https://datatalks.club/people/ella(wati)sahnan.html "Learn more about Ella (Wati) Sahnan")
* [Ellen König](https://datatalks.club/people/ellenkonig.html "Learn more about Ellen König")
* [Elle O'Brien](https://datatalks.club/people/elleobrien.html "Learn more about Elle O'Brien")
* [Emeli Dral](https://datatalks.club/people/emelidral.html "Learn more about Emeli Dral")
* [Emil Bogomolov](https://datatalks.club/people/emilbogomolov.html "Learn more about Emil Bogomolov")
* [Emmanuel Ameisen](https://datatalks.club/people/emmanuelameisen.html "Learn more about Emmanuel Ameisen")
* [Emmanuel Raj](https://datatalks.club/people/emmanuelraj.html "Learn more about Emmanuel Raj")
* [Engin Yöyen](https://datatalks.club/people/enginyoyen.html "Learn more about Engin Yöyen")
* [Eric Sims](https://datatalks.club/people/ericsims.html "Learn more about Eric Sims")
* [Ertugrul Mutlu](https://datatalks.club/people/ertugrulmutlu.html "Learn more about Ertugrul Mutlu")
* [Erum Afzal](https://datatalks.club/people/erumafzal.html "Learn more about Erum Afzal")
* [Eugene Yan](https://datatalks.club/people/eugeneyan.html "Learn more about Eugene Yan")
* [Evan Shellshear](https://datatalks.club/people/evanshellshear.html "Learn more about Evan Shellshear")
* [Fabiana Clemente](https://datatalks.club/people/fabianaclemente.html "Learn more about Fabiana Clemente")
* [Faisal Masood](https://datatalks.club/people/faisalmasood.html "Learn more about Faisal Masood")
* [Fernando Doglio](https://datatalks.club/people/fernandodoglio.html "Learn more about Fernando Doglio")
* [Filipa Castro](https://datatalks.club/people/filipacastro.html "Learn more about Filipa Castro")
* [Florian Hoenicke](https://datatalks.club/people/florianhoenicke.html "Learn more about Florian Hoenicke")
* [Gant Laborde](https://datatalks.club/people/gantlaborde.html "Learn more about Gant Laborde")
* [Geo Jolly](https://datatalks.club/people/geojolly.html "Learn more about Geo Jolly")
* [Giuseppe Bonaccorso](https://datatalks.club/people/giuseppebonaccorso.html "Learn more about Giuseppe Bonaccorso")
* [Gloria Quiceno](https://datatalks.club/people/gloriaquiceno.html "Learn more about Gloria Quiceno")
* [Gonçalo Sequeira](https://datatalks.club/people/goncalosequeira.html "Learn more about Gonçalo Sequeira")
* [Gráinne McKnight](https://datatalks.club/people/grainnemcknight.html "Learn more about Gráinne McKnight")
* [Greg Coquillo](https://datatalks.club/people/gregcoquillo.html "Learn more about Greg Coquillo")
* [Guillaume Lemaître](https://datatalks.club/people/guillaumelemaitre.html "Learn more about Guillaume Lemaître")
* [Guy Adams](https://datatalks.club/people/guyadams.html "Learn more about Guy Adams")
* [Hagop Dippel](https://datatalks.club/people/hagopdippel.html "Learn more about Hagop Dippel")
* [Hannes Hapke](https://datatalks.club/people/hanneshapke.html "Learn more about Hannes Hapke")
* [Hayden Liu](https://datatalks.club/people/haydenliu.html "Learn more about Hayden Liu")
* [Haziqa Sajid](https://datatalks.club/people/haziqasajid.html "Learn more about Haziqa Sajid")
* [Hélder Russa](https://datatalks.club/people/helderrussa.html "Learn more about Hélder Russa")
* [Hiba Jamal](https://datatalks.club/people/hibajamal.html "Learn more about Hiba Jamal")
* [Himanshu Upreti](https://datatalks.club/people/himanshuupreti.html "Learn more about Himanshu Upreti")
* [Hugo Bowne-Anderson](https://datatalks.club/people/hugobowneanderson.html "Learn more about Hugo Bowne-Anderson")
* [Igor Demidov](https://datatalks.club/people/igordemidov.html "Learn more about Igor Demidov")
* [Igor Susmelj](https://datatalks.club/people/igorsusmelj.html "Learn more about Igor Susmelj")
* [Ilia Ivanov](https://datatalks.club/people/iliaivanov.html "Learn more about Ilia Ivanov")
* [Illia Todor](https://datatalks.club/people/illiatodor.html "Learn more about Illia Todor")
* [Ilya Boytsov](https://datatalks.club/people/ilyaboytsov.html "Learn more about Ilya Boytsov")
* [Ioannis Mesionis](https://datatalks.club/people/ioannismesionis.html "Learn more about Ioannis Mesionis")
* [Irina Brudaru](https://datatalks.club/people/irinabrudaru.html "Learn more about Irina Brudaru")
* [Isabella Bicalho](https://datatalks.club/people/isabellabicalho.html "Learn more about Isabella Bicalho")
* [Itai Admi](https://datatalks.club/people/itaiadmi.html "Learn more about Itai Admi")
* [Ivan Bilan](https://datatalks.club/people/ivanbilan.html "Learn more about Ivan Bilan")
* [Ivan Brigida](https://datatalks.club/people/ivanbrigida.html "Learn more about Ivan Brigida")
* [Ivan Potapov](https://datatalks.club/people/ivanpotapov.html "Learn more about Ivan Potapov")
* [Jack Blandin](https://datatalks.club/people/jackblandin.html "Learn more about Jack Blandin")
* [Jacques Peeters](https://datatalks.club/people/jacquespeeters.html "Learn more about Jacques Peeters")
* [Jakob Graff](https://datatalks.club/people/jakobgraff.html "Learn more about Jakob Graff")
* [James Phoenix](https://datatalks.club/people/jamesphoenix.html "Learn more about James Phoenix")
* [Jamie Broomall](https://datatalks.club/people/jamiebroomall.html "Learn more about Jamie Broomall")
* [Janna Lipenkova](https://datatalks.club/people/jannalipenkova.html "Learn more about Janna Lipenkova")
* [Jan Schlicht](https://datatalks.club/people/janschlicht.html "Learn more about Jan Schlicht")
* [Jan Zawadzki](https://datatalks.club/people/janzawadzki.html "Learn more about Jan Zawadzki")
* [Jeanine Harb](https://datatalks.club/people/jeanineharb.html "Learn more about Jeanine Harb")
* [Jeff Katz](https://datatalks.club/people/jeffkatz.html "Learn more about Jeff Katz")
* [Jekaterina Kokatjuhha](https://datatalks.club/people/jekaterinakokatjuhha.html "Learn more about Jekaterina Kokatjuhha")
* [Jens Albrecht](https://datatalks.club/people/jensalbrecht.html "Learn more about Jens Albrecht")
* [Jesse Anderson](https://datatalks.club/people/jesseanderson.html "Learn more about Jesse Anderson")
* [Jessi Ashdown](https://datatalks.club/people/jessiashdown.html "Learn more about Jessi Ashdown")
* [Jessica Greene](https://datatalks.club/people/jessicagreene.html "Learn more about Jessica Greene")
* [Jessie Yaros](https://datatalks.club/people/jessieyaros.html "Learn more about Jessie Yaros")
* [Joe Reis](https://datatalks.club/people/joereis.html "Learn more about Joe Reis")
* [Johanna Bayer](https://datatalks.club/people/johannabayer.html "Learn more about Johanna Bayer")
* [Johannes Hötter](https://datatalks.club/people/johanneshotter.html "Learn more about Johannes Hötter")
* [Jonas Christensen](https://datatalks.club/people/jonaschristensen.html "Learn more about Jonas Christensen")
* [Jonathan Rioux](https://datatalks.club/people/jonathanrioux.html "Learn more about Jonathan Rioux")
* [Jon Skeet](https://datatalks.club/people/jonskeet.html "Learn more about Jon Skeet")
* [José María Sánchez Salas](https://datatalks.club/people/josemaria.html "Learn more about José María Sánchez Salas")
* [Josh Fischer](https://datatalks.club/people/joshfischer.html "Learn more about Josh Fischer")
* [Josh Tobin](https://datatalks.club/people/joshtobin.html "Learn more about Josh Tobin")
* [Joyce Kay Avila](https://datatalks.club/people/joycekayavila.html "Learn more about Joyce Kay Avila")
* [Juan Manuel Perafan](https://datatalks.club/people/juanmanuelperafan.html "Learn more about Juan Manuel Perafan")
* [Juan Orduz](https://datatalks.club/people/juanorduz.html "Learn more about Juan Orduz")
* [Juan Pablo](https://datatalks.club/people/juanpablo.html "Learn more about Juan Pablo")
* [Julia Ostheimer](https://datatalks.club/people/juliaostheimer.html "Learn more about Julia Ostheimer")
* [Justin Ryan](https://datatalks.club/people/justinryan.html "Learn more about Justin Ryan")
* [Katarzyna Foremniak](https://datatalks.club/people/katarzynaforemniak.html "Learn more about Katarzyna Foremniak")
* [Kate Ogochukwu Nwankwo](https://datatalks.club/people/kateogochukwunwankwo.html "Learn more about Kate Ogochukwu Nwankwo")
* [Katharine Jarmul](https://datatalks.club/people/katharinejarmul.html "Learn more about Katharine Jarmul")
* [Katie Bauer](https://datatalks.club/people/katiebauer.html "Learn more about Katie Bauer")
* [Ken Youens-Clark](https://datatalks.club/people/kenyouens-clark.html "Learn more about Ken Youens-Clark")
* [Kevin Huo](https://datatalks.club/people/kevinhuo.html "Learn more about Kevin Huo")
* [Khuyen Tran](https://datatalks.club/people/khuyentran.html "Learn more about Khuyen Tran")
* [Kim Falk](https://datatalks.club/people/kimfalk.html "Learn more about Kim Falk")
* [Kishan Manani](https://datatalks.club/people/kishanmanani.html "Learn more about Kishan Manani")
* [Konrad Banachewicz](https://datatalks.club/people/konradbanachewicz.html "Learn more about Konrad Banachewicz")
* [Kranti K. Parisa](https://datatalks.club/people/krantik-parisa.html "Learn more about Kranti K. Parisa")
* [Krzysztof Ograbek](https://datatalks.club/people/krzysztofograbek.html "Learn more about Krzysztof Ograbek")
* [Krzysztof Szafanek](https://datatalks.club/people/krzysztofszafanek.html "Learn more about Krzysztof Szafanek")
* [Ksenia Legostay](https://datatalks.club/people/ksenialegostay.html "Learn more about Ksenia Legostay")
* [Kyle Shannon](https://datatalks.club/people/kyleshannon.html "Learn more about Kyle Shannon")
* [Lalit Pagaria](https://datatalks.club/people/lalitpagaria.html "Learn more about Lalit Pagaria")
* [Lars Albertsson](https://datatalks.club/people/larsalbertsson.html "Learn more about Lars Albertsson")
* [Larysa Visengeriyeva](https://datatalks.club/people/larysavisengeriyeva.html "Learn more about Larysa Visengeriyeva")
* [Laurence Moroney](https://datatalks.club/people/laurencemoroney.html "Learn more about Laurence Moroney")
* [Lavanya Gupta](https://datatalks.club/people/lavanyagupta.html "Learn more about Lavanya Gupta")
* [Leandro von Werra](https://datatalks.club/people/leandrovonwerra.html "Learn more about Leandro von Werra")
* [Leonard Püttmann](https://datatalks.club/people/leonardputtmann.html "Learn more about Leonard Püttmann")
* [Leon Wei](https://datatalks.club/people/leonwei.html "Learn more about Leon Wei")
* [Lera Kaimashnіkova](https://datatalks.club/people/lerakaimashnikova.html "Learn more about Lera Kaimashnіkova")
* [Lewis Tunstall](https://datatalks.club/people/lewistunstall.html "Learn more about Lewis Tunstall")
* [Liesbeth Dingemans](https://datatalks.club/people/liesbethdingemans.html "Learn more about Liesbeth Dingemans")
* [Lina Weichbrodt](https://datatalks.club/people/linaweichbrodt.html "Learn more about Lina Weichbrodt")
* [Lindsay McQuade](https://datatalks.club/people/lindsaymcquade.html "Learn more about Lindsay McQuade")
* [Lior Barak](https://datatalks.club/people/liorbarak.html "Learn more about Lior Barak")
* [Lisa Cohen](https://datatalks.club/people/lisacohen.html "Learn more about Lisa Cohen")
* [Loïc Magnien](https://datatalks.club/people/loicmagnien.html "Learn more about Loïc Magnien")
* [Loris Marini](https://datatalks.club/people/lorismarini.html "Learn more about Loris Marini")
* [Luca Massaron](https://datatalks.club/people/lucamassaron.html "Learn more about Luca Massaron")
* [Luís Oliveira](https://datatalks.club/people/luisoliveira.html "Learn more about Luís Oliveira")
* [Luis Serrano](https://datatalks.club/people/luisserrano.html "Learn more about Luis Serrano")
* [Luke Whipps](https://datatalks.club/people/lukewhipps.html "Learn more about Luke Whipps")
* [Madiha Khalid](https://datatalks.club/people/madihakhalid.html "Learn more about Madiha Khalid")
* [Magdalena Konkiewicz](https://datatalks.club/people/magdalenakonkiewicz.html "Learn more about Magdalena Konkiewicz")
* [Magdalena Kuhn](https://datatalks.club/people/magdalenakuhn.html "Learn more about Magdalena Kuhn")
* [Mahmoud AbdelAziz](https://datatalks.club/people/mahmoudaziz.html "Learn more about Mahmoud AbdelAziz")
* [Manmohan Gosada](https://datatalks.club/people/manmohangosada.html "Learn more about Manmohan Gosada")
* [Manoj Kukreja](https://datatalks.club/people/manojkukreja.html "Learn more about Manoj Kukreja")
* [Marcello La Rocca](https://datatalks.club/people/marcellolarocca.html "Learn more about Marcello La Rocca")
* [Marco De Sa](https://datatalks.club/people/marcodesa.html "Learn more about Marco De Sa")
* [Maria Bruckert](https://datatalks.club/people/mariabruckert.html "Learn more about Maria Bruckert")
* [Marianna Diachuk](https://datatalks.club/people/mariannadiachuk.html "Learn more about Marianna Diachuk")
* [Mariano Semelman](https://datatalks.club/people/marianosemelman.html "Learn more about Mariano Semelman")
* [Maria Sukhareva](https://datatalks.club/people/mariasukhareva.html "Learn more about Maria Sukhareva")
* [Maria Vechtomova](https://datatalks.club/people/mariavechtomova.html "Learn more about Maria Vechtomova")
* [Marijn Markus](https://datatalks.club/people/marijnmarkus.html "Learn more about Marijn Markus")
* [Mario Lazo](https://datatalks.club/people/mariolazo.html "Learn more about Mario Lazo")
* [Mark Ryan](https://datatalks.club/people/markryan.html "Learn more about Mark Ryan")
* [Martin Kleppmann](https://datatalks.club/people/martinkleppmann.html "Learn more about Martin Kleppmann")
* [Martin Potančok](https://datatalks.club/people/martinpotancok.html "Learn more about Martin Potančok")
* [Mary Jane Dykeman](https://datatalks.club/people/maryjanedykeman.html "Learn more about Mary Jane Dykeman")
* [Marysia Winkels](https://datatalks.club/people/marysiawinkels.html "Learn more about Marysia Winkels")
* [Matt Harrison](https://datatalks.club/people/mattharrison.html "Learn more about Matt Harrison")
* [Matthew Housley](https://datatalks.club/people/matthewhousley.html "Learn more about Matthew Housley")
* [Matt Palmer](https://datatalks.club/people/mattpalmer.html "Learn more about Matt Palmer")
* [Maxime Labonne](https://datatalks.club/people/maximelabonne.html "Learn more about Maxime Labonne")
* [Maxim Lukichev](https://datatalks.club/people/maximlukichev.html "Learn more about Maxim Lukichev")
* [Max Schultze](https://datatalks.club/people/maxschultze.html "Learn more about Max Schultze")
* [Mehdi OUAZZA](https://datatalks.club/people/mehdiouazza.html "Learn more about Mehdi OUAZZA")
* [Meor Amer](https://datatalks.club/people/meoramer.html "Learn more about Meor Amer")
* [Merel Theisen](https://datatalks.club/people/mereltheisen.html "Learn more about Merel Theisen")
* [Merve Noyan](https://datatalks.club/people/mervenoyan.html "Learn more about Merve Noyan")
* [Meryem Arik](https://datatalks.club/people/meryemarik.html "Learn more about Meryem Arik")
* [Meysam Asgari-Chenaghlu](https://datatalks.club/people/meysamasgarichenaghlu.html "Learn more about Meysam Asgari-Chenaghlu")
* [Michael Munn](https://datatalks.club/people/michaelmunn.html "Learn more about Michael Munn")
* [Michael Taylor](https://datatalks.club/people/michaeltaylor.html "Learn more about Michael Taylor")
* [Micheal Lanham](https://datatalks.club/people/micheallanham.html "Learn more about Micheal Lanham")
* [Miguel Morales](https://datatalks.club/people/miguelmorales.html "Learn more about Miguel Morales")
* [Mihail Eric](https://datatalks.club/people/mihaileric.html "Learn more about Mihail Eric")
* [Mikhail Sveshnikov](https://datatalks.club/people/mikhailsveshnikov.html "Learn more about Mikhail Sveshnikov")
* [Mikio Braun](https://datatalks.club/people/mikiobraun.html "Learn more about Mikio Braun")
* [Mısra Turp](https://datatalks.club/people/misraturp.html "Learn more about Mısra Turp")
* [Moein Foroughi](https://datatalks.club/people/moeinforoughi.html "Learn more about Moein Foroughi")
* [Nadia Nahar](https://datatalks.club/people/nadianahar.html "Learn more about Nadia Nahar")
* [Nakul Bajaj](https://datatalks.club/people/nakulbajaj.html "Learn more about Nakul Bajaj")
* [Naomi Nguyen](https://datatalks.club/people/naominguyen.html "Learn more about Naomi Nguyen")
* [Nastasia Saby](https://datatalks.club/people/nastasiasaby.html "Learn more about Nastasia Saby")
* [Natalie Kwong](https://datatalks.club/people/nataliekwong.html "Learn more about Natalie Kwong")
* [Nataliya Portman](https://datatalks.club/people/nataliyaportman.html "Learn more about Nataliya Portman")
* [Nathan Wang](https://datatalks.club/people/nathanwang.html "Learn more about Nathan Wang")
* [Neal Lathia](https://datatalks.club/people/neallathia.html "Learn more about Neal Lathia")
* [Nemanja Radojkovic](https://datatalks.club/people/nemanjaradojkovic.html "Learn more about Nemanja Radojkovic")
* [Niall Murphy](https://datatalks.club/people/niallmurphy.html "Learn more about Niall Murphy")
* [Nick Bilozerov](https://datatalks.club/people/nickbilozerov.html "Learn more about Nick Bilozerov")
* [Nick Singh](https://datatalks.club/people/nicksingh.html "Learn more about Nick Singh")
* [Nicolas Rassam](https://datatalks.club/people/nicolasrassam.html "Learn more about Nicolas Rassam")
* [Nielsen Aileen](https://datatalks.club/people/nielsenaileen.html "Learn more about Nielsen Aileen")
* [Nikita Iserson](https://datatalks.club/people/nikitaiserson.html "Learn more about Nikita Iserson")
* [Nikita Kozodoi](https://datatalks.club/people/nikitakozodoi.html "Learn more about Nikita Kozodoi")
* [Nikola Maksimovic](https://datatalks.club/people/nikolamaksimovic.html "Learn more about Nikola Maksimovic")
* [Nikolay Smorchkov](https://datatalks.club/people/nikolaysmorchkov.html "Learn more about Nikolay Smorchkov")
* [Ning Wang](https://datatalks.club/people/ningwang.html "Learn more about Ning Wang")
* [Nishant Mohan](https://datatalks.club/people/nishantmohan.html "Learn more about Nishant Mohan")
* [Noah Gift](https://datatalks.club/people/noahgift.html "Learn more about Noah Gift")
* [Noel Kwan](https://datatalks.club/people/noelkwan.html "Learn more about Noel Kwan")
* [Nour Karessli](https://datatalks.club/people/nourkaressli.html "Learn more about Nour Karessli")
* [Oleg Novikov](https://datatalks.club/people/olegnovikov.html "Learn more about Oleg Novikov")
* [Oleg Polivin](https://datatalks.club/people/olegpolivin.html "Learn more about Oleg Polivin")
* [Olga Ivina](https://datatalks.club/people/olgaivina.html "Learn more about Olga Ivina")
* [Olga Petrova](https://datatalks.club/people/olgapetrova.html "Learn more about Olga Petrova")
* [Ondřej Bothe](https://datatalks.club/people/ondrejbothe.html "Learn more about Ondřej Bothe")
* [Ondřej Kubera](https://datatalks.club/people/ondrejkubera.html "Learn more about Ondřej Kubera")
* [Orell Garten](https://datatalks.club/people/orellgarten.html "Learn more about Orell Garten")
* [Orlando Hohmeier](https://datatalks.club/people/orlandohohmeier.html "Learn more about Orlando Hohmeier")
* [Padma Chitturi](https://datatalks.club/people/padmachitturi.html "Learn more about Padma Chitturi")
* [Parul Pandey](https://datatalks.club/people/parulpandey.html "Learn more about Parul Pandey")
* [Parvathy Krishnan](https://datatalks.club/people/parvathykrishnan.html "Learn more about Parvathy Krishnan")
* [Pastor Soto](https://datatalks.club/people/pastorsoto.html "Learn more about Pastor Soto")
* [Patricio Cerda Mardini](https://datatalks.club/people/patriciocerdamardini.html "Learn more about Patricio Cerda Mardini")
* [Pauline Clavelloux](https://datatalks.club/people/paulineclavelloux.html "Learn more about Pauline Clavelloux")
* [Paul Iusztin](https://datatalks.club/people/pauliusztin.html "Learn more about Paul Iusztin")
* [Paul Orland](https://datatalks.club/people/paulorland.html "Learn more about Paul Orland")
* [Pavel Chernetsov](https://datatalks.club/people/pavelchernetsov.html "Learn more about Pavel Chernetsov")
* [Philippe Saadé](https://datatalks.club/people/philippesaade.html "Learn more about Philippe Saadé")
* [Phil Winder](https://datatalks.club/people/philwinder.html "Learn more about Phil Winder")
* [Pier Paolo Ippolito](https://datatalks.club/people/pierpaoloippolito.html "Learn more about Pier Paolo Ippolito")
* [Polina Mosolova](https://datatalks.club/people/polinamosolova.html "Learn more about Polina Mosolova")
* [Prasoon Shukla](https://datatalks.club/people/prasoonshukla.html "Learn more about Prasoon Shukla")
* [Prateek Joshi](https://datatalks.club/people/prateekjoshi.html "Learn more about Prateek Joshi")
* [Rachael Tatman](https://datatalks.club/people/rachaeltatman.html "Learn more about Rachael Tatman")
* [Rachel Lim](https://datatalks.club/people/rachellim.html "Learn more about Rachel Lim")
* [Raghav Bali](https://datatalks.club/people/raghavbali.html "Learn more about Raghav Bali")
* [Rahul Jain](https://datatalks.club/people/rahuljain.html "Learn more about Rahul Jain")
* [Ramiro Aznar](https://datatalks.club/people/ramiroaznar.html "Learn more about Ramiro Aznar")
* [Ranjitha Kulkarni](https://datatalks.club/people/ranjithakulkarni.html "Learn more about Ranjitha Kulkarni")
* [Raphaël Hoogvliets](https://datatalks.club/people/raphaelhoogvliets.html "Learn more about Raphaël Hoogvliets")
* [Reem Mahmoud](https://datatalks.club/people/reemmahmoud.html "Learn more about Reem Mahmoud")
* [Rileen Sinha](https://datatalks.club/people/rileensinha.html "Learn more about Rileen Sinha")
* [Rishabh Bhargava](https://datatalks.club/people/rishabhbhargava.html "Learn more about Rishabh Bhargava")
* [Rob De Wit](https://datatalks.club/people/robdewit.html "Learn more about Rob De Wit")
* [Rob Zinkov](https://datatalks.club/people/robzinkov.html "Learn more about Rob Zinkov")
* [Roksolana Diachuk](https://datatalks.club/people/roksolanadiachuk.html "Learn more about Roksolana Diachuk")
* [Roman Grebennikov](https://datatalks.club/people/romangrebennikov.html "Learn more about Roman Grebennikov")
* [Rosona Eldred](https://datatalks.club/people/rosonaeldred.html "Learn more about Rosona Eldred")
* [Ross Brigoli](https://datatalks.club/people/rossbrigoli.html "Learn more about Ross Brigoli")
* [Roy Jafari](https://datatalks.club/people/royjafari.html "Learn more about Roy Jafari")
* [Rui Machado](https://datatalks.club/people/ruimachado.html "Learn more about Rui Machado")
* [Ruslan Shchuchkin](https://datatalks.club/people/ruslanshchuchkin.html "Learn more about Ruslan Shchuchkin")
* [Rustem Feyzkhanov](https://datatalks.club/people/rustemfeyzkhanov.html "Learn more about Rustem Feyzkhanov")
* [Sabina Firtala](https://datatalks.club/people/sabinafirtala.html "Learn more about Sabina Firtala")
* [Sadat Anwar](https://datatalks.club/people/sadatanwar.html "Learn more about Sadat Anwar")
* [Sadik Bakiu](https://datatalks.club/people/sadikbakiu.html "Learn more about Sadik Bakiu")
* [Sage Elliott](https://datatalks.club/people/sageelliott.html "Learn more about Sage Elliott")
* [Sally-Ann DeLucia](https://datatalks.club/people/sallyanndelucia.html "Learn more about Sally-Ann DeLucia")
* [Sandra Kublik](https://datatalks.club/people/sandrakublik.html "Learn more about Sandra Kublik")
* [Santona Tuli](https://datatalks.club/people/santonatuli.html "Learn more about Santona Tuli")
* [Sara EL-ATEIF](https://datatalks.club/people/saraelateif.html "Learn more about Sara EL-ATEIF")
* [Sarah Mestiri](https://datatalks.club/people/sarahmestiri.html "Learn more about Sarah Mestiri")
* [Sara Menefee](https://datatalks.club/people/saramenefee.html "Learn more about Sara Menefee")
* [Sara Robinson](https://datatalks.club/people/sararobinson.html "Learn more about Sara Robinson")
* [Saurav Maheshkar](https://datatalks.club/people/sauravmaheshkar.html "Learn more about Saurav Maheshkar")
* [Savaş Yıldırım](https://datatalks.club/people/savasyildirim.html "Learn more about Savaş Yıldırım")
* [Sean Sheng](https://datatalks.club/people/seansheng.html "Learn more about Sean Sheng")
* [Sebastian Ayala Ruano](https://datatalks.club/people/sebastianayalaruano.html "Learn more about Sebastian Ayala Ruano")
* [Sebastian Raschka](https://datatalks.club/people/sebastianraschka.html "Learn more about Sebastian Raschka")
* [Sedat Kapanoglu](https://datatalks.club/people/sedatkapanoglu.html "Learn more about Sedat Kapanoglu")
* [Sejal Vaidya](https://datatalks.club/people/sejalvaidya.html "Learn more about Sejal Vaidya")
* [Serena Haidar](https://datatalks.club/people/serenahaidar.html "Learn more about Serena Haidar")
* [Sergei Boitsov](https://datatalks.club/people/sergeiboitsov.html "Learn more about Sergei Boitsov")
* [Sergei Shaikin](https://datatalks.club/people/sergeishaikin.html "Learn more about Sergei Shaikin")
* [Serg Masis](https://datatalks.club/people/sergmasis.html "Learn more about Serg Masis")
* [Shachar Meir](https://datatalks.club/people/shacharmeir.html "Learn more about Shachar Meir")
* [Shir Meir Lador](https://datatalks.club/people/shirmeirlador.html "Learn more about Shir Meir Lador")
* [Shubham Saboo](https://datatalks.club/people/shubhamsaboo.html "Learn more about Shubham Saboo")
* [Sidharth Ramachandran](https://datatalks.club/people/sidharthramachandran.html "Learn more about Sidharth Ramachandran")
* [Simon Stiebellehner](https://datatalks.club/people/simonstiebellehner.html "Learn more about Simon Stiebellehner")
* [Simon Thompson](https://datatalks.club/people/simonthompson.html "Learn more about Simon Thompson")
* [Sivan Biham](https://datatalks.club/people/sivanbiham.html "Learn more about Sivan Biham")
* [Sofya Yulpatova](https://datatalks.club/people/sofyayulpatova.html "Learn more about Sofya Yulpatova")
* [Soledad Galli](https://datatalks.club/people/soledadgalli.html "Learn more about Soledad Galli")
* [Sonal Goyal](https://datatalks.club/people/sonalgoyal.html "Learn more about Sonal Goyal")
* [Soumik Rakshit](https://datatalks.club/people/soumikrakshit.html "Learn more about Soumik Rakshit")
* [Srivathsan Canchi](https://datatalks.club/people/srivathsancanchi.html "Learn more about Srivathsan Canchi")
* [Stefan Gudmundsson](https://datatalks.club/people/stefangudmundsson.html "Learn more about Stefan Gudmundsson")
* [Stefanie Molin](https://datatalks.club/people/stefaniemolin.html "Learn more about Stefanie Molin")
* [Stefan Jansen](https://datatalks.club/people/stefanjansen.html "Learn more about Stefan Jansen")
* [Supreet Kaur](https://datatalks.club/people/supreetkaur.html "Learn more about Supreet Kaur")
* [Susan Walsh](https://datatalks.club/people/susanwalsh.html "Learn more about Susan Walsh")
* [Santiago Valdarrama](https://datatalks.club/people/svpino.html "Learn more about Santiago Valdarrama")
* [Shawn Swyx Wang](https://datatalks.club/people/swyx.html "Learn more about Shawn Swyx Wang")
* [Tamara Atanasoska](https://datatalks.club/people/tamaraatanasoska.html "Learn more about Tamara Atanasoska")
* [Tammy Liang](https://datatalks.club/people/tammyliang.html "Learn more about Tammy Liang")
* [Tanya Berger-Wolf](https://datatalks.club/people/tanyabergerwolf.html "Learn more about Tanya Berger-Wolf")
* [Tatiana Gabruseva](https://datatalks.club/people/tatianagabruseva.html "Learn more about Tatiana Gabruseva")
* [Tatyjana Ankudo](https://datatalks.club/people/tatyjanaankudo.html "Learn more about Tatyjana Ankudo")
* [Tereza Iofciu](https://datatalks.club/people/terezaiofciu.html "Learn more about Tereza Iofciu")
* [Theofilos Papapanagiotou](https://datatalks.club/people/theofilospapapanagiotou.html "Learn more about Theofilos Papapanagiotou")
* [Thomas Nield](https://datatalks.club/people/thomasnield.html "Learn more about Thomas Nield")
* [Thomas Wolf](https://datatalks.club/people/thomaswolf.html "Learn more about Thomas Wolf")
* [Thom Ives](https://datatalks.club/people/thomives.html "Learn more about Thom Ives")
* [Timothy Davis](https://datatalks.club/people/timothydavis.html "Learn more about Timothy Davis")
* [Tobias Zwingmann](https://datatalks.club/people/tobiaszwingmann.html "Learn more about Tobias Zwingmann")
* [Todd Underwood](https://datatalks.club/people/toddunderwood.html "Learn more about Todd Underwood")
* [Tomasz Hinc](https://datatalks.club/people/tomaszhinc.html "Learn more about Tomasz Hinc")
* [Tomasz Lelek](https://datatalks.club/people/tomaszlelek.html "Learn more about Tomasz Lelek")
* [Tomaz Bratanic](https://datatalks.club/people/tomazbratanic.html "Learn more about Tomaz Bratanic")
* [Tomek Jamiński](https://datatalks.club/people/tomekjaminski.html "Learn more about Tomek Jamiński")
* [Tommy Dang](https://datatalks.club/people/tommydang.html "Learn more about Tommy Dang")
* [Uri Gilad](https://datatalks.club/people/urigilad.html "Learn more about Uri Gilad")
* [Vadim Smolyakov](https://datatalks.club/people/vadimsmolyakov.html "Learn more about Vadim Smolyakov")
* [Valeriia Kuka](https://datatalks.club/people/valeriiakuka.html "Learn more about Valeriia Kuka")
* [Valerii Babushkin](https://datatalks.club/people/valeriybabushkin.html "Learn more about Valerii Babushkin")
* [Vanessa Aguilar](https://datatalks.club/people/vanessaaguilar.html "Learn more about Vanessa Aguilar")
* [Verena Weber](https://datatalks.club/people/verenaweber.html "Learn more about Verena Weber")
* [Victoria Perez Mola](https://datatalks.club/people/victoriaperezmola.html "Learn more about Victoria Perez Mola")
* [Vijay Kiran](https://datatalks.club/people/vijaykiran.html "Learn more about Vijay Kiran")
* [Ville Tuulos](https://datatalks.club/people/villetuulos.html "Learn more about Ville Tuulos")
* [Vincent Tatan](https://datatalks.club/people/vincenttatan.html "Learn more about Vincent Tatan")
* [Vincent Warmerdam](https://datatalks.club/people/vincentwarmerdam.html "Learn more about Vincent Warmerdam")
* [Vin Vashishta](https://datatalks.club/people/vinvashishta.html "Learn more about Vin Vashishta")
* [Violetta Mishechkina](https://datatalks.club/people/violettamishechkina.html "Learn more about Violetta Mishechkina")
* [Vishwas BV](https://datatalks.club/people/vishwasbv.html "Learn more about Vishwas BV")
* [Vladimir Haltakov](https://datatalks.club/people/vladimirhaltakov.html "Learn more about Vladimir Haltakov")
* [Wendy Mak](https://datatalks.club/people/wendymak.html "Learn more about Wendy Mak")
* [Willem Pienaar](https://datatalks.club/people/willempienaar.html "Learn more about Willem Pienaar")
* [Will McGugan](https://datatalks.club/people/willmcgugan.html "Learn more about Will McGugan")
* [Will Russell](https://datatalks.club/people/willrussell.html "Learn more about Will Russell")
* [Xia He-Bleinagel](https://datatalks.club/people/xiahebleinagel.html "Learn more about Xia He-Bleinagel")
* [Yuan Tang](https://datatalks.club/people/yuantang.html "Learn more about Yuan Tang")
* [Yulia Pavlova](https://datatalks.club/people/yuliapavlova.html "Learn more about Yulia Pavlova")
* [Yury Kashnitsky](https://datatalks.club/people/yurykashnitsky.html "Learn more about Yury Kashnitsky")
* [Zhamak Dehghani](https://datatalks.club/people/zhamakdehghani.html "Learn more about Zhamak Dehghani")
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ksenia Legostay – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ksenia Legostay
Ksenia is currently working as a Manager/Data Scientist at momox GmbH. After 4 years working as a Project Manager, she turned her carrier towards data science. To do this, she spent 3 years researching fraud and anomaly detection and earned a degree in data analysis.
[](https://twitter.com/ksenialegostay)
[](https://linkedin.com/in/ksenialeg)
### Articles
* [Regularization in Regression](https://datatalks.club/blog/regularization-in-regression.html)
### Events
* Transitioning from Project Management to Data Science ([watch on youtube](https://www.youtube.com/watch?v=rBKezdb9jEc)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Transitioning-from-Project-Management-to-Data-Science---Ksenia-Legostay-euig2a)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Kyle Shannon – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Kyle Shannon
Kyle Shannon is a highly motivated professional with a strong background in Software Engineering, Data Analytics & Marketing Tech. Genuine love for all things data, learning and improving every day. Always looking for new ways to extract value from existing and future data. Enjoy coaching and mentoring analysts & engineers.
[](https://twitter.com/ktshannon)
[](https://linkedin.com/in/ktshannon)
[](https://github.com/ktshannon)
### Events
* Modern Data Stack for Analytics Engineering ([watch on youtube](https://datatalks.club/people/kyleshannon.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lalit Pagaria – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lalit Pagaria
Lalit is creator of Obsei: an AI powered automation tool. He is community maintainer of Haystack: a framework for building search systems. Previously he designed low latency highly scalable backend system while working with Careem (an Uber company) and Ola. He also played with embedded system and Linux kernel while working with Brocade communication and Gridbots. He love playing chess in his free time.
[](https://twitter.com/pagarialalit)
[](https://linkedin.com/in/lalitpagaria)
[](https://github.com/lalitpagaria)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lars Albertsson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lars Albertsson
Lars Albertsson is the founder of Scling, a data engineering startup based in Stockholm. Scling provides data-value-as-a-service - customer tailored data engineering, analytics, and data science. Lars is a frequent conference speaker on data engineering and data strategy. Before founding Scling, Lars has worked at Google, Spotify, Schibsted, and as an independent consultant, helping organisations create value with data processing.
[](https://twitter.com/lalleal)
[](https://linkedin.com/in/larsalbertsson)
[](https://github.com/lallea)
[](https://www.scling.com/)
### Events
* DataOps 101 ([watch on youtube](https://www.youtube.com/watch?v=vyF3yGsF6UY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/DataOps-101---Lars-Albertsson-ethsp1)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Larysa Visengeriyeva – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Larysa Visengeriyeva
Larysa is working at INNOQ and her current interest is the intersection between Sofware Engineering and Machine Learning - MLOps. She holds a PhD in the field of Augmented Data Cleaning.
[](https://twitter.com/visenger)
[](https://linkedin.com/in/larysavisenger)
[](https://github.com/visenger)
[](https://ml-ops.org/)
### Events
* DataTalks.Club Conference: ML in Production ([watch on youtube](https://www.youtube.com/watch?v=og1DG1KZ71c)
)
* 10 Foundational Practices of Machine Learning Engineering ([watch on youtube](https://www.youtube.com/watch?v=4gHSUjNUopc)
)
* Back-of-the-Envelope Calculation For Machine Learning Projects ([watch on youtube](https://www.youtube.com/watch?v=LFFdahY0w7A)
)
* Effective Domain-Driven Design for Machine Learning Products ([watch on youtube](https://www.youtube.com/watch?v=LPW7DDCUXHY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Laurence Moroney – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Laurence Moroney
Laurence Moroney leads AI Advocacy at Google, with a vision to make AI easy for developers and to widen access to ML careers for everyone. He’s written dozens of programming books, the most recent being ‘AI and ML for Coders’ at O’Reilly. Laurence believes that MOOCs are one of the greatest ways to learn, and is excited to create TensorFlow Specializations with DeepLearning.AI on Coursera. When not working with technology, he’s an active member of the Science Fiction Writers of America, and has authored several sci-fi novels, and comics books, and a produced screenplay. Laurence is based in Washington state, where he drinks way too much coffee.
[](https://twitter.com/lmoroney)
[](https://linkedin.com/in/laurence-moroney)
[](https://github.com/lmoroney)
[](http://www.laurencemoroney.com/)
### Books
* [AI and Machine Learning for Coders](https://datatalks.club/books/20210412-ai-and-machine-learning-for-coders.html)
(the book of the week from 12 Apr 2021 to 16 Apr 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lavanya Gupta – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lavanya Gupta
Lavanya is a Carnegie Mellon University (CMU) alumni of the Language Technologies Institute (LTI). She currently works as a Sr. AI/ML Applied Scientist at JPMorgan Chase in their specialized Machine Learning Center of Excellence (MLCOE) vertical.
In addition to having a strong industrial research background of 5+ years, she is also an enthusiastic technical speaker. She has delivered talks at events such as Women in Data Science (WiDS) 2021, PyData, Illuminate AI 2021, TensorFlow User Group (TFUG), and MindHack! Summit. She also serves as a reviewer at top international NLP conferences (NeurIPS 2024, ICLR 2025, NAACL 2025). Additionally, through her collaborations with various prestigious organizations, like Anita BOrg and Women in Coding and Data Science (WiCDS), she is committed to mentoring young and aspiring machine learning enthusiasts.
[](https://twitter.com/lavanya_gupta18)
[](https://linkedin.com/in/lgupta18)
[](https://lava18.medium.com/)
### Events
* Build a Strong Career in Data ([watch on youtube](https://www.youtube.com/watch?v=ekG5zJioyFs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Leandro von Werra – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Leandro von Werra
Leandro von Werra is a machine learning engineer in the open source team at Hugging Face. He has several years of industry experience bringing NLP projects to production by working across the whole machine learning stack, and is the creator of a popular Python library called TRL that combines transformers with reinforcement learning.
[](https://twitter.com/lvwerra)
[](https://linkedin.com/in/lvwerra)
[](https://github.com/lvwerra)
### Books
* [Natural Language Processing with Transformers](https://datatalks.club/books/20220425-natural-language-processing-with-transformers.html)
(the book of the week from 25 Apr 2022 to 29 Apr 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Leonard Püttmann – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Leonard Püttmann
Leonard Püttmann is a Data Scientist at Kern AI, where they build an amazing platform to accelerate your LLM adoption. He is interested in all things Machine Learning, especially when it comes to Natural Language Processing. He’s also a passionate tea drinker and always happy to talk about tech over a hot cup of tea!
[](https://linkedin.com/in/leonard-p%C3%BCttmann-4648231a9)
### Events
* RAG in Action: Next-Level Retrieval Augmented Generation ([watch on youtube](https://datatalks.club/people/leonardputtmann.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Leon Wei – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Leon Wei
Leon Wei is the founder of instamentor.com, a career coaching platform that connects young professionals to tech industry veterans.
Before launching instamentor, he was a senior machine learning manager at Apple. He led teams of world-class AI researchers and engineers to use machine learning to empower Apple’s multi-billion-dollar businesses.
He also worked as a research scientist at Amazon, where he worked on a pricing engine responsible for millions of products’ prices in real-time.
Leon studied advanced mathematics from the College of William Mary, where he graduated in 2008 with a master’s degree in science.
[](https://twitter.com/theleonwei)
[](https://linkedin.com/in/theleonwei)
[](https://github.com/theleonwei)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lera Kaimashnіkova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lera Kaimashnіkova
I’m an e-commerce Product Owner with a focus on site search optimization, analytics, team management, and product development. I work on creating better user experiences and boosting conversion rates by implementing features like autocomplete, and search filters, and improving site speed.
I collaborate with developers, QA, BA, backend, frontend, analytics, and UX designers. My communication skills and experience in project management help me get projects done efficiently and successfully.
As a product manager, I develop roadmaps and strategies that align with the goals of the business and the needs of customers. I’m skilled in Python, SQL, Google Sheets, and Power BI, which help me analyze data and track performance metrics. I’m also familiar with HTML, CSS, JavaScript, and Figma, allowing me to contribute to a variety of projects in e-commerce.
[](https://linkedin.com/in/leracaiman)
### Events
* From Marketing to Product Owner in Search ([watch on youtube](https://www.youtube.com/watch?v=-HbQQ_bVdfE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lewis Tunstall – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lewis Tunstall
Lewis Tunstall is a machine learning engineer at Hugging Face. He has built machine learning applications for startups and enterprises in the domains of NLP, topological data analysis, and time series. Lewis has a PhD in theoretical physics and has held research positions in Australia, the USA, and Switzerland. His current work focuses on developing tools for the NLP community and teaching people to use them effectively.
[](https://twitter.com/_lewtun)
[](https://linkedin.com/in/lewis-tunstall)
[](https://github.com/lewtun)
### Books
* [Natural Language Processing with Transformers](https://datatalks.club/books/20220425-natural-language-processing-with-transformers.html)
(the book of the week from 25 Apr 2022 to 29 Apr 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Liesbeth Dingemans – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Liesbeth Dingemans
Liesbeth Dingemans is an accomplished strategy and AI leader passionate about using technology to drive sustainable and meaningful impact. Over the past six years, she has worked at the intersection of business strategy, artificial intelligence, and sustainability—helping organizations translate bold visions into measurable results. Currently leading Dingemans Consulting, she provides fractional leadership and advisory services in strategy, fundraising, and go-to-market development for deeptech and AI startups. Her work emphasizes ethical innovation and long-term value creation, bridging the gap between vision and execution.
Previously, Liesbeth served as VP of Revenue and Chief of Staff at Source.ag, an AI-native agritech company focused on sustainable and climate-resilient food production. There, she led commercial growth and fundraising efforts, closing a €25M Series A and driving 8x revenue growth within six months. Before Source.ag, Liesbeth was Head of AI Strategy and Innovation Projects at Prosus, where she helped portfolio companies identify and scale AI opportunities responsibly. She began her career as a strategy consultant at McKinsey & Company, advising leading organizations on digital and analytics transformations across multiple sectors.
With an academic foundation that uniquely combines Physics and Art History, Liesbeth has always sought to unite analytical problem-solving with creative thinking. She continues to champion technology that advances sustainability, equity, and human progress—helping organizations grow with purpose in an increasingly AI-driven world.
[](https://linkedin.com/in/liesbeth-dingemans)
### Events
* Innovation and Design for Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=tcqBfZw41FM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lina Weichbrodt – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lina Weichbrodt
Generalist machine learning developer with an interest in prototyping data driven products and bringing them live. She likes to cover the whole process from design, implementation to A/B test and operations. Stack: AWS, Java, Python, Spark.
[](https://twitter.com/rmminusrslash)
[](https://linkedin.com/in/lina-weichbrodt-344a066a)
[](https://github.com/rmminusrslash)
### Events
* DataTalks.Club Summer Marathon: Machine Learning in Production ([watch on youtube](https://www.youtube.com/watch?v=jQDkBpzK-7w)
)
* Humans in the Loop ([watch on youtube](https://www.youtube.com/watch?v=o50j_Ndx2Hg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Humans-in-the-Loop---Lina-Weichbrodt-e14npgp)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lindsay McQuade – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lindsay McQuade
With over 20 years of experience spanning management consulting, higher education, and tech, Lindsay McQuade is a Transformational Coach who helps professionals navigate change, gain promotions, and reach their full potential.
Before launching her private coaching practice, she spent four years as Senior Career and Development Coach at SPICED Academy, where she designed and delivered a new training program for over 500 participants annually—earning a 94% “very good” or “excellent” rating. She also championed a dedicated communication skills module that was later adopted across multiple programs due to its strong impact.
Earlier in her career, she led Careers and Professional Development at Cass Business School, where she transformed the MBA and Masters program feedback ratings from 2.4 to 4.8. She also co-founded two successful ventures, including Veo Elite Ltd, which delivered high-quality corporate events for clients like GE, De’Longhi, and Electrolux.
Starting her professional journey at Accenture, she rose from junior consultant to senior manager, leading projects for major organizations such as the BBC, London Stock Exchange, and Thames Water.
Today, she combines her corporate insight, coaching expertise, and entrepreneurial mindset to help clients build confidence, resilience, and leadership presence—particularly in fast-paced, high-stakes environments.
[](https://linkedin.com/in/lindsay-mcquade)
### Events
* Career Coaching ([watch on youtube](https://www.youtube.com/watch?v=_U8GrYJvmJM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Loïc Magnien – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Loïc Magnien
Loïc is the Lead Data at Mylight150 and before that, had 10 years experience in the data space in various rôles : database management, data engineering, product owner, tech lead. data architect and lead data.
[](https://linkedin.com/in/https://www.linkedin.com/in/loicmagnien/)
### Events
* Building a Group Wide Data Lakehouse for Data Science & BI ([watch on youtube](https://datatalks.club/people/loicmagnien.html)
)
* From Data Manager to Data Architect ([watch on youtube](https://www.youtube.com/watch?v=qWG--iYO2uc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lisa Cohen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lisa Cohen
Lisa Cohen is a Director of Data Science for Twitter. She leads an organization of 70 data scientists, focused on driving the strategy and direction of the Twitter product. Her team uses machine learning, experimentation and causal analyses, and partners closely with Engineering, Product, Design and Research. Before Twitter, Lisa led the Azure Customer Growth Analytics organization as part of Microsoft Cloud Data sciences. Her team was responsible for analyzing OKRs, informing data-driven decisions, and developing data science models to help customers be successful on Azure. Lisa worked at Microsoft for 17yrs, and also helped develop multiple versions of Visual Studio. She holds Bachelor and Masters degrees from Harvard in Applied Mathematics. You can follow Lisa on LinkedIn and Medium.
[](https://twitter.com/lisafeig)
[](https://linkedin.com/in/cohenlisa)
### Events
* Designing a Data Science Organization ([watch on youtube](https://www.youtube.com/watch?v=F_rJ4fg5ZEA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Lior Barak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Lior Barak
Lior Barak is the author of “Data is Like a Plate of Hummus” a book about setting a data strategy for non-business functions, the co-host of “WHAT the Data?!” Podcast talking about data and business growth, and the founder of Tale About Data.
Lior has experience of over 12 years in building data teams, leveraging data for growth, and a lot of pains and learnings which brought him to write his book and advocate for a change in the way companies use data.
[](https://twitter.com/liorb)
[](https://linkedin.com/in/liorbarak)
[](https://www.taleaboutdata.com/)
### Events
* Effective Communication with Business for Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=gqroEsTyLD0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Effective-Communication-with-Business-for-Data-Professionals---Lior-Barak-e1002rm)
)
* Mindful Data Strategy: From Pipelines to Business Impact ([watch on youtube](https://www.youtube.com/watch?v=B76J4QkZPWs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Loris Marini – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Loris Marini
Loris Marini is the CEO and founder of Discovering Data and he is on a mission to build a bridge between business leaders and data leaders. Loris hosts the Discovering Data podcast - a show for business leaders and for data professionals that want to turn data into outcomes. Discover the mindset, skills, and strategies of the most successful leaders in the world and learn how to maximize the impact of your work.
[](https://twitter.com/lorismarini)
[](https://linkedin.com/in/lorismarini)
[](https://github.com/LorisMarini)
### Events
* Business Skills for Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=xMYRUiTu960)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Luca Massaron – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Luca Massaron
Having joined Kaggle over 10 years ago, Luca Massaron is a Kaggle Grandmaster in discussions and a Kaggle Master in competitions and notebooks. In Kaggle competitions he reached no. 7 in the worldwide rankings. On the professional side, Luca is a data scientist with more than a decade of experience in transforming data into smarter artifacts, solving real-world problems, and generating value for businesses and stakeholders. He is a Google Developer Expert(GDE) in machine learning and the author of best-selling books on AI, machine learning, and algorithms.
[](https://twitter.com/lucamassaron)
[](https://linkedin.com/in/lmassaron)
[](https://github.com/lmassaron)
### Books
* [The Kaggle Book](https://datatalks.club/books/20220919-kaggle-book.html)
(the book of the week from 19 Sep 2022 to 23 Sep 2022)
* [Machine Learning for Tabular Data](https://datatalks.club/books/20250505-machine-learning-for-tabular-data.html)
(the book of the week from 05 May 2025 to 09 May 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Luis Serrano – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Luis Serrano
Luis G. Serrano is a research scientist in quantum artificial intelligence at Zapata Computing. He has worked previously as a Machine Learning Engineer at Google, as a Lead Artificial Intelligence Educator at Apple, and as the Head of Content in Artificial Intelligence and Data Science at Udacity. Luis has a PhD in mathematics from the University of Michigan, a bachelor’s and master’s in mathematics from the University of Waterloo, and worked as a postdoctoral researcher at the Laboratoire de Combinatoire et d’Informatique Mathématique at the University of Quebec at Montreal. Luis maintains a popular YouTube channel about machine learning with over 75,000 subscribers and over 3 million views, and is a frequent speaker at artificial intelligence and data science conferences.
[](https://twitter.com/luis_likes_math)
[](https://linkedin.com/in/luisgserrano)
[](https://github.com/luisguiserrano)
[](http://serrano.academy/)
### Books
* [Grokking Machine Learning](https://datatalks.club/books/20210809-grokking-machine-learning.html)
(the book of the week from 09 Aug 2021 to 13 Aug 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Luís Oliveira – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Luís Oliveira
I am a data engineer working with data since 2017. Even not having a software engineer degree I was able to start my career as an IT professional. This was possible due to lots of persistent, e-learning courses and reading articles.
My main goal is to be a top data engineer with intensive learning and support to new professionals in this area.
[](https://linkedin.com/in/https://www.linkedin.com/in/lgsoliveira/)
[](https://github.com/https://github.com/guoliveira)
[](https://medium.com/@lgsoliveira)
### Articles
* [How to run PostgreSQL and PgAdmin with Docker](https://datatalks.club/blog/how-to-run-postgresql-and-pgadmin-with-docker.html)
* [Guidelines to Get a Data Engineer Job Against the Odds](https://datatalks.club/blog/guidelines-to-get-data-engineer-job-against-odds.html)
* [Important SQL Fact That Everyone Should Know](https://datatalks.club/blog/important-sql-fact-that-everyone-should-know.html)
* [Do You Know the Golden Rules While Working With Data?](https://datatalks.club/blog/do-you-know-golden-rules-while-working-with-data.html)
* [How to Setup a Lightweight Local Version for Airflow](https://datatalks.club/blog/how-to-setup-lightweight-local-version-for-airflow.html)
* [Data Engineers Aren't Plumbers](https://datatalks.club/blog/data-engineers-arent-plumbers.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Luke Whipps – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Luke Whipps
Luke Whipps is a co-founder of Neural.AI – a company specializing in recruiting data scientists and other AI professionals. He has 8+ years of experience working as a recruiter. Luke is also the host of the AI Game Changer podcast.
[](https://linkedin.com/in/lukewhipps)
### Events
* Standing out as a Data Scientist ([watch on youtube](https://www.youtube.com/watch?v=Sb4CJlonB3c)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Standing-out-as-a-Data-Scientist---Luke-Whipps-envr7e)
)
* Preparing for a Data Science Interview ([watch on youtube](https://www.youtube.com/watch?v=NnZjlMowkWA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Madiha Khalid – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Madiha Khalid
Madiha is an experienced data engineer working as a Fractional Data Engineering Lead. Her role involves helping startups build their data platforms at different stages. She inspires them to recognize the potential of data platforms in the age of AI, especially in their early stages. She is helping an early-stage startup manage and lead a scalable end-to-end data platform that integrates AI and ML models to prepare for launch this year.
Madiha has spent the last ten years building performant, trust-worthy, and reliable data platforms, including data warehouses and data lakes for companies such as Jimdo, Eurecat, and Jazz (Telecom Enterprise). She is also a mentor for aspiring engineers, focusing on empowering fresh graduates and juniors, especially women, to kick-start their journey as engineers.
[](https://linkedin.com/in/madihakh)
### Events
* Unlocking the Door to Your Kick-Start Data Engineering Career ([watch on youtube](https://datatalks.club/people/madihakhalid.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Magdalena Konkiewicz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Magdalena Konkiewicz
Magdalena is a Data Evangelist at Toloka, a global data labeling company serving around 2,000 large and small businesses worldwide. Magdalena holds a Master’s degree in Artificial Intelligence from Edinburgh University. She’s worked as an NLP Engineer, Developer, and Data Scientist for businesses in Europe and America. She now teaches and mentors Data Scientists, and regularly contributes to publications like Towards Data Science.
[](https://twitter.com/MagdalenaKonki1)
[](https://linkedin.com/in/magdalena-konkiewicz-644678125)
[](https://medium.com/@konkiewicz.m)
### Events
* Monitoring Model Performance with Crowdsourcing ([watch on youtube](https://www.youtube.com/watch?v=sFh7F7pJ6JI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Magdalena Kuhn – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Magdalena Kuhn
Magdalena is a Senior Machine Learning Engineer @ BMW
Formerly a pure data scientist, Magdalena transitioned to a Machine Learning Engineer role during her last job at Delivery Hero in 2020. Now, at BMW she works as a senior machine learning engineer on building, operating and scaling a Machine Learning Platform. Thereby, she benefits from her knowledge in data science and DevOps to push and spread MLOps principles within BMW. Outside of work, she loves both exploring vibrant big cities as well as enjoying calm and picturesque places of nature.
[](https://linkedin.com/in/magdalenadeschner)
### Events
* Building and Scaling a Machine Learning Platform ([watch on youtube](https://www.youtube.com/watch?v=bNrBJwiLBWU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mahmoud AbdelAziz – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mahmoud AbdelAziz
Mahmoud has 8+ years of spearheading projects & products in many fields of Machine Vision, Robotics, Artificial intelligence, Smart Manufacturing, Quality Control and Software development. He started his first startup QEYE that builds Machine vision solutions for the textile industry. Then, he founded DevisionX for building quality inspection systems using a mix of Machine Vision &AI that can be applied in many industries. Also he was a partner at Digified, that is using computer Vision and AI in FinTech for digital identity verification.
[](https://linkedin.com/in/mahmoudaziz)
### Events
* DataTalks.Club Conference: ML Use Cases ([watch on youtube](https://www.youtube.com/watch?v=jvqS1_GnLsk)
)
* Build Your AI Machine Vision System by Yourself ([watch on youtube](https://www.youtube.com/watch?v=GPeJKcBvKrw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Manmohan Gosada – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Manmohan Gosada
Manmohan Gosada is a seasoned professional with a proven track record in the dynamic field of data science. With a comprehensive background spanning various data science functions and industries, Manmohan has emerged as a leader in driving innovation and delivering impactful solutions. He has successfully led large-scale data science projects, leveraging cutting-edge technologies to implement transformative products.
With a postgraduate degree, he is not only well-versed in the theoretical foundations of data science but is also passionate about sharing insights and knowledge. A captivating speaker, he engages audiences with a blend of expertise and enthusiasm, demystifying complex concepts in the world of data science.
[](https://linkedin.com/in/manmohan-gosada-491556160)
### Books
* [Data-Centric Machine Learning with Python](https://datatalks.club/books/20240408-data-centric-machine-learning-with-python.html)
(the book of the week from 08 Apr 2024 to 12 Apr 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Marco De Sa – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Marco De Sa
Marco is the Chief Data Officer at OLX Group. Before that, he worked at Spotify, Twitter, and Facebook.
[](https://linkedin.com/in/marcodesa)
### Events
* Chief Data Officer ([watch on youtube](https://www.youtube.com/watch?v=IdaZOD46FEw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Chief-Data-Officer---Marco-De-Sa-e16hm4t)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Marcello La Rocca – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Marcello La Rocca
Marcello is a senior software engineer at Tundra.com. He’s the author of “Algorithms and Data Structures in Action”. His work and interests focus on graphs, optimization algorithms, genetic algorithms, machine learning, and quantum computing. In his career he has contributed to large-scale web applications at companies like Twitter, Microsoft and Apple, also working on data infrastructure and applied research in both academia and industry. He authored the Neatsort adaptive sorting algorithm.
[](https://twitter.com/mlarocca)
[](https://linkedin.com/in/marcellolarocca)
[](https://github.com/mlarocca)
[](https://dev.to/mlarocca)
### Events
* Mastering Algorithms and Data Structures ([watch on youtube](https://www.youtube.com/watch?v=RiQa-9LguW8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Mastering-Algorithms-and-Data-Structures---Marcello-La-Rocca-e16s7lf)
)
### Books
* [Advanced Algorithms and Data Structures](https://datatalks.club/books/20210531-advanced-algorithms-and-data-structures.html)
(the book of the week from 31 May 2021 to 04 Jun 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Maria Bruckert – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Maria Bruckert
Maria-Liisa Bruckert is the Co-Founder and Co-CEO of the Health Tech company SQIN. She is skilled in Digital Strategy, Strategic Planning, Business Development, Start-up Consulting, and Digitization. Additionally, she is a strong business development professional focusing on Business and Strategy Development.
Maria-Liisa is also a recipient of the Google Female Founder Immersion 2020, the Google Play Best of 2020 Award with the AI (Machine Learning) based application SQIN, and one of Industry Era’s 10 most admired women leaders 2021.
[](https://twitter.com/BruckertMaria)
[](https://linkedin.com/in/mariabruckert)
### Events
* AI for Digital Health ([watch on youtube](https://www.youtube.com/watch?v=whpkDmVVGUE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Manoj Kukreja – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Manoj Kukreja
Manoj Kukreja is a Principal Architect at Northbay Solutions who specializes in creating complex Data Lakes and Data Analytics Pipelines for large-scale organizations such as banks, insurance companies, universities, and US/Canadian government agencies. Previously, he worked for Pythian, a large managed service provider where he was leading the MySQL and MongoDB DBA group and supporting large-scale data infrastructure for enterprises across the globe. With over 25 years of IT experience, he has delivered Data Lake solutions using all major cloud providers including AWS, Azure, GCP, and Alibaba Cloud. On weekends, he trains groups of aspiring Data Engineers and Data Scientists on Hadoop, Spark, Kafka and Data Analytics on AWS and Azure Cloud.
[](https://linkedin.com/in/manoj-kukreja-cloud-data-architect-and-data-scientist-8225604)
[](https://github.com/mkukreja1)
### Books
* [Data Engineering with Apache Spark, Delta Lake, and Lakehouse](https://datatalks.club/books/20220314-data-engineering-with-apache-spark-delta-lake-and-lakehouse.html)
(the book of the week from 14 Mar 2022 to 18 Mar 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Marianna Diachuk – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Marianna Diachuk
Marianna is a data scientist at Restream and Data Science Lead and mentor in the local branch of Women Who Code community. Former data scientist in DataRobot and former lead of a data science team in a fintech startup so she loves to share not just her technical experience. She is passionate about science, traveling and writing.
[](https://twitter.com/dark_matter88_)
[](https://linkedin.com/in/marianna-diachuk-53ba60116)
[](https://github.com/DarkMatter88)
### Events
* Introducing Data Science in Startups ([watch on youtube](https://youtube.com/watch?v=KMSE9GkU2mE)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Introducing-Data-Science-in-Startups---Marianna-Diachuk-e17rc4i)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mariano Semelman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mariano Semelman
Mariano is head of data science at OLX Group. He has over 13 years of experience, with almos
[](https://linkedin.com/in/msemelman)
### Events
* Becoming a Data Science Manager ([watch on youtube](https://www.youtube.com/watch?v=qOLR84-KHoY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-Science-Manager---Mariano-Semelman-e1cbrf7)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Maria Sukhareva – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Maria Sukhareva
Maria Sukhareva is a principal key expert in Artificial Intelligence in Siemens with over 15 years of experience at the forefront of generative AI technologies. Known for her keen eye for technological innovation, Maria excels at transforming cutting-edge AI research into practical, value-driven tools that address real-world needs. Her approach is both hands-on and results-focused, with a commitment to creating scalable, long-term solutions that improve communication, streamline complex processes, and empower smarter decision-making. Maria’s work reflects a balanced vision, where the power of innovation is met with ethical responsibility, ensuring that her AI projects deliver impactful and production-ready outcomes.
[](https://twitter.com/sukharevamaria)
[](https://linkedin.com/in/maria-sukhareva-a532b1100)
### Events
* AI in Industry: Trust, Return on Investment and Future ([watch on youtube](https://www.youtube.com/watch?v=bT7-HRNCltk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Maria Vechtomova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Maria Vechtomova
I’m passionate about MLOps and think about it like an organizational challenge rather than a technological challenge.
[](https://linkedin.com/in/maria-vechtomova)
### Events
* Pragmatic and Standardized MLOps ([watch on youtube](https://www.youtube.com/watch?v=q3DTR3Od1MA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Marijn Markus – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Marijn Markus
Marijn Markus is an AI Lead and Managing Data Scientist at Capgemini, where he helps organizations harness artificial intelligence and data for social and business transformation. Trained as both a sociologist and a data scientist, Marijn brings a human-centered perspective to AI, striving to build systems that are not only innovative but also ethical, inclusive, and impactful.
With more than five years of experience in machine learning, NLP, and data strategy, Marijn has led multiple high-impact projects across industries. Most notably, he helped create Project FARM, an AI and Big Data platform designed to support more than 30,000 smallholder farmers in Kenya. The project earned recognition as a European finalist in the IBM Watson Build competition, was featured at World Summit AI and IBM THINK, and won the La Niaque NL Award for innovation.
Beyond his technical achievements, Marijn is a passionate communicator and public speaker who regularly shares insights on the intersection of AI, ethics, and human behavior. Known for his engaging talks, visuals, and coaching, he aims to inspire others to see AI not as science fiction—but as a tool to make life better, fairer, and more meaningful through the responsible use of data.
[](https://twitter.com/MarijnMarkus)
[](https://linkedin.com/in/marijnmarkus)
### Events
* Doing Good with Data ([watch on youtube](https://www.youtube.com/watch?v=esv5-4HRXvI)
)
* Hacking Your Data Career ([watch on youtube](https://www.youtube.com/watch?v=RhSg8ill1So)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mario Lazo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mario Lazo
Howdy! Meet Mario Lazo, an Enterprise Transformation Architect with over 20 years of experience making technology work better for businesses. Mario has a gift for bringing together technical innovation and business value. He’s an expert in areas like intelligent automation, robotic process automation (RPA), cloud SaaS ERP, and AI governance.
Mario’s tech journey started at Accenture, Information Resources, and the University of Chicago. There, he sharpened his skills in various roles, including developer, product owner, and technical architect. He mastered systems thinking and data-driven approaches, excelling in systems data integration, eCommerce, data warehousing, product management, and cloud ERP solutions.
His career journey led to senior positions at NetSuite (now Oracle), where he managed global practices across the US, the Philippines, Czech Republic, India, and Uruguay. Mario built data integrations and AI-driven solutions for retail, manufacturing, finance, and healthcare sectors.
At UiPath and Blue Prism, Mario shined as a Customer Success Director and Senior Program Leader. He organized impactful innovation events for top executives and created comprehensive guides for citizen developers. Now, he leads a global portfolio of cloud-based AI and RPA implementations, achieving significant cost savings and risk reduction.
Mario’s educational background includes an MBA in Finance and Marketing from Loyola University Chicago and a Bachelor’s in Management Information Systems from Ateneo de Manila University. He’s also a certified Project Management Professional (PMP) and Certified Scrum Product Owner (CSPO), showing his dedication to systems excellence.
When he’s not diving into tech, Mario loves photography, traveling the world, mentoring, and giving back to the Philippines. He also enjoys spending quality time with his loved ones and his four cats.
Mario’s unique perspective and compassionate approach make his work a must-read for anyone interested in navigating the world of AI, business transformation and data privacy.
[](https://linkedin.com/in/mariolazo)
### Books
* [AI Data Privacy and Protection](https://datatalks.club/books/20240715-ai-data-privacy-and-protection.html)
(the book of the week from 15 Jul 2024 to 19 Jul 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mark Ryan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mark Ryan
Mark Ryan has 20 years of experience leading technical teams in the areas of relational database and machine learning. Mark is a Data Science Manager at Intact.
[](https://twitter.com/MarkRyanMkm)
[](https://linkedin.com/in/mark-ryan-31826743)
[](https://github.com/ryanmark1867)
### Books
* [Deep Learning with Structured Data](https://datatalks.club/books/20210118-deep-learning-structured-data.html)
(the book of the week from 18 Jan 2021 to 22 Jan 2021)
* [Deep Learning with fastai Cookbook](https://datatalks.club/books/20211206-deep-learning-with-fastai-cookbook.html)
(the book of the week from 06 Dec 2021 to 10 Dec 2021)
* [Machine Learning for Tabular Data](https://datatalks.club/books/20250505-machine-learning-for-tabular-data.html)
(the book of the week from 05 May 2025 to 09 May 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Martin Kleppmann – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Martin Kleppmann
Martin Kleppmann is a researcher in distributed systems at the University of Cambridge. Previously he was a software engineer and entrepreneur at Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure. In the process he learned a few things the hard way, and he hopes this book will save you from repeating the same mistakes.
Martin is a regular conference speaker, blogger, and open source contributor. He believes that profound technical ideas should be accessible to everyone, and that deeper understanding will help us develop better software.
You can find him as [@martinkl](https://twitter.com/martinkl)
on Twitter, and his blog is at [martin.kleppmann.com](http://martin.kleppmann.com/)
.
[](https://twitter.com/martinkl)
[](https://linkedin.com/in/martinkleppmann)
[](https://martin.kleppmann.com/)
### Books
* [Designing Data-Intensive Applications](https://datatalks.club/books/20210308-designing-data-intensive-applications.html)
(the book of the week from 08 Mar 2021 to 12 Mar 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Martin Potančok – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Martin Potančok
Martin Potančok supports data-inspired decisions in commercial and research projects. He has experience in software development and analytics. He has worked as a business analyst and project manager on software projects delivering mainly budgeting and reporting systems for international companies. Recently, he has been working as a business analyst on data and analytics projects. At the Prague University of Economics and Business, he is in charge of data activities and research projects organized in cooperation with the Faculty of Informatics and Statistics.
[](https://linkedin.com/in/martinpotancok)
### Books
* [Data Analytics Initiatives](https://datatalks.club/books/20220801-data-analytics-initiatives.html)
(the book of the week from 01 Aug 2022 to 05 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Marysia Winkels – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Marysia Winkels
Marysia completed a Master’s Degree in Artificial Intelligence at the University of Amsterdam, with a focus on data-efficient deep learning. She now works as a Lead Data Scientist at GoDataDriven, where she find out. She also has a strong interest in educating and teaching, both as part of her current role at GoDataDriven and as co-organizer of PyData Amsterdam and PyData Global
[](https://twitter.com/artemiish)
[](https://linkedin.com/in/marysia-winkels)
[](https://marysia.nl/)
### Events
* Data-Centric AI ([watch on youtube](https://www.youtube.com/watch?v=t3HDdVWQzNM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mary Jane Dykeman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mary Jane Dykeman
Mary Jane is co-founder of Canari AI, an AI risk impact solution.
Mary Jane Dykeman is a managing partner at INQ Law. In addition to data law, she is a long-standing health lawyer. Her data practice focuses on privacy, artificial intelligence (AI), cyber preparedness and response, and data governance. She regularly advises on use and disclosure of identifiable and de-identified data. Mary Jane applies a strategic, risk and innovation lens to data and emerging technologies. She helps clients identify the data they hold, understand how to use it within the law, and how to innovate responsibly. In her health law practice, Mary Jane focuses on clinical and enterprise risk, privacy and information management, consent, capacity and substitute decision-making, and counseling through difficult situations, including at the end of life. She currently acts as VP Legal, Chief Legal/Risk to a Toronto teaching hospital; and was instrumental in the development of Ontario’s health privacy legislation.
Mary Jane regularly consults on large data initiatives and use of data for health research and quality purposes. Her consulting work extends to modernizing privacy legislation and digital societies, and she works with Boards, CEOs and CIOs on the emerging risks and trends in data. Mary Jane regularly speaks on AI, cyber risk and how to better engage and build trust with clients and customers whose data is at play. She is also a frequent speaker and writer on health law and data law.
[](https://twitter.com/mjdykeman)
[](https://linkedin.com/in/mary-jane-dykeman-80ab1b22)
### Events
* Building the End to end Roadmap for your Data ([watch on youtube](https://www.youtube.com/watch?v=xiKcS1c0sBk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Matt Harrison – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Matt Harrison
Matt Harrison is a corporate trainer, consultant, and author who has worked with various startups architecting data science, business intelligence, storage, open source stack support, search and high availability. My posts tend to be on subjects related to these. I also might post on my hobbies, which include frisbee (mostly ultimate and goaltimate), gardening, and outdoor activities (skiing, trail running, hiking, climbing, biking, etc). I have a computer science degree from Stanford.
[](https://twitter.com/__mharrison__)
[](https://linkedin.com/in/panela)
[](https://github.com/mattharrison)
[](https://hairysun.com/)
### Books
* [Effective Pandas](https://datatalks.club/books/20220131-effective-pandas.html)
(the book of the week from 31 Jan 2022 to 04 Feb 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Matthew Housley – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Matthew Housley
Matt Housley is a data engineering consultant and cloud specialist. After some early programming experience with Logo, Basic and 6502 assembly, he completed a PhD in mathematics at the University of Utah. Matt then began working in data science, eventually specializing in cloud based data engineering. He co-founded Ternary Data with Joe Reis, where he leverages his teaching experience to train future data engineers and advise teams on robust data architecture. Matt and Joe also pontificate on all things data on The Monday Morning Data Chat.
[](https://twitter.com/doctorhousley)
[](https://linkedin.com/in/housleymatthew)
### Books
* [Fundamentals of Data Engineering](https://datatalks.club/books/20220815-fundamentals-of-data-engineering.html)
(the book of the week from 15 Aug 2022 to 19 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Matt Palmer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Matt Palmer
Matt Palmer is a developer relations engineer at Replit, with a background in product analytics and data engineering. In his free time, he enjoys writing, climbing, & hiking. He lives outside San Francisco and spends most of his free weekends exploring California.
[](https://twitter.com/mattppal)
[](https://linkedin.com/in/matt-palmer)
### Events
* Make Data Magical with Mage ([watch on youtube](https://www.youtube.com/watch?v=JKALtxziBG0)
)
### Books
* [Understanding ETL](https://datatalks.club/books/20240415-understanding-etl.html)
(the book of the week from 15 Apr 2024 to 19 Apr 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Maxime Labonne – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Maxime Labonne
Maxime Labonne is Head of Post-Training at Liquid AI. He holds a Ph.D. in Machine Learning from the Polytechnic Institute of Paris and is a Google Developer Expert in AI/ML. He has made significant contributions to the open-source community, including the LLM Course, tutorials on fine-tuning, tools such as LLM AutoEval, and several state-of-the-art models like NeuralDaredevil. He is the author of the best-selling books “LLM Engineer’s Handbook” and “Hands-On Graph Neural Networks Using Python”.
[](https://twitter.com/maximelabonne)
[](https://linkedin.com/in/maxime-labonne)
[](https://mlabonne.github.io/blog/)
### Books
* [LLM Engineer's Handbook](https://datatalks.club/books/20241104-llm-engineer-s-handbook.html)
(the book of the week from 04 Nov 2024 to 08 Nov 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Maxim Lukichev – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Maxim Lukichev
Max is a co-founder and CTO of Telmai (telm.ai). He holds a Ph.D. in Computer Science (Database theory) and spent over a decade building SaaS products with focus on big data analytics and ML in companies like Veeva, Reltio, SignalFx, and Splunk. Throughout his career, Max observed transformations of the companies from very early stages to acquisition and IPO.
[](https://linkedin.com/in/maximlukichev)
### Events
* Building Streaming Analytics: The Journey and Learnings ([watch on youtube](https://www.youtube.com/watch?v=DN_p_tFSw6c)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Max Schultze – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Max Schultze
Max Schultze is a data engineering manager working on building a data lake at Zalando, Europe’s biggest online platform for fashion. His focus lies on building data pipelines at petabytes scale and productionizing Spark and Presto on Delta Lake inside the company. He is a public advocate of the Data Mesh paradigm, contributing to its development through conference appearances and online trainings. Max originally graduated from the Humboldt University of Berlin, actively taking part in the university’s initial development of Apache Flink.
[](https://twitter.com/mcs1408)
[](https://linkedin.com/in/max-schultze-b11996110)
### Events
* Data Mesh in Practice ([watch on youtube](https://www.youtube.com/watch?v=ekEc8D_D3zY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mehdi OUAZZA – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mehdi OUAZZA
I’m a data geek entrepreneur who’s passionate about Big data, Data science, Web App and Music. With more than 7 years of experience, I got the opportunity to work on multiple aspect regarding data engineering, that includes data pipelines (stream/batch), data modeling, orchestration, infrastructure and from time to time analytical reports using dashboarding tools.
I always keep my feet on the ground and take some times to do technology watch: continuous learning is my motto. I’m also a startup entrepreneur, meetup lover and blogger: sharing is caring.
[](https://twitter.com/mehd_io)
[](https://linkedin.com/in/mehd-io)
[](https://github.com/mehd-io)
[](https://mehd.io/)
### Articles
* [What Open Source Can Do For Your Data Career](https://datatalks.club/blog/what-open-source-can-do-for-your-data-career.html)
### Events
* Growing Data Engineering Team in a Scale-Up ([watch on youtube](https://www.youtube.com/watch?v=acJ6sVqKOUk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Meor Amer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Meor Amer
Meor Amer is an educator, author, and Developer Relations professional at Cohere, where he helps enterprises and developers integrate large language models into real-world applications. His journey into AI began in 2010 after his son was born with a limb difference, inspiring him to study neurotechnology at Imperial College London, where he earned an MSc with distinction. Since then, Meor has dedicated his career to making artificial intelligence approachable through visuals, storytelling, and interactive learning.
Before joining Cohere, Meor spent over a decade in the telecom analytics industry, providing data-driven solutions and training to clients in more than 15 countries. He later founded kDimensions and Edsquare, platforms focused on simplifying AI and data science concepts through visual learning and gamified education. His acclaimed book, A Visual Introduction to Deep Learning, is used by readers from leading companies like Apple, Meta, and Wells Fargo.
Meor’s work bridges AI education, creative communication, and technical enablement. Through his visual-first approach, he empowers learners, educators, and organizations to understand the fundamentals of machine learning and large language models—turning complex ideas into clear, engaging stories that inspire curiosity and confidence in AI.
[](https://twitter.com/MeorAmer1)
[](https://linkedin.com/in/meoramer)
[](https://kdimensions.com/)
### Events
* Visualising Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=OuCuk-7RHjM)
)
### Books
* [A Visual Introduction to Deep Learning](https://datatalks.club/books/20220214-a-visual-introduction-to-deep-learning.html)
(the book of the week from 14 Feb 2022 to 18 Feb 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Merel Theisen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Merel Theisen
I am a Software Engineer working with the Kedro team at QuantumBlack. Before I joined the Kedro team I worked on our in-house experiment and model performance tracking tool. During my career as software engineer I’ve developed tools in various languages (Java, Scala, and Python), for various use cases from social networking for academia to targeted advertising. I’ve always been interested in machine learning and love contributing to the space from a software engineering perspective. I really enjoy creating things and helping others.
[](https://linkedin.com/in/merel-theisen-30087b52)
[](https://github.com/MerelTheisenQB)
[](http://www.mereltheisen.com/about)
### Events
* Building Machine Learning Pipelines with Kedro ([watch on youtube](https://www.youtube.com/watch?v=AUmDliHzWp0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Merve Noyan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Merve Noyan
Merve Noyan is a Google Developer Expert on Machine Learning and a graduate student in Data Science. She currently works in iamyiam as a Machine Learning Engineer with an NLP focus. She streams on twitch, giving workshops on computer science, statistics, and data science. She is also the host of the Inference Podcast, hosting professionals in data science & ML.
[](https://twitter.com/mervenoyann)
[](https://linkedin.com/in/merve-noyan-28b1a113a)
[](https://github.com/merveenoyan)
[](https://mervenoyan.me/)
### Events
* Conversational AI ([watch on youtube](https://www.youtube.com/watch?v=Jrigg7n-bt8)
)
* Developer Advocacy Engineer for Open-Source ([watch on youtube](https://www.youtube.com/watch?v=SnEYvF-Ztb8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Meryem Arik – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Meryem Arik
Meryem is a recovering physicist and the co-founder of TitanML - TitanML is an NLP development platform that focuses on deployability of LLMs - allowing businesses to build smaller and cheaper deployments of language models with ease.
[](https://linkedin.com/in/meryemarik)
### Events
* LLMs for Everyone ([watch on youtube](https://www.youtube.com/watch?v=6dn6uZFkk04)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Michael Munn – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Michael Munn
Mike is an ML solutions engineer in Google Cloud. He helps customers design, implement, and deploy machine learning models and teaches the ML Immersion Program in Google’s Advanced Solutions Lab.
[](https://linkedin.com/in/munnm)
### Books
* [Machine Learning Design Patterns](https://datatalks.club/books/20210208-ml-design-patterns.html)
(the book of the week from 08 Feb 2021 to 12 Feb 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Michael Taylor – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Michael Taylor
I graduated with a master’s degree in Economics, then worked across a number of growth roles at startups, before co-founding a marketing agency and growing it to 50 people. After exiting I created courses on LinkedIn and Udemy taken by over 350,000 people, wrote a self-published book called Marketing Memetics, and a book for O’Reilly on Prompt Engineering for Generative AI. Today, I’m focused on building AI products through Brightpool.
[](https://linkedin.com/in/https://www.linkedin.com/in/mjt145/)
### Books
* [Prompt Engineering for Generative AI](https://datatalks.club/books/20240701-prompt-engineering-for-generative-ai.html)
(the book of the week from 01 Jul 2024 to 05 Jul 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Micheal Lanham – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Micheal Lanham
Micheal Lanham is a best-selling author, innovator, and AI engineer based in Calgary, Canada. His work spans games, graphics, GIS, enterprise software, and machine learning. He has published over ten technical books, including Evolutionary Deep Learning, Hands-On Reinforcement Learning for Games, and AI Agents in Action.
Micheal has worked as a lead AI developer, architect, and manager across industries from oil and gas to fintech, and today focuses on building intelligent systems with deep reinforcement learning, evolutionary methods, and generative AI.
[](https://linkedin.com/in/micheal-lanham-189693123)
### Events
* Lessons from Two Decades of AI ([watch on youtube](https://www.youtube.com/watch?v=DSxqUlumM3A)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Meysam Asgari-Chenaghlu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Meysam Asgari-Chenaghlu
Meysam Asgari-chenaghlu is an AI Manager at Carbon Consulting and is also a PhD candidate at the University of Tabriz. He has worked on one of Turkey’s leading telecommunication companies Chatbot NLU project and on Turkey national Search Engine project.
[](https://linkedin.com/in/meysam-ac)
[](https://github.com/MeysamAsgariC)
### Books
* [Mastering Transformers](https://datatalks.club/books/20211011-mastering-transformers.html)
(the book of the week from 11 Oct 2021 to 15 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Miguel Morales – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Miguel Morales
Miguel Morales works on reinforcement learning at Lockheed Martin, Missiles and Fire Control, Autonomous Systems, in Denver, Colorado. He recently published the book Grokking Deep Reinforcement Learning with Manning Publications. He is a part-time Instructional Associate at the Georgia Institute of Technology, where he teaches the graduate course in Reinforcement Learning and Decision Making. Miguel has worked for Udacity as a reviewer, mentor, and more recently, as a Content Developer for the Deep Reinforcement Learning Nanodegree. He graduated from Georgia Tech with a Master’s in Computer Science, specializing in Interactive Intelligence.
[](https://twitter.com/mimoralea)
[](https://linkedin.com/in/mimoralea)
[](https://www.youtube.com/c/mimoralea)
### Books
* [Grokking Deep Reinforcement Learning](https://datatalks.club/books/20210517-grokking-deep-reinforcement-learning.html)
(the book of the week from 17 May 2021 to 21 May 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mihail Eric – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mihail Eric
Mihail Eric is the founder of Pametan Data Innovation, a machine learning consultancy focused on helping organizations build data-driven systems to solve their toughest business problems. He’s also a founder of Confetti AI, the premier educational platform for practitioners learning the skills to succeed in their machine learning and data science careers.
He built his first machine learning model in 2012 and has never turned back. Over the years, Mihail’s career has spanned machine learning industry, research, and engineering across domains such as conversational AI and self-driving vehicles. He has published papers at the top conferences in the world including ACL, AAAI, and NeurIPS and has helped start teams at innovative companies like RideOS and Amazon Alexa. Systems that he has architected are used by hundreds of thousands of people globally.
Mihail also believes in paying things forward and actively blogs/speaks about complex topics in engineering, data science, and anything else he’s learning.
[](https://twitter.com/mihail_eric)
[](https://linkedin.com/in/mihaileric)
[](https://github.com/mihail911)
[](https://www.mihaileric.com/)
### Events
* What Researchers and Engineers Can Learn from Each Other ([watch on youtube](https://www.youtube.com/watch?v=d9xVXqKq3sU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-Researchers-and-Engineers-Can-Learn-from-Each-Other---Mihail-Eric-e1854bj)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mikhail Sveshnikov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mikhail Sveshnikov
Mikhail Sveshnikov is an AI engineer at Evidently.ai with 10+ years in ML and MLOps, focused on building developer tools for reliable and measurable AI in production.
[](https://linkedin.com/in/mike0sv)
[](https://github.com/evidently)
[](https://www.evidentlyai.com/)
### Events
* Automated Prompt Optimization with Evidently AI ([watch on youtube](https://www.youtube.com/watch?v=uMNYVw4jh-8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mikio Braun – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mikio Braun
Mikio Braun has worked as both a ML researcher and in various industry positions to put machine learning products into production. Previously he has worked at two of Europe’s unicorn startups, Zalando and GetYourGuide. He is currently consulting companies on how to make the best use of machine learning, looking at the whole picture from technical infrastructure to product development.
[](https://twitter.com/mikiobraun)
[](https://linkedin.com/in/mikiobraun)
[](https://github.com/mikiobraun)
[](https://margint.blog/)
### Events
* DataTalks.Club Conference: ML in Production ([watch on youtube](https://www.youtube.com/watch?v=og1DG1KZ71c)
)
* Putting Data Science in Production ([watch on youtube](https://www.youtube.com/watch?v=gFuEgIeZzIo)
)
* Freelancing in Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=HfF791e0HR8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Freelancing-in-Machine-Learning---Mikio-Braun-e166n7r)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mısra Turp – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Mısra Turp
Mısra Turp is a Data Scientist and content creator. After years working as a data scientist for international companies in the Netherlands, she has started her own platform for teaching data science and guiding new-comers into the field. her website “So you want to be a data scientist?” has grown into a full-blown platform of videos, blog posts and online courses. On top of this, she also works as a developer advocate at AssemblyAI, a company that makes a state-of-the-art Speech-to-text API.
[](https://twitter.com/misraturp)
[](https://linkedin.com/in/misraturp)
[](https://github.com/misraturp)
[](https://www.soyouwanttobeadatascientist.com/)
### Events
* Data Scientists at Work ([watch on youtube](https://www.youtube.com/watch?v=oUycqtMoYr8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Moein Foroughi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Moein Foroughi
Moein Foroughi is a DevOps engineer focused on automation and scalable systems, with a professional interest in applying AI and modern technologies to improve engineering workflows and operational efficiency.
[](https://linkedin.com/in/moein-foroughi-ce)
### Events
* n8n: From Fundamentals to Building Intelligent Automation Pipeline ([watch on youtube](https://www.youtube.com/watch?v=KR9ApZXsV8g)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nadia Nahar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nadia Nahar
Nadia Nahar is a Software Engineering Ph.D. Student, Institute for Software Research, School of Computer Science, Carnegie Mellon University
[](https://twitter.com/NadiaIIT)
[](https://linkedin.com/in/nadia-nahar-iit)
[](https://sites.google.com/view/nadianahar)
### Events
* SE4ML - Software Engineering for Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=35Ch8xL2SA8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nakul Bajaj – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nakul Bajaj
Nakul Bajaj is a data scientist, machine learning operations engineer, educator and a mentor, helping junior data scientists and data engineers to navigate their data journey.
He has over eight years experience in the data science and technology domain, working on growing number of projects. He led a customer analytics team for bank’s corporate and institutional customers for a year. He was involved in building a custom blockchain solution to reduce cash flow lead time for agriculture customer of the bank. During his banking career, his most valuable contribution was building a scalable and self service Business Intelligence software on the cloud. This solution will provide custom reporting and insights, to over six thousand business customers.
Post banking he has been involved in building Machine learning(ML) engineering and Machine Learning Operations (MLOPS) practices in the health industry. He has tutored Data Science graduates at Deakin University in 2017 and The University of Queensland in 2022. Every year Nakul volunteers his time to deliver a lecture on “Starting in Data Science” and other data topics to graduates at RMIT University in Melbourne. He is quite active in the data community and mentors students from different universities across Australia.
Nakul holds a Masters degree in Information Systems from Deakin, with a Business Analytics Specialisation as well as Bachelors degree in Information Systems from Monash University. He also holds a nano degree in Data Engineering from Udacity E-learning platform.
He has been a business manager and left the family business in India to kickstart a career in analytics in Australia. His business management career made him learn about pain points, when humans are involved in manual and repetitive processes, which led him to chase technology and automation.
He is currently leveraging his passion to reduce complexity from machine learning use-cases in business, by making data science and data engineering, easy to do and understand across the organisation. He believes, that by raising data science and Artificial Intelligence awareness world can be a better place.
[](https://linkedin.com/in/nakul-bajaj)
### Books
* [Data-Centric Machine Learning with Python](https://datatalks.club/books/20240408-data-centric-machine-learning-with-python.html)
(the book of the week from 08 Apr 2024 to 12 Apr 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Naomi Nguyen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Naomi Nguyen
A machine learning practitioner with emphasis on model evaluation and imbalance class problem. Unusual events, model overfitting, and outdated modeling due to changing underlying distribution of new data are some of my top area of concern. I’m always a fast and curious learner, an eager problem-solver, and a believer in communication, transparency, and constructive criticism.
I enjoy teaching mathematics and statistics as well as having discussions on new fields.
[](https://linkedin.com/in/naomi7777)
[](https://github.com/naominguyen7)
### Events
* The War of Gradient Boosted Trees ([watch on youtube](https://www.youtube.com/watch?v=croe7mMze6s)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Natalie Kwong – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Natalie Kwong
Natalie is a Growth Product Manager at Airbyte focused on building the user experience and overseeing analytics. She previously held Analytics and Operations roles at Harness, KeepTruckin, and AppDynamics. Her expertise is in scaling analytics teams and systems from the ground up. She currently resides in New York, and while she’s not at work, you can find her creating pottery, plant shopping, or playing recreational dodgeball.
[](https://linkedin.com/in/nataliekwong)
### Events
* Making Sense of Data Engineering Acronyms and Buzzwords ([watch on youtube](https://www.youtube.com/watch?v=t9Z1S3OYnJU)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Making-Sense-of-Data-Engineering-Acronyms-and-Buzzwords---Natalie-Kwong-e177303)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nataliya Portman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nataliya Portman
Nataliya received her Doctoral Degree in Applied Mathematics from the University of Waterloo in 2010, followed by postdoctoral training at the Neurological Institute in Montreal. Following her postdoctoral assignment, she developed a novel approach to brain tissue classification in early childhood brain MRIs using modern Computer Vision pattern recognition and perceptual image quality models. Nataliya has worked in many industries including neuroscience, biotech, the public sector, and various start-up software companies. Throughout her career, she has applied her expertise in Mathematics to develop numerous models including but not limited to machine learning algorithms, computationally efficient algorithms for model validation, and neural networks. She is the co-inventor of “Bid-Assist”, a strategy for setting up an initial bidding amount to discourage low bidding behaviour, and “AutoVision”, a mobile app that allows automatic taking of pictures of vehicle views and damages recognized by an image classifier. Nataliya paved a new way for Data Science in incentives industry. She developed predictive analytics tools that help channel leaders maximize the return on investment of their channel incentive programs. In January 2021, Nataliya joined the Cineplex Digital Media as a Senior Data Scientist committed to the development of media content recommendation systems.
[](https://linkedin.com/in/nataliyaportman)
[](https://github.com/nportman)
### Events
* Recommender Systems for Digital Advertising ([watch on youtube](https://www.youtube.com/watch?v=96HudrBW3rU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nathan Wang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nathan Wang
Nathan Wang is the Director of Data Science & Analytics at Twilio Segment, where he has been having a ball of a time harnessing Segment’s Customer Data Platform to build data products that help go-to-market partners drive positive business outcomes.
Prior to Segment, Nathan spent three years on the Data Science & Analytics team at Opendoor where he worked on pricing optimization and risk mitigation. Perhaps somewhat unusually, Nathan started his career as a Macroeconomist, but then decided that instead of passively describing how ‘total factor productivity’ had been stagnating, that he should be part of the real economy and become a growth-driver.
Outside of work, Nathan enjoys traveling and going on new adventures with his wife and two-year-old son.
[](https://linkedin.com/in/nathanyinghengwang)
### Events
* Identity Resolution Essentials from a Data Scientist ([watch on youtube](https://www.youtube.com/watch?v=hHvvXagqY9k)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nastasia Saby – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nastasia Saby
Nastasia used to be a software backend developer. Now, She’s a machine learning engineer specialised in productionizing models. Her role is to help others to productionize safely predictive systems.
She has a blog named Machine Learning in Real Life and a newsletter focused on data science in business, DataOps, and MLOps.
[](https://twitter.com/saby_nastasia)
[](https://linkedin.com/in/nastasia-saby-b4220132)
[](https://github.com/NastasiaSaby)
[](https://mlinreallife.github.io/)
### Events
* Productionizing ML Systems without Fear nor Heroism ([watch on youtube](https://www.youtube.com/watch?v=KkOGMaz4Xws)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Neal Lathia – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Neal Lathia
Neal Lathia is Director of Machine Learning at Monzo in London. He and his team are focusing on building machine learning systems to help make money work for everyone.
[](https://twitter.com/neal_lathia)
[](https://linkedin.com/in/nlathia)
[](https://github.com/nlathia)
[](https://nlathia.github.io/)
### Events
* Machine Learning for Customer Service ([watch on youtube](https://www.youtube.com/watch?v=ohjSvKdUumY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nemanja Radojkovic – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nemanja Radojkovic
Nemanja, originally from Belgrade, Serbia, has been living in Leuven, Belgium since 2014. His professional journey spans from being an Electrical Engineer to transitioning into roles as a Data Scientist and MLOps Engineer. Despite dropping out of a Ph.D. program, Nemanja has diverse experience as a former salesman and consultant in both Big4 and boutique firms. Alongside his main roles, he engages in teaching Data Science, Machine Learning, and Python programming, contributing two courses to DataCamp. In 2011, he humorously mentions being “bit by a radioactive Python.” Outside of work, Nemanja holds a purple belt in Brazilian Jiu-Jitsu and is a proud father of three.
[](https://linkedin.com/in/radojkovic)
### Events
* Machine Learning Engineering in Finance ([watch on youtube](https://www.youtube.com/watch?v=Nl4aibeFwiI)
)
* MLOps in Corporations and Startups ([watch on youtube](https://www.youtube.com/watch?v=DX9c__a4jzg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Niall Murphy – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Niall Murphy
Niall Richard Murphy is an award-winning author, speaker, technologist, and executive leader. He is best known as the instigator, editor, and co-author of the best-selling and industry-defining Site Reliability Engineering book, published with O’Reilly, as well as its successor volume The Site Reliability Workbook. He has devoted over twenty years of mission-critical engineering roles to the craft of software, large-scale service operations, teaching, and diversity in computing.
[](https://twitter.com/niallm)
[](https://linkedin.com/in/niallm)
### Books
* [Reliable Machine Learning](https://datatalks.club/books/20221121-reliable-machine-learning.html)
(the book of the week from 05 Dec 2022 to 09 Dec 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nick Bilozerov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nick Bilozerov
Originally from Ukraine. When was a child dreamed about building computer games, but after finishing masters in Computer Science ended up building shopping cards and booking systems. Overal 9 years of experience as a Software Engineer. In the last 5 years focused on Data and ML field. Was working for Flixbus and a retail startup similar to Zalando. Last 3 years working for Delivery Hero in Product Data and ML team, part of Global Data Department. At Delivery Hero focused on data and ML infrastructure, data pipelines and helping out DS to deliver ML solutions for our customers.
[](https://linkedin.com/in/nick-bilozerov)
### Events
* Notebooks in Production - Fun or not Fun? ([watch on youtube](https://www.youtube.com/watch?v=yy88MqWQ-eM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nick Singh – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nick Singh
Nick Singh started his career as a Software Engineer on Facebooks’ Growth Team, and most recently, worked at SafeGraph, a location analytics startup. He graduated from the University of Virginia with a degree in Systems Engineering, and a minor in Computer Science and Applied Math. In college, he interned at Microsoft and on the Data Infrastructure team at Google’s Nest Labs.
[](https://linkedin.com/in/nipun-singh)
[](https://www.acethedatascienceinterview.com/)
### Events
* Ace Non-Technical Data Science Interviews ([watch on youtube](https://www.youtube.com/watch?v=tRdLVUKU7Bo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Ace-Non-Technical-Data-Science-Interviews---Nick-Singh-e1a5qtd)
)
### Books
* [Ace The Data Science Interview](https://datatalks.club/books/20211115-ace-the-data-science-interview.html)
(the book of the week from 15 Nov 2021 to 19 Nov 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nicolas Rassam – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nicolas Rassam
Nicolas Rassam is a seasoned technical recruiter and talent partner with over ten years of experience hiring for some of Europe’s leading AI and technology organizations. Currently a Senior Talent Acquisition Partner at Helsing, he leads recruitment for artificial intelligence and software engineering roles, helping the company attract top talent in defense technology and applied AI. Nicolas also advises startups and scale-ups through Fledge, where he supports hiring strategy, talent branding, and team-building initiatives.
Before joining Helsing, Nicolas played a key role in scaling AI and engineering teams at Onfido and Criteo, where he was among the first members of the R&D talent acquisition team. At Criteo, he helped build Criteo AI Lab, recruiting globally across Europe, North America, and Asia for top machine learning researchers, software engineers, and data scientists. His hands-on technical understanding, combined with his experience across in-house and consulting roles, allows him to connect deeply with both hiring managers and candidates.
Driven by a belief that innovation thrives in diverse environments, Nicolas advocates for inclusive hiring practices and equitable access to opportunities in tech. Whether at AI conferences like NeurIPS or RecSys or within fast-moving internal teams, he’s known for his approachable communication style, international mindset, and ability to bring people and purpose together in the world of AI recruitment.
[](https://twitter.com/n_rassam)
[](https://linkedin.com/in/nicolasrassam)
### Events
* Recruiting Data Engineers ([watch on youtube](https://www.youtube.com/watch?v=hylxiu4VGTo)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nielsen Aileen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nielsen Aileen
Aileen has worked in corporate law, physics research labs, and, most recently, a variety of NYC tech startups. Her interests range from defensive software engineering to UX designs for reducing cognitive load to the interplay between law and technology. Aileen is currently working at an early-stage NYC startup that has something to do with time series data and neural networks. She also serves as chair of the New York City Bar Associationâs Science and Law committee, which focuses on how the latest developments in science and computing should be regulated and how such developments should inform existing legal practices. In the recent past, Aileen worked at mobile health platform One Drop and on Hillary Clinton’s presidential campaign. She is a frequent speaker at machine learning conferences on both technical and sociological subjects. She holds an A.B. from Princeton University and is A.B.D. in Applied Physics at Columbia University.
[](https://github.com/AileenNielsen)
### Books
* [Practical Fairness](https://datatalks.club/books/20220523-practical-fairness.html)
(the book of the week from 23 May 2022 to 27 May 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nikita Iserson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nikita Iserson
Nikita is a Lead Machine Learning Engineer at S&P Global with over 10 years of experience in software engineering, data warehouse development, data analytics, and machine learning. He has built demand forecasting, network analysis, recommender systems, digital twins, and much more covering a wide range of industries, including telecom, retail, and banking.
[](https://linkedin.com/in/nikita-iserson)
### Events
* Fraud Detection with Graph Features and GNN ([watch on youtube](https://www.youtube.com/watch?v=ykVq8PemUsU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nikita Kozodoi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nikita Kozodoi
Nikita Kozodoi is Applied Scientist at AWS, where he works on building Generative AI solutions to solve diverse business problems for customers across industries. He holds PhD in Machine Learning from Humboldt University of Berlin, and has earned 18 medals in different Kaggle competitions.
[](https://linkedin.com/in/kozodoi)
[](https://kozodoi.me/)
### Events
* Five Techniques for Improving RAG Chatbots ([watch on youtube](https://www.youtube.com/watch?v=xPYmClWk5O8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nikola Maksimovic – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nikola Maksimovic
TBA
[](https://linkedin.com/in/nikola-maksimovic-40188183)
### Events
* From Digital Marketing to Analytics Engineering ([watch on youtube](https://www.youtube.com/watch?v=GawJ7mG5ElQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nikolay Smorchkov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nikolay Smorchkov
Nikolay Smorchkov, MBA, is a seasoned product and technology leader with over a decade of experience in software and product development, as well as team management. He has served as CTO and co-founder of a web agency, leading diverse client projects from concept to deployment. Currently, he works as a Payment Software Leader at a B2B payments provider in Germany, combining his technical know-how with a strong business mindset. Holding an MBA and certifications in ITIL, AWS, ISO 27001, and Business Continuity Management, Nikolay has built and led teams across multiple European countries, gaining firsthand insight into cross-cultural collaboration and global product delivery.
Nikolay is passionate about fostering strong team practices, clear communication, and robust documentation to achieve the highest performance in software development. He regularly shares insights on AI, security, User Experience (UX), and global team dynamics. From dissecting AI-generated code risks to exposing poor digital form design.
Nikolay brings real-world examples and practical frameworks to his writing. His blend of hands-on development experience, management acumen, and business mindset makes him uniquely qualified to guide readers through the complexities of modern software product development.
[](https://linkedin.com/in/nikolay-smorchkov-mba)
[](http://www.smorchkov.com/)
### Books
* [Software Development at Rocket Speed](https://datatalks.club/books/20251006-software-development-at-rocket-speed.html)
(the book of the week from 06 Oct 2025 to 10 Oct 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ning Wang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ning Wang
Ning Wang is an Apache Committer, and part of the project management committee for the Apache Heron distributed stream processing engine. Ning is also a software engineer at Amplitude building real-time data pipelines. He was a key contributor of Apache Heron in Twitter’s Real-time Compute team.
[](https://linkedin.com/in/ningwang)
[](https://github.com/nwangtw)
### Books
* [Grokking Streaming Systems](https://datatalks.club/books/20220704-grokking-streaming-systems.html)
(the book of the week from 04 Jul 2022 to 08 Jul 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nishant Mohan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nishant Mohan
Nishant Mohan is a Data Science enthusiast with more than 4 years of experience in customer behavior and product analytics, leading a team of analysts to derive actionable outcomes from data to drive business.
[](https://linkedin.com/in/mohannishant)
[](https://github.com/mohannishant6)
### Articles
* [How I Landed a Job As a Product Analyst](https://datatalks.club/blog/landing-product-analyst-job.html)
* [Customer Segmentation with RFM+ and K-Means: 7 Segments from Gaming Data](https://datatalks.club/blog/segmentation.html)
### Events
* Customer Segmentation 2.0 ([watch on youtube](https://www.youtube.com/watch?v=pWqD7SGuihs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Noah Gift – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Noah Gift
Noah Gift is the founder of Pragmatic AI Labs. Noah Gift lectures at MSDS, at Northwestern, Duke MIDS Graduate Data Science Program, and the Graduate Data Science program at UC Berkeley and the UC Davis Graduate School of Management MSBA program, and UNC Charlotte Data Science Initiative. He is teaching and designing graduate machine learning, A.I., Data Science courses, and consulting on Machine Learning and Cloud Architecture for students and faculty. These responsibilities include leading a multi-cloud certification initiative for students.
[](https://linkedin.com/in/noahgift)
[](https://github.com/noahgift)
[](https://noahgift.com/)
### Events
* Becoming a Solopreneur in Data ([watch on youtube](https://www.youtube.com/watch?v=gCLUY37HGtw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Solopreneur-in-Data---Noah-Gift-e19gqbr)
)
### Books
* [Practical MLOps](https://datatalks.club/books/20210830-practical-mlops.html)
(the book of the week from 30 Aug 2021 to 03 Sep 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Noel Kwan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Noel Kwan
Noel works as a Database Kernel Engineer at RisingWave, building a cost-effective and scalable database solution designed to streamline data processing and query serving.
He Focuses on Performance and reliability and has worked on fuzzing tools such as SqlSmith and performance-related features such as Backfill. And previously studied at NUS, with a focus on Programming Languages
[](https://linkedin.com/in/noelkwan)
[](https://github.com/kwannoel)
### Events
* Stream Processing ([watch on youtube](https://datatalks.club/people/noelkwan.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Nour Karessli – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Nour Karessli
Nour is a senior applied scientist at Zalando focused on computer vision and machine learning-driven solutions to tackle the complex challenge of delivering size and fit advice to millions of customers. In parallel, she’s on the organizing committee of Zalando Data Science Community (DSC) and Zalando Women in Tech Employee Resource Group (ZWT ERG). Outside of Zalando, Nour co-organized Women in Computer Vision (WiCV) workshop at the CVPR conference aiming to raise the visibility of female researchers and sharing experiences between students and professionals. She completed a Master degree in Computer Science from Saarland University - Max Planck Institute for Informatics. In her free time, she enjoys walking in nature, video games, and cooking.
[](https://twitter.com/nour_karessli)
[](https://linkedin.com/in/nour-karessli-8a35b739)
### Events
* AI in Fashion - Size & Fit ([watch on youtube](https://datatalks.club/people/nourkaressli.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Oleg Novikov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Oleg Novikov
Oleg Novikov is the creator of Next Round - a free mock interview service that provides personalized feedback and learning materials to help you identify your strengths and move your career to the next level. Before Next Round, Oleg was a data science manager at Uber and worked as a data and software engineer in startups.
[](https://linkedin.com/in/olegnovikov)
[](https://www.nextround.cc/)
### Events
* What I Learned After Interviewing 300 Data Scientists ([watch on youtube](https://www.youtube.com/watch?v=AYi7b-8GPm4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-I-Learned-After-Interviewing-300-Data-Scientists---Oleg-Novikov-e10ctbs)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Oleg Polivin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Oleg Polivin
Data Scientist
[](https://linkedin.com/in/polivin)
[](https://github.com/olegpolivin)
[](https://medium.com/@olegpolivin)
### Articles
* [Python CI/CD with GitHub Actions: Pre-commit, Linters, and Pytest Guide](https://datatalks.club/blog/practical-guide-better-code.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Olga Ivina – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Olga Ivina
Olga Ivina leads is a Delivery Data Science Director at Microsoft, where she leads a team of exceptionally talented Data Scientists across EMEA. She joined Microsoft from Deloitte Consulting in Germany, where, amongst others, she helped develop and led the delivery practice for applied AI. Olga works in AI for over 16 years, and she has taken different roles in her career: from researcher to project manager, to people manager. She speaks several languages, and she firmly believes in the power of communication and human interaction. Olga holds a Ph.D. in Experimental Sciences and Sustainability, and she claims to be a true nerd, a forever scholar, and a boring bookworm.
[](https://twitter.com/olgaivina)
[](https://linkedin.com/in/olgaivina)
[](https://github.com/olgaivina)
### Events
* Hiring Data Science Talent ([watch on youtube](https://www.youtube.com/watch?v=Af9t9r2b0z0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Olga Petrova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Olga Petrova
A former theoretical physicist turned machine learning engineer, Olga is now building a smart data annotation platform at Scaleway as a technical product manager. On the community side, she enjoys blogging about the latest advancements in AI both in and out of working hours
[](https://twitter.com/0lga_petrova)
[](https://linkedin.com/in/olga-p-petrova)
[](https://github.com/opetrova)
[](https://medium.com/@olgapetrova_92798)
### Events
* Introduction to Transformers for NLP ([watch on youtube](https://www.youtube.com/watch?v=hGNycrko5kc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ondřej Bothe – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ondřej Bothe
Ondřej Bothe has spent his whole life working in analytics. He has worked in many different roles: from hands-on developer to manager leading a team responsible for project implementation and IT analytical landscape operation. He was on the customer side, receiving advice from a consultant, but he has also worn the consultant hat, advising clients on analytical issues across sectors. Thanks to his economic background, he has a unique view of analytical projects. He can combine the aspect of technology, the delivery approach, and an understanding of data with an economist’s perspective to ensure the best possible results for analytical insight consumers.
[](https://linkedin.com/in/obothe)
### Books
* [Data Analytics Initiatives](https://datatalks.club/books/20220801-data-analytics-initiatives.html)
(the book of the week from 01 Aug 2022 to 05 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ondřej Kubera – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ondřej Kubera
Ondřej Kubera has spent most of his career in IT delivery and consulting, focusing on analytics, including areas such as business intelligence, information management, and data governance. He is passionate about bridging the gap between the business and technical perspectives in the data analytics domain. He graduated from the Czech Technical University in Prague and has experience from a variety of hands-on engineering, as well as client-facing and managerial roles. He has led, designed, and consulted analytics initiatives in major consulting firms, a boutique data intelligence company, and a global pharma company.
[](https://twitter.com/ondrejkubera)
[](https://linkedin.com/in/ondrejkubera)
[](https://ondrejkubera.com/)
### Books
* [Data Analytics Initiatives](https://datatalks.club/books/20220801-data-analytics-initiatives.html)
(the book of the week from 01 Aug 2022 to 05 Aug 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Padma Chitturi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Padma Chitturi
Padma works with Indeed as Senior Manager of Enterprise Data Analytics, with a focus on building scalable data science platforms and a deep passion for addressing the business needs leveraging data driven decisioning. She has extensive experience as Data Engineer, Data Scientist and as data strategy consultant in her previous roles. She has the blend of engineering and product skills and constantly thrives for opportunities to test and build innovative data solutions in highly dynamic environment with bias towards action.
### Events
* Helping People Get Jobs with Data ([watch on youtube](https://www.youtube.com/watch?v=wVwTMkRS1VM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Orlando Hohmeier – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Orlando Hohmeier
Orlando has been developing complex, highly scalable software products for various industries, from automotive to telecommunications, for over 17 years. As a recognized full-stack expert for distributed systems, he is responsible for the engineering team. He strives for excellent solutions that make AI applicable in manufacturing.
[](https://twitter.com/orlandohohmeier)
[](https://linkedin.com/in/orlandohohmeier)
[](https://github.com/orlandohohmeier)
[](https://www.orlandohohmeier.com/)
### Events
* Federated Learning to the Rescue ([watch on youtube](https://www.youtube.com/watch?v=GtJsn3es2kA)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Orell Garten – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Orell Garten
Orell graduated in 2018 with a degree in Electrical Engineering, where he focused on simulation algorithms. He then began a PhD, but when COVID arrived, he decided to leave academia and look for new challenges. After that, he joined a government-funded startup program to explore how to turn scientific research into real products. Although that project didn’t go as planned, it taught him what he loves most: applying rigorous simulation methods to solve practical problems.
[](https://twitter.com/orgarten.bsky.social)
[](https://linkedin.com/in/orgarten)
[](https://github.com/orgarten)
[](https://orellgarten.com/)
### Events
* From Simulation Algorithms to Production-Grade Data Systems ([watch on youtube](https://www.youtube.com/watch?v=pkcpH5N-GP8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Parul Pandey – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Parul Pandey
Parul is currently working as a Data Scientist & Evangelist at H2O.ai, an Automated Machine learning-based start-up based headquartered in California. Academically, she is an electrical engineer having previously worked in the field of Power Distribution at Tata Power, India. She is also the founder of a non-profit called Women in Coding & Data Science (WiCDS), whose mission is to support & promote women & gender minorities in data science, machine learning, and AI space. Parul is also a Kaggle Grandmaster in the notebooks category and was one of Linkedin’s Top Voice in the Software Development category in 2019.
[](https://twitter.com/pandeyparul)
[](https://linkedin.com/in/parulpandeyindia)
[](https://github.com/parulnith)
[](https://parulpandey.com/)
### Events
* DataTalks.Club Conference: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=ltFkvoiA57M)
)
* Career Transitioning into Data Science ([watch on youtube](https://www.youtube.com/watch?v=slczbYNn_Kg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Parvathy Krishnan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Parvathy Krishnan
Parvathy is CTO at Analytics for a Better World, a non-profit organisation aiming to use analytics techniques to contribute to Sustainable Development Goals (SDGs)/ Parvathy and holds a Professional doctorate in data science from Eindhoven university of technology
[](https://twitter.com/parvathykkrish)
[](https://linkedin.com/in/parvathykrishnank)
[](https://github.com/parvathykrishnank)
### Events
* Analytics for a Better World ([watch on youtube](https://www.youtube.com/watch?v=b6x5zZ3C6sQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Pastor Soto – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Pastor Soto
Pastor Soto is a Machine Learning Engineer and mentor with a practical bias toward shipping. While studying medicine at the university, he took on side work that pulled him into machine learning. The inflection point was ML Zoomcamp: by publishing exercises and projects publicly, he attracted interviews and job offers.
Pastor’s work centers on production ML: moving data, wiring APIs, and feeding LLMs so applications run. He mentors with DeepLearning.AI, leads sections for Stanford’s Code in Place, and writes about focused learning and small, finished steps that compound.
[](https://twitter.com/PastorSotoB1)
[](https://linkedin.com/in/pastorsoto)
[](https://github.com/sotoblanco)
[](https://substack.com/@pastorsoto)
### Events
* From Medicine to Machine Learning: How Public Learning Turned into a Career ([watch on youtube](https://www.youtube.com/watch?v=5km62e4nDaw)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Patricio Cerda Mardini – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Patricio Cerda Mardini
Patricio Cerda-Mardini, Machine Learning Research Engineer @MindsDB. Since joining MindsDB in 2020, Patricio has become the core maintainer of its in-database predictive ML engine. His research interests include deep learning forecasting and model-agnostic calibration methods. His academic degree is focused on human-robot interaction and recommendation systems, areas in which he holds several publications.
[](https://twitter.com/paxcema)
[](https://linkedin.com/in/paxcema)
### Events
* Effective Machine Learning Inside Your Database ([watch on youtube](https://www.youtube.com/watch?v=hfIOh_r8NEE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Pauline Clavelloux – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Pauline Clavelloux
Pauline is a data science manager and a consultant at IBM where she has worked for 8 years. She’s also an indie hacker and is working on a few side hustles in crypto and generative AI. One of her latest projects allows uploading a few selfies and it generates a lot of cool and stylistic pictures.
[](https://twitter.com/Pauline_Cx)
[](https://linkedin.com/in/paulineclavelloux)
[](https://wintopy.io/)
### Events
* Indie Hacking ([watch on youtube](https://www.youtube.com/watch?v=KsV_SVXlTo8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Paul Iusztin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Paul Iusztin
Paul Iusztin is a senior ML and MLOps engineer with over seven years of experience building GenAI, Computer Vision and MLOps solutions. His latest contribution was at Metaphysic, where he served as one of their core engineers in taking large neural networks to production. He previously worked at CoreAI, Everseen, and Continental.
He is the Founder of Decoding ML, an educational channel on production-grade ML that provides posts, articles, and open-source courses to help others build real-world ML systems.
[](https://twitter.com/iusztinpaul)
[](https://linkedin.com/in/pauliusztin)
[](https://github.com/decodingml)
### Events
* [AI Engineering: Skill Stack, Agents, LLMOps, and How to Ship AI Products](https://luma.com/fx3wplwn)
on 26 Jan 2026
### Books
* [LLM Engineer's Handbook](https://datatalks.club/books/20241104-llm-engineer-s-handbook.html)
(the book of the week from 04 Nov 2024 to 08 Nov 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Pavel Chernetsov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Pavel Chernetsov
Pavel grew up in the United States and has been a professional translator and copywriter for over 6 years. He currently resides near the Black sea, in the mountains of Georgia, with his lovely wife. Pavel enjoys reading and writing about all topics related to data science, machine learning, and coding. In his free time, he likes to ride jet-skis and can make a mean shish kabob.
### Articles
* [The Hiring Process for Data Professionals](https://datatalks.club/blog/hiring-process-for-data-professionals.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Paul Orland – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Paul Orland
Paul Orland is a programmer, software entrepreneur, and math enthusiast. He is co-founder of Tachyus, a start-up building predictive analytics software for the energy industry. You can find him online at [www.paulor.land](https://paulor.land/)
.
[](https://linkedin.com/in/paul-orland-4b293a58)
[](https://github.com/orlandpm)
[](https://paulor.land/)
### Books
* [Math for Programmers](https://datatalks.club/books/20210215-math-for-programmers.html)
(the book of the week from 15 Feb 2021 to 19 Feb 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Philippe Saadé – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Philippe Saadé
Philippe Saadé is the AI/ML project manager at Wikimedia Deutschland. His current work focuses on making Wikidata accessible to AI application with projects like the Wikidata vector database and the Wikidata Model Context Protocol.
[](https://linkedin.com/in/philippesaade1998)
### Events
* [Fact-Checking with Wikidata](https://luma.com/7fs5v7os)
on 20 Jan 2026
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Phil Winder – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Phil Winder
Dr. Phil Winder is a multidisciplinary Software Engineer and Data Scientist. As the CEO of Winder Research, a Cloud-Native Data Science consultancy based in the UK, he helps startups and enterprises utilise Data Science. Through a combination of consulting and development they are able to grow and scale their business by improving their products and platforms. For the past 5 years, Phil has taught thousands of Engineers about Data Science in his range of Data Science training courses at conferences, in public, in private and on the online Safari learning platform. In these courses Phil focuses on the practicalities of using Data Science in industry on a wide range of topics from cleaning data all the way through to deep reinforcement learning.
[](https://twitter.com/drphilwinder)
[](https://linkedin.com/in/drphilwinder)
[](https://github.com/philwinder)
[](https://winderresearch.com/)
### Events
* DataTalks.Club Conference: ML Use Cases ([watch on youtube](https://www.youtube.com/watch?v=jvqS1_GnLsk)
)
* Industrial Applications of Reinforcement Learning ([watch on youtube](https://www.youtube.com/watch?v=ih6IEv89IV4)
)
### Books
* [Reinforcement Learning](https://datatalks.club/books/20210111-reinforcement-learning.html)
(the book of the week from 11 Jan 2021 to 15 Jan 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Prasoon Shukla – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Prasoon Shukla
Prasoon is a Data Scientist at Stripe where he works to optimize Stripe’s payment flows and reduce fraud. He has experience in fintech and traditional finance companies building credit, identification, forecasting and anomaly detection models.
[](https://linkedin.com/in/prasoondshukla)
[](https://github.com/prasoon2211)
### Events
* Introduction to Bayesian Inference for Parameter Estimation ([watch on youtube](https://www.youtube.com/watch?v=q1B6YwINFvc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Pier Paolo Ippolito – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Pier Paolo Ippolito
Pier Paolo Ippolito is a Data Scientist at SAS and MSc in Artificial Intelligence graduate with an interest in research areas such as Data Analytics, Machine Learning, and Cloud Development. Aside from his work activities, he is a freelancer and technical writer for Towards Data Science.
[](https://twitter.com/Pier_Paolo_28)
[](https://linkedin.com/in/pierpaolo28)
[](https://github.com/pierpaolo28)
[](https://pierpaolo28.github.io/)
### Events
* Paradoxes in Data Science ([watch on youtube](https://www.youtube.com/watch?v=bIgGMfuctXQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Polina Mosolova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Polina Mosolova
I am a data scientist at SAP, passionate about bringing the full potential of current machine learning research to business applications. I am interested in creative combinations of statistical and machine learning methods for use cases addressing real-world problems. In my PhD dissertation, I created an applied machine learning framework for churn prediction, enhanced by organisational trust theory and explainable machine learning methods.
[](https://linkedin.com/in/polina-mosolova)
### Events
* Interpretable AI and ML ([watch on youtube](https://www.youtube.com/watch?v=EQcY83VA0Us)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Prateek Joshi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Prateek Joshi
Prateek Joshi is the founder and CEO of Plutoshift. He is an author of 13 books on Machine Learning and his blog has readership in 200+ countries. He has been featured on Forbes, CNBC, TechCrunch, and Bloomberg. He has been a speaker at conferences such as TEDx, Global Big Data Conference, and Machine Learning Developers Conference. You can visit prateekj.com to learn more about him.
[](https://twitter.com/prateekvjoshi)
[](https://linkedin.com/in/prateek-joshi-91047b19)
[](https://github.com/prateekjoshi565)
[](https://www.prateekj.com/)
### Books
* [Artificial Intelligence with Python](https://datatalks.club/books/20220509-artificial-intelligence-with-python.html)
(the book of the week from 09 May 2022 to 13 May 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rachael Tatman – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rachael Tatman
Dr. Rachael Tatman is a senior developer advocate at Rasa, where she helps developers use open-source software to build great conversational AI projects. Before that, she was at Kaggle (part of Google) and doing a PhD in Linguistics. If you hear a dog during the talk, his name is Benson.
[](https://twitter.com/rctatman)
[](https://linkedin.com/in/rachael-tatman-500a323a)
[](https://github.com/rctatman)
[](http://www.rctatman.com/)
### Events
* Make Your First ML Chatbot ([watch on youtube](https://www.youtube.com/watch?v=YwjauyAflmc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rachel Lim – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rachel Lim
I am a urban data scientist dedicated to creating liveable cities through the innovative use of data. With a background in geography, and a masters in urban data science, I blend qualitative and quantitative analysis to tackle urban challenges. I strive to integrate data driven techniques with urban design to foster sustainable and equitable urban environments.
[](https://linkedin.com/in/rachel-lim-xin-rong)
[](https://github.com/rxl204)
### Events
* Using Data to Create Liveable Cities ([watch on youtube](https://www.youtube.com/watch?v=VXQIGHUWeL0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Raghav Bali – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Raghav Bali
Raghav Bali is a Senior Data Scientist at Optum (UnitedHealth Group), one the world’s largest health care organizations. He has over ten years experience in the research and development of enterprise-level solutions using machine learning, deep learning, and natural language processing. He is the author of multiple machine learning papers, books, and patents.
[](https://twitter.com/rghv_bali)
[](https://linkedin.com/in/baliraghav)
[](https://github.com/raghavbali)
### Books
* [Transfer Learning in Action](https://datatalks.club/books/20211004-transfer-learning-in-action.html)
(the book of the week from 04 Oct 2021 to 08 Oct 2021)
* [Generative AI with Python and TensorFlow 2](https://datatalks.club/books/20211108-generative-ai-with-python-and-tensorflow-2.html)
(the book of the week from 08 Nov 2021 to 12 Nov 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rahul Jain – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rahul Jain
Rahul Jain is a seasoned data and AI professional with over 15 years of experience driving enterprise data transformations. As Senior Solutions Engineer at Snowflake, he partners with sales, product, and engineering teams to help global clients adopt the Snowflake Data Cloud and modernize their data ecosystems. Rahul has been instrumental in designing proof-of-concept implementations, running workshops and training sessions, and helping organizations align their data strategies with business growth.
Before his current role, Rahul served as Data Engineering Manager at Snowflake, leading a high-performing team that built internal data products for sales, finance, and IT. He also played a key role in promoting Snowflake@Snowflake use cases, speaking at global conferences such as Data Cloud World Tour and BUILD, and developing the internal workshop series AI for Everyone.
Rahul’s previous experience includes leading data platform initiatives at Siemens, where he helped optimize industrial operations through predictive analytics, and managing large-scale data engineering projects at Saama, First Republic Bank, and Tata Consultancy Services. His expertise spans Snowflake, AWS, dbt, Informatica, and advanced data modeling. Rahul continues to advocate for the power of data and AI to create measurable business value across industries.
[](https://twitter.com/rahulj51)
[](https://linkedin.com/in/rahul-jain-83055b45)
[](https://rahulj51.github.io/)
### Events
* Mentoring ([watch on youtube](https://www.youtube.com/watch?v=LQvwTNQbPg4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Mentoring---Rahul-Jain-eo7cmu)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ramiro Aznar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ramiro Aznar
Ramiro Aznar is a Environmental Biologist working as a Geospatial Data Engineer Manager at Planet for more than 3 years, previously he was a Solutions Engineer at CARTO.
[](https://twitter.com/ramiroaznar)
[](https://linkedin.com/in/ramiroaznar)
### Events
* Building the Modern Geospatial Data Stack ([watch on youtube](https://www.youtube.com/watch?v=VntcKnMGdhM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ranjitha Kulkarni – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ranjitha Kulkarni
Ranjitha Gurunath Kulkarni is a Staff Machine Learning Engineer at NeuBird.ai. Previously, she built LLM- and agent-powered product capabilities at Dropbox Dash and worked on speech recognition, language modeling, online metrics, and assistant evaluation at Microsoft. Her publications span voice query reformulation and automatic online evaluation of intelligent assistants, and her patents include automated closed captioning using temporal data and hyperarticulation detection. Ranjitha holds a master’s from Carnegie Mellon University (Language Technologies Institute).
[](https://linkedin.com/in/ranjitha-gurunath-kulkarni)
### Events
* Building reliable AI products in the era of Gen AI and Agents ([watch on youtube](https://www.youtube.com/watch?v=x2AAjqz2XmM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Raphaël Hoogvliets – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Raphaël Hoogvliets
Raphaël Hoogvliets is a notable figure in the field of MLOps, known for his expertise as a data scientist and machine learning engineer. He is particularly recognized for his leadership and team-building skills within the technology sector. Raphaël is active in sharing his knowledge and insights on platforms like LinkedIn and Substack, where he regularly posts, and runs a newsletter. He shares learnings on MLOps from a variety of perspectives, including CxO-level strategies and beginner-level coding.
Currently, Raphaël is leading a team of 12 engineers at Eneco, a leading sustainable energy company that provides energy to 2m households and businesses.
[](https://linkedin.com/in/hoogvliets)
[](https://github.com/hoogvliets)
### Events
* MLOps as a Team ([watch on youtube](https://www.youtube.com/watch?v=rMq63r3zi4c)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Reem Mahmoud – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Reem Mahmoud
Reem Mahmoud is the Director of Data Science at intervu.ai
[](https://linkedin.com/in/reemmahmoud)
### Events
* Building Machine Learning Products ([watch on youtube](https://www.youtube.com/watch?v=m45tNY-8gY8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rileen Sinha – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rileen Sinha
Experienced Computational Biologist with over a decade of experience in Cancer Genomics, including working within the TCGA consortium as part of the integrative genomics team while at the Sander lab in MSKCC. Particularly adept at interdisciplinary collaborations, with extensive experience of collaborating with Wet-Lab Biologists, Physician Scientists, and Computational Biologists.
Extensive research on evaluating cell lines as tumor models using genomic profiles, including two first-author papers published in Nature Communications (PMID: 23839242, PMID: 28489074), and a third first-author paper published in Cell Reports Methods (https://www.sciencedirect.com/science/article/pii/S2667237521000849).
[](https://twitter.com/RileenSinha)
[](https://linkedin.com/in/rileen-sinha-a644692)
[](https://github.com/Optimistix)
### Events
* Advancing Cancer Genomics with Machine Learning ([watch on youtube](https://datatalks.club/people/rileensinha.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rishabh Bhargava – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rishabh Bhargava
Rishabh has worked with analytics and ML teams for 7+ years. Most recently, he led Sales Engineering at a data infra company called Datacoral (acquired by Cloudera) helping analytics teams with their data pipelines. Previous to that, he was an early employee at Primer.ai where he built and deployed ML models for multiple natural language applications. He also writes a newsletter that discusses challenges with ML in production: [https://mlopsroundup.substack.com](https://mlopsroundup.substack.com/)
.
Rishabh has a Masters in CS from Stanford and completed his undergraduate studies at the University of Cambridge.
[](https://twitter.com/rish_bhargava)
[](https://linkedin.com/in/bhargavarishabh)
[](https://github.com/rishabh-bhargava)
[](https://mlopsroundup.substack.com/)
### Events
* Similarities and Differences between ML and Analytics ([watch on youtube](https://www.youtube.com/watch?v=rMRUa8WxDz4)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Similarities-and-Differences-between-ML-and-Analytics---Rishabh-Bhargava-e18rcam)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rob De Wit – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rob De Wit
Rob is a developer advocate at Iterative AI. He’s got a background in information sciences, and experience in data analytics and engineering. Previously he worked in FinTech, and right now he’s learning a whole lot about MLOps. He’s got first-hand experience with the gap between data scientists and engineers, and particularly interested in bridging that gap.
[](https://linkedin.com/in/rcdewit)
[](https://robdewit.nl/)
### Events
* GitOps for ML: Converting Notebooks to Reproducible Pipelines ([watch on youtube](https://www.youtube.com/watch?v=t92ISBh4y_E)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rob Zinkov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rob Zinkov
Rob Zinkov is a machine learning engineer and data scientist. My work covers how to more efficiently specify and train deep generative models as well as how to more effectively discover a good statistical model for your data. Previously I was a research scientist at Indiana University where I was the lead developer of the Hakaru probabilistic programming language.
[](https://linkedin.com/in/robzinkov)
[](https://github.com/zaxtax)
### Events
* Bayesian Modeling and Probabilistic Programming ([watch on youtube](https://www.youtube.com/watch?v=kcKvUSInm-M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Roksolana Diachuk – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Roksolana Diachuk
Roksolana works as a Big Data Engineer at Captify and Diversity & Inclusion ambassador in the Ukrainian region. Also, she is a speaker, one of the Women Who Code Kyiv leads and mentors. She is passionate about Big Data, Scala, and Kubernetes – those are the topics she often chooses for her talks. Other topics of interest for her are diversity & inclusion and women in tech. Her hobbies include building technical topics around fairytales and discovering new cities.
[](https://twitter.com/dead_flowers22)
[](https://linkedin.com/in/roksolanadiachuk)
[](https://github.com/roksolana-d)
[](https://roksolanadiachuk.wixsite.com/roksolana-d)
### Events
* DataTalks.Club Summer Marathon: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
* Big Data Engineer vs Data Scientist ([watch on youtube](https://www.youtube.com/watch?v=yg3d1lFd7Uo)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Big-Data-Engineer-vs-Data-Scientist---Roksolana-Diachuk-e139sl8)
)
* Modern Data Pipelines in AdTech - Life in the Trenches ([watch on youtube](https://www.youtube.com/watch?v=DCcRE1m6960)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Roman Grebennikov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Roman Grebennikov
Roman is a seasoned DS engineer and startup enthusiast writing ML applications for food
[](https://twitter.com/public_void_grv)
[](https://linkedin.com/in/romangrebennikov)
[](https://github.com/shuttie)
[](https://dfdx.me/)
### Events
* Reinforcement Learning for Search ([watch on youtube](https://www.youtube.com/watch?v=oVkD-2xbqKM)
)
* Building an Open-Source Feature Store with Apache Flink ([watch on youtube](https://www.youtube.com/watch?v=BskjQPkrYec)
)
* Practical Learning-to-Rank: Deep, Fast, Precise ([watch on youtube](https://www.youtube.com/watch?v=oXfFqAKf4Ac)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rosona Eldred – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rosona Eldred
Rosona Eldred is a trained mathematician working in the “data space” for the last 6 years, the last three with industrial data. She is currently a machine learning engineer with a technical leadership role around synthetic tabular data in an AI Innovation team.
She’s particularly intrigued by Industrial R&D problems, especially around dealing with tiny data.
Originally from the US, after finishing her Ph.D., she moved to Germany in 2012 as a postdoctoral researcher and has stayed there ever since. If the ampel coalition ever finishes the debate over dual citizenship, she’ll officially become German.
[](https://linkedin.com/in/rosona-eldred)
### Events
* Navigating Industrial Data Challenges ([watch on youtube](https://www.youtube.com/watch?v=rwuud5wr3J4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ross Brigoli – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ross Brigoli
I am a consulting architect at Red Hat focused on delivering cutting-edge solutions to enterprises. I am originally from the Philippines. I have previously worked in the UK and France and am currently living in Singapore.
I am a father of a little girl who loves dinosaurs. That makes me an expert in dinosaurs too.
I write about different topics, mostly hacks, and DIY projects, on my blog: https://blog.rossbrigoli.com.
[](https://twitter.com/rossbrigoli)
[](https://linkedin.com/in/rossbrigoli)
[](https://github.com/rossbrigoli)
### Books
* [Machine Learning on Kubernetes](https://datatalks.club/books/20221107-machine-learning-on-kubernetes.html)
(the book of the week from 07 Nov 2022 to 11 Nov 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Roy Jafari – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Roy Jafari
Roy Jafari, Ph.D. is an assistant professor of business analytics at the University of Redlands.
Roy has taught and developed college-level courses that cover data cleaning, decision making, data science, machine learning, and optimization.
Roy’s style of teaching is hands-on and he believes the best way to learn is to learn by doing. He uses active learning teaching philosophy and readers will get to experience active learning in this book.
Roy believes that successful data preprocessing only happens when you are equipped with the most efficient tools, have an appropriate understanding of data analytic goals, are aware of data preprocessing steps, and can compare a variety of methods. This belief has shaped the structure of this book.
[](https://twitter.com/JafariRoy)
[](https://linkedin.com/in/roy-jafari-marandi-84077932)
[](https://roy-jafari.com/)
### Books
* [Hands-On Data Preprocessing in Python](https://datatalks.club/books/20220228-hands-on-data-preprocessing-in-python.html)
(the book of the week from 28 Feb 2022 to 04 Mar 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rui Machado – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rui Machado
Rui Machado currently works at Fraudio as VP of Technology and has a background in Information Technologies and Data Science. Has over a decade of relevant experience in the architecture and implementation of data warehouses, data lakes, and decision support systems in industries such as Retail, Ecommerce, Supply Chain, Healthcare, and Social Networks. Has led Engineering and Analytics teams at Jumia, Nike, and Facebook. He is also co-founder and CEO of ShopAI.co. He has previously collaborated with Synfusion in publishing three technical books on Powershell, SSIS, and BizTalk Server.
[](https://linkedin.com/in/rpmachado)
### Books
* [Analytics Engineering with SQL and DBT](https://datatalks.club/books/20231106-analytics-engineering-with-sql-and-dbt.html)
(the book of the week from 06 Nov 2023 to 10 Nov 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ruslan Shchuchkin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ruslan Shchuchkin
Ruslan is a Data Scientist living in Berlin who has tried lots of healthy (and not so healthy) things to stay focused and productive. From getting depression just by looking at Data science studying roadmap to landing a job as a DS, he found some set of techniques, protocols and concepts about our brain, body and mind that might be helpful to others in Data community.
[](https://twitter.com/r_shchuchkin)
[](https://linkedin.com/in/ruslanshchuchkin)
### Events
* Biohacking for Data Scientists and ML Engineers ([watch on youtube](https://www.youtube.com/watch?v=uyxUBADZYpU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Rustem Feyzkhanov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Rustem Feyzkhanov
Rustem is a AWS Machine Learning hero. He’s passionate about the use of cloud infrastructure for AI/ML applications. He has experience with architecting and deploying deep learning training and inference pipelines.
[](https://twitter.com/ryfeus)
[](https://linkedin.com/in/ryfeus)
[](https://github.com/ryfeus)
[](https://www.ryfeus.io/)
### Events
* Building Scalable End-to-End Deep Learning Pipelines in the Cloud ([watch on youtube](https://www.youtube.com/watch?v=Nq8-VdBEY98)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sabina Firtala – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sabina Firtala
Sabina works on Frontline’s AI product development. She’s involved in all aspects of model development, from data wrangling and visualisation to statistical tests, model training and validation. She’s got a background in Natural Sciences and previously worked as a data analyst in finance and SaaS companies. As a freelance data analyst, she takes on challenging projects for mission-driven companies, being especially interested in social impact, healthcare, and accessibility.
[](https://linkedin.com/in/sabina-firtala)
[](https://github.com/sabinagio)
### Events
* Building a Domestic Risk Assessment Tool ([watch on youtube](https://www.youtube.com/watch?v=CpWlBAmD9ok)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sadat Anwar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sadat Anwar
A Data Science Manager who enjoys working with his people and team at the centre. I firmly believe in taking a people centric approach to building great engineering teams that conquer even greater engineering challenges.
A Software Engineer, passionate about building products to solve problems. Oftentimes the most complex problems can be solved by simple and elegant solutions.
I love exploring data, and try to have an unbiased view on the story it has to tell. I speak Java, Scala, Python and English.
[](https://twitter.com/CodeCrunchedd)
[](https://linkedin.com/in/syed-sadat-anwar)
[](https://github.com/SadatAnwar)
### Events
* From Software Engineer to Data Science Manager ([watch on youtube](https://www.youtube.com/watch?v=xyTfqIWeKf8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sadik Bakiu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sadik Bakiu
Sadik is co-founder and ML Engineer consultant at data-max.io. He is focused in developing solutions to bring ML to production.
Since the early beginning of his career, more than a decade ago, he was fascinated by Data and Information management systems and has been working with them ever since. Sadik also writes occasionally about technology topics.
[](https://linkedin.com/in/sbakiu)
[](https://github.com/sbakiu)
### Events
* A Practical Guide to A/B Testing in MLOps ([watch on youtube](https://www.youtube.com/watch?v=k9CqduJ7ha4)
)
* Machine Learning with Ray: Supercharge Your GPU Clusters ([watch on youtube](https://datatalks.club/people/sadikbakiu.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sally-Ann DeLucia – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sally-Ann DeLucia
Driven machine learning enthusiast with a master’s degree in Applied Data Science possessing strong attention to detail, exceptional analytic skills, and the ability to solve complex problems. A deeply technical young professional with a creative outlook striving to develop solutions that are not only technically sound but also socially responsible.
[](https://linkedin.com/in/sally-ann-delucia-59a381172)
[](https://arize.com/)
### Articles
* [Building an AI Agent that Thrives in the Real World](https://datatalks.club/blog/building-ai-agent-that-thrives-in-real-world.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sandra Kublik – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sandra Kublik
Sandra Kublik is an AI entrepreneur, evangelist, and community builder, fostering AI business innovation in her work. She has served as a mentor and coach to AI-first companies, co-founded the world’s first independent AI acceleration program for startups, and grew and successfully scaled a global hackathon community of AI professionals and enthusiasts. She is an active spokeswoman on the subjects of NLP and synthetic media. She runs a YouTube channel where she interviews ecosystem stakeholders and discusses groundbreaking AI trends with fun and educational content.
[](https://twitter.com/sandra_kublik)
[](https://linkedin.com/in/sandrakublik)
[](https://github.com/SandraKublik)
### Events
* The Good, the Bad and the Ugly of GPT ([watch on youtube](https://www.youtube.com/watch?v=bM6AR4A-f98)
)
### Books
* [GPT-3](https://datatalks.club/books/20230306-gpt-3.html)
(the book of the week from 06 Mar 2023 to 10 Mar 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sage Elliott – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sage Elliott
Sage Elliott has worked in hardware and software engineering roles at several startups for over a decade. Through hands-on workshops and mentoring at hackathons, he has taught thousands of people how to get started with programming, machine learning, and computer vision. As a technical evangelist at WhyLabs, Sage enjoys breaking down the barrier to AI observability and talking to amazing people in the Robust & Responsible AI community.
[](https://twitter.com/sagecodes)
[](https://linkedin.com/in/sageelliott)
[](https://github.com/sagecodes)
[](https://sageelliott.com/)
### Events
* Three Stages of Real-Time Data Monitoring ([watch on youtube](https://www.youtube.com/watch?v=gZZjirywiUI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Santona Tuli – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Santona Tuli
Santona began her data journey through fundamental physics—searching through massive event data from particle collisions at CERN to detect rare particles. She’s since extended her machine learning engineering to natural language processing, before switching focus to product and data engineering for data workflow authoring frameworks. As a python engineer, she started with the programmatic data orchestration tool, Airflow, helping improve its usability for data science and machine learning pipelines. Currently at Upsolver, she leads data engineering and science, driving research for the declarative workflow authoring framework in SQL. Santona is passionate about building, as well as empowering others to build, end-to-end data and ML pipelines, scalably.
[](https://linkedin.com/in/santona-tuli)
### Events
* From MLOps to DataOps ([watch on youtube](https://www.youtube.com/watch?v=kSTfhQ_SZgc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sara EL-ATEIF – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sara EL-ATEIF
Sara El-Ateif, Google Developer Expert in Machine Learning, Google PhD Fellow, Co-Founder of AI Wonder Girls, and Evercoach Certified Business Coach/Consultant, is on a mission to demystify AI and innovation to empower individuals with tools and mindsets required to build solutions that matter to their community/to humanity.
[](https://linkedin.com/in/sara-el-ateif)
[](https://github.com/elateifsara)
### Events
* Make an Impact Through Volunteering Open Source Work ([watch on youtube](https://www.youtube.com/watch?v=aHdaIwOEI8Q)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sarah Mestiri – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sarah Mestiri
Sarah is a Data Scientist and a Certified Career & Interview Coach. She has 6 years+ of experience in Tech, working at international companies, financial companies (FIS), and startups. Sarah’s experience of going through a career transition and her natural drive to support other women in following their professional dreams inspired her to become a certified career coach. She is on a mission to support more women to find the right job for them and achieve their full potential. As a career coach, She has been supporting women returning to work after a career break and transitioning to the data field.
[](https://linkedin.com/in/sarahmestiri)
### Events
* Accelerating The Job Hunt for The Perfect Job in Tech ([watch on youtube](https://www.youtube.com/watch?v=PchwbIs0tOg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sara Menefee – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sara Menefee
Sara is a product manager at Meroxa, a company building a data platform that helps software teams orchestrate and integrate data into their data-driven applications. In a past life, she worked as a product designer for companies including Sora, Checkr, Change.org, and Zendesk.
[](https://linkedin.com/in/saramenefee)
### Events
* Becoming a Data Product Manager ([watch on youtube](https://www.youtube.com/watch?v=nt__pVuuC-k)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Becoming-a-Data-Product-Manager---Sara-Menefee-e1arc4a)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sara Robinson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sara Robinson
Sara is a Developer Advocate for Google Cloud, focusing on machine learning. She inspires developers and data scientists to integrate ML into their applications through demos, online content, and events.
Before Google, she was a Developer Advocate on the Firebase team. Sara has a Bachelor’s degree from Brandeis University. When she’s not writing code, she can be found on a spin bike or eating frosting.
[](https://twitter.com/SRobTweets)
[](https://linkedin.com/in/sara-robinson-40377924)
[](https://github.com/sararob)
[](https://sararobinson.dev/)
### Events
* DataTalks.Club Conference: ML in Production ([watch on youtube](https://www.youtube.com/watch?v=og1DG1KZ71c)
)
* Machine Learning Design Patterns ([watch on youtube](https://www.youtube.com/watch?v=aL_zQfi7lDI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Saurav Maheshkar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Saurav Maheshkar
Budding Data Scientist. Researching Attention and Representation Learning. Attention is all you need!
[](https://twitter.com/MaheshkarSaurav)
[](https://github.com/SauravMaheshkar)
[](https://sauravmaheshkar.github.io/blog/)
### Articles
* [NER with Reformer in Trax: End‑to‑End Tutorial on a Kaggle Dataset](https://datatalks.club/blog/ner-reformers.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Savaş Yıldırım – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Savaş Yıldırım
Savaş Yıldırım is an associate professor at the Istanbul Bilgi University and is also a visiting researcher at Ryerson University. He also teaches NLP courses at the university and consults ChatBot and NLP projects of a tech company
[](https://linkedin.com/in/savasy)
[](https://github.com/savasy)
### Books
* [Mastering Transformers](https://datatalks.club/books/20211011-mastering-transformers.html)
(the book of the week from 11 Oct 2021 to 15 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sean Sheng – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sean Sheng
Sean leads the engineering team and oversees the product development and technical innovation at BentoML. Before joining BentoML, Sean worked as a Senior Engineering Manager at LinkedIn, leading multiple teams developing the service frameworks and infrastructure that powered all of LinkedIn’s distributed service architecture. The scale of infrastructure spanned three geographic co-locations with a quarter million service instances, supporting all of LinkedIn’s infrastructure teams and product lines. Sean has worked in the cloud infrastructure space for over ten years in both IC and leadership roles. Sean holds a bachelor’s degree in Computer Science from the University of Waterloo.
[](https://linkedin.com/in/ssheng)
### Events
* Building an ML Service Platform from the Ground Up ([watch on youtube](https://www.youtube.com/watch?v=8h7vIN2WzT4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sebastian Ayala Ruano – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sebastian Ayala Ruano
Sebastian Ayala Ruano is a bioinformatics software developer whose work bridges biotechnology and computational biology. He has contributed to open-source tools including MicW2Graph, VueGen, and VueCore, designed to simplify multi-omics data analysis for researchers. Previously, he worked on projects in cheminformatics, peptide discovery, and network-based analysis, and has developed educational bioinformatics tools for open science communities. Sebastian is currently pursuing a Master’s degree in Systems Biology at Maastricht University, where he focuses on integrating machine learning and network science into biological research. He shares his projects and insights through his personal website and GitHub.
[](https://linkedin.com/in/sayalaruano)
[](https://sayalaruano.github.io/project/)
### Events
* From Biotechnology to Bioinformatics Software ([watch on youtube](https://www.youtube.com/watch?v=ZFrcrTtnB1Q)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sebastian Raschka – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sebastian Raschka
Sebastian Raschka, PhD, has been working in machine learning and AI for more than a decade. In addition to being a researcher, Sebastian has a strong passion for education. He is known for his bestselling books on machine learning with Python and his contributions to open source. Sebastian is a staff research engineer at Lightning AI, focusing on implementing and training LLMs. Before his industry experience, Sebastian was an assistant professor in the Department of Statistics at the University of Wisconsin-Madison, where he focused on deep learning research. You can learn more about Sebastian at https://sebastianraschka.com.
[](https://twitter.com/rasbt)
[](https://linkedin.com/in/sebastianraschka)
[](https://github.com/rasbt)
[](https://sebastianraschka.com/)
### Books
* [Build a Large Language Model (From Scratch)](https://datatalks.club/books/20241017-build-large-language-model-from-scratch.html)
(the book of the week from 14 Oct 2024 to 20 Oct 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sedat Kapanoglu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sedat Kapanoglu
Sedat Kapanoglu is a self-taught programmer with more than 25 years of experience, including a stint at Microsoft as a software engineer. He is the creator of Ekşi Sözlük, the number one social media platform for Turkish-speaking audiences.
[](https://twitter.com/esesci)
[](https://linkedin.com/in/kapanoglu)
[](https://github.com/ssg)
### Books
* [Street Coder](https://datatalks.club/books/20210322-street-coder.html)
(the book of the week from 22 Mar 2021 to 26 Mar 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sejal Vaidya – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sejal Vaidya
Sejal Vaidya is an experienced data engineer with more than 10 years of experience in software development. She’s proficient in data platforms, infrastructure, and cloud architecture, and she’s interested in machine learning engineering and MLOps.
[](https://twitter.com/sejalv_)
[](https://linkedin.com/in/vaidyasejal)
[](https://github.com/sejalv)
### Articles
* [Deploy ML Models on AWS Lambda with Docker Containers and SAM](https://datatalks.club/blog/ml-deployment-lambda.html)
### Events
* Deploying Serverless Machine Learning with AWS ([watch on youtube](https://www.youtube.com/watch?v=79B8AOKkpho)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sergei Boitsov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sergei Boitsov
Sergei Boitsov is a Senior Data Engineer at JetBrains, based in Berlin. He enjoys driving and playing the bass guitar in his spare time.
[](https://twitter.com/sergboec)
[](https://linkedin.com/in/sergboec)
### Events
* Foundations of Data Engineering Instruments ([watch on youtube](https://www.youtube.com/watch?v=H3sxOea9qD0)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Serena Haidar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Serena Haidar
Serena Haidar is a Computer and Communications Engineer with a self-taught passion for AI/ML, building tech that drives accessibility and sustainability. A lifelong problem solver, she continually advances her skills through online courses and hands-on projects, mastering Python, Java, Arduino, and other programming languages. Her learning-by-doing approach turns every hour of exploration into an investment, because making a positive impact is what matters most.
[](https://linkedin.com/in/serena-haidar)
[](https://github.com/Serena-github-c)
### Articles
* [Key Lessons from ML Zoomcamp: Serena Haidar](https://datatalks.club/blog/key-lessons-from-ml-zoomcamp-serena-haidar.html)
* [How to Build a Waste Classifier: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sergei Shaikin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sergei Shaikin
Sergei Shaikin is the CTO at Artillex, a UK-based fintech startup, with a rich background leading data engineering teams, leading data products delivery, and hands-on experience in data engineering. Sergei is a Microsoft Azure Certified Architect and the author of a book on Apache Carbondata, an open-source, high-performance, scalable storage format. Committed to the growth of the data engineering community, Sergei mentors on ADPList, offering his insights and guidance to fellow data professionals.
[](https://linkedin.com/in/sergey-shaykin-88078026)
### Events
* System Design in Data Engineering ([watch on youtube](https://datatalks.club/people/sergeishaikin.html)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Serg Masis – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Serg Masis
My aspiration is to provide the often-missing link between data and decision-making. After a long career in web/app development and entrepreneurship, I realized my passion lay in the ability to turn large amounts of information into knowledge, to identify patterns, to uncover relationships, to solve problems, and to formulate predictions.
To that end, although I had been making models and engaging with data for almost two decades, four years ago I decided to go back to school to get a graduate degree in data science. Since then I have been providing that link, by combining my extensive experience in computer science, data science, decision science, and business thinking.
[](https://twitter.com/smasis)
[](https://linkedin.com/in/smasis)
[](https://github.com/smasis001)
[](https://www.serg.ai/)
### Events
* Mitigating Bias with the XAI Toolbox ([watch on youtube](https://www.youtube.com/watch?v=mP_9zEFwrho)
)
### Books
* [Interpretable Machine Learning with Python](https://datatalks.club/books/20210719-interpretable-machine-learning-with-python.html)
(the book of the week from 19 Jul 2021 to 23 Jul 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Shachar Meir – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Shachar Meir
Shachar has 20+ years of experience in data and analytics leadership.
He has built and led data and analytics teams in established startups (WorldRemit, Pontis), and large companies (PayPal, Meta).
In his last role at Meta, Shachar was the Director of Data Engineering for Facebook/Instagram/Messenger on Trust and Safety. Prior to that Shachar led a Data Engineering team in Facebook’s Ads and Business Platform.
Shachar was also the London Data Engineering site lead, building and growing the data engineering function from 4 data engineers to more than 200.
These days, Shachar is a Data Advisor, supporting companies and individuals in their data journey.
Outside of work, Shachar loves spending time with his family, flying airplanes and helicopters recreationally, and cooking.
[](https://linkedin.com/in/shacharmeir)
### Events
* How to Boost Your Impact as A Data Professional ([watch on youtube](https://www.youtube.com/watch?v=eNmoRCwFXM8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Shir Meir Lador – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Shir Meir Lador
Shir Meir Lador, Shir Meir Lador, data science group manager at intuit US (Mountain View, California), develops machine and deep learning models for document intelligence for products like TurboTax and quickbooks.
[](https://twitter.com/shirmeir86)
[](https://linkedin.com/in/shir-meir-lador-98b4692a)
### Events
* The Secret Sauce of Data Science Management ([watch on youtube](https://www.youtube.com/watch?v=gcxP0qRO-MY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Shubham Saboo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Shubham Saboo
Shubham Saboo has played multiple roles from a data scientist to an AI evangelist at renowned firms across the globe, where he was involved in building organization-wide data strategies and technology infrastructure to create and scale data science and machine learning practice from scratch. His work as an AI evangelist has led him to build communities and reach out to a broader audience to foster the exchange of ideas and thoughts in the burgeoning field of artificial intelligence. As part of his passion for learning new things and sharing knowledge with the community, he writes technical blogs on the advancements in AI and its economic implications. In his spare time, you can find him traveling across the country which enables him to immerse in different cultures and refine his worldview based on his experiences.
[](https://twitter.com/Saboo_Shubham_)
[](https://linkedin.com/in/shubhamsaboo)
[](https://github.com/Shubhamsaboo)
### Books
* [GPT-3](https://datatalks.club/books/20230306-gpt-3.html)
(the book of the week from 06 Mar 2023 to 10 Mar 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sidharth Ramachandran – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sidharth Ramachandran
Sidharth currently leads a team of data scientists at GfK helping to build data products for the consumer goods industry. He has over 10 years of experience in software engineering and data science across telecom, banking and marketing industries. Sidharth also co-founded WACAO, a smart personal assistant on Whatsapp which was also featured on Techcrunch. He holds an undergraduate engineering degree from IIT Roorkee and an MBA from IIM Kozhikode. Sidharth is passionate about solving real problems through technology and loves to hack through personal projects in his free time.
[](https://linkedin.com/in/sidharthramachandran)
### Books
* [Blueprints for Text Analytics Using Python](https://datatalks.club/books/20211018-blueprints-for-text-analytics-using-python.html)
(the book of the week from 18 Oct 2021 to 22 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Simon Stiebellehner – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Simon Stiebellehner
Simon Stiebellehner has been building ML Platforms for over half a decade. Currently, he is Lead MLOps Engineer at Transaction Monitoring Netherlands (TMNL), a worldwide unique initiative of the big 5 banks of the Netherlands that tackles advanced analytics for Anti Money Laundering in an unprecedented way. Next to his work at TMNL, Simon is also university lecturer for Data Mining & Data Warehousing.
[](https://linkedin.com/in/simonstiebellehner)
### Events
* From Scratch to Success: Building an MLOps Team and ML Platform ([watch on youtube](https://www.youtube.com/watch?v=CB1YIsxQRtc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Simon Thompson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Simon Thompson
Simon Thompson has spent 25 years developing AI systems. He led the AI research program at BT Labs in the UK, where he helped pioneer Big Data technology in the company and managed an applied research practice for nearly a decade. Simon now works delivering Machine Learning systems for financial services companies in the City of London as the Head of Data Science at GFT Technologies.
[](https://twitter.com/AiSimonThompson)
[](https://linkedin.com/in/simon-thompson-025a7)
### Books
* [Managing Machine Learning Projects](https://datatalks.club/books/20221010-managing-machine-learning-projects.html)
(the book of the week from 10 Oct 2022 to 14 Oct 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sivan Biham – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sivan Biham
Sivan Biham is a Computer Vision Researcher and Algorithm Developer in Healthy.io, where she works on healthcare-related products, taking them all the way from the algorithm design stage to a fully implemented product with thousands of users across the world.
Sivan holds an M.Sc. in computer science from Weizmann Institute with a specialization in Computer Vision and Deep Learning, and a B.Sc. in both Computer Science and Neuroscience from Bar Ilan University.
She is enthusiastic about using her algorithmic skills and knowledge for improving people’s health and life. As part of her daily work, Sivan places emphasis on designing and implementing maintainable and modular algorithms, with software architecture principles in mind. She always strives to play an integral part in the problem-solving process in her own domain as well as others. In her spare time, she loves to run and practice yoga.
[](https://linkedin.com/in/sbiham)
[](https://www.sbiham.com/)
### Events
* The Unspoken Relationship - Product Managers and Data Scientists ([watch on youtube](https://www.youtube.com/watch?v=4FmApxkyoXQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sofya Yulpatova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sofya Yulpatova
Sofya Yulpatova is the Founder and CEO of Fit Tails, a PetTech startup developing an activity and health tracker for pets. With a background in computer science and product management, she blends technical skill with hands-on experience in building digital products. Before founding Fit Tails, she managed product and project delivery at FixParts, an international automotive parts distributor. Sofya is also an alumna of the Sales and Marketing Programme at the Stockholm School of Economics in Riga.
[](https://linkedin.com/in/sofya-yulpatova)
### Events
* Building Pet Health Tech: ML, Sensors, and Dog Behavior Data ([watch on youtube](https://www.youtube.com/watch?v=4bl2TSHD_Fc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Sonal Goyal – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Sonal Goyal
Sonal is the founder of Zingg (https://www.zingg.ai), building an ML-powered identity resolution and entity resolution framework for a single source of truth for customers and suppliers. Zingg is open-sourced at https://github.com/zinggAI/zingg. Sonal has 23 years of experience building data products for investment banking, telecom, gaming, insurance and other industries. She is a repeat speaker at conferences like Databricks Data and AI, Strata, Data Con LA, and GIDS. She holds a BTech from IIT Delhi and lives in India.
[](https://twitter.com/sonalgoyal)
[](https://linkedin.com/in/sonalgoyal)
[](https://github.com/sonalgoyal)
[](https://www.zingg.ai/)
### Events
* Large-Scale Entity Resolution ([watch on youtube](https://www.youtube.com/watch?v=lpjffCOPxlY)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Soumik Rakshit – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Soumik Rakshit
Soumik is a Machine Learning Engineer working on end-to-end deep learning pipelines as part of the Growth team at Weights & Biases. His research and development interests include Generative Machine Learning Models, Video Enhancement Systems, Neural Radiance Fields, and NLP-based solutions to improve and automate student evaluation in the education sector. He enjoys implementing research papers in machine learning and computer graphics and communicating such ideas using end-to-end notebooks, and articles. He’s also working on training machine learning models on TPUs using Jax, Flax, Tensorflow, and Keras.
[](https://twitter.com/soumikRakshit96)
[](https://linkedin.com/in/soumikrakshit)
### Events
* MLOps Zoomcamp - Experiment Tracking with Weights and Biases ([watch on youtube](https://www.youtube.com/watch?v=yNyqFMwEyL4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Soledad Galli – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Soledad Galli
Soledad Galli is a lead data scientist and founder of [Train in Data](https://www.trainindata.com/)
. She developed machine learning models in finance and insurance, received a Data Science Leaders Award in 2018 and was selected “LinkedIn’s voice” in data science and analytics in 2019.
Sole is passionate about sharing knowledge and tools and helping others succeed in data science. She teaches [7 online courses on Machine Learning](https://www.trainindata.com/courses)
at udemy.com, she is the author of a [Book on Feature engineering](https://www.packtpub.com/product/python-feature-engineering-cookbook/9781789806311)
and also the creator and maintainer of the Python open-source package [Feature-engine](https://feature-engine.readthedocs.io/en/latest/index.html)
, which is downloaded ~30k monthly.
Sole is passionate about empowering people to step into and excel in data science. She mentors data scientists, writes articles online and speaks at data science meetings.
As a data scientist in finance and insurance companies, Sole researched, developed and put in production machine learning models to assess Credit Risk, Insurance Claims and to prevent Fraud, leading in the adoption of machine learning in the organizations.
Sole has an MSc in Biology, a PhD in Biochemistry and 8+ years of experience as a research scientist in well-known institutions like University College London and the Max Planck Institute. She has scientific publications in various fields such as Cancer Research and Neuroscience, and her research was covered by the media on multiple occasions.
[](https://twitter.com/Soledad_Galli)
[](https://linkedin.com/in/soledad-galli)
[](https://github.com/solegalli)
[](https://www.trainindata.com/)
### Events
* Feature Selection in Machine Learning with Python ([watch on youtube](https://www.youtube.com/watch?v=blvmNWbcPDo)
)
### Books
* [Python Feature Engineering Cookbook](https://datatalks.club/books/20210920-python-feature-engineering-cookbook.html)
(the book of the week from 20 Sep 2021 to 24 Sep 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Srivathsan Canchi – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Srivathsan Canchi
Srivathsan is the head of engineering for Intuit’s real time streaming data infrastructure. He was most recently responsible for the Machine Learning platform at Intuit - his team was behind the open soucing of the MLOps tools mlctl and baklava. The ML Platform at Intuit powers 100s of ML models and performs 100s of millions of ML predictions every day. Prior to Intuit, he was responsible for building the cloud platform at eBay. His team built the cloud platform-as-a-service, enabling eBay to run 1000s of services handling 10s of Billions of calls every day. His team was one of the early contributors and adopters of Kubernetes in an on-premise environment.
[](https://twitter.com/srivathsanvc)
[](https://linkedin.com/in/srivathsancanchi)
[](https://github.com/srivathsanvc)
### Events
* Orchestrating Enterprise ML Workload Jobs Across Clouds ([watch on youtube](https://www.youtube.com/watch?v=ocDP-94YFjI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Stefan Gudmundsson – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Stefan Gudmundsson
Stefan Gudmundsson is known for turning advanced machine learning into real world impact across gaming and healthcare. As Director of Data, Analytics, and AI at CCP Games in Reykjavik, he leads the design of scalable data platforms, builds predictive and clustering models for player understanding, and delivers actionable insights to leadership and development teams. He chairs the CCP AI committee, setting AI strategy and governance with a focus on production velocity, quality, and ethical best practices.
Before CCP Games, Stefan served as VP of AI and Engineering at modl.ai, where he guided research and engineering for player bots and automated quality assurance. He previously built the data science and AI organization at Sidekick Health, launched partner facing data products, and advanced machine learning that led to patentable products. Earlier, he led AI research collaborations at H and M and managed teams that developed high performance recommender systems, and at King he directed research and development on game AI, human like playtesting, and player modeling. Stefan has shared his work at industry conferences including GDC, FDG, and CIG, and his research includes widely cited studies on human like playtesting and level difficulty prediction.
With a background that spans banking risk analytics, university lecturing in mathematics, and decades of hands on programming, Stefan brings deep technical breadth together with clear communication and team leadership. His core skills include data engineering, statistical modeling, generative AI, large language models, SQL, and modern data warehouse architecture. As an advisor and people leader, he builds high trust teams, aligns innovation with business strategy, and ships AI systems that improve player experience and health outcomes.
[](https://linkedin.com/in/stefanfreyrgudmundsson)
### Events
* Machine Learning and Personalization in Healthcare ([watch on youtube](https://www.youtube.com/watch?v=IDzhmmKeNG4)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Stefanie Molin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Stefanie Molin
Stefanie Molin is a software engineer and data scientist at Bloomberg in New York City, where she tackles tough problems in information security, particularly those revolving around data wrangling/visualization, building tools for gathering data, and knowledge sharing. She is also the author of Hands-On Data Analysis with Pandas, which is currently in its second edition. She holds a bachelor’s of science degree in operations research from Columbia University’s Fu Foundation School of Engineering and Applied Science. She is currently pursuing a master’s degree in computer science, with a specialization in machine learning, from Georgia Tech. In her free time, she enjoys traveling the world, inventing new recipes, and learning new languages spoken among both people and computers.
[](https://twitter.com/StefanieMolin)
[](https://linkedin.com/in/stefanie-molin)
[](https://github.com/stefmolin)
### Books
* [Hands-On Data Analysis with Pandas - Second Edition](https://datatalks.club/books/20220718-hands-on-data-analysis-with-pandas.html)
(the book of the week from 18 Jul 2022 to 22 Jul 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Stefan Jansen – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Stefan Jansen
Stefan is the founder and CEO of Applied AI. He advises Fortune 500 companies, investment firms, and startups across industries on data & AI strategy, building data science teams, and developing end-to-end machine learning solutions.
Before his current venture, he was a partner and managing director at an international investment firm, where he built the predictive analytics and investment research practice. He was also a senior executive at a global fintech company with operations in 15 markets, advised Central Banks in emerging markets, and consulted for the World Bank.
He holds Master’s degrees in Computer Science from Georgia Tech and in Economics from Harvard and Free University Berlin, and is a CFA Charterholder. He has worked in six languages across Europe, Asia, and the Americas and taught data science at Data camp and General Assembly.
[](https://linkedin.com/in/applied-ai)
[](https://github.com/stefan-jansen)
### Books
* [Machine Learning for Algorithmic Trading](https://datatalks.club/books/20210222-ml-algotrading-2ed.html)
(the book of the week from 22 Feb 2021 to 26 Feb 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Supreet Kaur – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Supreet Kaur
Supreet is an AVP at Morgan Stanley in Data Strategy and Products Team, a Data Science Mentor at Columbia University and Rutgers University, also an advisory board member at Ithaca College. She is also the founder of DataBuzz, a volunteer driven community that engages audience who want to pivot in the field of technology. She is also an ardent writer and speaker hence communicates her Data Science views at different platforms.
[](https://linkedin.com/in/supreet-kaur1995)
### Events
* Responsible and Explainable AI ([watch on youtube](https://www.youtube.com/watch?v=8Eb5mG-pC3o)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Susan Walsh – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Susan Walsh
With nearly a decade of experience fixing your dirty data, Susan Walsh is The Classification Guru.
She brings clarity and accuracy to data and procurement; helps teams work more effectively and efficiently; and cuts through the jargon to address the issues of dirty data and its consequences in an entertaining and engaging way.
Susan is a specialist in data classification, supplier normalisation, taxonomy customisation, and data cleansing and can help your business find cost savings through spend and time management - supporting better, more informed business decisions.
Susan has developed a methodology to accurately and efficiently classify, cleanse and check data for errors which will help prevent costly mistakes and could save days, if not weeks of laborious cleansing and classifying.
Susan is passionate about helping you find the value in cleaning your ‘dirty data’ and raises awareness of the consequences of ignoring issues through her blogs, vlogs, webinars and speaking engagements.
You can contact her on susan@theclassificationguru.com.
[](https://linkedin.com/in/susanewalsh)
[](https://www.theclassificationguru.com/)
### Events
* DataTalks.Club Conference: Product and Process ([watch on youtube](https://www.youtube.com/watch?v=dvzPU43tqFM)
)
* Dangers of Dirty Data ([watch on youtube](https://www.youtube.com/watch?v=N6994_LkdaI)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Santiago Valdarrama – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Santiago Valdarrama
Santiago is a Computer Scientist focusing on applied Machine Learning. He has more than two decades of experience building software to solve exciting and —sometimes— hard problems.
He cares deeply about unlocking the power of technology for individuals and businesses, so they can use computers in a way that was previously unrealistic for them. He finds joy in ambiguity and feels more engaged when working on problems that can’t be solved by merely searching the web or reading a book. Bonus points if he can collaborate with a team of like-minded engineers!
[](https://twitter.com/svpino)
[](https://linkedin.com/in/svpino)
[](https://github.com/svpino)
[](http://digest.underfitted.io/)
### Events
* DataTalks.Club Summer Marathon: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
)
* From Software Engineering to Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=xVYOdRrN7hw)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/From-Software-Engineering-to-Machine-Learning---Santiago-Valdarrama-e139s63)
)
### Books
* [Everyday ML Questions](https://datatalks.club/books/20220516-everyday-ml-questions.html)
(the book of the week from 16 May 2022 to 20 May 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Shawn Swyx Wang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Shawn Swyx Wang
swyx is passionate about Developer Tooling and Developer Communities. He currently works as a Senior Developer Advocate for AWS Amplify and recently published the Coding Career Handbook for Junior to Senior developer careers. In his free time he teaches React, TypeScript, Storybook and Node.js CLI’s at Egghead.io, and helps run the Svelte Society community of meetups.
[](https://twitter.com/swyx)
[](https://github.com/sw-yx)
[](https://www.swyx.io/)
### Events
* How to Market Yourself (without Being a Celebrity) ([watch on youtube](https://www.youtube.com/watch?v=tkBCPqWKCL8)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/How-to-Market-Yourself-without-Being-a-Celebrity---Shawn-Swyx-Wang-e11ai8t)
)
### Books
* [The Coding Career Handbook](https://datatalks.club/books/20210510-the-coding-career-handbook.html)
(the book of the week from 10 May 2021 to 14 May 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tammy Liang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tammy Liang
Tammy Liang is the Chief of Data at Platanomelón, a digital sex toy brand that empowers people to break taboos on sexuality. Together with her passionate and creative team, they infuse data analytics into the company’s day-to-day operation and decision-making. Driven by sustainability causes, Tammy is also the co-host of the podcast Data for Future, where she interviews people who are in the field of data or sustainability to curate a community for open conversation and collaboration.
[](https://linkedin.com/in/tammy-cen-liang)
[](https://dataforfuture.org/)
### Events
* Building and Leading Data Teams ([watch on youtube](https://www.youtube.com/watch?v=kI4V2iBbaH0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-and-Leading-Data-Teams---Tammy-Liang-e18efdl)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tanya Berger-Wolf – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tanya Berger-Wolf
Tanya is a Computational ecologist with research at the unique intersection of computer science, wildlife biology, and social sciences. Director of TDAI@OSU. Co-founder of Wildbook project and director of tech for conservation non-profit @Wild Me.
[](https://linkedin.com/in/tanyabw)
### Events
* AI for Ecology, Biodiversity, and Conservation ([watch on youtube](https://www.youtube.com/watch?v=30tTrozbAkg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tamara Atanasoska – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tamara Atanasoska
Tamara works on ML explainability, interpretability, and fairness as Open Source Software Engineer at :probabl.. She is a maintainer of Fairlearn, contributor to scikit-learn and skops. Tamara has both a Computer Science Software Engineering and a Computational Linguistics (NLP) background.
[](https://linkedin.com/in/tamaraatanasoska)
[](https://github.com/TamaraAtanasoska)
[](https://holophrase.substack.com/)
### Events
* Linguistics and Fairness ([watch on youtube](https://www.youtube.com/watch?v=sXU9vMDBjmk)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tatiana Gabruseva – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tatiana Gabruseva
Tatiana Gabruseva is a Сomputer Vision/DL Engineer and Kaggle Competitions Master. She currently works as a Senior ML Engineer at Cork University Hospital.
[](https://twitter.com/tatigabru)
[](https://linkedin.com/in/tatigabru)
[](https://github.com/tatigabru)
[](http://tatigabru.com/)
### Events
* From Physics to Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=wJPi6Ip9PX0)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/From-Physics-to-Machine-Learning---Tatiana-Gabruseva-e10r4pl)
)
* Staff AI Engineer ([watch on youtube](https://www.youtube.com/watch?v=_xr1_xb736E)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tatyjana Ankudo – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tatyjana Ankudo
I possess 20+ years of Research and Development experience, primarily in the Biotechnology and Pharmaceuticals sectors. Passionate about programming, and I have resent experience in data science and my aim is to become a valuable specialist at the crossroads of biotechnology and computational science.
[](https://linkedin.com/in/ankudo)
[](https://github.com/tankudo)
### Articles
* [Starting a Career in Data Science at 45](https://datatalks.club/blog/starting-career-in-data-science-at-45.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Theofilos Papapanagiotou – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Theofilos Papapanagiotou
Theofilos has been practicing systems engineering for 20 years, mostly in telcos. He’s currently building tools to support companies run effectively their ML workloads. Passionate about Kubeflow.
[](https://twitter.com/theofpa)
[](https://linkedin.com/in/theofpa)
### Events
* The Rise of MLOps ([watch on youtube](https://www.youtube.com/watch?v=-i0fVp0ntYA)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/The-Rise-of-MLOps---Theofilos-Papapanagiotou-ept67o)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tereza Iofciu – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tereza Iofciu
Tereza Iofciu is an experienced data practitioner, having worked as a data science manager, data scientist, data engineer, and product manager. She is leading a coaching team and teaching data science and neuefische in Hamburg. She is a co-organizer of the PyLadies Hamburg group, hosting regular meetups with the community. She is also a PSF Code of Conduct and Diversity & Inclusion working groups member and has been awarded the Python Software Foundation 2021 Q1 community service award. Recently she has joined the DISC Steering Committee team.
[](https://twitter.com/terezaif)
[](https://linkedin.com/in/tereza-iofciu)
[](https://github.com/terezaif)
### Events
* Decoding Data Science Job Descriptions ([watch on youtube](https://www.youtube.com/watch?v=bqxBiIwtmX4)
)
* Inclusive Data Leadership Coaching ([watch on youtube](https://www.youtube.com/watch?v=Z4vOTgzLkJQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Thomas Nield – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Thomas Nield
Thomas Nield is the founder of Nield Consulting Group as well as an instructor at O’Reilly Media and University of Southern California. He enjoys making technical content relatable and relevant to those unfamiliar or intimidated by it. Thomas regularly teaches classes on data analysis, machine learning, mathematical optimization, and practical artificial intelligence. At USC he teaches AI System Safety, developing systematic approaches for identifying AI-related hazards in aviation and ground vehicles. He’s authored two books, including Getting Started with SQL (O’Reilly) and Learning RxJava (Packt).
He is also the founder and inventor of Yawman Flight, a company developing universal handheld flight controls for flight simulation and unmanned aerial vehicles.
[](https://twitter.com/thomasnield76)
[](https://linkedin.com/in/thomasnield)
[](https://github.com/thomasnield)
### Books
* [Essential Math for Data Science](https://datatalks.club/books/20220829-essential-math-for-data-science.html)
(the book of the week from 29 Aug 2022 to 02 Sep 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Thomas Wolf – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Thomas Wolf
Thomas Wolf is Chief Science Officer and co-founder of Hugging Face Inc. He and his teams are on a mission to catalyse and democratise responsible ML and AI research by creating large scale open-source and open-science projects. Prior to founding HuggingFace, he gained a Ph.D. in statistical and quantum physics, and later a law degree from Sorbonne University. He previously worked as a physics researcher and a European Patent Attorney in the USA, France, and the Netherlands where he currently reside with his family.
[](https://twitter.com/Thom_Wolf)
[](https://linkedin.com/in/thomas-wolf-a056857)
[](https://github.com/thomwolf)
### Books
* [Natural Language Processing with Transformers](https://datatalks.club/books/20220425-natural-language-processing-with-transformers.html)
(the book of the week from 25 Apr 2022 to 29 Apr 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Thom Ives – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Thom Ives
Thom Ives founded Integrated Machine Learning & AI, which is a very large group of data scientists that seek to grow and learn MORE TOGETHER.
He is a leading data scientist and has developed a wide range of analytical models using data, multi-physics, and experiments. While Thom loves predictive modeling, his real passion for the data science space is making sure that data is clean from collection to storage for achieving the greatest overall return on data for all.
Thom is married, has 9 kids = 4 bios + 5 internationally adopted. He also has an awesome son-in-law that he is close to, AND he also regularly adopts amazing people from around the world! He lives in Eagle, Idaho, USA.
[](https://linkedin.com/in/thomives)
[](https://github.com/ThomIves)
[](https://integratedmlai.com/)
### Events
* Building Business Acumen for Data Professionals ([watch on youtube](https://www.youtube.com/watch?v=pImYf9ML95Q)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Building-Business-Acumen-for-Data-Professionals---Thom-Ives-e19gq91)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Timothy Davis – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Timothy Davis
Tim is a Developer Advocate with years of experience in infrastructure, operations, DevOps, and data. He has spoken at many large global conferences and small community meetups alike, and has a plethora of written and recorded content online across numerous niche’s of the tech industry.
[](https://twitter.com/vtimd)
[](https://linkedin.com/in/vtimd)
### Events
* How to write data pipelines with Apache Airflow® 3.0 ([watch on youtube](https://www.youtube.com/watch?v=v_jbmhVWBvM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tobias Zwingmann – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tobias Zwingmann
Tobias Zwingmann is an experienced Senior Data Scientist and currently Co-Founder at RAPYD.AI, a Germany-based AI startup focused on prototyping AI solutions. His mission is to help companies adopt machine learning and AI solutions faster while creating meaningful business impact and reducing the risk of project failure. Before founding RAPYD.AI, he has worked for more than 15 years in a corporate setting where he has been responsible for creating a company-wide data strategy and building out data science use cases.
[](https://twitter.com/ztobi)
[](https://linkedin.com/in/tobias-zwingmann)
### Books
* [AI-Powered Business Intelligence](https://datatalks.club/books/20220606-ai-powered-business-intelligence.html)
(the book of the week from 06 Jun 2022 to 10 Jun 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Todd Underwood – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Todd Underwood
Todd Underwood is a Director at Google and leads Machine Learning SRE. He is also Site Lead for Google’s Pittsburgh office. ML SRE teams build and scale internal and external ML services, and are critical to almost every significant product at Google. Before working at Google, Todd held a variety of roles at Renesys (in charge of operations, security, and peering for Internet intelligence services) now part of Oracle’s Cloud, and before that he was Chief Technology Officer of Oso Grande, an independent Internet service provider in New Mexico.
[](https://twitter.com/tmu)
[](https://linkedin.com/in/toddunder)
### Books
* [Reliable Machine Learning](https://datatalks.club/books/20221121-reliable-machine-learning.html)
(the book of the week from 05 Dec 2022 to 09 Dec 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tomasz Hinc – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tomasz Hinc
Tomek Hinc is a DataOps, living in Poznań, Poland. After working in Product Analytics, Data Engineering, Data Science and ML, he fell in love with Operations. He finds peace in fixing poorly written IAM roles and teaching people.
[](https://linkedin.com/in/tomasz-h-0a1b90234)
### Events
* From Data Science to DataOps ([watch on youtube](https://www.youtube.com/watch?v=lem7knxqNzg)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tomasz Lelek – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tomasz Lelek
Tomasz Lelek has years of experience working with various production services, architectures, and programming languages. He has designed systems that handle tens of millions of unique users and hundreds of thousands of operations per second. Currently, he designs developer tools for DataStax, a company that builds products around Cassandra Database.
[](https://twitter.com/tomekl007)
[](https://linkedin.com/in/tomaszlelek)
[](https://github.com/tomekl007)
### Books
* [Software Mistakes and Tradeoffs](https://datatalks.club/books/20210906-software-mistakes-and-tradeoffs.html)
(the book of the week from 06 Sep 2021 to 10 Sep 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tomaz Bratanic – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tomaz Bratanic
Tomaž Bratanič is a network scientist at heart, working at the intersection of graphs and machine learning. He has applied these graph techniques to projects in various domains including fraud detection, biomedicine, business-oriented analytics, and recommendations.
[](https://twitter.com/tb_tomaz)
[](https://linkedin.com/in/tomaz-bratanic-a58891127)
[](https://github.com/tomasonjo)
### Books
* [Graph Algorithms for Data Science](https://datatalks.club/books/20220926-graph-algorithms-for-data-science.html)
(the book of the week from 26 Sep 2022 to 30 Sep 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tomek Jamiński – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tomek Jamiński
Tomek Jamiński, Head of Data Science, OLX Group
Data native servant leader with broad experience in Data Science, Business Intelligence and Research. Fluent both in the ecommerce as well as the FMCG domain. Uniquely understands the mechanics and cultures of both worlds.
Loves taking on challenges that connect data and business. Passionate in building cross-functional teams. Enjoys working with critically thinking leaders. Focused on developing the strengths of every person.
Holds a MSc in Electronics and Telecommunications from the Poznań University of Technology and a postgraduate degree in Corporate Management from the Poznań University of Economics.
Privately husband and father for whom taking care of the family and homeliness are top priority. In his free time Tomek enjoys aikido and music.
[](https://linkedin.com/in/tomekjaminski)
### Events
* Data Monetization with Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=CoO8PGcOgtE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tommy Dang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Tommy Dang
Tommy Dang is a self-taught coder who grew up in the San Francisco Bay Area and attended UC Berkeley. He spent over five years at Airbnb, working as a software engineer and data engineer, where he developed data tools for developers. In 2021, Tommy embarked on a new venture called Mage, with the goal of making data engineering more accessible. His mission is to provide developers with the necessary data tools to create innovative and enchanting products.
[](https://twitter.com/TommyDANGerouss)
[](https://linkedin.com/in/dangtommy)
### Events
* Data Plumbing without the poop ([watch on youtube](https://www.youtube.com/watch?v=nUfAqM2Sguc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Uri Gilad – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Uri Gilad
Product Manager, Data Governance, Google Cloud
Uri is leading the Data Governance efforts, within the Data Analytics area in Google Cloud. As part of his role, Uri is spearheading a cross-functional effort to create the relevant controls, management tools and workflows that enable a GCP customer to apply Data Governance policies in a unified fashion wherever your data may be in your cloud deployment.
Prior to Google, Uri served as an executive in multiple Data Security companies: most recently as the VP of product in MobileIron, a public Zero Trust/Endpoint security platform. Uri was an early employee and a manager in CheckPoint and Forescout - two well known Security brands. Uri holds an M.sc from Tel Aviv University and a B.sc from the Technion, Israel’s Institute of Technology.
[](https://twitter.com/ugilad)
[](https://linkedin.com/in/ugilad)
[](https://www.amazon.com/Uri-Gilad/e/B08W6QPM69)
### Events
* Data Governance ([watch on youtube](https://www.youtube.com/watch?v=tJ3v8h7A7RY)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Data-Governance---Jessi-Ashdown--Uri-Gilad-e12jmoo)
)
### Books
* [Data Governance: The Definitive Guide](https://datatalks.club/books/20210524-data-governance-the-definitive-guide.html)
(the book of the week from 24 May 2021 to 28 May 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vadim Smolyakov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vadim Smolyakov
Vadim Smolyakov is a Machine Learning Engineer working on Copilot AI in the Business and Industry Copilot (BIC) team at Microsoft. He is a former PhD student in AI at MIT CSAIL with research interests in Bayesian inference and deep learning. Prior to joining Microsoft, Vadim developed machine learning solutions in e-commerce space. In his current role, he is interested in developing Gen AI products at scale and leading diverse teams.
[](https://twitter.com/vsmolyakov)
[](https://linkedin.com/in/vsmolyakov)
[](https://github.com/vsmolyakov)
### Events
* From Algorithms to Agents: Lessons from Building Copilot ([watch on youtube](https://www.youtube.com/watch?v=s8kyzy8V5b8)
)
### Books
* [Machine Learning Algorithms in Depth](https://datatalks.club/books/20250908-machine-learning-algorithms-in-depth.html)
(the book of the week from 08 Sep 2025 to 12 Sep 2025)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Valeriia Kuka – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Valeriia Kuka
Valeriia Kuka is a Content Manager at DataTalks.Club. She focuses on making AI/ML accessible through clear, practical writing. She built a 60K+ AI audience with reposts from Stanford NLP, Amazon Research, and Hugging Face, has worked with AI/ML newsletters and global communities (100K+), and writes concise explainers and historical pieces.
[](https://linkedin.com/in/valeriia-kuka)
[](https://github.com/kavaivaleri)
### Articles
* [15 Free Data Engineering Courses + 5 Paid Courses: Complete Guide](https://datatalks.club/blog/free-data-engineering-courses.html)
* [AI Dev Tools Zoomcamp: Free Course to Master AI Tools for Developers](https://datatalks.club/blog/ai-dev-tools-zoomcamp-2025-free-course-to-master-coding-assistants-agents-and-automation.html)
* [LLM Zoomcamp: Free LLM Engineering Course and Certification](https://datatalks.club/blog/llm-zoomcamp.html)
* [Free DataTalks.Club Courses: ML, Data Engineering, MLOps, LLM & AI Dev Tools Zoomcamps](https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html)
* [MLOps Zoomcamp: Free MLOps Course and Certification](https://datatalks.club/blog/mlops-zoomcamp.html)
* [Data Engineering Zoomcamp: Free Data Engineering Course and Certification](https://datatalks.club/blog/data-engineering-zoomcamp.html)
* [ML Zoomcamp: Free Machine Learning Engineering Course and Certification](https://datatalks.club/blog/machine-learning-zoomcamp.html)
* [20+ Best Free Machine Learning Courses: Learn from Stanford, MIT and Google Without Paying Tuition](https://datatalks.club/blog/free-machine-learning-courses.html)
* [20+ Best Data Science Slack Communities to Join in 2025](https://datatalks.club/blog/slack-communities.html)
* [How to Build a Blood Cell Classifier for Cancer Prediction: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-blood-cell-classifier-for-cancer-prediction-case-study-from-ml-zoomcamp.html)
* [Building Discipline in Machine Learning with ML Zoomcamp](https://datatalks.club/blog/building-discipline-in-machine-learning-with-ml-zoomcamp.html)
* [Key Lessons from ML Zoomcamp: Serena Haidar](https://datatalks.club/blog/key-lessons-from-ml-zoomcamp-serena-haidar.html)
* [How to Build a Waste Classifier: A Case Study from ML Zoomcamp](https://datatalks.club/blog/how-to-build-waste-classifier-case-study-from-ml-zoomcamp.html)
* [DataTalks.Club Community Demographics](https://datatalks.club/blog/datatalks-club-community-demographics.html)
* [How Do Data Professionals Use Data Engineering Tools and Practices?](https://datatalks.club/blog/how-do-data-professionals-use-data-engineering-tools-and-practices.html)
* [How Do Data Professionals Use MLOps Tools and Frameworks?](https://datatalks.club/blog/how-do-data-professionals-use-ml-and-mlops-tools-and-practices.html)
* [How Do Professionals Use Data Engineering Tools and Practices?](https://datatalks.club/blog/how-do-data-professionals-use-data-engineering-tools-and-practices.html)
* [How Do Professionals Use LLM Tools and Frameworks?](https://datatalks.club/blog/how-do-professionals-use-llm-tools-and-frameworks.html)
* [How Do Professionals Use AI Tools for Personal Productivity?](https://datatalks.club/blog/ai-tools-for-personal-productivity.html)
* [Winning Solutions from the LLM Zoomcamp 2024 Competition](https://datatalks.club/blog/winning-solutions-from-llm-zoomcamp-2024-competition.html)
* [8 Newsletters for Data Science, AI, and ML Enthusiasts](https://datatalks.club/blog/8-newsletters-for-data-science-ai-and-ml-enthusiasts.html)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Valerii Babushkin – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Valerii Babushkin
Valerii Babushkin is a leading AI and data science executive with over 15 years of experience building and scaling data-driven organizations. Currently serving as Senior Director of Data, Analytics, and AI at BP, he leads global initiatives across trading, shipping, and customer products. Before BP, Valerii held leadership roles at Blockchain.com, Meta (WhatsApp Integrity team), Alibaba Russia, X5 Retail Group, and Yandex, where he designed and deployed machine learning systems that significantly improved business outcomes and operational efficiency.
A Kaggle Competitions Grandmaster and author of Machine Learning System Design, Valerii is widely respected for his deep expertise in ML infrastructure, system architecture, and applied data science. He has built and led multidisciplinary teams covering machine learning, computer vision, NLP, A/B testing, and data engineering—often transforming legacy systems into modern AI ecosystems.
Beyond industry, Valerii has shared his knowledge as a university lecturer at the Higher School of Economics and the National Bank of Kazakhstan. His work bridges the gap between theory and production, helping organizations adopt AI responsibly and at scale.
[](https://linkedin.com/in/venheads)
[](https://github.com/VENHEADs)
### Events
* Machine Learning System Design Interview ([watch on youtube](https://www.youtube.com/watch?v=0RsmRjar66E)
)
* Why Machine Learning Design is Broken ([watch on youtube](https://www.youtube.com/watch?v=6YBMU6475KQ)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vanessa Aguilar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vanessa Aguilar
Vanessa Aguilar (@veernacular) is a Mexican-American Site Reliability Engineer living and working in Berlin. When she isn’t in front of her computer, she is cuddling with her Pug “Connie”, cooking Oaxacan food, listening to music, or doing all three at once. Through engineering, she aims to create ways to empower her community and share her love for technology.
[](https://twitter.com/veernacular)
[](https://linkedin.com/in/vanessa-aguilar-b749ab182)
[](https://github.com/Vinesse)
### Events
* Observing Python Applications Using Prometheus ([watch on youtube](https://www.youtube.com/watch?v=Jb5ikhhttxc)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Verena Weber – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Verena Weber
Verena Weber believes that GenAI is going to transform the way we work and interact with devices. Her mission is to help companies prepare for this transformation. She has strong expertise in NLP and over 7 years of experience in Machine Learning. She recently quit her job as a Research Scientist at Alexa AI (Amazon) to work as a freelancer. Her background is in statistics. She loves to balance the hours spent in front of the laptop with yoga, meditation, sound baths and outdoor sports.
[](https://linkedin.com/in/verena-weber-134178b9)
### Events
* From a Research Scientist at Amazon to a Machine learning/AI Consultant ([watch on youtube](https://www.youtube.com/watch?v=4RargY8iOaE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Victoria Perez Mola – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Victoria Perez Mola
Born in Argentina, graduated in Systems Engineering, currently working as an Analytics Engineer at Tier in Berlin. Victoria has over 5 years of experience working with ERP systems, reporting and databases, acting as the intersection between business and technology.
She is passionate about using technology to help people make their daily tasks easier. In her free time, she likes to encourage people to enter the IT world by volunteering, teaching and mentoring.
[](https://linkedin.com/in/victoriaperezmola)
[](https://github.com/Victoriapm)
### Events
* Analytics Engineer: New Role in a Data Team ([watch on youtube](https://www.youtube.com/watch?v=C5UcxBwdCEg)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Analytics-Engineer-New-Role-in-a-Data-Team---Victoria-Perez-Mola-e131e3n)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vijay Kiran – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vijay Kiran
Vijay Kiran is a Principal Software Engineer skilled in Big Data Engineering with a passion for Scala, Clojure, and Agile Methodologies. He works at Soda as the lead for the open-source data reliability tools. Vijay lives in the Netherlands with his wife Neha and dog, Bowerick Wowbagger. He holds an Executive MBA from the Rotterdam School of Management, Erasmus University.
[](https://twitter.com/vijaykiran)
[](https://linkedin.com/in/vijaykiran)
[](https://github.com/vijaykiran)
[](http://vijaykiran.com/)
### Events
* Data Quality and Reliability with Soda Core ([watch on youtube](https://www.youtube.com/watch?v=CSqHZ1eJ5is)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ville Tuulos – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Ville Tuulos
Ville Tuulos has been developing tools and infrastructure for data science and machine learning for over two decades. At Netflix, he designed and built Metaflow, a full-stack framework for data science. Currently, he is the CEO of a startup focusing on data science infrastructure.
[](https://twitter.com/vtuulos)
[](https://linkedin.com/in/villetuulos)
[](https://github.com/tuulos)
### Books
* [Effective Data Science Infrastructure](https://datatalks.club/books/20210927-effective-data-science-infrastructure.html)
(the book of the week from 27 Sep 2021 to 01 Oct 2021)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vincent Tatan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vincent Tatan
Vincent fights phishing with machine learning at Google. He uses advanced ML algorithms and MLOps to protect Chrome, Gmail and Android users.
[](https://twitter.com/vincenttatan)
[](https://linkedin.com/in/vincenttatan)
[](https://github.com/VincentTatan)
[](https://medium.com/@vincentkernn)
### Events
* Unboxing Design Docs for Data Scientists ([watch on youtube](https://www.youtube.com/watch?v=vO3Rr50hWSU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vincent Warmerdam – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vincent Warmerdam
Currently Vincent works as a Research Advocate at Rasa where he collaborates with the research team to explain and understand conversational systems better. One of his activities is maintaining a youtube playlist that explains NLP algorithms over at “the algorithm whiteboard”. Before, he was a consultant working on production algorithms for a large range of companies.
Vincent is a firm believer in open source and certainly doesn’t mind to evangalise from time to time. He has a blog about the slightly less obvious aspects in the world of data science over at https://koaning.io. He’s also working on a project that tries to help people get started over at https://calmcode.io.
The future is pretty awesome, all we have to do is build it.
[](https://twitter.com/fishnets88)
[](https://linkedin.com/in/vincentwarmerdam)
[](https://github.com/koaning)
[](https://koaning.io/)
### Events
* Getting Started with Open Source ([watch on youtube](https://www.youtube.com/watch?v=IxV9EH-tphQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Getting-Started-with-Open-Source---Vincent-Warmerdam-epk60j)
)
* Working in Open Source - Probabl.ai and sklearn ([watch on youtube](https://www.youtube.com/watch?v=UPlIETGwTg8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vin Vashishta – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vin Vashishta
Vin works in Applied Machine Learning. He has built and brought products to market with ARR in the $100’s of millions. He is an Engineer Strategist.
His current focus is monetizing machine learning which covers revenue and pricing strategies, model reliability, defining and hiring research/architecture/product management roles, and the path to production.
[](https://twitter.com/v_vashishta)
[](https://linkedin.com/in/vineetvashishta)
[](https://databyvsquared.com/)
### Events
* DataTalks.Club Conference: Career in Data ([watch on youtube](https://www.youtube.com/watch?v=ltFkvoiA57M)
)
* New Roles and Key Skills to Monetize Machine Learning ([watch on youtube](https://www.youtube.com/watch?v=xCjzA_8S4kI)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/New-Roles-and-Key-Skills-to-Monetize-Machine-Learning---Vin-Vashishta-escer6)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Violetta Mishechkina – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Violetta Mishechkina
Violetta is a Python Developer and Solution Engineer at dltHub, the company behind the dlt (data load tool).
[](https://linkedin.com/in/violetta-mishechkina)
### Events
* Data Ingestion From APIs to Warehouses and Data Lakes ([watch on youtube](https://www.youtube.com/watch?v=pgJWP_xqO1g)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vishwas BV – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vishwas BV
a Data Scientist, AI researcher and Sr. AI Consultant, Currently living in Bengaluru(INDIA). His highest qualification is Master of Technology in Software Engineering from Birla Institute of Technology & Science, Pilani, and his primary focus and inspiration is Data Warehousing, Big Data, Data Science (Machine Learning, Deep Learning, Timeseries, Natural Language Processing, Reinforcement Learning, and Operation Research). He has over seven years of IT experience currently working at Infosys as Data Scientist & Sr. AI Consultant. He has also worked on Data Migration, Data Profiling, ETL & ELT, OWB, Python, PL/SQL, Unix Shell Scripting, Azure ML Studio, Azure Cognitive Services, and AWS.
[](https://linkedin.com/in/vishwas-bv-865849120)
### Books
* [Hands-on Time Series Analysis with Python](https://datatalks.club/books/20230612-hands-on-time-series-analysis-with-python.html)
(the book of the week from 12 Jun 2023 to 16 Jun 2023)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Vladimir Haltakov – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Vladimir Haltakov
Vladimir is a self-driving car engineer with a lot of experience in machine learning and computer vision. He loves helping people learn new things by explaining complicated things in a simple and accessible manner. He believes that the best way to learn is by building.
[](https://twitter.com/haltakov)
[](https://linkedin.com/in/haltakov)
[](https://github.com/haltakov)
### Books
* [Everyday ML Questions](https://datatalks.club/books/20220516-everyday-ml-questions.html)
(the book of the week from 16 May 2022 to 20 May 2022)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Wendy Mak – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Wendy Mak
I am currently a data scientist working for a esports platform in London. In previous jobs, I have done a mix of other roles including full stack software engineer, data engineer, data viz engineer and being the ‘data person’ at a startup. In my spare time I like to ‘tinker’ with random data/ML related side projects and cycling
[](https://github.com/wwymak)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Willem Pienaar – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Willem Pienaar
Willem Pienaar is an engineering lead at Tecton and the creator of Feast
[](https://twitter.com/willpienaar)
[](https://linkedin.com/in/willempienaar)
[](https://github.com/woop)
[](https://feast.dev/)
### Events
* Feature Stores: Cutting through the Hype ([watch on youtube](https://www.youtube.com/watch?v=FQYTb4uWljQ)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/Feature-Stores-Cutting-through-the-Hype---Willem-Pienaar-ept6m8)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Will McGugan – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Will McGugan
Will is a software engineer and author, living in Edinburgh Scotland. He is an enthusiastic developer of Open Source software, and is the creator of some very popular Python packages, including PyFilesystem, Rich and Textual. The latter led to him founding Textualize, a company dedicated to producing software for terminals.
[](https://twitter.com/willmcgugan)
[](https://linkedin.com/in/willmcgugan)
[](https://github.com/willmcgugan)
[](https://www.willmcgugan.com/)
### Events
* From Open-Source Maintainer to Founder ([watch on youtube](https://www.youtube.com/watch?v=bwfR9dyxf1M)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Will Russell – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Will Russell
Will Russell is a Developer Advocate at Kestra, known for his videos on workflow orchestration. Previously, Will built open source education programs to help up and coming developers make their first contributions in open source. With a passion for developer education, Will creates technical video content and documentation that makes technologies more approachable for developers.
[](https://twitter.com/wrussell1999)
[](https://linkedin.com/in/wrussell1999)
[](https://github.com/wrussell1999)
[](https://wrussell.co.uk/)
### Events
* From Hackathons To Developer Advocacy ([watch on youtube](https://www.youtube.com/watch?v=vXbMUfHE1OE)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Xia He-Bleinagel – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Xia He-Bleinagel
From full-time stay-at-home mom to Head of Data & Cloud, my journey has been one of clear goals, continuous learning, and resilience.
As a Data Generalist turned Data Leader, my expertise spans machine learning, NLP, data engineering, and AWS cloud architecture, and now extends to helping organizations build impactful Data & AI strategies that drive real business value.
I’m passionate about coaching and mentoring, especially supporting women returning to work or entrepreneur journey. My leadership style combines technical depth, strategic vision, and empathy, empowering teams to grow and perform at their best.
[](https://linkedin.com/in/xia-he-bleinagel-51773585)
[](https://github.com/Data-Think-2021)
[](https://datathinker.de/)
### Events
* Reinventing a Career in Tech ([watch on youtube](https://www.youtube.com/watch?v=D2rw52SOFfM)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Yuan Tang – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Yuan Tang
Yuan Tang is a project lead of Argo and Kubeflow, maintainer of TensorFlow and XGBoost, and author of numerous open source projects.
[](https://twitter.com/terrytangyuan)
[](https://linkedin.com/in/terrytangyuan)
[](https://github.com/terrytangyuan)
### Books
* [Distributed Machine Learning Patterns](https://datatalks.club/books/20240115-distributed-machine-learning-patterns.html)
(the book of the week from 15 Jan 2024 to 19 Jan 2024)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Yulia Pavlova – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Yulia Pavlova
Yulia is working as a Senior Data Scientist at Applied Innovation team at Reuters News agency. For the past year and a half, she’s based in Toronto, Canada. Her work is focused on artificial intelligence applications in news and media industry: in particular, insights extraction from video and image content, continuous speech transcription, and the creation of customized solutions for captioning. Previously, Yulia has been working on big data analytics and computer vision applications in wildlife monitoring. Yulia is Adjunct Professor in Environmental Economics at University of Helsinki, Finland. She has a PhD in Scientific Computing from University of Jyväskylä (Finland) and a PhD Discrete Mathematics and Cybernetics from St. Petersburg University (Russia).
[](https://linkedin.com/in/yuliapavlovaphd)
[](https://github.com/Lumia720)
### Events
* AI-Powered Computer Vision Applications in Media Industry ([watch on youtube](https://www.youtube.com/watch?v=bINeBnWqFwo)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Yury Kashnitsky – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Yury Kashnitsky
Yury’s got a Ph.D. degree in applied math and a Kaggle competitions master tier.
Looking for a balance between academic and industrial ML, he’s currently working as a Senior ML Scientist at Elsevier.
He’s also leading [mlcourse.ai](https://mlcourse.ai/)
– an open Machine Learning course. Yury lives in the Hague, the Netherlands.
[](https://twitter.com/ykashnitsky)
[](https://linkedin.com/in/kashnitskiy)
[](https://github.com/Yorko)
[](https://yorko.github.io/)
### Events
* What Data Scientists Don’t Mention in Their LinkedIn Profiles ([watch on youtube](https://www.youtube.com/watch?v=c6dK1LWpv4g)
, [listen on anchor.fm](https://anchor.fm/datatalksclub/episodes/What-Data-Scientists-Dont-Mention-in-Their-LinkedIn-Profiles---Yury-Kashnitsky-e125jjl)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Zhamak Dehghani – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
 Zhamak Dehghani
Zhamak Dehghani is a director of technology at Thoughtworks, focusing on distributed systems and data architecture in the enterprise. She’s a member of multiple technology advisory boards including Thoughtworks. Zhamak is an advocate for the decentralization of all things, including architecture, data, and ultimately power. She is the founder of data mesh.
[](https://twitter.com/zhamakd)
[](https://linkedin.com/in/zhamak-dehghani)
### Events
* Data Mesh 101 ([watch on youtube](https://www.youtube.com/watch?v=346N_pCtYZU)
)
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Machine Learning Bookcamp – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Machine Learning Bookcamp
-------------------------
#### by [Alexey Grigorev](https://datatalks.club/people/alexeygrigorev.html)
##### The book of the week from 14 Dec 2020 to 18 Dec 2020
Machine Learning Bookcamp: learn machine learning by doing projects and get the skills needed to work as a data scientist or machine learning engineer.
* [Book's page on Manning](http://bit.ly/mlbookcamp)
* [Book's page](https://mlbookcamp.com/)
* [Book's GitHub repository](https://github.com/alexeygrigorev/mlbookcamp-code)
Questions and Answers
---------------------
**Vladimir Finkelshtein**
I’ll take the honor of asking the first question.
Why so many introductory ML books, including yours, choose not to cover timeseries? It is certainly more basic than neural networks and, I imagine, is encountered more often in entry level jobs.
**Alexey Grigorev**
Oh actually my first outline included a chapter on time series, tight after classification.
There were a couple more chapters that we ended up excluding, e.g. working with text and large scale machine learning. There was concern about the side of the book, so we needed to remove something and ended up removing these things.
The main reason: they are not essential for an ML engineer / data scientist, and rather nice-to-have. Most of the time (in my opinion) we deal with classification problems, and sometimes with regression.
And it’s also possible to pick this topic up after reading the book: the book (hopefully) gives enough foundation to read an article about time series and understand what’s going on.
**Wendy Mak**
I would like to ask about testing in ml code– from the table of contents it doesn’t seem like your book covers it? (yet I think this is an important but often missed topic… )
**Alexey Grigorev**
What do you mean by testing? Unit and integration testing or something else?
**Wendy Mak**
yeah, unit/integration testing
**Alexey Grigorev**
In my opinion, these topics are not super specific to ML, and there are other good books that cover these topics (maybe somebody could suggest some?)
So that’s why I didn’t include it. I might be wrong - of course my biases and my own experience influenced a lot the table of contents
**Alexey Grigorev**
I’d be happy to be wrong about it!
**Elle O'Brien**
i think testing for ML code/models/data is young! i think there will be ML-specific tests eventually as part of the development process, but they’re not yet standardized or widespread. Check out this blog: [https://www.jeremyjordan.me/testing-ml/](https://www.jeremyjordan.me/testing-ml/)
**Elle O'Brien**
I have a discussion video with the author of the testing blog, Jeremy Jordan: [https://youtu.be/k0naEYedv5I](https://youtu.be/k0naEYedv5I)
**Neal Lathia**
:question: What motivated you to write a book? Alexey Grigorev
**Alexey Grigorev**
Oh I’m not sure I have a simple answer for that!
It’s not my first book. I previously wrote two books - “Mastering Java for Data Science” and “TensorFlow Projects” (I was a co-author there, with 4 other people)
The first wasn’t really successful - Java is not that popular for ML and I also didn’t invest much time in promoting it
The second was interesting - I really liked the approach of learning different concepts through projects
So when Manning reached out to me saying that they want to write a book with project-based learning (like the TF projects one), I agreed - I wanted to use this approach, but this time using a language more popular than Java :smiley:
**Alexey Grigorev**
That’s only a part of the story of course, I also like writing, I was blogging for a while - in Russian, this website is abandoned now. But I really like this feeling of sharing knowledge. And writing also helps me personally to learn things really well.
You probably know that as well, that’s why you’re also blogging
**Alexey Grigorev**
And last but not least, Luca Massaron’s influence was a really important factor when I decided to write a book. I’ve always admired Luca’s work and also wanted to write books like him. So, thanks, Luca!
**Neal Lathia**
Amazing!
**Neal Lathia**
I also did my earliest ML work with java :see\_no\_evil: - back in the days of using the weka library :joy:
**George Melvin**
Hello! First, let me say that I think that Book of the Week is a fantastic idea :clap::skin-tone-2:
Here’s my question: I’ve recently been thinking a lot about versioning in the ML lifecycle - data versioning, model versioning, feature engineering code,… - and wondered how you solve The Versioning Problem to ensure effective model monitoring, debugging, updating?
**Alexey Grigorev**
That’s a complex topic, and I’m trying to take a pragmatic approach to versioning.
In my opinion, most of the projects, especially at the beginning, don’t need versioning. I’m mostly talking about data versioning because typically you need to use some tools for that. So it adds an extra layer of complexity for projects. If a project dies, spending time on adding this complexity is wasted.
When it comes to model versioning, it’s a similar situation. I typically do simple versioning e.g. with a timestamp which I include in the response, but there are probably better ways of doing it.
**Alexey Grigorev**
Elle O’Brien probably you might want to add something :smiley:
**Elle O'Brien**
Of course I have thoughts on this! As Alexey Grigorev mentions, the need for versioning is proportional to the complexity of your project- meaning, number of people involved, how fast your data/modeling pipeline is changing, and how big the space of models you want to explore is.
When it’s a small project with a static dataset, you can very well get away with Alexey’s approach. More power to you.
DVC (note: i am part of this project and have a vested financial stake in it! so i’m biased) is one of several tools for versioning datasets, code, and models. It basically extends Git versioning and makefiles to datasets and models. Even though we say “data version control” in the name, many people actually use it for models :slightly\_smiling\_face: If your dataset is static, or is only modified via append-only operations that are easy to filter by timestamp, there is less need for true “data versioning”. Keeping track of the way data is transformed to features and then to a model is where Git versioning + makefiles really shines, in my opinion.
I hope that helps some. Curious what others think!
**George Melvin**
Alexey Grigorev Thanks for your thoughts around this topic - it is a challenging problem and a tricky one to get right. I certainly agree about waiting until the R&D stage is more mature and there’s a better sense of the viability of a model.
**George Melvin**
Elle O’Brien Thanks for sharing your thoughts - I agree with what you say on relating the need for versioning to diff velocity in pipeline management. I’ve never quite realised the issue in these terms though so thanks for the insight!
I have used DVC in the past and really like the functionality, especially when working on experiments. I’ve not leveraged it for deployments/rollbacks, however, which is where the biggest challenges (for me) lie. Would be interested to hear from others if they’ve had success.
**Elle O'Brien**
I have a question: do you think classical hypothesis-testing statistics (using p-values to assess statistical significance, for example) is an important skill for most data scientists? Does it have a role in most modern data science jobs?
**Elle O'Brien**
I guess I see it as… a lot of ML involves linear regression (or variants on it). If you only care about prediction, you don’t need hypothesis testing. But if you care about understanding what relationships in your data are meaningful, you probably do. How often does this really come up, though?
**Alexey Grigorev**
Not really often. Typically we need this in experiments
E.g. if we have a new recommender system, we want to test it in online settings - do A/B test or A/B/C test, etc.
The data isn’t necessarily normally distributed, so sometimes we need to do a Mann-Whitney U test instead of the usual T-test. So knowing these kinds of things might be important for that
On the other hand, we have a stats engine for running experiments, so actually this kind of knowledge is not required on a daily basis. We just needed to implement it once and that was enough
So it’s not something we need often, it’s more like a nice-to-have skill. We also don’t typically check these kind of things during the interviews - at least for data scientists.
Data analysts, on the other hand, might actually get a few stats questions during the interviews
**Alexey Shvets**
Questions:
1. What do you like the most in your book?
2. What part of the book would you like to improve/include if you had more time/energy?
**Alexey Grigorev**
Hey Alexey, thanks for your questions
\> What do you like the most in your book?
I like that it’s project-based. For me personally this approach works better than “theory first, application second”. Instead, I first introduce the problem and then show how to solve it
\> What part of the book would you like to improve/include if you had more time/energy?
First of all, I need to find the energy to finish it :sweat\_smile:
When it comes to improving, there are a lot of small things that could be rephrased, edited, etc, but it takes too much time.
Probably instead I’d rather spend this time writing something new once I finish this one
**Alexey Shvets**
Alex thanks for your answers! Honestly I love your approach and your book! I will definitely buy a hard copy as soon as it will be released!
**Alper Demirel**
How do you get the motivation to work with the book? How do you focus?
**Alexey Grigorev**
My content development editor is doing a great job pinging me every week and asking about the progress. That definitely helps.
But what helps even more is the positive feedback I’m getting about the book. After reading some of the messages on LinkedIn, Twitter or here in this Slack, I feel very motivated to work on it.
With focus it’s more difficult though - my kid makes sure I don’t get any focus time :slightly\_smiling\_face: so the only time I can really focus is after 10 pm. Also sometimes I wake a bit earlier and work on the book while everyone is asleep
**Alper Demirel**
Thanks sir for reply, love to your child :star-struck:
**Karthy**
I have seen many books about machine learning. How is your book different from other books? What value do you bring to the machine learning community with your book?
**Alexey Grigorev**
First, I have a very specific audience in mind - software engineers who want to get into machine learning. Being a developer myself, I know the kinds of problems devs have when transitioning, so I try to teach ML in a way that will work for them.
I do that by keeping the book very practical. The amount of theory is kept to the minimum, and I cover the stuff that is really required at work. For example, I talk about model deployment quite early in the book. I also spend a lot of time talking about evaluation metrics.
The format is project-based, so we first start with a problem and then find a solution to the problem. This helps to keep the book focused.
Interestingly, this way worked also for other kinds of audiences - I got quite positive feedback from data analysts who want to transition to data science and even from product managers.
**Alexey Grigorev**
Okay I really like my answer, I’m going to copy it to notion and use it as my elevator pitch when somebody asks me what makes my book stand out :smile:
**Karthy**
Thanks for your replies. I am looking forward to read your book.
**Sejal Vaidya**
This book introduces machine learning to engineers like me really well, and gives a good insight on designing workflows.
Do you plan to write any other book after this, that could cover Machine Learning at Scale?
Here, I refer to some of the questions asked above regarding Versioning, Monitoring, Testing, as well as use of more Distributed technologies.
**Alexey Grigorev**
I don’t think I have enough experience with these kinds of things to write a book about them.
But I do have some ideas of what the next thing may look like.
I want to write about the architectural design of ML services, like how to design and architect a recommender system and things like that.
Some topics I brainstormed about included these:
* A/B testing
* Vandalism detection
* Unsupervised near-duplicate detection
* Duplicate detection in online marketplaces
* Serving deep learning models at scale
* RTB online learning
* Personalized newsfeed ranking
* E-commerce search
* Recommender engine
* Visual search
* Fraud detection in marketplaces
* Matching in two-side marketplaces
* Dynamic pricing
I’ll probably narrow it down to 8-10 topics. Then start with a course first and see how it goes. If I like it, I might convert it to a bunch of articles, and if it works out and I don’t lose motivation, I’ll convert them into a book.
That’s just an idea, let’s see if I actually decide to do it after finishing with ML bookcamp
**Sejal Vaidya**
Thank you for answering and sharing your plan! That’s an interesting list of use-cases. Would love to see more architectural content on those topics.
**Ashutosh Sanzgiri**
Hi Alexey, I am curious as to how you selected the datasets to illustrate the various ML techniques in your book. It is always a challenge to pick datasets that are more than “toy”, and somewhat “real world”. I see that a few datasets from Kaggle (did you select contests in which you can continue to make submissions?). Do you provide next steps for readers to continue with their learning. Sorry, I have only skimmed the book and seen the sections that Manning decodes for you.
**Alexey Grigorev**
When selecting datasets I was thinking about these things:
* how practical is the problem? How close is it to real world?
* how often this dataset is used in other books and tutorials?
I wanted to use something practical, but not something super common
So I didn’t want to use iris, titanic, mnist or Boston housing. Also I decided not to include the flights delay dataset, but was pretty close to doing it
**Alexey Grigorev**
As for the next steps, at the end of each chapter, I suggest to
* try the same thing with some other datasets
* explore a few extra things on your own
I recommend to do these exercises to take most out of the book
**Alexey Grigorev**
The data is not from competitors, but from kaggle datasets, so it’s not possible to submit the solution to kaggle
**Ashutosh Sanzgiri**
Thanks for your replies, Alexey!
**Octavian Mocanu**
Hi Alexey, my question is what kind of deployments do you present, e.g. using SageMaker, KubeFlow… Maybe also some comparatives? Thx :slightly\_smiling\_face:
**Alexey Grigorev**
I was thinking about SageMaker, but there’s already too much AWS in my book :smiley:
So I have these options:
* in chapter 5, there’s a simple flask+docker webapp deployed with aws elastic beanstalk
* in chapter 8, which I’m writing right now, I show how to do it with AWS Lambda and Kubeflow
Oh and it was so nice of AWS to release an update when I was writing chapter 8, so now I need to adjust it to use Docker :sweat\_smile:
Yes, I’ll include a short comparison, when to use which option
**Vladimir Finkelshtein**
Since you mentioned that projects are the focus of your book, have you considered a chapter on how the project is born? How does one arrive to formulating the questions asked in the project? Maybe some examples of bad questions that could be asked about dataset. How does one choose the ML metrics and relate them to the business goals? How is the project planned? I noticed that all ML books tend to focus on the technical side, and later the data scientist are always criticized for their lack of soft skills.
**Alexey Grigorev**
I do a bit of that in the first chapter - I describe CRISP-DM. I’m not sure if I should do extensive coverage of it and have a separate chapter… I probably won’t. It’s too late to change the outline anyways.
But I do want to have a separate chapter on data collection - that will be the last chapter of the book (chapter 9)
**Alexey Grigorev**
I took a note of what you asked, it’s definitely a good topic - perhaps we can cover these topic in more details in some of our events, or a blog post
**Alexey Grigorev**
Thank you!
**dreyy**
For someone new to machine learning and have followed some tutorials. I noticed most of the time is spent of processing the data and how the algorithms work isn’t really explained. Do you consider this when writing the book?
**Alexey Grigorev**
Yes! The datasets that I use are somewhat prepared already, but I also include some data preprocessing steps as well. The focus is on developing the intuition behind the ML algorithms, but it’s not possible to completely avoid data preparation
**Alexey Grigorev**
Btw, you can also check the code of the book to see how much data prep is there - [https://github.com/alexeygrigorev/mlbookcamp-code](https://github.com/alexeygrigorev/mlbookcamp-code)
**Octavian Mocanu**
Maybe it was already mentioned, but I think another aspect that is not always considered in ML books is A/B testing ML models, which in production setups is really important. (Here, too, SM offers this possibility :slightly\_smiling\_face: , but, for sure, it’s not the only framework). WDYT?
**Alexey Grigorev**
I think you’re right! Could be a nice chapter to add, and indeed not so many books cover it (my previous one about Java does cover it btw :smile:)
I somehow had to make this decision and not include it because it felt to me that it’s not a crucial skill for ML engineers / data scientists. And aslo often it’s not even tested during the interviews (probably because the interviewers also have no clue about it)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mastering Machine Learning Algorithms - Second Edition – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Mastering Machine Learning Algorithms - Second Edition
------------------------------------------------------
#### by [Giuseppe Bonaccorso](https://datatalks.club/people/giuseppebonaccorso.html)
##### The book of the week from 25 Jan 2021 to 29 Jan 2021
Mastering Machine Learning Algorithms, Second Edition helps you harness the real power of machine learning algorithms in order to implement smarter ways of meeting today’s overwhelming data needs. This newly updated and revised guide will help you master algorithms used widely in semi-supervised learning, reinforcement learning, supervised learning, and unsupervised learning domains.
* [Book's page on Packt](https://www.packtpub.com/product/mastering-machine-learning-algorithms-second-edition/9781838820299)
* [Book on Amazon](https://www.amazon.com/Mastering-Machine-Learning-Algorithms-understanding/dp/1838820299)
* [Book's GitHub repository](https://github.com/packtpublishing/mastering-machine-learning-algorithms)
Questions and Answers
---------------------
**Sara Lane**
Hello Giuseppe Bonaccorso and thank you for taking the time to answer our questions! I looked at your book on Amazon and it looks really fascinating and comprehensive.
If you could rewrite it today, what algorithms would you include (that weren’t included) and which algorithms would you exclude (that were included), and why?
**Giuseppe Bonaccorso**
Hi Sara Lane,
the book was meant as a “continuation” of Machine Learning Algorithms (2nd ed. too), which contains more fundamentals.
In another project, I’d probably completely remove the Deep Learning part (which can be expanded in a separate book) and focus more on:
* All evaluation metrics with pros and cons
* A deeper (it’s already quite complex) emphasis on statistical learning
* Probabilistic graphical models (more complex methods and examples)
* Time-series analysis (also in this case, with many more details)
The reason is to write a more “complete” book focused on “classic” ML. This is not a limitation, considering the number of applications and the usage of these concepts in the context of DL.
**Sara Lane**
Also, I see that you go through all the different areas of machine learning (or at least most of them) and explain the various algorithms for each of them. Which algorithms, overall, do you think are the most overlooked?
**Giuseppe Bonaccorso**
Dimensionality reduction (both linear and non-linear), component analysis, and, of course, DL models.
**Sara Lane**
Why do you say that DL models are overlooked? In what way do you mean?
**Giuseppe Bonaccorso**
It’s more a marketing problem. In general DL practitioners search for books that emphasize them. This book is about ML (more generically), but many people enjoyed the DL part :)
**Sara Lane**
Thank you for taking the time to answer my questions.
**Wendy Mak**
Hi, my questions are:
* are there any algorithms that you think many people make mistakes when they’re using it? (e.g. not understanding the underlying assumptions of a particular algorithm about data, use it in an unsuitable context etc)
* what are good ways of becoming more familiar with algorithms in depth? (since it can be really dry reading about them, and it’s also not all the interesting to write it from scratch to work on a toy dataset, or at least I don’t ;))
**Giuseppe Bonaccorso**
For example, component analysis is a tricky part, that is often misunderstood. Another area where it’s even too easy to make mistakes is clustering. Many newbies have no idea about the concept of distance measures and tend to use default approaches even when they are completely inappropriate.
In general, every algorithm has been developed with a purpose. That should be the starting point. Why is this algorithm different? Which peculiarities does it have? Answering these questions allows knowing the meaning of all hyperparameters and how to tune them up in real cases. It’s also important to compare the performances, trying to focus the attention on the differences. The expertise can be obtained only starting from the foundations of the algorithm. That’s why, in many cases, it’s also helpful to read the original papers, where the authors explained the context that led them to develop a new algorithm.
**Evren Unal**
Hi Giuseppe Bonaccorso
As far as I see, your book contains wide range of topics.
Congratulations. It must have require quite effort to gather up all that topic in 1 book.
My question is;
To get the most out of your book, Should the reader have any prior knowledge of ml or math?
**Giuseppe Bonaccorso**
Yes. The book requires math knowledge to understand the theory behind the algorithms. In some cases, the paragraphs are self-contained, but, in many others, it’s necessary to give background knowledge for granted. As I often suggest, this can be a good way to learn something new. When you meet a concept you don’t understand, “bookmark” it, go, for example, on Wikipedia (or a text-book), and study what you don’t know. In this way, the gap will be slowly filled.
**Evren Unal**
Thank you for the answer 👍
**Alper Demirel**
Hello Giuseppe Bonaccorso,
First of all, thank you for your time!
* How is it different from other books on the market?
* At the end of the book, what kind of intuition did you want the reader to create about algorithms?
**Giuseppe Bonaccorso**
Hi Alper Demirel,
* The book has been “designed” to focus both on theory (with proofs and mathematical explanations) and practice. All paragraphs contain examples to show how the algorithms work and how to implement them in real cases.
* The goal is to let the reader understand how machine learning can be achieved and which mathematical frameworks have been developed to do so. Every new step in this field requires not to forget the foundations, to avoid the mistake of thinking that progress has been obtained “by magic”.
**Alper Demirel**
I understand sir, thank you very much for your answers! Now I’m starting to wonder more about the book.
**Ghaith Sankari**
Hello Giuseppe Bonaccorso :
My questions are as the following:
1. How you categorize the book: is it for beginners, or for experts, and what is the best usage of it… do you suggest to study it fully .. or to use when data scientist face scenarios and he want to find what is the best way to address them.
2. in case of studying this the book, what is next .. i think it is depend on the interest of the learner.. but I am asking what is next from your side, what is your next book or plan?
thank you
**Giuseppe Bonaccorso**
Hi Ghaith Sankari,
* The book is for mid-level and advanced users. Everybody can choose the best way to use it. In general, I jump directly to the topic I’m interested in, but this is not a general rule.
* I don’t have any plan to write a new book. Every reader can pick the topics that most capture her attention and look for more detailed resources. In general, I add references for this reason.
**Ghaith Sankari**
thank you very much
**Livsha Klingman**
Hello Giuseppe Bonaccorso,
Thank you so much for your time and allowing this opportunity to ask you questions!
ML algorithms are many and there parameters, even more.
* Is there a logical way of choosing an algorithm over another, other than a process of elimination and trial and error?
* Once picking your algorithm, and applying the various parameters suitable for your given project, is there a methodology in the finer tuning rather than memory, trial and error?
* Mastering successful algorithm implementation and developing expert intuition, is achieved from acquiring more knowledge and understanding (logical) or achieved from building on past experiences - successes and failures (deductible)? Or both?
**Giuseppe Bonaccorso**
Hi Livsha Klingman,
* Unguided trial and error is definitely not a good strategy. I always invite the reader to acquire a “basic” awareness and to create a subset of really appropriate algorithms (possibly covering the peculiar aspects of the dataset). For example, in a clustering task, it’s possible always to include K-Means, but if we know that the data is fragmented in irregular clusters (like in a geographic dataset), it’s also helpful to evaluate the performances of algorithms like DBSCAN.
* Hyperparameter tuning can start from default values, but it must go on according to each specific scenario. Considering again DBSCAN, for example, if there are too many noisy points, it’s easy to understand that the radius is too small. Of course, it’s possible also to employ methods like grid search or Bayesian selection. Again, searching without an understanding of the effect of every hyperparameter can result in a waste of time and unacceptable results.
* Definitely both
**Vladimir Finkelshtein**
I noticed that you included VC dimension in the ML fundamentals chapter. I thought it is a notion that lives only in academia. Are there ways one can actually use it? For example, can one use it to anticipate the complexity of algorithm required to solve a classification problem on a given dataset? If so, are there algorithms to evaluate it?
**Giuseppe Bonaccorso**
I agree that VC is a theoretical concept and there’s only a paragraph to explain to the readers the efforts made to evaluate the capacity of a model.
The complexity of this theory is very high and it requires the usage of lots of maths. I referenced books and papers dedicated to this topic, but all practical applications are extremely difficult.
I included this concept for completeness, discussing an easy example. In practice (in particular DL), it’s almost impossible to make a correct evaluation (unless we default in the universal approximation approach). I hope my answer is satisfactory.
**Vladimir Finkelshtein**
Thanks for clarification. I was just surprised to see that it is even mentioned, never saw it in “practical ML” book before. I am myself a mathematician, and I keep being reminded that the industry doesn’t have much reason to care about understanding algorithms or theoretical concepts.
After all, people just need to know which algorithm works better under certain conditions, and in most cases this is summarized in one line in any book/blog with common interview questions.
Was hoping there is a use of knowing what VC dimension is, but I guess there is not much…
**Sara Lane**
This book looks like a serious work that required a lot of research. What gap were you looking to fill when you wrote it?
**Giuseppe Bonaccorso**
It’s indeed a very long work that required a lot of time… The gap I had to fill is the one normally present in papers (where everything is almost given for granted) and practical books (where there are only examples). I wanted to include theory contextualized with practical examples, avoiding too many “holes”.
**Livsha Klingman**
Thank you so much for answering me previously!
Could I ask you another question?
My past ML experience was almost totally through knowledge gaining through asking others and trial and error, and though I got very successful results, my personal understanding of the specific hyperparameters and the ‘weights’ that they provide to each individual algorithm I felt was very limited, but I also did not find adequate resources to lead me to the clarity that I was looking for.
I visibly saw that depending on a given algorithm depended on the value and the effect of a given hyperparameter and not necessary was uniform to the same hyperparameter in other algorithms. Is that correct? And is your book targeting the loophole in information that I am looking for?
**Giuseppe Bonaccorso**
Algorithms are presented together with their peculiar hyperparameters, so it’s relatively easy to make your own personal experiments.
Absolutely, the effect of hyperparameters can be very different when changing algorithms (in some cases, they might not exist at all). But if the algorithm is described together with its hyperparameters, the selection work can be easier.
Of course, trial and error can be helpful, but, at least, you know that a hyperparameter can have an effect or another.
Just to summarize, some experience is necessary, but you should know that, e.g., an L1 penalty will induce sparsity. I hope to be clear.
**Livsha Klingman**
Thank you so much - I’m assuming then that your book will give valuable insight into mapping through the maze of algorithms! Thank you again for your time!
**Vladimir Finkelshtein**
Giuseppe Bonaccorso do you think hyperparameters could be made adaptive in the models? For example, is it possible to adjust the (l1 or l2)-regularization constant during the training of linear regression, instead of doing gridsearch? For examples, some optimizers can somehow adjust the learning rate while training, if they don’t like the progress.
**Sara Lane**
Vladimir Finkelshtein Have you ever looked into Azure ML? It has both AutoML and a Hyperdrive option where you can specify parameters (like the learning rate) and use one of 3 types of parameter sampling: Random, Grid or Bayesian. It sounds like you’re talking about Bayesian Sampling.
[https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters)
**Vladimir Finkelshtein**
i am not familiar exactly with this, but it seems that they replace greedy gridsearch with sampling, but they still run a training session for each choice of hyperparameters. I am wondering if one can adjust those parameters during one session of training (like some optimizers do). Another simple example of adaptive behavior is early stopping, one can think of number of epochs as hyperparameter, but during the training it can change if some conditions are met…
**Sara Lane**
Yes, you’re right - they run a separate training session for each choice of hyperparameters. Interesting idea for the parameters to adapt according to the performance - if it hasn’t been done yet maybe you’ll be the one to do it!
**Rosona**
Hi Giuseppe Bonaccorso! Thanks for taking our questions. I have two I’d like to ask.
1. Do you handle working with datasets with unbalanced labels (e.g. 20 bad labelsfor 50 000 objects)?
2. Do you have a kind of meta algorithm for how you decide the dataset does not contain enough information to answer the question as stated? Or a method for trying to suss that out before trying every algorithm in your book? :)
**Giuseppe Bonaccorso**
Hi Rosona,
1. The techniques to manage unbalanced datasets (like SMOTE) are discussed in the book Machine Learning Algorithms (2nd ed.). In this book, I discuss different semi-supervised algorithms to work with partially labeled datasets.
2. No. I rely on evaluation metrics to understand whether a model is working properly or not. XAI techniques (like SHAP) can help understand how the features are contributing to the outcomes, so a domain expert might check whether the algorithm is working properly or not.
**Samuel O. Alfred**
Hello Giuseppe Bonaccorso I do like your book as it is a good mix of theory and practical. I’m usually concerned when books contain sample code as we all know, libraries like scikit-learn periodically make changes to their tools. How do you react to this ? Follow the tide by making a new edition? Experienced users don’t have a problem making these changes as they know how to source for answers from GitHub and stackoverflow. Beginners become stuck as they don’t have enough experience to know that a change has occurred.
**Giuseppe Bonaccorso**
I start from the assumption that a user that understands the theoretical part, can check the documentation to know, for example, if a parameter has been renamed. Of course, it’s impossible to guarantee complete future compatibility, but I never refer to package, functions, or parameter names, but rather to the mathematical parameter.
**Samuel O. Alfred**
Alright, thank you
**Alexey Grigorev**
Hi Giuseppe!
I know that your book is used as a course textbook by some universities. How did it happen? Do you have a list of courses that use your books as a reference?
**Giuseppe Bonaccorso**
Yes, the books (both Machine Learning Algorithms and Mastering Machine Learning Algorithms) have been used as textbooks. Some time ago I posted on LinkedIn a list (I hope to find it quickly). I never promoted to university, but I have quite a good number of references in academic papers. That’s maybe the reason. I don’t know more :)
**Alexey Grigorev**
That’s nice! Thanks for sharing!
**Vladimir Finkelshtein**
Since you mentioned Shapley values in one of the answers, when do you think interpretability techniques will become a part of standard ML curriculum? Books I have seen rarely mention anything beyond feature importance for decision trees (and even that without explanation of how it works or without mentioning its caveats).
**Giuseppe Bonaccorso**
XAI is a field that still requires a lot of basic research. I think many methods are already part of some advanced programs (like LIME or SHAP), but, considering the importance of their application (e.g., medical imaging), it’s still necessary some time to find out solutions that have the same solidity as the DT/Random forest feature importance. However, interpretability is essential to create engagement and increase confidence, in particular when black-box applications must be employed in critical sectors.
**Sara Lane**
Has working on this book inspired you to develop your own algorithms?
**Giuseppe Bonaccorso**
Working on the examples was an extremely helpful exercise. In fact, I had to find out those elements in the algorithms that had to be emphasized. From this viewpoint, I also become more mentally flexible when working on new algorithms. In particular, in all those contexts where it’s necessary to find “unique” solutions and different aspects of several algorithms must be joined together.
**Sara Lane**
Sounds pretty fascinating!
**Alexey Grigorev**
Hi Giuseppe!
Not sure if this list of books is complete, but it’s amazing! How did you manage to write so many? What keeps you motivated?
**Giuseppe Bonaccorso**
Yes, more or less, it’s complete! I wrote a lot in the past 3 years. Now I’m taking a break. I always liked the idea of expressing the concepts I loved using my language and experience. Therefore I started writing. Every new book is a sorta new step because I keep on learning from mistakes and I discover new possibilities to expand what I’ve already discussed. However, it’s hard work and, when you have a “regular” job, it can become very demanding. That’s why I decided to slow down a little bit and restart when fully refreshed.
**Alexey Grigorev**
Indeed, it’s not easy to do it when you have a job. Do you have some sort of routine that helps you stay on track?
**Giuseppe Bonaccorso**
Discipline. And a lot of working weekends…
**Alexey Grigorev**
Discipline - that’s something I definitely need. Thanks a lot!
**Doink**
Giuseppe Bonaccorso What are the benefits an author get’s by writing a book apart from Monetary benefits?
**Giuseppe Bonaccorso**
Excluding the monetary benefit (which is almost negligible), writing helps to improve all ML skills as it’s necessary to think the concepts from different viewpoints (in particular learner’s one, which is generally one of the most difficult to manage).
**Ufuk Eskici**
Giuseppe Bonaccorso Hello, my question is: What is the difference of this book from other similar books? Ther are so many ML books in the market. Thanks in advance!
**Giuseppe Bonaccorso**
Hi Ufuk Eskici,
as said in other answers, my main goal is to join theory and practice without sacrificing the former for the latter or vice versa. Every paragraph starts with a complete theoretical discussion (sometimes more or less complex) that should help the reader understand how the algorithm works and continues with a practical example. In this way, it’s easier to employ any other framework.
**Ufuk Eskici**
I appreciate for your reply. Thank you!
**Ufuk Eskici**
I appreciate for your reply. Thank you!
**Alexey Grigorev**
Good morning!
Can you tell us about Bonaccorso’s Law? What is it? And how did the name appear? 🙂
**Giuseppe Bonaccorso**
It started from a joke because I used to repeat that it’s possible to learn what is already somehow encoded in the data. A friend of mine suggested me to call it “Bonaccorso’s law”. However, I think the concept is very important because nowadays so many people tend to think that ML is a sort of magic that can invent from nothing.
**Alexey Grigorev**
It is definitely important!
**Sansom Lee**
Not sure if this is the right channel but came across this short paper identifying a bunch of real life technical debt we face daily in ML: [paper](https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf)
**Alexey Grigorev**
This is a great paper! Probably `#engineering` is the best channel to discuss such papers
**Rishabh Bhargava**
One of my favorite papers ever!
**Sara Lane**
How do you think the machine learning world is going to change over the next decade?
**Giuseppe Bonaccorso**
The field that is going to change a lot is certainly is deep learning. Lower and lower hardware prices and more and more powerful systems allow training huge models with tons of data.
There are fields (like the human brain project) that can benefit this research, but the business world is more interested in systems that can be monetized someway. So, today’s “fancy” will probably become more “classical”. Moreover, the diffusion of several automation tools will probably reduce the expertise required by many companies (while it will increase for cutting-edge ones). I don’t know if data science will be the sexiest job for a long time, but I’m sure it will have more and more tools to express its power.
**Sara Lane**
Thank you for your response!
**Alexey Grigorev**
Good morning!
In which order do you think we should learn ML algorithms?
Do we first learn logistic regression and then decision trees? Or first decision trees and then logistic regression?
**Rosona**
Relatedly, is there a red thread (Leitfaden) through the book other than the order it is laid out?
**Giuseppe Bonaccorso**
There’s no specific red line. In particular, considering the different families of algorithms that share only a few basic elements. I generally suggest following the path that best suits everyone’s needs. Sometimes, it’s necessary to “jump back” if a concept is missing, but normally this process works fine.
To answer your question Alexey Grigorev, I don’t think there’s a reason to select one algorithm as the first one. From a statistical viewpoint, logistic regression is indeed a regression, therefore it’s often studied before any other ML algorithm. On the other side, DTs are very easy to understand and they can be presented also to profanes. Considering my personal experience, logistic regression is generally explained before any other algorithm, simply because it’s linear and the logic behind it is mathematically extremely simple. However, there are courses, when DTs are explained first because the “technicalities” can be limited to just a few purity criteria. I don’t think there’s a golden rule.
**Alexey Grigorev**
My personal preference is to do it immediately after linear regression because we can build on top of that.
But this was recently challenged, so I wanted to know what you think about it.
Thanks!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Reinforcement Learning – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Reinforcement Learning
----------------------
#### by [Phil Winder](https://datatalks.club/people/philwinder.html)
##### The book of the week from 11 Jan 2021 to 15 Jan 2021
Reinforcement learning (RL) will deliver one of the biggest breakthroughs in AI over the next decade, enabling algorithms to learn from their environment to achieve arbitrary goals. This exciting development avoids constraints found in traditional machine learning (ML) algorithms. This practical book shows data science and AI professionals how to learn by reinforcement and enable a machine to learn by itself.
* [Book's page on O'Reilly](https://www.oreilly.com/library/view/reinforcement-learning/9781492072386/)
* [Book's page](https://rl-book.com/)
Questions and Answers
---------------------
**Dmitry Yemelyanov**
Hello, Phil Winder! First of all, thanks for kindly agreeing to share your knowledge with us :muscle:
:question: So my question is:
Could it be possible to improve performance of RL agent doing humanoid motions by virtual demonstrations of a person wearing a mocap suit and what, in your opinion, would be TOP challenges in order to do this?
**Phil Winder**
:100: yes. This is a perfect example where Behaviour cloning/Imitation RL will be useful. In fact, this reminds me of a paper that I read a while ago… Here: [https://bair.berkeley.edu/blog/2020/04/03/laikago/](https://bair.berkeley.edu/blog/2020/04/03/laikago/)
Gif for example. 1) Motion capture, 2) no IRL, 3) with IRL.
**Alexey Grigorev**
Apart from multi-armed bandits, what are the other RL techniques that are getting wide adoption in the industry?
**Phil Winder**
Good question but it’s hard to obtain any real numbers on this. From my research/reading, most people tend to follow the media. If a particular algorithm gets media attention then it’s then quite popular in the frameworks which then leads to adpotion.
In general though, the tried and tested, simple models tend to remain the most popular. From basic Q-learning based algorithms, to simple policy gradient algorithms like SAC.
There’s no one-size fits all “best algo” though, like in ML, the “no free lunch” theorem. So you have to evaluate and experiment for your particular application.
**Phil Winder**
Morning all. I’ll be online all week and responding to questions in threads. Be sure to tag me so I don’t miss the message. Thanks in advance!
**Alexey Grigorev**
Good morning!
**Sara Lane**
Good morning Phil Winder!
Which industries do you see being most affected by advancements in reinforcement learning? More specifically, in which industries do you think it will prove the most useful and also will be open to actually implementing the necessary changes?
**Phil Winder**
Hi SL,
For reference, see page 5-7 of the book.
“Industry” is a tricky word because it is broad and out-dated. It’s similar to asking what industry could make use of software. Of course, all of them could. There are opportunities everywhere.
With that said, it’s a valid question. So far, robotics seems to be the number 1 use case. Simply because it’s hard to derive control programs for complex tasks. It’s easier to learn them.
Pricing/bidding/recommendations/advertising/etc. are largely similar tasks and have also had a lot of press.
The finance industry are going to be big users. I’ve spoken to people already that are using it.
Healthcare and specifically personalised medicine is a perfect match, although the regulatory requirements are likely to prevent this from taking off.
The Tech industry can leverage it to much greater extents for automation. E.g. ML, auto-ML, neural architecture search, etc. Lots of mundane automation like Alexa, email control, etc.
And lots more… :smile:
**Sara Lane**
Thanks for the clear response!
I know that all industries could technically use this technology, but I’ve seen that many businesses are hesitant to take on new things. Many of them live by the adage, if it ain’t broke, don’t fix it. So I’m curious more where you see it taking off practically, not theoretically.
What you said about healthcare is interesting, why would regulatory requirements prevent reinforcement learning from improving things there?
As far as pricing/bidding/recommendation/advertising, why is reinforcement learning superior to other forms of machine learning for this?
Thanks!
**Phil Winder**
Healthcare == people’s lives. So there’s lots of rules and regulations to prevent accidents. This means there’s a very high barrier to entry. (I’m talking from a UK/EU perspective by the way :wink: - there may be fewer regulations in, say, the US for e.g.)
Better depends on your application. Testing, experimentation and evidence will prove whether it’s better. But in general, any application that involves mult-step decision making could be improved by RL. ML makes one-shot decisions which are unlikely to be optimal in the long run.
See page 5 in the book.
**Sara Lane**
Phil Winder I just read pages 5-7, fascinating! As you wrote, I always associated RL with robotics and not much else. Thanks for clarifying and I look forward to reading more.
And I believe that you’re correct, in the USA there aren’t as many regulations for these things and I wonder if we will indeed see RL being used in healthcare in the coming years.
**Neal Lathia**
:question: Phil Winder What are your top tips for debugging RL algos?
**Phil Winder**
Great question. Check out chapter 11 for more detail on this.
Here’s some random thoughts off the top of my head:
1. Visualise what is going on (like any data-related task)
2. If you are given the environment, start with the simplest algorithm and work up (e.g. random/CEM).
3. If you have control over the environment/simulation, make that as simple as possible and solve that first. Then make the environment/simulation more complex.
4. Split the tech. If you’re working with deep models, attempt to decouple the training of the deep NN from the RL. Not always optimal, but makes development much easier. For example, use autoencoders, train the autoencoder first and verify it works. Then pass the much lower-dimensional state into the RL algo. It will train much faster (possibly less optimally) and it will be easier to figure out issues.
5. Split the problem. Try and halve the problem. Halve it again. Solve each quarter independently.
6. Consider hierarchical policies (similar to 5). If you can manually design the hierarchy, even better for understanding/explainability. But you can automate that process too.
7. Good old debugging techniques. print’s are your friend.
8. Assert expected array sizes
9. Don’t overcomplicate the reward function.
And more and more…
**Dmitry Yemelyanov**
Great answer! :thumbsup:
**Jayesh Garg**
Are there customers already live with RL algorithms?
**Phil Winder**
Hi Jayesh. Oh if only my sales team had access to that information. :smile:
So, there’s a variety of things that I’ve heard. Some public, some not. Let me try and recall some:
* [Covariant AI](https://covariant.ai/)
demoed a super cool RL-driven pick and place robot.
* I’ve spoken to engineers that have used RL to improve their recommendations.
* I’ve spoken to leaders that have deployed RL as part of a continuous-learning strategy for their ML models.
* I spoke to another leader that managed to reduce the size of the ML team running their core recommendations algorithm by using RL.
And there’s loads of use cases reported in the papers. But of course, whether you call that production or not depends on what they are doing. Many are pure research. But lots are for research on current production systems. For example, this one from the [YouTube team](https://rl-book.com/applications/2019_reinforcement_learning_for_slatebased_recommender_systems_a_tractable_decomposition_and_practical_methodology/)
,
More on [https://rl-book.com/applications/](https://rl-book.com/applications/)
and in the book.
Of course if you know anyone that wants to develop production RL algorithms, let me know. :wink:
**Vladimir Finkelshtein**
What would be an example of environment with which one can experiment at home? I have neither robotic hand at home nor trading partners willing to make biding wars. The card/text/video games are covered in much detail in the books. It will be more interesting to play with something resembling a commercial use case.
**Phil Winder**
Hi Vladimir,
The world really is your oyster here. You can create your own in a domain that you want more experience in (that’s a great way to gain experience). Or you can search through the thousands of gyms other people have created.
For example:
* [https://github.com/topics/openai-gym](https://github.com/topics/openai-gym)
* [https://rlenv.directory/](https://rlenv.directory/)
**Rishabh Bhargava**
Phil Winder what are the most common use cases in industry where problems are framed as supervised learning (or ranking) problems, but you would reframe them as RL problems? what would be the evidence you would give such teams?
**Phil Winder**
Hi Rishabh. Really great question and one that deserves a much more comprehensive and evidence-based answer.
But, if I had to try and fit it in a chat window….
I’d summarise the dilemma by reminding you of the Markov Decision Process (MDP - page 35 of the book).
If you have an environment that has state that can be mutated, if it can be observed, if you can alter the state through your agent’s actions, and if you have a business problem that where it pays to move the environment into a certain state, then by definition you have an RL problem.
To the first part of your question, common use cases masquerading as supervised ML…
Any recommendations task. I think that’s broad enough for you! I would suggest that the vast majority of cases where people use recommendations are optimising for the wrong thing. The goal is to help the user find things as easily as possible so that they value the functionality and keep coming back/buying unnecessary plastic stuff.
A standard solution (I’m grossly simplifying here) would build a model, in a supervised manner, that maps user intent to products, quantified by click through rate or something.
That’s entirely the wrong metric. You could use RL and train over full customer lifecycles. You could train on raw profit. Or the amount of time individual users spend on the site. Or whatever is most applicable for your problem.
So the action is the recommendation (lots of research available on this). The environment is user and possibly the business/products. The observation is the product catalogue, user demographics, past history, information, the weather, etc. The reward is customer lifetime value or whatever.
Look up any of the RL recommendations papers for an academic argument as to why RL is better suited.
**Rishabh Bhargava**
Thanks for the great answer.
On the topic of recommender systems: if the metric that is being optimized might be off, do you think this is more down to Product Managers not setting the metrics correctly or ML engineers choosing to solve for the simpler problem (even though it might not be the right formulation)? Given that this is such a technical domain, who should be pushing for RL adoption?
**Phil Winder**
Like most things in life, I suspect there’s no easy or right answer. I’m no expert in management, but I think POs or PMs should be steering product development, but decisions should be agreed/discussed as a team. Ideas, solutions, metrics, everything, have to be defined by “the team” because no one person can know everything and get everything right.
I have the same argument with people that have the word “architect” in their title. :wink:
**Ashutosh Sanzgiri**
Hi Phil Winder - I am curious as to why RL techniques are not widely used as a means to improve on supervised learning problems - by this I mean both in model training (hyperparameter optimization, architecture search, ensembling models etc.) and in model deployment (measuring model drift, correcting for it etc.)
**Phil Winder**
\> why RL techniques are not widely used as a means to improve on supervised learning problems
Why? I guess it’s some complex combination of attention, media, ease of use, advice, reading, media and something with the word OpenAI or Google in the name. :smile:
I mean, it’s there, it’s possible. Maybe it’s just waiting for someone to wrap it or market it better than the last person? Hint hint, nudge nudge. If you have a spare 6 months on your hands. :slightly\_smiling\_face:
To be fair there are things out there already. For example I’ve used Optuna for hyperparameter optimisation, which has an RL solution in there.
But they’re not selling the fact that it’s using RL. They’re selling the fact that it automatically does hyperparameter tuning for you.
Same with Kubeflow’s Katib. That has an RL mode too.
That’s the thing about engineering in general. People don’t care how the sausage is made. It’s the product that counts. And it’s why UI engineers take all the glory!
**A McCauley**
Hi Phil Winder , congratulations on the book! I have a question on the topic.
Is reinforcement learning considered a crucial approach in robotics (or do you have an opinion on its use for this)?
With a robot learning behaviours through ‘trial and error’ of their interactions. It seems like RL would have many advantages which ML just couldn’t solve in this case - becoming useful with dynamically changing constrains, or environments, they would experience.
**Phil Winder**
Hi A,
Crucial. Hmmm. Depends on how you define the word. I wouldn’t say it’s CRUCIAL, in capital letters, no. You can create perfectly adaquate solutions using simple stuff like PID controllers and inverse kinematics.
The threshold is complexity. Once you need to do something remotely complex, like more complex than just “move to coordinates x,y” or as soon as it involves a non-trivial number of interacting components, then yes, RL is probably necessary.
But I think that’s missing the point slightly. The great thing about RL is the interface. The MDP. It’s a way of defining problems, not solutions. And it can be applied to any project, simple or complex. If the interface is the same then you can use the same processes, the same techniques to solve a wide variety of problems. It scales from simple to mind-bendingly complex, very few ML techniques and say the same.
For example, if you worked for a robotics company and you sold a bomb-disposal robot and a floor-cleaning robot, you’d have to develop completely different architectures, systems, code, solutions, etc. But if you’re using RL, it’s the same. Define the environment, define what you’re trying to do, try lots of actions and learn which ones maximise the reward.
Sorry, rambling a bit. But yes, I think we’re on the same page!
**Alexey Grigorev**
Phil Winder you mentioned in a thread that RL was used for “reducing the size of the ML team running their core recommendations algorithm”.
I’m curious to learn more about it. If it’s possible, can you give more details about this case? How did they actually do it?
And another question - when RL will replace all the data scientists? :sweat\_smile:
**Phil Winder**
Haha. Thanks Alexey. I knew someone would pick up on that.
This is not another clickbait “we all won’t have jobs next year”. :smile:
No I can’t I’m afraid, it’s not public knowledge.
To summarise, consider:
a) a team of 10+ highly educated, very expensive smart people tweaking neural network architectures and running massive expensive experiments (for example). This is what large tech companies do to solve heavily used, data-intensive systems.
vs.
b) an RL algorithm, with a decent reward function, that trains itself over the long term, to solve the actual business metric that the business is keen on improving.
RL can easily match and with effort surpass the performance of that team quite quickly.
To be clear, the engineering challenge doesn’t go away, it shifts. Now these people are curators. Guardians of the RL algorithm that is actually doing the number crunching. There’s still a lot of engineering work that goes into building a system like that, but it’s not pure data science any more.
I’m being intentionally vague and speculative here, but you can see it happening.
edited for clarity.
**Alexey Grigorev**
Makes sense. I like the “guardian” metaphor! Thank you
**Alexey Grigorev**
Good morning!
Phil Winder you mentioned that “the engineering challenge doesn’t go away, it shifts”.
I’m curious to learn about these engineering challenges that come with training and deploying RL algorithms. How are they different from “classical” ML and DL models? What are the typical tools for training and deploying?
**Phil Winder**
Morning Alexey.
First, bear in mind that there isn’t much industrial experience of running RL in production, yet. It’s not like ML, where there’s now years worth or experience to leverage. But I can speculate.
One of the key issues with RL is state. By definition the MDP loop is constantly evolving. New observations, new models, new actions. In particular, if you’re running an algorithm which is actively learning (most, but not all implementations), which means that the underlying state of the model (the trained parameters) are changing ALL the time.
One of the definitions of “modern” software is immutability and software that is free of side effects. By definition, an actively learning RL algorithm is mutable and most definitely has side effects!
So over the next few years I predict that there is going to be industrial research (i.e. new frameworks/blog posts/presentations/etc.) into how to run mutable RL algorithms in a robust way. I imagine that under the hood there will be a strategy to do some kind of checkpointing to make it pseudo-immutable.
On the training side, there’s loads. I can’t keep up. I did a review a long time ago and I’ve been meaning to update it (here: [https://rl-book.com/rl-frameworks/](https://rl-book.com/rl-frameworks/)
). Take your pick.
On the deployment side, less so. Many of the frameworks above have some kind of serving mode, but I get the impression that most people have to roll their own serving infrastructure and tooling.
**Alexey Grigorev**
Thank you! Looking forward to seeing how this field develops
**Ritobrata Ghosh**
Phil Winder, can this book be treated as a primary textbook of Reinforcement Learning or a reference book for studying Reinforcement Learning?
**Phil Winder**
Hi Ritobrata,
Sorry I don’t quite understand your question. Can you explain what you are looking for?
I wrote this book from an industrial perspective. It contains more “advice” that you would expect from an academic reference. It also contains less mathematics than you would expect from academia.
My goal was to try and be a bridge between the industrial, software-driven world and academic research.
**Samuel O. Alfred**
Phil Winder I actually like this question. I have read the popular reinforcement learning book by Bartto and Sutton. I don’t have access to your book. So, is your book an extension or looking at things from a different perspective with the same underlying principles?
**Phil Winder**
Yes, the underlying principals are the same. We’re both talking about RL in the context of the MDP and build up from there.
My book is far more focussed towards industry. I cover more modern algorithms and talk a LOT more about how to do RL in industry. Sutton/Barto’s book is more formal, has a lot more maths, talks less about industrial concerns. In short, Sutton/Barto’s is a textbook. Mine is an O’Reilly book. :slightly\_smiling\_face:
**Phil Winder**
Sutton/Barto’s book is excellent for what it is, by the way. I recommend getting both. :smile:
**Phil Winder**
You can find more info on the main page of the website: [https://rl-book.com/](https://rl-book.com/)
I might add some pages from the preface there too, to answer this question outright.
Thanks!
**Ritobrata Ghosh**
Phil Winder, thanks for the reply. Appreciate it. Look forward to reading your book. I suggest Sutton and Barto’s book to everyone who asks me. Many came back to me looking for an alternative. While not a substitute, Thomas Simonini’s tutorials do offer a different attempt in learning RL. So I was asking you if learners could read your book to gain a fair level of knowledge in RL before eventually graduating towards Sutton and Barto.
**Phil Winder**
Yeah I’d agree with that. Most engineers in industry are probably going struggle a bit with sutton’s because it’s too academic. So yes, I’d definitely recommend reading mine first. :blush:
**Ritobrata Ghosh**
Phil Winder Yes, you are right. Even with my background in Physics, I found Deep Learning textbooks such as Goodfellow’s to be easier to read than Sutton, Barto’s book. I have my answer, thanks. Look forward to reading this book!
**Leonid Kholkine**
Hello Phil Winder!
Happy to see a book more focused on the industry. I know that more and more companies are exploring the application of RL, but, at least in Portugal, it is still in a very embryonary stage. I’m wondering how do you see the adaption of RL by the industry?
**Phil Winder**
Hi Leonid,
Like you said, nascent at this point. But it is moving. I don’t think it will be anywhere as big as the generic ML/analytics industry, which in turn isn’t as big as the software industry.
But as you probably know already, these are just tools in your tool belt. The trick is to pick the right tool for the job.
In terms of adoption, I think it’s being adopted already. It’s just a matter of size. I think it will cascade as more “normal” use cases come into popular industrial culture. And as frameworks/libraries start to offer easy to use and robust RL serving, natively.
In short, we’re fighting against low-hanging fruit here. Quite often something very simple is good enough and/or better than nothing. It takes quite a lot to jump up through the hoops of full ML to full RL.
This probably means that it’s going to be larger companies that adopt first. Smaller ones (at least in the non-tech industry) will probably have to wait.
**Phil Winder**
Yeah, to be clear, I see RL taking a slice of the ML industry. So RL depends on the underlying size of the ML and software industries.
**Leonid Kholkine**
That leads me to another thought, will there be then more out of the box tools as it happens now more and more with ML?
I think that also might shorten this gap
**Leonid Kholkine**
And a more interesting question for me, it’s how do you see RL being applied besides the classical cases such as recommender systems, Auto ML, finances, robotics, etc… :slightly\_smiling\_face:
**Phil Winder**
Hi again Leonid,
I’m afraid I’m going to have to resort to: `${insert any use case here}`. :smile:
Sorry, I couldn’t help it. :stuck\_out\_tongue: It has a very broad applicability. In fact, you could technically use it anywhere you currently use ML. Although it may not technically be more performance. But it many cases it could be.
I’ve been trying to collate use cases here ([https://rl-book.com/applications/](https://rl-book.com/applications/)
). There’s lots already, but I’m already well out of date.
Check out some of the other answers here too: [https://rl-book.com/learn/faq/frequently\_asked\_questions/](https://rl-book.com/learn/faq/frequently_asked_questions/)
Apologies for the generic answer but the real answer is really broad.
**Leonid Kholkine**
That’s a perfect answer, I did miss those use cases :slightly\_smiling\_face:
**Alexey Grigorev**
Good morning!
Phil Winder I know you also have a lot of interest in MLOps. Is there any connection between it and RL in your work?
**Phil Winder**
Great question.
You’re right. I am very interested and we’ve gained a lot of experience delivering MLOps projects.
The connection to RL is the operational part. RLOps, if you will. Just like in ML, data scientists probably aren’t that interested in spending massive amounts of time messing about with infra/tooling. They’re job and responsibility is extracting value from data, not building supporting infra.
The same is true in RL too. The value is delivering the algorithm that optimises the business metric. The Ops part is irrelevant. The business doesn’t care how it happens, just that it does.
But the business certainly does care how long it takes and whether it is operationally viable. They’d be the first to complain if it breaks.
So there is value in the supporting tech/infra, but it’s not directly tied to the business objective. The value is “making it easier for other people to do their job”.
Since RL is hard to do well, and very difficult to operationalise/productionise, RLOps certianly has a very important role to play.
**Alexey Grigorev**
Great, thank you! Excited to see how “RLOps” is going to develop
**Alexey Grigorev**
I was checking the book on Amazon and noticed this:
Best Sellers Rank:
* #136 in Machine Theory (Books)
* #155 in Minecraft Guides
* #165 in Artificial Intelligence (Books)
The second category is quite an interesting one. I’m curious how it ended up there? :slightly\_smiling\_face:
Do you use Minecraft as one of the examples?
**Phil Winder**
Haha. Yeah I saw that too. Hilarious.
The US metrics are more stable, because there’s been more sales there.
But yeah, Minecraft. Someone needs to do some NLP consulting to Amazon to fix their broken catagorisation algorithm!
**Phil Winder**
I’ve just grepped the book and I never mention the word minecraft. So I can only assume that there is some overlap in embedding-space between my content and other minecraft books.
**Ritobrata Ghosh**
No, Amazon should never fix it. Think about it. You get to brag about writing a bestselling book about Minecraft! :wink:
**Phil Winder**
Hahaha. Yay! :joy: Can you imagine…
Media person: “… and here to talk about minecraft is bestselling author…”
Me: “errrm…. blocks and stuff?”
**Ritobrata Ghosh**
Just bragging rights to teenagers! Don’t go deep into it, or you’d be caught!
**Ritobrata Ghosh**
Phil Winder, in your opinion, what factors have prevented the wide adoption of Reinforcement Learning in the industry as opposed to Machine Learning and to some extent, Deep Learning as academic fields widely adopted in the industry?
**Phil Winder**
Hi Ritobrata,
Good question. Probably just a combination of time, media exposure, market size, processing power, low-hanging fruit.
You say
\> ML widely adopted in the industry.
But statistics, and therefore ML, has existed forever. Only recently (i.e. a decade, maybe) has ML “taken off”. So you could argue that it took 200 years for ML to be adopted.
RL originated around the 90’s, so wait until 2290, then ask your question again. :smile:
So the real answer is market size and perception. It goes like this:
IT -> Software -> ML -> DL -> RL.
Because they are applied by/for:
Everyone -> Companies -> One-shot decisions -> Complex decisions -> Strategic/long-term decisions.
Each time you’re reducing the market size. And when you do that you are reducing media exposure. So it might seem like ML has been adopted and RL hasn’t, but in fact the market is just smaller. When normalised the perceived adoption is the same.
With that said, I do think you’re right, it’s not been adopted yet. Mainly because there’s a lack of books like mine and well defined use cases. We’ll get there…
**Ritobrata Ghosh**
Thanks for the detailed answer. :slightly\_smiling\_face:
**Leonid Kholkine**
Phil Winder A bit more of a generic question, but how do you see the field of RL fold out in the next 3-5 years?
**Phil Winder**
The correct answer to this is probably more boring than you were hoping for.
It will expand, it will get used more. It will become easier to use and will become more obvious where to use it (because you can use off the shelf open-source solutions).
Then RLOps will become a thing.
Then people will perceive it as being “adopted”.
Then something else will take the limelight.
If I put my marketing hat on it would sound similar except with more hyperbole! :smile:
**Timothy Wolodzko**
Phil Winder RL is still niche of ML, there’s much less books & courses on it as compared to general ML. Besides your book, what would you suggest for someone interested in learning it? Where to start? What on focus first? Moreover, I have a feeling that online you can find either trivial examples, or the very complicated applications like AlphaGo, with not much intermediate ones. So any suggestions for planning the learning journey further?
**Phil Winder**
It’s the same as anything technical IMO. There is stuff to be learnt, and you can do that by reading. Read all the books and papers you can.
But the real learning experience is… experience. Do it for real. Do it in your company. Do it at work. Then and only then do you learn what you need to learn to do your job.
Yeah, that’s the point. Doing your job is nothing like anybody else’s job. I could tell you to do certain projects but it wouldn’t make sense for your unique situation. Your first challenge is finding a problem that is valuable and makes sense for RL. Then work on that. Start simple. Start with software, then ML, then RL. Work your way up.
What you need is a RL driven learning curriculum that delivers training that suits your unique needs. :smile:
**Alexey Grigorev**
Can one use RL to come up with the best curriculum to study it?
**Alexey Grigorev**
Sounds like a good project :sweat\_smile:
**Ritobrata Ghosh**
Timothy Wolodzko, the author has answered. I would like to add a few things. You could try University of Alberta’s RL Specialization in Coursera, and definitely read the Sutton, Barto’s book. I highly recommend David Silver’s Lectures. There are other lectures from DeepMind as well. There’s Spinning Up from OpenAI. You could also read Thomas Simonini’s indroductory RL blogs. There’s plethora of beginners’ and intermediate stuff out there for RL, not as much as DL and ML, but enough for an individual to learn.
**Timothy Wolodzko**
Ritobrata Ghosh & Phil Winder thanks! I already have some of those books etc, just wanted to learn if there’s anything more I should look for.
**Vladimir Finkelshtein**
I think one obstacle for learning is the lack of plug and play libraries like for more classical ML. Even with openai.gym, some of the recent books have code that doesn’t compile, because the libraries are still being developed and change too often. It is certainly an obstacle for people who are less experienced in programming.
**Phil Winder**
Hi Vladimir Finkelshtein,
Although I have seen people/companies try to do data science without software experience/capabilities, I would recommend that gaining software engineering experience is as important as ML/RL experience.
Software is the language of applications, so if you want to build useable ML/RL, you need software. Of course this doesn’t apply to everything. E.g. you can just about get by with hosted tools for an analytics project and larger companies can hire multiple people with different skils (this is the general solution, by the way). But sooner or later you’ll need to code :slightly\_smiling\_face:
I have sympathy for your despair, however. I think the main issue is the complexity. Any complex system has a million ways to fail and it sounds like you’ve found most of them. :slightly\_smiling\_face:
**Ashutosh Sanzgiri**
Phil Winder Do you think that the field needs a new name or branding? Maybe the word “reinforcement” is not catchy enough or does not have the right connotations? Also when do you think we will start seeing “self-help” apps (weight loss etc.) that claim to be powered by RL?
**Phil Winder**
Not necessarily a new name, no. But I would like to see RL become more mainstream in the ML toolbox.
I’d like to be at the point where people say (at the most general level) “we’re working on a data project and we might need to dip into our toolbox, rummage around, and we might need to pick RL for the job”.
**Phil Winder**
And the term RL represents quite a small spectrum of techniques. You could use the words for all the sub-techniques too if you want to be more specific (e.g. imitation RL, inverse RL, curriculum RL, etc. etc.).
“Claims” are powered by marketing/advertising. So that’s entirely powered by marketing.
I’d suspect at some point someone in marketing will hear the term, go “oh that’s cool, is that like AI?” and then they’ll run with it. “The first app to use RL…”
Then there will be a domino effect, then users will get confused annoyed, and then people will stop using it again and move on to the next thing.
This is why I tend to try to avoid predicting marketing hype cycles. They are so fickle. The core technologies and concepts are useful in certain applications and that is why it will stick around for a long time.
**George Melvin**
Phil Winder Hi Phil, Pavlov’s famous experiments (:bell:) are a great example of reinforcement learning for non-machines. I’m interested to know: do you foresee any interplay between reinforcement learning for biological/machine entities in the future? e.g. do you expect to see research insights from (machine) reinforcement learning having application in psychology, and vice-versa?
**Phil Winder**
Great question. I think the answer depends on how deep you want to go.
At a superficial level, yes, definitely, health apps in particular. RL driven, personalised nudges to help you loose weight, get fit, learn a new subjects, etc. are an obvious use case.
At a slightly deeper level, the introduction of RL in core front-line healthcare, like personalised medicine, shows strong signs.
But at the full-on I’ve-had-too-many-beers-deep level, you could imagine RL providing “life” strategies. Like a personalised, optimal route to getting a job that you want. Or “automated relationships”.
Haha. I need that. Imagine not having to remember anniversaries, the perfect present automatically ordered.
**Phil Winder**
And in pshychological wellness. Yes, definitely.
“Hi Dave, you look sad Dave.”
**Phil Winder**
[https://media.giphy.com/media/11wzSz5pm4WktW/giphy.gif](https://media.giphy.com/media/11wzSz5pm4WktW/giphy.gif)
**Phil Winder**
TIL there’s a poor selection of Red Dwarf gifs available on the internet…
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Deep Learning with Structured Data – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Deep Learning with Structured Data
----------------------------------
#### by [Mark Ryan](https://datatalks.club/people/markryan.html)
##### The book of the week from 18 Jan 2021 to 22 Jan 2021
Deep learning offers the potential to identify complex patterns and relationships hidden in data of all sorts. Deep Learning with Structured Data shows you how to apply powerful deep learning analysis techniques to the kind of structured, tabular data you’ll find in the relational databases that real-world businesses depend on. Filled with practical, relevant applications, this book teaches you how deep learning can augment your existing machine learning and business intelligence systems.
* [Book's page on Manning](https://www.manning.com/books/deep-learning-with-structured-data)
* [Book's GitHub repository](https://github.com/ryanmark1867/deep_learning_for_structured_data)
Questions and Answers
---------------------
**Wendy Mak**
I haven’t read the book, so apologies if these are already covered in the book. I have a few questions:
* Are there any real world industrial use cases of DL for tabular data that you are aware of? (ie company x is using it to solve problem y etc)
* in many instances DL does not offer any significant boost to ‘traditional’ methods such as boosted trees etc for structured data, is there any problems where DL is a clear winner like in computer vision and NLP?
* if you have both structured and unstructured data as attributes for modelling, how would you combine DL/ traditional methods for best results, or would you use DL for the whole pipeline?
**Mark Ryan**
Hi Wendy Mak - great questions:
* Companies can be a bit coy about exactly what model they are using for particular applications, but I have heard that companies that have to detect patterns of fraud (for example, in credit card transactions) have been using deep learning with tabular data. Tabnet (nice description here [https://towardsdatascience.com/implementing-tabnet-in-pytorch-fc977c383279](https://towardsdatascience.com/implementing-tabnet-in-pytorch-fc977c383279)
) would suggest that Google is using deep learning for tabular data, but again I don’t know for sure in what areas. Nvidia has some active research on DL with tabular data, but I expect that is directed at finding new markets for GPUs rather than to solve their own internal work.
* You’re absolutely right that traditional methods have been the “go to” for tabular data. In the book I have a significant part of one chapter comparing an XGBoost solution of the major coding example in the book with the DL solution. In that comparison, XGBoost comes out slightly ahead in raw performance (accuracy and avoiding false negatives), and DL & XGBoost are roughly tied in terms of code complexity and training time. Where DL comes out ahead is flexibility - DL could tackle problems with tabular data that includes columns with unstructured text, for example, where traditional ML methods would be at best clunky, and I believe it’s much easier to create a DL system that can handle a very broad range of tabular data.
* I think that one of the reasons that traditional ML has outshone DL for tabular data is that DL has a reputation for being “hard”. I argue in the book that improvements in DL frameworks (e.g. integration of Keras w. TensorFlow as of TF 2.0), along with much better DL training for non-specialists (such as the [fast.ai](http://fast.ai/)
DL courses and the [deeplearning.ai](http://deeplearning.ai/)
DL curriculum) eliminate that objection.
* For tackling a problem that has both structured and unstructured data, I would argue to keep an open mind about DL for the whole pipeline. I mentioned tables with free-form text columns as a simple example. Columns with JSON structures is another example. For more use cases with, say, images and tabular data, I don’t have direct experience, but I think that such use cases would lend themselves to DL end-to-end as well.
* Finally, I wouldn’t advocate DL for all tabular (or all mixed tabular/unstructured) problems. If there isn’t enough data (at least 10s of thousands of records), or if the tabular structure is really simple, I don’t see DL being beneficial. I also think there are many applications where DL and traditional ML may be closely matched and it may boil down to what’s the simplest engineering problem to solve. The case I try to make in the book is to keep an open mind about DL being a valid option for problems involving tabular data.
**Mark Ryan**
Thanks for your questions and I hope the responses are useful.
**Wendy Mak**
yeap, thanks for the insight :))
**Sara Lane**
Hi Mark,
Thanks for making yourself available for our questions.
What inspired you to write a book about deep learning with structured data?
**Mark Ryan**
Hi Sara - when I was learning about deep learning, almost all of the examples used in the teaching materials had nothing to do with problems from my actual job. Like many people, my job is all about tabular data. A sample DL application to classify images isn’t relevant to my work. NLP is a bit more relevant, but what I really wanted was examples that I could apply to my job. Further, when I asked experts why DL wasn’t suitable for tabular data, the answers they gave me weren’t all that convincing, so I went looking for better answers, and that ultimately led to the book.
**Sara Lane**
Also, do you see deep learning eventually surpassing machine learning (ex. XGBoost) in terms of raw performance with tabular data? If so, can you specify what changes you think are going to happen to cause that?
**Mark Ryan**
Will DL surpass non-DL (e.g. XGBoost) in terms of performance? Maybe. I think it’s a research area that isn’t getting a lot of attention. However, I think that the real question is overall cost of implementing a model in production. If XGBoost requires tons of feature engineering and needs tweaking when the table structure changes, while DL can produce an adequate model without so much feature engineering, then DL may be better for an application even if its raw performance in terms of accuracy isn’t as good as XGBoost. XGBoost may continue to win Kaggle competitions over DL but DL may be better to solve real-world tabular data problems for businesses.
**Vladimir Finkelshtein**
Deep learning is known to be vulnerable to adversarial attacks. The main focus until now was on the image domain, and there the ability of adversaries to execute the attacks is limited (e.g. they usually can’t change individual pixels on your camera, etc..). But this is not the case with the tabular data. Aren’t better security and interpretability of the shallow model almost always more important than extra few points in the accuracy?
**Mark Ryan**
Hi Vladimir Finkelshtein model security and interpretability are important. For model security, I haven’t seen any comparisons between XGBoost and DL on tabular data, though I have seen some work on DL on tabular data vulnerability to adversarial attacks ([https://arxiv.org/abs/1911.03274](https://arxiv.org/abs/1911.03274)
). For interpretability, that’s an area that’s getting a lot of focus in DL in general, so I expect it won’t be a bottleneck for DL on tabular data in particular. As for accuracy, I don’t think it will be the deciding point either way. Overall “cost of ownership” - how much feature engineering is required and how robust is the overall application to changes in schema/table structure - will decide whether DL is better than XGBoost for a particular real-world application.
**Vladimir Finkelshtein**
Thanks for the example. Hopefully, there will be more success with defenses soon.
If everything boils down to “cost of ownership”, wouldn’t AUTOML be the best choice for the majority of the use cases? In particular, it will be the one to make the choice of the technology :)
**Mark Ryan**
AutoML may indeed reach the point where us mere humans won’t have to make choices between DL and something else for tabular datasets. I don’t think AutoML is near that yet for real-world problems.
**Sara Lane**
AutoML isn’t there yet. I’m working on a project now comparing Azure AutoML to Azure hyperdrive (hyperparameter tuning) and I’m getting better results with the hyperdrive. Admittedly, I think that usually the AutoML is better, but clearly that’s not always the case.
**Alper Demirel**
Hello Mark,
First of all, thank you for offering us such an opportunity.
* How effective do you think DL is for regression solutions?
* Is there information about hyperparameter tuning in the book?
**Mark Ryan**
Hi Alper Demirel and thanks for your questions. Insightful question about regression. I’ve had better luck with classification than regression on DL with tabular data, particularly with data sets that are on the lower end of “good enough” for DL (10’s of thousands of records). I expect that DL would be OK for regression given enough data to get a signal, but I only have one example of my own to go on, and that was for a dataset of about 1.5 M records.
**Mark Ryan**
The book does have a section about hyperparameter tuning in the context of the major code example, how the parameters were chosen and which ones a reader may want to tweak as they expand on the major code example.
**Alper Demirel**
Thank you very much for your answers sir
**Alper Demirel**
I’m looking forward to reading your book
**Vladimir Finkelshtein**
So according to your previous answers, the main advantage of DL in structured data is automatic “feature engineering”/non-linear representation of the data. Do you think we can see in the future some pre-trained embeddings for different domains (like the ones for text/images)? For example, one could collect all the health related features from many big data sets, and run it through an autoencoder. I know it’s very vague and speculative, but can such a thing possibly be beneficial?
**Mark Ryan**
I think the advantages of DL for tabular data are less feature engineering as well as flexibility to deal with a broad variety of tabular data (able to deal with tables that colums that have free-form text or encompass BLOBs). Another benefit that could be useful is using DL to exploit the metadata in database catalogs - I think there’s potential to do some interesting things like using the catalog to crawl all the tables in a database and then using DL to automatically generate models on the subset of column/table combinations that are “interesting” (with the definition of “interesting” TBD). Such a scenario, where the table structures are not known ahead of time, requires the flexibility of DL.
I think what you are suggesting is a kind of transfer learning for tabular data. That’s an interesting idea - as far as I know, nobody has investigated that yet.
**Vladimir Finkelshtein**
After googling a bit, it seems like some folks are trying to use transfer learning, but in somewhat nonnatural way - by first transforming tabular data to image and then doing transfer learning on images. I wonder if this is because CNN is more powerful in finding features than other architectures.
I do think that in some instances such trick may be natural. For example timeseries with seasonality can be represented as a nice image.
**Mark Ryan**
Thanks Vladimir Finkelshtein - I hadn’t heard of people taking that approach (tabular -> image -> transfer learning on the images). I wonder what the source of the transfer learning was in this use case.
**Vladimir Finkelshtein**
[https://towardsdatascience.com/fast-and-accurate-learning-with-transfer-learning-on-tabular-data-how-and-why-dfe4e752bb2d](https://towardsdatascience.com/fast-and-accurate-learning-with-transfer-learning-on-tabular-data-how-and-why-dfe4e752bb2d)
for example here it seems they just used ImageNet, so I guess anything can work.
**Vladimir Finkelshtein**
here is the paper:
**Vladimir Finkelshtein**
[https://arxiv.org/pdf/1903.06246.pdf](https://arxiv.org/pdf/1903.06246.pdf)
**Mark Ryan**
thanks Vladimir Finkelshtein!
**Rishabh Bhargava**
Hi Mark, thanks for writing this book - looking forward to checking it out soon. In the meantime, a couple of questions for you:
* I know you mentioned [TabNet](https://arxiv.org/pdf/1908.07442.pdf)
earlier, but are there any neural net architectures that you see becoming the de-facto standard for structured data problems? As an example (and generally speaking), CNNs are the go-to architectures in Computer Visions and RNNs/Transformers in NLP.
* Are there specific industries or use cases involving structured data where Deep Learning should be used more, but isn’t? I’m trying to understand if there are cases when folks are spending a lot of time feature engineering, but one can rely on DL to save the day 🙂
**Mark Ryan**
Hi Rishabh Bhargava - for architectures for DL with tabular data, I’ve seen LSTMs used for examples in financial services. The extended example in my book uses an architecture where each class of column (continuous, categorical, or text) gets a distinct set of layers, so in that example there are really 3 architectures, one for each kind of column, wrapped into a single model. For the future I wouldn’t be surprised to see transformers used in some DL for tabular data applications. Transformers seem to be eating the world, so why not tabular data?
For your second question, I think it’s not so much a case of DL not being used for tabular data as not being considered for tabular data. Regulated industries (where model changes need to be submitted to a standards/regulatory body before going into production) are shy about DL because of its reputation for not being interpretable. There’s great work being done to make DL more transparent and thus more palatable for regulated industries, but I don’t think the research in that area has reached the point where it can convince all the stakeholders that DL is ready for regulated industries.
**Rishabh Bhargava**
Would love to see Transformers for these problems!
**Ritobrata Ghosh**
Mark Ryan, can you provide some insights why Deep Learning, when applied to tabular data, doesn’t always give better results than ensemble methods such as Random Forests?
**Mark Ryan**
Hi Ritobrata Ghosh - this article has a good summary of why DL can lag behind other methods (including columns often being correlated in tabular data, making it harder to tease out the features that are really independent): [https://towardsdatascience.com/the-unreasonable-ineffectiveness-of-deep-learning-on-tabular-data-fd784ea29c33](https://towardsdatascience.com/the-unreasonable-ineffectiveness-of-deep-learning-on-tabular-data-fd784ea29c33)
**Ritobrata Ghosh**
Thanks for pointing out the resource. I will give it a read.
**Vladimir Finkelshtein**
I don’t fully understand the claims in the article. Imbalances in data can be addressed with class\_weight with reasonable effectiveness. Maybe I am wrong, but the main problem of correlations between the features is the convergence of the learning (also for shallow ML models), but I imagine that this could be addressed with regularization.
Missing values are mentioned as a reason, but it is not clear why xgboost deals with it better than deep learning.
The whole story about embeddings of features not being dense - that seems just to be a matter of network architecture. Tokenized text is also not embedded densely, so one needs to construct something that will find nice embeddings.
It seems to me that everything sums to: neural networks are just harder to tune and one has very little intuition how to do it. They rarely work out of the box, unlike decision trees.
**Matt Welke**
Mark Ryan I’m just getting into data science and ML now. I feel a bit overwhelmed with all the options out there for machine learning and deep learning. People throw around a lot of words that don’t mean anything to me. But I know my work has lots of data in tabular format. We have massive SQL data warehouse. Do you think your book would help me find useful things I could do with it when finished the book?
**Mark Ryan**
Hi Matt - I was in your shoes, getting into modern ML (I had a background in symbolic, early 90s style AI, but that had the distinct characteristic of not working) and trying to solve problems with data in relational databases. If you want to see how to apply ML to tabular data and you haven’t done any ML, I suggest starting with an ML overview (like Andrew Ng’s introductory ML course) and get some experience with Python. With that you should be able to try out a classic ML approach like XGBoost on some problems with tabular data. I would be delighted if you were to also get my book, but it does assume basic ML knowledge and the code example assumes some ability with Python.
When you’re ready to try out deep learning, the [fast.ai](http://fast.ai/)
intro course [https://course.fast.ai/](https://course.fast.ai/)
is a great starting point.
I hope this is helpful. You probably have a treasure trove of potential applications of ML in that data warehouse.
**Matt Welke**
Thanks for the reply. My work reimburses me for educational books, so I’ve actually already purchased yours. 😛 I’m just planning my path. I started by taking a Coursera course on GCP data engineering, since that’s what I do at work. It introduced me to a few of the GCP ML products. Then, I got more familiar with Python. Right now, I’m reading the Manning book “Machine Learning Bookcamp” and I’m liking it so far. The first project you code along with in the book used used tabular data.
**Matt Welke**
I’m thinking your book on ML with structured data is my next book.
**Matt Welke**
Thanks for the tips. I might take that course too. I definitely fit into the “without a PhD” category. I only have a college diploma. I spent some time yesterday on Wikipedia learning what normal distributions were lol.
**Matt Welke**
One step at a time, eh.
**Mark Ryan**
That sounds like a great plan. Thanks very much for buying the book and I hope that you find it useful.
**Alexey Grigorev**
hey Matt Welke glad to hear that you like ML Bookcamp :) if you have any questions about it, you can use `#ml-bookcamp` or `#books` or just write me in DM
(Sorry Mark for hijacking your thread)
**Matt Welke**
Oh true that’s how I found this Slack server. 😛
**Matt Welke**
Yeah I’ve only completed the first two chapters I think. I’m super busy this month finishing up studying for my GCP data engineer cert (I take the exam on the 30th). So I’m just doing that for now, and I’ll sink my teeth back into ML in February.
**Ritobrata Ghosh**
Mark Ryan, How will you rate and rank DL frameworks and methods available for working with tabular data (NVIDIA Rapids, [fast.ai](http://fast.ai/)
Tabular, TabNet, etc.) keeping in mind these important factors- effectiveness, ease of use, and resource efficiency?
**Mark Ryan**
Hi Ritobrata Ghosh - I’ve used fastai and Keras for DL on tabular problems, and done some experiments with Rapids. I have looked at TabNet but not used it to tackle a problem. Between fastai and Keras, fastai is easier to get started with because it provides a bunch of support for tabular data, like automatically identifying categorical and continuous columns. The advantage of Keras is that it’s better documented and there’s a larger community of developers working with it (overall, not just on tabular problems). Rapids is essentially a way to do what you would do with Pandas but with much better performance thanks to exploiting GPUs. I got great perf results with Rapids but when I ran into some install / config issues trying to use it in different environments. For example, I could get it to work in Paperspace Gradient, but not for multiple sessions, and it worked in Colab but not consistently. Since perf wasn’t critical path for what I was working on, I didn’t pursue it.
**Mark Ryan**
Overall, I would recommend fastai for somebody starting with DL for tabular data, and starting with Keras for a more production-oriented application, but assuming fastai gets used more, it could be good for production as well.
**Mark Ryan**
Here’s a high-level overall comparison of fastai & Keras: [https://youtu.be/3d6rGGyPR5c](https://youtu.be/3d6rGGyPR5c)
**Ritobrata Ghosh**
Thanks for the suggestions, I will go through them.
**Ritobrata Ghosh**
Actually, I am a heavy user of PyTorch and its ecosystem and derived tools. I would like to stick with it.
**Ritobrata Ghosh**
With [fast.ai](http://fast.ai/)
’s tutorials and documentation, I did not have any issues getting started with it for tabular data.
**Ritobrata Ghosh**
Mark Ryan, what do you think of PyTorch Lightning (and its API and efficiency) with respect to working with tabular data?
**Mark Ryan**
Hi Ritobrata Ghosh - I have not used Lightening, but from what I understand, its relationship to vanilla PyTorch (from a user perspective) has some similarities to Keras’ relationship to TensorFlow. I have used & like fastai (see responses above), and it seems to have some overlap in use cases with Lightening. This exchange includes some contrasts between Lightening and fastai: [https://forums.fast.ai/t/fastai2-vs-pytorch-lightening-pros-and-cons-integration-of-the-two/71341/11](https://forums.fast.ai/t/fastai2-vs-pytorch-lightening-pros-and-cons-integration-of-the-two/71341/11)
**Ritobrata Ghosh**
Mark Ryan, would you recommend RAPIDS AI for non-Deep Learning tasks? This question is kind of naive. I mean, for really large datasets, even datasets with 10 gigs of data, Pandas doesn’t work out of the box. You have to tune it, tweak function parameters, etc. if you don’t have large memory at your disposal. So, People use Dask, or DataTable (from [H2O.ai](http://h2o.ai/)
) for these tasks. Do you recommend continue using those tools or move to RAPIDS, or would using RAPIDS for non-DL tasks would be an overkill?
**Mark Ryan**
Hi Ritobrata Ghosh if you have GPU capacity and a large dataset RAPIDS seems to be a good fit even if you’re not doing deep learning. I saw really good performance with RAPIDS compared to Pandas, when I was able to get RAPIDS working consistently. Paperspace has a RAPIDS-enabled Gradient notebook environment that makes it fairly painless to experiment with RAPIDS: [https://gradient.paperspace.com/integrations/nvidia-rapids](https://gradient.paperspace.com/integrations/nvidia-rapids)
**Ritobrata Ghosh**
That’s what I wanted to know. And I didn’t notice Paperspace having a ready-made RAPIDS environment. I will certainly head over there soon and get my hands dirty. Thanks a bunch.
The nextt time I handle tabular data for a project, I will seriously consider RAPIDS over DataTable.
**Alexey Grigorev**
Some time ago I took active parts in Kaggle competitions and for tabular data the winners usually use xgboost or some other GBT method. However, there’s always one or two people in the gold who used a neural net for their solution
So clearly it’s possible to get great performance with neural nets, but it’s not the usual choice. Why do you think it’s the case?
Is it more difficult to do it with neural nets, or just xgboost is the to-go method for tabular data, so other models don’t get so much attention?
**Mark Ryan**
Hi Alexey Grigorev - I think DL is not considered for tabular data problems because (a) the Kaggle success of non-DL that you refer to (b) somewhat dated perception that DL is more difficult to develop with than non-DL (c) model intepretability - a legit concern, but becoming less so, (d) unlike other use cases where DL has blown the doors off alternatives in terms of performance, DL and non-DL have similar performance in many tabular data problems.
What needs to change for DL to be taken more seriously as an option for tabular data problems? Better understanding in business of research on making DL more interpretable and a more realistic assessment of how Kaggle results do (and do not) apply to real-world applications with non-curated, “wild” datasets.
I hope that my book contributes to a more balanced approach to applying DL to tabular datasets.
**Alexey Grigorev**
Thank you!
**Alexey Grigorev**
When it comes to interpretability, xgboost is as black-boxy as neural nets in my opinion
**Ashutosh Sanzgiri**
Alexey Grigorev Could you elaborate on why you think xgboost is “black-boxy”? Tools such as LIME and SHAP work fine with xgboost models.
**Ritobrata Ghosh**
Alexey Grigorev, anything that goes beyond Lin Reg and Log Reg, is not fully interpretable, IMO.
And to solve pressing real world problems, I don’t think we should limit ourselves to using algorithms based only on interpretability. IMO, we should shift our focus more to problem-solving than interpratability.
I also understand that in some fields, interpretability is crucial or even mandatory. We have to stick with Log Reg in those fields for a while.
**Alexey Grigorev**
Yes, I agree that we don’t have to stick to logreg for the rest of our lives
I’d also add decision trees to that list of out-of-the-box interpretable models
**Doink**
I think all classical ML Algos are interpretable right?
**Alexey Grigorev**
by classical you mean linear? or not-deep-learning?
**Doink**
By classical I mean Linear, Logistic Regression, SVM, KNN, Naive Bayes, Decision Tree, Random Forest, Tree Ensembles. Not Neural Nets
**Sara Lane**
Can you tell us about your favorite DL-with-structured-data project that you worked on and why it was your favorite?
**Mark Ryan**
Hi Sara Lane - I really enjoyed the project described in the book because it involves a subject area (transit) that I’m very interested in, and the dataset was incredibly real-world & messy. However, I have to say my favourite DL with tabular data project was one I did when I was back at IBM. I was responsible for the support team for Db2 relational database, and I wanted to predict from past support tickets when a client would escalate (make a duty manager call). The dataset (support ticket records) was interesting, and I was motivated to make the solution work because I fewer duty manager calls meant fewer interruptions for me on weekends. I describe the project in more detail in this article: [https://medium.com/@markryan\_69718/deep-learning-on-structured-data-part-3-9bff73cc77c4](https://medium.com/@markryan_69718/deep-learning-on-structured-data-part-3-9bff73cc77c4)
**Sara Lane**
I read the article - fascinating, thanks for sharing! Curious - can you venture a guess as to which fields made the difference?
**Mark Ryan**
Hi Sara Lane - the subject column (a free-form text description of the problem entered by the client who opened the ticket) had a surprising impact. When I started doing models with this dataset I expected this column would be pretty useless because it varied so much (from “help - db crashed” to an SQL error code to a long summary of symptoms) and a third of the time it wasn’t even English text. But with even with rudimentary NLP on this column it ended up helping the model quite a bit, given a sufficient volume of ticket data to train on.
**Sara Lane**
Wow! So interesting! You’re right, I wouldn’t have expected that, especially since like you said a third of the time it wasn’t even English. Now I’ll think twice when typing in the subject for a support ticket…
**Sara Lane**
I’m thinking about this more, I’m wondering if maybe the ability to clearly and concisely express one’s issue in the subject column is connected to communications skills in general. Meaning that if someone can clearly state their problem in the subject column, it’s more likely that the issue will be resolved without reaching a crisis level. But if someone has poor communication skills, that will reflect in the subject column and will also more likely result in a Duty Manager call.
This is probably an over-simplification, but I think it’s an example of how deep learning can often demonstrate connections that we were never aware of.
**Ritobrata Ghosh**
Mark Ryan, in your opinion, what are the paths forward to alleviate the problem of ineffectiveness of DL in tabular data?
**Mark Ryan**
Hi Ritobrata Ghosh - despite the title of this article [https://towardsdatascience.com/the-unreasonable-ineffectiveness-of-deep-learning-on-tabular-data-fd784ea29c33](https://towardsdatascience.com/the-unreasonable-ineffectiveness-of-deep-learning-on-tabular-data-fd784ea29c33)
- I think the problem isn’t so much the universal ineffectiveness of DL for tabular data. I think the problem is the common perception that DL isn’t even worth considering for tabular data. How to alleviate that problem? Keep an open mind, take advantage of the frameworks that exist now to do proofs of concept with DL on tabular data, and don’t assume that what works best for winning Kaggle competitions will work best for production solutions for businesses.
**Ritobrata Ghosh**
I have asked this from a research perspective. What should be DL practitioners’ approach to alleviate the ineffectiveness of DL in case of tabular data?
**Mark Ryan**
Thanks - that’s a great way to look at it. From a research perspective I think there are two approaches that could yield results. The first is to look at how the metadata about tables that exists in relational database catalogs could be harnessed. I think catalog information is a huge, untapped resource - every table has it and there’s a core of catalog info that is common across just about all database vendors. Second, and I think this is harder, I think there is more work that could be done on formalizing what combinations of columns / characteristics of tabular data lead to better outcomes w. DL. For example, in his book on fastai, in the section about DL w. tabular data, Jeremy Howard states that it works well when you have categorical columns with lots of distinct values. Enhancing heuristics like this with a more comprehensive analysis would be very useful.
**Ritobrata Ghosh**
Thanks so much for the response. I have never thought about using relations in an RDB can be seen as inputs to a DL model. That’s a great idea. I have not come across a paper that does this.
And I remember Jeremy Howard’s video and also the chapter involving tabular data. He asks the learner to rely on heuristics before putting the data through a DL model. And that really works well.
**Mark Ryan**
Thanks to everybody who participated this week and thanks very much to Alexey Grigorev for providing the venue. I really enjoyed the questions and the exchange of ideas.
If you get the book I encourage you to leave a review on Amazon - thanks!: [https://www.amazon.com/Deep-Learning-Structured-Data-Mark/dp/1617296724/ref=sr\_1\_1?c\[…\]+learning+with+structured%2Cstripbooks-intl-ship%2C312&sr=1-1](https://www.amazon.com/Deep-Learning-Structured-Data-Mark/dp/1617296724/ref=sr_1_1?crid=2NBP30UIUGEOS&dchild=1&keywords=deep+learning+with+structured+data&qid=1611328706&s=books&sprefix=deep+learning+with+structured%2Cstripbooks-intl-ship%2C312&sr=1-1)
**Alexey Grigorev**
Thank you Mark for agreeing to take part in this event and finding time to answer our questions!
**Alper Demirel**
Thank you very much for your answers 🤩
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Teams – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Teams
----------
#### by [Jesse Anderson](https://datatalks.club/people/jesseanderson.html)
##### The book of the week from 01 Feb 2021 to 05 Feb 2021
Are you starting a data team and don’t know where to start? Are your data teams working but not producing, and you don’t know why? I’ve written this book to share my extensive experience helping companies create value with data.
Data Teams goes in-depth on the unified model for creating successful data teams. Being successful includes ensuring you have the three fundamental teams: data science, data engineering, and operations. Without all three teams, the data teams won’t achieve their highest and best output. For some organizations, the team model is there but isn’t working. Data Teams helps you diagnose the problems so you get the right teams in the right places.
* [Book's page on Apress](https://www.springer.com/us/book/9781484262276)
* [Book's Page](https://www.datateams.io/)
Questions and Answers
---------------------
**Wendy Mak**
Hi Jesse Anderson, I have a few questions:
for data teams, when would you choose a more ‘centralised’ mode (central data science team where all the business teams go to ) vs a ‘distributed’ mode (each relevant business team having one or two DS people working with them)
**Jesse Anderson**
I’m looking at this once we’re clear on best practices and doing them is second nature. Also, I’m verifying that we aren’t trying to move out of a centralized team because the data teams aren’t giving enough time to the relevant business team. Those issues don’t magically go away.
**Wendy Mak**
How would you encourage business people to do more data analysis for themselves, e.g. if there’s an in house metrics dashboards/analysis tool, how would you encourage biz people to use that and not just ask the data team to get the data for them?
**Jesse Anderson**
Through good data democratization and training. Make sure that you’ve given a place to query data while having the skills to do it.
**Wendy Mak**
How would you timebox/plan a proof of concept type project where it’s harder to gauge how long things would take
**Jesse Anderson**
If a project is that difficult and the team isn’t experienced enough, I’m looking for outside help. This outside help wouldn’t be writing the code but helping to gauge the difficulty relative to the team’s skills. That would give you a better understanding.
**Wendy Mak**
If you are in a more senior role in a DS team, how do you balance time between working on your own working and helping other team members and generally making sure all the projects are on track?
**Jesse Anderson**
Look at the StitchFix interview. I think they had an interesting way of dealing with this by having working managers. I’d be making sure that your help is actually improving others. E.g. do they consistently make the same mistake? If so, this may be a skill or knowledge gap that needs training rather than help.
**Filipa Castro**
Hi Jesse Anderson! Which method would you recommend for managing the work from a data team? I’ve seen people using sprints, Kaban, CRISP, but I’m not yet convinced…
**Jesse Anderson**
There hasn’t been much written on this subject. It was part of why I included each interviewee’s project management framework. I prefer scrum or Kanban when I’m working with a team.
**Wendy Mak**
do you have any recommendations for best practices around collaboration (git and so on?)
**Jesse Anderson**
Git, wikis, good documentation in both code and architecture, and fomenting a culture of sharing and communication.
**Alessandro Lavelli**
Hi Jesse Anderson , in data focused team data versioning is crucial and it is recommended to keep the same version ID both for code (feature extractor and models) and data sets. How manage this when multiple people work on the same source or when there are many data sources? Thanks a lot!
**Jesse Anderson**
There are software projects that address this issue. I think Datatron does this and some others too. IMHO, this is a mix of technical and business process that’s breaking down.
**Alessandro Lavelli**
Thanks!
**Preetdeep Kumar**
Hi Jesse Anderson - As a manager of a data team, how do you define role of product owner (PO). I prefer a PO who is also aware of SQL and familiar with tools like AWS Athena or Presto.
Second how we define a boundary for Data teams or their exist overlaps and ambiguity which can’t be done with.
**Jesse Anderson**
Brian O’Neill and I did a podcast about this [https://designingforanalytics.com/resources/episodes/053-creating-and-debugging-successful-data-product-teams-with-jesse-anderson/](https://designingforanalytics.com/resources/episodes/053-creating-and-debugging-successful-data-product-teams-with-jesse-anderson/)
. I ideally want a PO who is very close to the business problem and is technical enough to validate the data product meets the needs of the business.
**Jesse Anderson**
For the boundary, I think that’s a team maturity issue. I’d be focusing on the symbiotic relationships between the individual data teams. This will help them solve an ambiguity quickly or improve the times when the teams need to work together to solve a problem.
**Jesse Anderson**
Be sure to check out the extras I created for the book. I created a series of videos to complement the book. I did one more case study with Criteo that was really interesting. Get them at [https://www.datateams.io](https://www.datateams.io/)
.
**Ricky McMaster**
Hi Jesse Anderson - Considering that data teams generally receive ad hoc, unplanned requests on a regular basis, do you think this can be accommodated structurally within Scrum? Or is it more realistic to opt for Kanban?
**Jesse Anderson**
For analytics, lots of ad hoc tasks are unavoidable. If all tasks are ad hoc, that’s a sign of another problem. This especially applies to data engineering. For ad hoc that’s correct, I usually recommend Kanban.
**Ricky McMaster**
Great, thanks a lot. The proportion of ad hoc stuff is usually ~15-20%, so the challenge is to balance this with the strategic, planned tasks in a way that works for the team and doesn’t create too much admin overhead for them.
So far, the best idea we’ve come up with is 2 boards - one for Scrum, one for Kanban. The Scrum board could contain ‘placeholder’ tickets for the expected Kanban commitment (effectively timeboxing), so that this anticipated effort is budgeted in advance alongside the strategic issues.
**Jesse Anderson**
Another way would be to divide the team. One doing ad hoc and the other long-term. Have the team members rotate through this so they’re not getting bored.
**Ricky McMaster**
Yep, we do that to an extent - I’m thinking of making each ad hoc ‘shift’ shorter though, max 2 days, for that very reason (boredom). Anyway thanks a lot for your responses! Super helpful for me.
**Sara Lane**
There are many ways to manage data teams. How did you go about doing research for this book?
**Jesse Anderson**
I’ve been consulting and working with data teams for the past 9 years. There’s a great deal of my experience and comparing notes with others. The majority of the book is my observations of best practices and how I consult with my own clients. I didn’t want the book to be just my own views. I had others contribute viewpoints and conducted interviews. The interviewees were chosen based on me asking my network who would they hold out as examples of well-functioning teams.
**Rosona**
Hey Jesse Anderson! Excited about this topic. Thoughts on how to take a group of siloed independent researchers (university backgrounds, now in industry in the same team) who each use a different programming language and with a different focus area and make them a “team”?
**Jesse Anderson**
This will be a really difficult task. There are three things in your question that make me worried: “researchers”, “different languages”, and an apparent lack of data engineering. You want to be careful of this.
To your question, there are likely some skill gaps. You’d want to standardize on 1, maybe 2 languages. The others will need to learn that language. You’ll want to explain why there is a need for teams. IME researchers aren’t as used to being in a team. Show them how being in a team will improve the situation. e.g. if something breaks while they’re on vacation, it can be fixed rather than trying to call them.
**Rosona**
Yeah, I was thinking what is really needed is projects where people are forced to work together. Not sure how to encourage that, but I’m also not the manager or pseudo group lead. Thank you for your thoughts.
**Jesse Anderson**
that will be even more difficult for you to enact change without more power. try to get your management to read the book.
**Rosona**
And another question for Jesse Anderson: if there is no active leadership in your team (no dailies/weeklies, no one tracking progress concretely, plans only on the level of “we should try to do x in this quarter”) – what can a team member do to encourage cohesion? Is it necessary to effectively take the reins? Are there lessons from scrum-style “servant leadership” that can be brought to bear?
I see this as a data teams theme since the group mentioned is sort of research flavored, people are developing models at whatever pace and creating proof of concepts to try to sell the notion of using data science and its value. I can’t imagine this is an uncommon kind of side problem, that too much leeway is then given.
**Jesse Anderson**
Pairing your two questions together, this team’s dynamics are a big worry. I winced while reading your question because this won’t improve without concerted effort and buy-in from the management. In your situation, I’d be trying to convince management of the need to improve. This could manifest as a real train wreck if it isn’t already (sorry). Servant leadership will only get you so far on this one. This situation isn’t uncommon because I’ve already seen it. They become total free-for-alls that don’t get anything out the door. Take an honest look if this is going to improve or not.
**Ash Smith**
Hey Jesse Anderson,
Great book! Great timing for me in the stage of career too.
I’m a PO in a data team and feel I protect the team too much from the business and stakeholders and want them to get out into the business more (dogfooding more 😄 )
Any thoughts on this? Some engineers just want to write code but never want to understand how the analysis actually wants to use the data.
Thanks!
**Jesse Anderson**
The engineers have to do this. If you use Agile, the engineers were always supposed to work with the business too but could get by without talking to them. Now as data engineers, it isn’t an option anymore. You should figure why the data engineers don’t want to. Is it laziness, too much work, personality conflicts? Once you’ve figured that out, you can start working with the individuals on those flaws.
**Vic**
Hi Jesse Anderson
How would you approach the distribution of seniority in a data team in a start up? Having too many senior people could mean they end up doing boring job and having too many juniors could end up generating low quality work or using up too much of the senior people’s time.
**Jesse Anderson**
Data teams tend to skew senior. If I see a team mostly junior people, that’s a problem. The boring jobs come with the territory. The senior people should be able to finish them much quicker by using the right tool for the job.
**Sara Lane**
What are the key components to creating a successful data team?
**Jesse Anderson**
Having all three teams present and with the right skills/people on the teams. You should be working with the business to deliver value from the data.
**Ashutosh Sanzgiri**
Jesse Anderson - what metrics do you think are appropriate for measuring the success of a data team? are there ways you can use data science techniques e.g. monitoring team communications, code checkins etc. to provide feedback and improve the efficiency of data teams?
**Jesse Anderson**
I talk about these metrics in the book. The overall metric is are you creating usable data products that the business can use and those data products generate business value.
**Sara Lane**
Do you see an advantage to teams working together on-site or do you think they can work equally well remotely?
**Jesse Anderson**
There is a certain comfort and experience with on-site teams. It’s simply what we’ve been used to. With the right changes and communication, they can be equally effective. Done right, I think remote teams can be more effective and have an easier time with hiring.
I talk more about this in my interview with [Criteo](https://www.datateams.io/extras/)
. Also, see the [survey results](https://www.jesse-anderson.com/2020/12/data-teams-survey-results/)
about data teams and COVID.
**Sara Lane**
Thanks!
**Rishabh Bhargava**
Hi Jesse Anderson, thanks for writing the book and answering the questions here.
My question is around specialization. How do you think data teams should think about specializing in tools/skills? For example, should everyone know how Airflow works, or should there be a go-to guy or gal for implementing pipelines? Alternatively, should everyone be reasonably proficient in the core skills/tools that are important to the data team? Or should this be thought through on a project-to-project basis?
I ask because there are so many tools (and everyday there’s a new one) that folks work with, and each and every one of them has its own unique quirks.
**Jesse Anderson**
For the base or foundational technologies, everyone should know them. For others, there should be at least three people. This way, you can have vacations without worrying about coverage. The issue of so many tools is the nature of the data engineering beast. It isn’t going to change any time soon.
**Vic**
Hi again Jesse Anderson , and thanks for your replies so far.
How should we address ownership in a data team? it’s very easy and natural by silos. But, some models affect more than one siklo, for example, finance and commercial. How to define ownership in such cases?
**Jesse Anderson**
Having two of something results in all kinds of problems. I think there is a team that’s responsible for the data product that is actively working with other teams to add or improve it.
**Jeanine Harb**
Hi Jesse Anderson!
Thanks for writing the book. Came here because I was looking for a book about Managing Data Engineering Projects, and your book was suggested to me!
I am working on a project where I was tasked to convert some un-versioned enormous SQL query, into a full-fledged data pipeline. We are 2.25 engineers on the project, and not having managed projects before, I am finding it hard to work in a scrum-like way, where we define tasks and organize our work into stories. A lot of the code is being built from scratch, with a clear end goal though.
Do you have any advice on how to tackle organization challenges in a data environment? What should be the main focus and guiding point for every iteration?
Thanks a lot!
**Jesse Anderson**
Firstly, I’d be checking if those 2.25 engineers meet the skills qualifications to fix this. From the sounds of it, this sort of project should lend itself well to scrum. You’d have to give me more context to give a suggestion.
**Alexey Grigorev**
If somebody wanted to follow your path and get into consulting - especially consulting companies about establishing data teams. What would you recommend them to do?
**Jesse Anderson**
Join me and together we will rule the galaxy as father and son
**Rosona**
Alexey Grigorev you should follow up on this. :)
**Alexey Grigorev**
Let’s say that somebody wants to rule the galaxy alone 😅
Are there any recommendations for wannabe-solo-consultants?
**Jesse Anderson**
Start cross-training heavily in marketing, sales, and business. Get really comfortable talking about money and contracts. Create a strong brand and good recognition. Build up your LinkedIn connections.
**Alexey Grigorev**
Thanks! What kind of training would you recommend for business? Something like an online mini MBA?
**Jesse Anderson**
A big realization in my entrepreneur journey is that MBAs teach you to run other people’s business rather than start your own. Focus on ones that teach you entrepreneurship. My favorites is the Million Dollar Consulting series.
**Alexey Grigorev**
thanks!
**Alexey Grigorev**
Also, what do you think about MLOps? You covered DataOps in the book a bit, but I’m curious to know your opinion about MLOps and how is it related to DataOps
**Jesse Anderson**
MLOps is an important part of the equation. I’m going to be doing more content and research on this area this year.
**Jesse Anderson**
It could become part of DataOps or stay separate. IMHO it will be a specialization of operations teams.
**Alexey Grigorev**
You can start your research with our two last podcast episodes about MLOps and feture stores (haha sorry for the plug)
I do agree it looks like a specialization of ops
Thanks!
**Alexey Grigorev**
Quote from the book:
\> Data engineering and—especially—operations won’t get the credit they deserve unless management makes a concerted effort to educate others. Failing to garner praise will make other teams think that a person or an entire team isn’t necessary to the success of the project. The reality is that key people often take their own competence for granted and don’t know how to call attention to their accomplishments.
\>
It’s often a big problem and a demotivating factor for data engineers and ops people. I’ve experienced that as well.
What are the best ways to address it? Shouting out to the engineering teams every time data scientists make a demo? Ask them to take part in demos? Something else?
**Jesse Anderson**
I think it starts with management taking an active role in calling out data engineering and operations to other managers. During a demo or conference talk, I think data scientists should acknowledge and call out the contributions of other teams. Even better is to get data engineers and operations up on stage or in the demo too.
The reason for this is that listeners just assume the data scientists did it all. When the listeners go to implement this feature, etc they don’t know why they fail.
**Surya G**
This is the main reason I am planning to move away from DE to more customer focused team. No one ever gets a promotion for writing
“ Successfully replicated 10TB/day data into datalake/snowflake”
in their self review.
No matter how much we stress the importance of DE/Ops, ppl holding the purse string will always consider it to be a cost center that they would rather not have. They have no way to know how hard or easy is it to replicate data or if even thats sort of an accomplishment.
**Jesse Anderson**
If you put “Successfully replicated 10TB/day data into datalake/snowflake” some of the shame is on you too. You’re forcing someone to figure out the value you created instead of telling them the value you created. What you do is say “Improved data lake replicated to reduce latency by 10 minutes” or “Added new data product to the data lake that allow business to reduce costs by 30%”. Engineers are good at engineering but terrible at marketing. TBH you may find yourself in the same boat in a customer-focused team if you don’t start marketing/positioning your value better.
**Surya G**
thats a very good point Jesse Anderson
**Surya G**
`Added new data product to the data lake that allow business to reduce costs by 30%".` problem is that I alone didn’t add a new data product to datalake. I just did a part of that task.
**Surya G**
there were 10 other ppl working on that project. Would be a little weird to say my contribution reduced the business costs by 30%
**Jesse Anderson**
you’re saying the data product does that
**Jesse Anderson**
you helped create that data product. you may have been part of a team but you’ll want to take some concrete credit for the value created individually or as a team
**Surya G**
for example there was an existing product driving off of bunch of streams/nosql/postgres databases. new project,
1. ETL-ed data from databases,streams into snowflake.
2. Created datamodels from raw tables in snowflake
3. migrated existing codebase to use snowflake instead of bunch of random datasources.
I did part of 1 as a support person for the product team doing 3. I have no insight into budgets/spends ect for existing projects, i really have no idea how much money my contribution saved the company.
**Jesse Anderson**
if you’re a support person, your value comes from uptime and reliability. for example, did the reliability increase now? how much? what is that increased reliability worth? you’ll have some leg work to figure this out but it’s worth it. also, this is what your resume bullet points should look like. in the longer term, this is what will get to a next-level position.
**Surya G**
yea agreed Jesse Anderson. I’ve always looked at my contributions form engineering pov like you mentioned.
I will def strive to think in terms of overall value created going forward. Point noted.
**Surya G**
thank you
**Matt Welke**
lol I get this a lot at my company, except people somehow subconsciously understand the data engineering team is important. They say things like “someday I’ll understand what the team does”, but at the same time we get a lot of questions sent our way.
**Jesse Anderson**
Send them the book! This is a big reason I wrote the book. I cut out the technical jargon and tried to make it management-friendly.
**Matt Welke**
I think your book really relates to my company right now. We’ve had data science and ML in one silo and data engineering and all other engineering in another silo. Literally two offices. In two countries.
**Matt Welke**
We desperately need to get everyone working together.
**Jesse Anderson**
That is something we do to help companies
**Matt Welke**
Data engineering seems to be a cross cutting concern.
**Jesse Anderson**
It’s a different animal and it sound like your company understands it at some level
**Noa Tamir**
I try to often call the data engineering teams “unsung heroes”. I heard another manager say it once and it clicked for me. So whenever I feel like giving stakeholders or other teams some perspectives I plug it into the conversation and people often agree.
**Alexey Grigorev**
But maybe engineers actually want to hear songs about their accomplishments? 🙂
**Jesse Anderson**
+1 on singing their songs. “Throw a coin to your data engineer”
**Noa Tamir**
Jesse Anderson thanks for answering the questions so far - I really enjoy going back and reading your answers. I haven’t gotten through them all, so please forgive me if I’m repeating someone else.
What is your experience and/or take on having embedded DS in a DE team and vice versa? I found it very useful in a setup where we had distinct teams for each function, but also wanted to give each team enough in-team context and skill to handle small issues on the go.
**Jesse Anderson**
I cover that in the DataOps section. It’s an advanced setup where you can accelerate data products.
**Alexey Grigorev**
In my (quite limited) experience, cross-functional teams are a lot more effective in terms of delivering useful things - with respect to product impact
But from reading the book I get an impression that the setup you suggest is separate teams.
In your opinion, what’s the best way to select the setup that will work better for a company? When to choose which approach? What kind of things we should take into account when deciding?
**Jesse Anderson**
I suggest separate teams initially. IMHO DataOps configurations are a more difficult organizational type. You look at DataOps once your biggest problem is organizational or team friction and not technical or beginners issues.
**Alexey Grigorev**
Okay so basically start with the three teams and switch to a more complex setup as the org grows?
**Jesse Anderson**
not grows, matures
**Alexey Grigorev**
Got it, thanks!
**Timothy Wolodzko**
Jesse Anderson do you think it is a good idea to have mixed teams of data scientists who build the models & ML engineers & developers who move them to production, or those should be split? There seem to be pros & cons of both approaches.
**Jesse Anderson**
I’ve seen this happen. Those teams need to be working with data scientists to level up their programming skills.
IMHO a data scientist who can’t put something into production isn’t a data scientist. They’re still a mathematician or statistician. The data scientist does need to know how to program.
It is common for MLE and data engineers to help harden data scientist’s code.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Machine Learning Design Patterns – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Machine Learning Design Patterns
--------------------------------
#### by Valliappa Lakshmanan, Sara Robinson, [Michael Munn](https://datatalks.club/people/michaelmunn.html)
##### The book of the week from 08 Feb 2021 to 12 Feb 2021
The design patterns in this book capture best practices and solutions to recurring problems in machine learning. The authors, three Google engineers, catalog proven methods to help data scientists tackle common problems throughout the ML process. These design patterns codify the experience of hundreds of experts into straightforward, approachable advice.
In this book, you will find detailed explanations of 30 patterns for data and problem representation, operationalization, repeatability, reproducibility, flexibility, explainability, and fairness. Each pattern includes a description of the problem, a variety of potential solutions, and recommendations for choosing the best technique for your situation.
* [Book's page on O'Reilly](https://www.oreilly.com/library/view/machine-learning-design/9781098115777/)
* [Book's Github Page](https://github.com/GoogleCloudPlatform/ml-design-patterns)
Questions and Answers
---------------------
**Denis Lepchev**
Hi Mike Munn - thank you and your colleagues for such a great book!
I have a question regarding the “continued model evaluation” design pattern. As you mention in the book, usually we act on the model’s predictions - as an example trying to prevent customer’s churn.
How companies like Google deal with the feedback loops, when acting on the model’s prediction changes the outcome, making it impossible to observe the ground truth? Many books and articles mention this problem, but the solution is not discussed.
One of the solutions I am aware of is to have a small (untouched) hold-out group as a basis for model evaluation and for re-training, but it comes with its own set of problems. Is there an alternative method(s)?
**Mike Munn**
Hi Denis Lepchev thanks!
You bring up a really good point about ‘feedback loops’ when dealing with continued model evaluation. There is a lot written about this. I remember reading an article about a company called Stripe that discussed this problem for them in some detail for a credit fraud model they had developed.
We discuss this briefly in the subsection called “Capturing ground truth” as well as other considerations. You mention a good approach. Another technique to consider is to use causal impact models. Causal impact uses Bayesian techniques to understand how a certain “treatment” might have affected a population. It’s commonly used in scenarios where it’s not possible (or not ethical) to have a hold out or control group. This is commonly used at Google and that team has open sourced a CausalImpact package you might want to check out.
**Denis Lepchev**
Thank you for the answer, Mike.
**Alper Demirel**
Hello Mike Munn, thank you for writing this beautiful book and coming to this channel 👏
* What is the most important difference from similar books on the market?
* Why did you need to write such a book with your friends?
**Mike Munn**
Thanks so much Alper Demirel. I’m glad you enjoy it. 🙂
* In my mind, this book would come after an intro book or reference on ML. We certainly talk about how you would do things (and there is a github repo with code implementing the ideas in code) but this book is focused more on common problems and their solutions that we’ve learned in working with customers along the years. Likely a lot of patterns will be familiar to you, if you’re a practicing ML/AI engineer. So this book is a kind of formal collection of “notes from the field” and solutions to common real world problems of building and productionizing ML models.
* Following on a bit from my point above, another reason we wanted to write this book (in addition to having a collection of common problems/solutions) was to create a shared language for these ML patterns. Like any pattern, you find yourself using the same tricks over and over. And in talking with other colleagues we wanted to create a single term or phrase that describes that problem/solution. So, for example, the “Bridged Schema” pattern gives a name to the common problem that arises when dealing with changing data fields. This way, we can try to have a shared language of common patterns for everyone to use.
**Alper Demirel**
Thank you so much for the answers, now I’m more excited to read the book 🤩🔝
**Aleix**
Hi there, Mike Munn! Pleasure talking to you :)
What has been the most recurring problem to an ML engineer like you when working at Google?
**Mike Munn**
Thanks Aleix glad to have the opportunity to talk to you as well. 🙂
That’s a good question. It’s hard to say what the most common problem would be. I work a lot with Google Cloud customers. We typically work on engagements that span the entire ML work lifecycle. The project specific problems or patterns that arise are so use case dependent. So I guess the most recurring problems then would be the problems that relate to initial or final stages of the lifecycle (since they are broad and similar across multiple use cases). Those would be the patterns addressing data issues or dealing with Responsible AI and working with business stakeholders. Typically the most common problem is in framing the problem and understanding issues with the data. That seems to be consistent across all projects I’ve worked on.
**Matthew Emerick**
Hey, Mike Munn! Appreciate a moment of your time. Your book is on my (rather large) Amazon wishlist. I do have a question for you. Are the patterns organized by difficulty level?
**Mike Munn**
hi Matthew Emerick! Happy to be able to stop by 🙂 I know too well about large book to-read lists lol.
The patterns aren’t arranged by difficulty. Instead we arranged them loosely with the different stages of a typical ML workflow or ML lifecycle. So we start with patterns related to Data Representation, then move to Problem Representation, then talk about patterns related to Resiliency and then Reproducibility and end with Responsible AI and stakeholder management.
**Matthew Emerick**
I like that. Is there anything about the complexity (ie bubble sort is easy to do, but slow) for each pattern?
**Mike Munn**
kind of! Each pattern is broken down into four main sections: Problem, Solution, Why It Works and Tradeoffs & Alternatives. The Tradeoffs & Alternatives section discusses exactly that. We touch on any ‘gotchas’ or complexity issues, or common pitfalls or just other related approaches you might want to consider.
**Matthew Emerick**
Okay, so the expected style for a pattern book. Makes it easier anyone who has used a pattern book in the past. Good plan. Organization is something I touch on in my book reviews, both macro and micro. A well written book with great and important content can hit a wall if it’s not organized well. Great job!
**George Melvin**
Hey Mike Munn, thanks for sharing your time with this community! I’m excited to dive deeper into the book. My question is around company size: are certain patterns more conducive to smaller/larger teams/organisations? In particular, I’d be interested to hear about your experience (re)learning patterns when moving between teams/organisations that are comprised of larger/smaller teams. Thanks!
**Mike Munn**
hi George Melvin thanks for having me
Good question. The patterns are more focused on implementation. So I can see how they would apply to both smaller/larger orgs. That being said, for some smaller orgs or smaller teams there is less focus on some MLOps aspects. For example, we have a section on Feature Stores. This is a super useful pattern and important to know about; however, I can image for a smaller org the need for a feature store may be less imperative than for a large organization where you have dedicated teams to engineer features for the models that another dedicated team is building.
But, I’d say the patterns in general are relevant for small or large teams and orgs. Just some may resonate more than others.
**George Melvin**
Great, thanks for your answer! I ’m looking forward to learning more 🤓
**Neal Fultz**
Thanks Mike Munn for answering questions. Mine is: what are the most dangerous anti-patterns that you have encountered?
**Mike Munn**
Hi Neal Fultz and thanks!
Ohhh that’s a good one. A lot of things come to mind. I think the most dangerous (and I may be biased in the type of work I usually do and some bad past experiences) would be anti-patterns related to reproducibility and resiliency. So much of ML development is experimentation (with data, with pipelines with models, etc) and reproducibility is crucial to running thorough experiments. Also, as I work with more companies interested in productionizing their models, anti-patterns related to resiliency really end up being more and more detrimental
**Mike Munn**
hello Everyone. Thanks for having me visit the channel and the opportunity to talk about the book 🙂
**Dr Abdulrahman**
Thank you Mike Munn for the book and for your time answering questions.
I have three questions:
1. From your experience, how would you assess the awareness of these patterns among senior practitioners. Would you say the majourity of seniors are not aware.
**Mike Munn**
hi Dr Abdulrahman very happy to join the channel this week.
1. I’d imagine that for most senior ML practitioners would be aware of the majority (if not all) of these patterns. Though they might not know it by the name we used. We have code for each pattern and use a variety of tools, some of them more recently developed than others. So even senior practitioners might be able to learn something new from the book.
2. That’s a good question. I’m not sure. I imagine it would likely come from constraints either in their infrastructure or project timing. For example, someone may understand the importance and need for Continued Model Evaluation but it can be difficult to implement carefully and require too much overhead for the business objective.
3. I wouldn’t say a complement book is needed, but it could certainly help. This book touches on patterns across the entire ML lifecycle so a book more focused on one aspect (say data engineering or ML pipelines) could be helpful for more implementation details.
**Dr Abdulrahman**
Second question: What drives many seniors not to follow these patterns when developing their ML models?
**Dr Abdulrahman**
Third one: Would we need a complement book about data engineering (pipeline) design patterns.
**Simon Steinkamp**
Thank you Mike Munn for taking the time answering questions here 🙂
The idea of design patterns is actually very new to me and it was very interesting to have a quick look at those. Having heard the term “copy-paste” data scientist recently, I have the feeling that some patterns probably evolve naturally, especially through worked examples for tensorflow, keras or scikit-learn.
Knowing that your book goes beyond the model fitting step, I was wondering what your opinion on these patterns is?
**Mike Munn**
Hi Simon Steinkamp thanks for joining with questions 🙂 I’m not familiar with the term “copy-paste” data scientist, but I think I can guess what it’s referring to. In some sense, the ideas are similar. However, these design patterns are more like ways of thinking when designing solutions or building ML systems. This book would follow a course on how to effectively build ML model, so we don’t get into the weeds about those details. There are already really great books out there that address that. The idea of patterns is more focused on ways of thinking and recognizing common solutions to common problems. The solutions can’t typically be copy-pasted directly (since each case is a bit different) but the idea is certainly repeated.
**Simon Steinkamp**
Great, thank you for your answer, I will have a closer look 🙂
**Poornima**
Hi Mike Munn first of all congratulations on your book release which will help so many ML practitioners on their professional and/or learning journey.
Like to ask two questions:
* What was your motivation behind coming up with the idea of writing a book on design patterns in ML.
**Mike Munn**
hi Poornima Thank you! I hope you enjoy it.
1. The motivation came from lots of customer engagements at Google Cloud in our roles. When designing ML systems or thinking about common ML approaches, we saw that there were some common patterns in the field. The book attempts to collect those pattens into a single easy to reference source.
2. There wasn’t any monolithic use case or project but we did try to use similar datasets throughout the book when illustrating a concept. This way, the code solutions would be less about the data or model and more about the pattern itself.
**Poornima**
Thank you much Mike Munn for answering. Looking forward to gain great knowledge from the book. Hoping to see more such excellent works from you in future. All the very best 🙂
**Poornima**
* Is there any sort of monolithic usecase/project you are illustrating to showcase the optimal design patterns for each stage of ML lifecycle for that particular case.
Thank you.
**Poornima**
Thank you much Mike Munn for answering. Looking forward to gain great knowledge from the book. Hoping to see more such excellent works from you in future. All the very best 🙂
**Alexey Grigorev**
I remember when I was learning design patterns (the GOF ones), I tried to use them everywhere, and in most cases, it was doing more harm than good. With experience, I learned when the patterns should be used and when they shouldn’t.
Do you think something like that can also happen with ML design patterns? If yes, what advice would you give to beginners to get started with ML design patterns?
**Mike Munn**
Alexey Grigorev yeah for sure! It’s like the typical hammer/nail scenario. For beginners, I think it would be helpful to first just read through the patterns to be aware of what’s out there. Then revisit certain patterns based on the individual use case and really question of it will help solve the problem _you_ have. The final chapter of the book gives an overview of all the patterns, breaking them down in a simple chart of problem/solution/description. If you think a pattern might be a good solution, take a critical look at the section that discusses that one in detail. And see if it fits or not.
**Alexey Grigorev**
Makes sense, thanks!
**Justin Neumann**
Dear Mike Munn,
I read parts of your book with my O’Reilly learning account and I really liked it so far! I particularly like the idea of the “Reframing” concept. Do you think it is applicable to a single, global (gradient boosted tree) regression model that captures multiple time series (which is the problem I deal with on a daily basis)? I can’t think of a way to apply the “target binning” regression -> classification approach to such a global model, unfortunately.
Thanks in advance for the answer!
PS: I have the book on O’Reilly, so no need to give me a book 😉
Best, Justin
**Mike Munn**
Thanks Justin Neumann great question. The Reframing pattern is an interesting one. It’s focus is less on the model itself and more on how the labels are represented. So, for example, the regression -> classification reframing discusses how you can modify a typical regression problem and frame the problem as learning a probability distribution via classification and binning. One scenario where it is useful is when there may be training examples which have the same features but different labels (I think in the book we used the dataset of birthweights). This way you are learning a probability distribution for a feature input and it can be less confusing for the model. So you’re really just changing your labels, and you would modify your gradient boosted tree to be classification instead of regression. The pattern isn’t model dependent. Does that make sense?
One last point to mention is that this approach of reframing may just not be beneficial to your data or use case. As with most solutions, it’s very use case dependent.
Lastly, I’m glad to hear you have the book. I really hope you enjoy it!
**Justin Neumann**
Thanks for the reply. Your answer makes all sense to me and you’re right, this pattern may just be not applicable to my problem. I’ll keep reading! 😉 Best, Justin
**Rosona**
Hi Mike Munn! Thanks for taking questions. Who do you see as the perfect reader (slash ideal audience) of this book? Do you see it as essential even for the raw beginners, or something to ingest after e.g. a year of experience or more?
**Mike Munn**
Hi Rosona Thanks for joining in. Our target audience was someone that knows the basics of ML and building models. We kind of pitched it as a follow up to a beginning ML book, like Geron’s book “Hands On ML with SciKit and TF” or Muller/Guido’s book “Intro to ML with python” or after a semester course in ML. With more experience, I image the reader would be able to appreciate (or recognize) more of the patterns.
**Alexey Grigorev**
How would you rank patterns in terms of applicability and usefulness for applied machine learning in industrial settings?
Which patterns come first to mind and we should use them in most of the projects?
**Mike Munn**
Alexey Grigorev yeah good question. When I think about applied machine learning or ML projects we’ve done with our cloud customers, I think the most useful and common patterns are those that show up in our chapters on Repeatability and Resiliency, patterns like ML Pipelines, Continued Model Evaluation, Batch Serving, Keyed Predictions, Batch Predictions, Feature Store, Transform, and Model Versioning. Also, the Responsible AI patterns are also really important for applied machine learning projects.
**Alexey Grigorev**
Thank you! And what is the least used one? You mentioned feature stores as something that smaller teams/companies usually don’t need - maybe this one? or some other one?
**Mike Munn**
yeah some of the patterns are more use case driven, so I’ve personally see those less. And yes, patterns like Feature Store seem to be much more important for industrial teams. In my experience, smaller, more agile teams might not benefit or need to have a feature store set up. They can be incredibly useful and necessary for some situations, but can also require a lot of technical overhead.
**Alexey Grigorev**
Got it, thank you!
**Rishabh Bhargava**
Hi Mike Munn - thanks for writing this book! So invaluable as we figure out good practices for shipping ML systems. My question is: what were some design patterns (if any) that you would have liked to include, but that had to get edited out? 🙂
**Mike Munn**
Hi Rishabh Bhargava thanks for joining with questions. It’s funny, that question has come up before, and honestly I think we were able to catch all the patterns we had in mind. Our initial list didn’t come out to such a round number of 30, but as we were writing a couple topics we thought would be covered as one pattern ended up getting their own section, or the opposite (some individual patterns ended up getting grouped together naturally). So in the end, we covered the ones we wanted….though, now that it’s been put in my mind I’m already thinking of new patterns we could have added 🙂
**Rishabh Bhargava**
I have a 2 parter question as a follow up.
1. Do you think these design patterns will need significant updates in the near future, or these patterns represent some general truths about how ML systems should be built?
2. If you think updates are likely, do you think the second edition will be needed in 6 months, 1 year, 5 years or 10 years?
**Mike Munn**
yeah good questions,
1. i don’t think these patterns themselves will need much updates..though, for some, the code or implementations could change over time and be updated or improved as tools continue to develop as solutions to address these common problems
2. I don’t foresee the need for an second edition soon, but if code updates do become necessary, they’ll make it into the book’s github with the code examples much sooner.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Math for Programmers – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Math for Programmers
--------------------
#### by [Paul Orland](https://datatalks.club/people/paulorland.html)
##### The book of the week from 15 Feb 2021 to 19 Feb 2021
To score a job in data science, machine learning, computer graphics, and cryptography, you need to bring strong math skills to the party. Math for Programmers teaches the math you need for these hot careers, concentrating on what you need to know as a developer. Filled with lots of helpful graphics and more than 200 exercises and mini-projects, this book unlocks the door to interesting–and lucrative!–careers in some of today’s hottest programming fields.
* [Book's page on Manning](https://www.manning.com/books/math-for-programmers)
* [Book's Github Page](https://github.com/orlandpm/math-for-programmers)
Questions and Answers
---------------------
**Wendy Mak**
Hi Paul, my questions are:
Are there topics that you really want to cover in the book, but decided against in the end?
**Paul Orland**
There are a lot of topics I wish I could have included! Originally, I wanted to write about abstract algebra and functional programming. This is a bit more advanced, and so I decided with my publisher to focus on topics that were more accessible and interesting to a broader audience. Also, there’s not much on probability or statistics in this one – I hope to write another book on “statistics for programmers” soon.
**Wendy Mak**
Most of the topics in the list of contents seems more or less fairly general, except the chapter on gravity simulation– what made you add that particular topic?
**Paul Orland**
I tried to have at least one application in each chapter – the new math in that chapter is the concept of partial derivatives and the gradient. I think that force fields are a useful example to think of when learning about the gradient, which is why I picked it!
**Wendy Mak**
In the intro, you mentioned that you wrote it partly with the idea of ‘scientific software engineer’ in mind– it’s not a job description I see very often, outside of universities, where do these jobs come up? And what sort of skills do they usually look for? (ie do they assume you’ll have a background in the science as well?)
**Paul Orland**
Job titles vary, but some software engineer job postings require more math than others. I would say there are a lot of data science and quantitative finance jobs out there that require math, as well as software eng. jobs at high-tech companies (e.g. aerospace, biotech, hardware)
**Evren Unal**
Hi Poul,
The book’s cover is interesting.
But particularly the lady’s hand sign is prominent.
İs there a meaning in it or is it aimless?
**Paul Orland**
I wish I could tell you! My publisher (Manning) has a tradition of putting people in old-fashioned dress on their covers. This woman is wearing traditional dress from Finland from a few hundred years ago. I picked this one from three options, because I like her smirk! No idea what she is doing with her hand… but it almost looks to me like she’s gesturing at a blackboard
**A McCauley**
Hey Paul Orland , great book 👏🏼, this is definitely something I want to expand my understanding in some more.
I like that you’ve added exercises at the end of the topics; How did you get your ideas for the exercises you listed? Are they short questions to test understanding or small practical project styled etc?
**Paul Orland**
Glad you’re enjoying it so far! Good question. I’ve designed most of the exercises to be doable in 1-15 minutes
**Paul Orland**
They are supposed to be relatively quick checks of your understanding, to help you make sure you are keeping up with the material. Then, if one takes longer than a few minutes to figure out, it’s a good sign you should reread the chapter and think through the topic again.
**Paul Orland**
Anything more involved is marked as a “mini project”, indicating that it will take what you’ve learned and go a bit beyond the material introduced in the text.
**Paul Orland**
The way I come up with these is roughly: I think of some small example I could work through in a paragraph, and rather than explaining it, I try to give a carefully worded hint so you can basically connect the dots and figure it out yourself.
**Vladimir Finkelshtein**
Will there be a book called “Programming for mathematicians”? Asking for a friend.
**Paul Orland**
Maybe not with that title, but yes 🙂 Later this year I’m releasing a book relating abstract math and functional programming, and I think it is the right way for mathematicians to learn programming!
**Poornima**
Hi Paul Orland, glad to converse with you. Are you covering the mathematical fundamentals needed for computer vision ( I saw you were covering 3D graphics) such as various image transforms. Thanks.
**Paul Orland**
Hmm… I don’t know much about computer vision in general, so I can’t say for sure. I bet many of the topics in the book are used in computer vision. And, the last chapter is on image recognition using neural networks!
**Poornima**
That sounds great !🙂 Will give it a try and thank you for answering.
**Matthew Emerick**
Hello, Paul Orland! Thanks for doing this. My question(s): if you were going to expand your book, what would you add? What would you expand? What would you cut?
**Paul Orland**
I would definitely add some content on probability and statistics! I think it is increasingly important math for applications, and I’m sad it didn’t fit in this book. It’s also not my area of expertise. I think most of the big topics in the current book are good, but if I could do it again, I might trim down some of the les useful sections and subsections (not thinking of any off the top of my head, but there are surely some!)
**Matthew Emerick**
I agree that statistics and probability are very important. I also think that too many rely on them without full understanding. Do you think machine learning engineers, data scientists, and AI practitioners in general need to focus more on math?
**Paul Orland**
I suppose most engineers, data scientists, etc. know the math they “need” to do their jobs, but I suspect there is room for most people to think more deeply about the math they use. In my book, I’ve tried to emphasize the philosophy behind each topic, as in why it is the way it is, and how it fits into the grand scheme of things. This perspective, I think, is similar to a software engineer taking the time to understand how the compiler of their favorite language works. It gives a deeper perspective on how your tools work.
**Doink**
Hi Paul Orland do you plan to have a separate book on the lines of Maths for Data Scientists? Slightly more deeper and covering topics on numerical optimization,infinite series etc but not in the typical classical math textbook format.
**Paul Orland**
I don’t, but I am trying to convince one of my friends to write such a book! One of the challenges is including enough relevant example to make the book interesting. Will keep you posted!
**Alexey Grigorev**
Hi Paul!
Which learning plan would you recommend for software engineers who want to get into machine learning? What kind of math do they need to focus on - to make the learning most effective? And would your book be enough to cover the math fundamentals?
**Paul Orland**
Alexey Grigorev good question! I think Math for Programmers has a lot of math necessary for ML – namely linear algebra and multivariable calculus. The main topic not included in my book which is necessary for ML is probability. I suggest buying my book (obviously 😇 ) and taking Andrew Ng’s free coursera course on ML . If you work thorough the exercises in my book and this ML course, you should be prepped to do a deeper dive in any area of ML you want
**Alexey Grigorev**
Thanks! Andrew Ng’s course is awesome!
In your opinion, how a programmer should approach learning probability and stats? You mentioned you might write a sequel for your current book - I guess you already thought about this question
**Paul Orland**
I would say, try to learn whatever you want to do in ML or elsewhere, and read about anything unfamiliar. A lot of what I wrote about in Math for Programmers is stuff I learned for some application (e.g. Physics). having a goal/application in mind is a great way to motivate yourself to learn something, and focus on the most important topics.
**Paul Orland**
I wish I had a good enough 50,000ft view of probability to give more specific advice. This is part of why I’d like to research and put together a book on it.
**Alexey Grigorev**
Makes sense, thank you!
**Vladimir Finkelshtein**
What’s your favorite example, that shows importance of understanding math behind ML? Most of the examples I can think of don’t really justify a deep dive…
**Paul Orland**
I am kind of a novice, but I think gradient descent is very important!
**Paul Orland**
It is good to think of learning as an optimization problem, and gradient descent is one optimization algorithm you can use pretty widely
**Vladimir Finkelshtein**
I agree that it is important to learn what an optimization problem is and to frame machine learning as one. But it seems like the focus of machine learning is on what to optimize, rather than how.
To me gradient descend feels kind of “academic”. I imagine that not many ever write their own optimizer for machine learning. Probably for most people reading two nontechnical paragraphs explaining the intuition behind gradient descent will be more than enough.
**Paul Orland**
Yeah, I guess it’s a matter of opinion. I think deriving the backpropagation formulas for a MLP (using gradient descent) and implementing them is a worthwhile exercise, because it demystifies machine learning. I totally agree that ML in practice is more about deciding what to optimize.
**Doink**
Paul Orland How is this book different from Think Stats?
**Paul Orland**
I don’t know that book, but presumably the difference is that it covers stats while my book does not 🙂
**Ben Wilson**
Paul Orland I have absolutely and thoroughly enjoyed reading your book. Not only are the progressions throughout the examples highly engaging and approachable to a wide audience, but the accompanying explanations are very clever and entertaining. I’m curious to know if you’re planning on ever applying your teaching style and examples to the world of geospatial processing (2D and 3D) algorithms? I think you’d do a rather great job at making a topic such as that approachable to the rapidly growing community of practitioners that are entering that specific space.
**Paul Orland**
Glad you liked it! I don’t know much about geospatial algorithms, but sounds like it could be a cool application for 2D and 3D geometry. Can you recommend any good references for me to get started?
**Wendy Mak**
Ben Wilson do you mean projection algorithms? or stuff like spatial indexes, spatial search and so on?
**Ben Wilson**
great circles intersection, polygonal collision, distance measurements with z-components, et. al. I think that with your presentation style you could write an effective primer to basically GIS ( 🙂 ) [http://wiki.gis.com/wiki/index.php/New\_to\_GIS#Related\_links](http://wiki.gis.com/wiki/index.php/New_to_GIS#Related_links)
**Wendy Mak**
nice :)) I used to do a lot of spatial stuff (postgis mainly but also a bit of python). Would be nice to have a deeper dive into some of the maths definitely
**Alexey Grigorev**
I know great developers who want to get into machine learning, but they are scared of math. The moment they see a formula, their mind goes blank.
What would you recommend them to do? How can they overcome this fear?
**Alexey Grigorev**
I think the fear comes from school/university when teachers just force math down our throat without really explaining why we should learn it
**Roman G**
this may help to manage fear
**Evren Unal**
Alexey Grigorev same problem goes for me
**Roman G**
It was the same for me, but if you take your time to break a complicated formula to parts and dig into each of them, then after a couple of iterations it won’t be that scary
**Evren Unal**
I usually learn difficult topics by finding good explanations
**Paul Orland**
Roman G I have found there is a delicate tradeoff between lowering inhibition and keeping ones mental faculties intact with the method you propose 🙂
**Alexey Grigorev**
This one? [https://xkcd.com/323/](https://xkcd.com/323/)
**Paul Orland**
I like this question because it is a problem I still have – sometimes I find myself looking at a page in a math textbook and thinking to myself this looks hopelessly complex.
**Paul Orland**
Alexey Grigorev yes exactly!
**Paul Orland**
I would say, start by writing down the formulas you do know or drawing diagrams (related to the topic), without looking at a book or other resources. That should show you where the holes in your understanding are. Then go back to the book and try to fill these holes as needed. Over time, you should get some more confidence!
**Alexey Grigorev**
Makes sense, thank you!
**Paul Orland**
I would say, start by writing down the formulas you do know or drawing diagrams (related to the topic), without looking at a book or other resources. That should show you where the holes in your understanding are. Then go back to the book and try to fill these holes as needed. Over time, you should get some more confidence!
**Alexey Grigorev**
Why knowing math is a good idea for programmers? Might it be useful for developers even if they don’t need it at work?
**Ennio M.A.**
I would say yes, I believe that algorithms are basically about breaking mathematical constructs into simple step by step processes or approximations, at school too much emphasis is given to paper math, I think more emphasis should be put into computational thinking in all engineering degrees not only computer science.
**Paul Orland**
A lot of ideas transfer directly from math to computer programming: logic, functions, sets, etc. I’m a strong believer that math skills help you think more rigorously about software (of any kind) that you build
**Alexey Grigorev**
Also I’m curious what you think about the way math is taught at schools and universities. Is there something you’d change?
**Paul Orland**
Yes, I would change some things! I think math classrooms should be flipped, meaning record lectures so students can consume the content at their own pace, independently. Then save class time for exploration and problem solving. I would also say that most people who take math in college are not going to become mathematicians, so there should be a greater emphasis on applied math and using technology to solve math problems (e.g. Python or even Excel)
**Alexey Grigorev**
Nice! How would you motivate students to watch the content at their own pace before the classes?
**Alexey Grigorev**
That’s a problem I also have when teaching adults =)
**Paul Orland**
Obviously some students are more self-motivated than others, but I think this would increase the quality of the learning experience and have most people more engaged overall.
**Alexey Grigorev**
Yes, makes sense! Thank you!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Machine Learning for Algorithmic Trading – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Machine Learning for Algorithmic Trading
----------------------------------------
#### by [Stefan Jansen](https://datatalks.club/people/stefanjansen.html)
##### The book of the week from 22 Feb 2021 to 26 Feb 2021
The explosive growth of digital data has boosted the demand for expertise in trading strategies that use machine learning (ML). This thoroughly revised and expanded second edition enables you to build and evaluate sophisticated supervised, unsupervised, and reinforcement learning models.
This edition introduces end-to-end machine learning for the trading workflow, from the idea and feature engineering to model optimization, strategy design, and backtesting. It illustrates this workflow using examples that range from linear models and tree-based ensembles to deep-learning techniques from the cutting edge of the research frontier.
* [Book's page on Packt](https://www.packtpub.com/product/machine-learning-for-algorithmic-trading-second-edition/9781839217715)
* [Book's page](https://ml4trading.io/)
* [Book on Amazon](https://www.amazon.com/gp/product/1839217715/)
* [Book's GitHub repository](https://github.com/stefan-jansen/machine-learning-for-trading)
* [ML4Trading Community](https://datatalks.club/books/20210222-ml-algotrading-2ed.html)
Questions and Answers
---------------------
**Doink**
How good are the strategies mentioned in the book good enough to make decent money in real time? Vladimir Finkelshtein basically completed my question, absolutely valid points with the whole Gamestock Saga and BTC price pump with just one tweet and investment by Musk.
**Vladimir Finkelshtein**
One of the most successful trading strategies is market manipulation. How can one make sure that ML does not learn to do it? (because it will, since it tries to optimize the profits)
From the other side, how can one identify that the market is being manipulated? Nowadays, a single tweet can wreak havoc in the markets.
**Aleix**
Hi, Stefan Jansen! Nice to meet you.
It’s actually a great timing for me, since I am actually very interested in this topic and was thinking of developing my bachelor thesis around an algotrading system 😄
My question is: how would you describe the use of ML in the algotrading scenario? Has it been a game-changer and everyone uses it now? Or just another tool that sometimes works and sometiems don’t?
Thanks a lot!
**Hong-Ngoc Emily Tran**
Who should read the book? Can a normal banker with somewhat knowledge in Data Analytics use this book to understand what IT people are doing?
**Rica**
Hi Stefan, thanks for doing this. What type of securities does machine learning for algorithmic trading cover?
**Rica**
Can you share some success stories in using machine learning for algorithmic trading?
**Rica**
Last question: can machine learning for algorithmic trading detect or prevent what happened to Gamestop? Thank you for your time 🙂
**Denis Lepchev**
Hi Stefan, thanks for writing the book!
What in your opinion are most common pitfalls in backtesting and how to avoid them? To extend the previous question, what do you think about usage of synthetic data for backtesting - how reliable it is?Thank you.
**Oliver Wilkins**
Hi Stefan, it’d be great to see your thoughts on this!
**Wendy Mak**
Hi Stefan, my question is:
Does algorithmic trading (tbh all the ‘fancy’ financial instruments in general) serve any positive purpose beyond making a bunch of investment bankers very rich? ;)) – sorry it’s only distantly related to the main content in your book, but I’ve never really got the point of the fancier things you can do on the stock/commodities/etc markets…
**Vladimir Finkelshtein**
I imagine if one trains trading algorithms with neural networks, they will learn many spurious correlations. Is there some regularization for this?
**Vladimir Finkelshtein**
Once in a while there is an article about some academic algorithm that beats the market. But these algorithms always work only on the past data. Are there examples of ML algorithms that actually generalize to the future?
Is this question even meaningful? Since most of the trading today is algorithmic, maybe beating the market just means beating other algorithms?
**Vladimir Finkelshtein**
My statement about “most of the trading is algorithmic” refers to forex market (in which wiki says algo trading is 90%). You can disregard it, since it is probably not true for more volatile markets.
**Tino**
Hey Stefan! 👋 Thanks for sharing your knowledge! 🧠 How do you include the natural uncertainty when modelling stocks, trading, etc! Often individuals (Elon Musk, etc.) can have an unpredictable impact. Is there kind of a general approach to include this?
**Arni Westh**
Have you launched any algo trading systems trading live autonomously with real assets on the line? Imo, until this stage, all of these discussions tends to be rather academic - it’s when you go live you learn about all details you didn’t think about in simulation/backtesting…
**Stefan Jansen**
Alright, so here are a few points on your questions:
1. On Aleix question of how I would describe the use of ML for trading in the industry:
* Finance, of course, has very long history of using quantitative tools. This includes basic ML algos like good old linear regression.
* Just as elsewhere, more data drives more demand for better techniques to apply to the data. More specifically, together with the emergence of ‘alternative data’ ([https://alternativedata.org/](https://alternativedata.org/)
), there has been a lot if interest (and more need) to use data science / ML to extract value.
* This takes different forms depending on the time horizon:
◦ On one end of the spectrum, it includes more traditional investors that have started to use ML to forecast fundamentals (see e.g. interview with Michael Recce, now CDS at Neuberberger Berman who previously introduced this at Point 72: [https://www.investmentmagazine.com.au/2019/10/michael-recce-the-goldilocks-approach-to-neuroscience-ai-and-investing/](https://www.investmentmagazine.com.au/2019/10/michael-recce-the-goldilocks-approach-to-neuroscience-ai-and-investing/)
).
◦ On the other end, high-frequency trading also offers applications from optimal trade execution to alpha (see, e.g. survey of use cases: [https://www.cis.upenn.edu/~mkearns/papers/KearnsNevmyvakaHFTRiskBooks.pdf](https://www.cis.upenn.edu/~mkearns/papers/KearnsNevmyvakaHFTRiskBooks.pdf)
).
* The basic argument in favor of using ML is that most investors are already trying to project market trends in the short or long run from all sorts of data. It’s not unreasonable to assume that ML can add value to this process. Machines naturally have more of an edge for shorter horizons, whereas humans may be (still) better positioned to process the complex information involved in analyzing longer-term trends.
* It is important to keep in mind that financial markets are highly competitive and packed with sophisticated institutions operating at substantial scale. Clearly it takes a bit more than downloading some data from yahoo finance and installing scikit-learn to predict Apple’s (or Tesla’s..) stock price. Some healthy skepticism is certainly warranted and several funds have closed down their ML efforts.
* At the same time, you may have heard of Renaissance Technology (RenTek). I’d highly recommend ‘The man who solve the market’ about Jim Simons ([https://www.amazon.com/Man-Who-Solved-Market-Revolution/dp/073521798X](https://www.amazon.com/Man-Who-Solved-Market-Revolution/dp/073521798X)
) who started the - by far! - most successful hedge fund, achieving 50%+ per year since the late 80s on their (now only internal) Medallion Fund. Earlier and more focused than most, Rentek has built a highly-secretive quantitative fund run largely by scientists who have started collecting huge datasets before anybody else was doing this, and apply all sorts of proprietary algorithms to identify patterns. While nobody knows what they are actually doing, much would probably be broadly classified as ML that operates in a largerly autonomous fashion.
* Two Sigma, started by DE Shaw alums who also hired several ML & Python specialists (Anaconda provides consultancy, lead Pandas maintainer is VP). These are very substantial success stories, and have attracted numerous attempts to imitate with mixed results.
* However, financial markets are very large and diverse, extending way beyond the (recently) popular sensations surrounding Gamestop, Tesla or Bitcoin. There are many niches in less actively traded areas where even smaller players can do well.
**Rica**
I’m curious has Rentek ever publish about its models?
**nate8020**
What a high-quality response Stefan. This alone has made me infinitely curious about your work.
**Aleix**
man, that’s what I call a response! thanks a lot for the huge details and clarity, really appreaciated!
**Stefan Jansen**
Rica nothing, nada, niente. airtight NDAs. the book is really fun to read, you’ll see. the folks who work there are as good as it gets.
**Doink**
why is alternativedata giving their datasets for free?
**Stefan Jansen**
On market manipulation: it’s hard to know. Manipulation is illegal and regulators are watching. Not sure predicting returns better than by chance which is already useful would count as or be confounded with market manipulation. I would also distinguish between the famous Elon Musk etc examples and the much larger volume that trades in less spectacular environments. Since I’m not sure that the bulk of trading activity is related to market manipulation, I’m not sure ML would automatically ‘learn how to do it’. Also, manipulation typically requires some influence - either the ML would first have to figure out how to attract as many twitter followers and general real world success as Elon Musk, or how to rally a bunch of reddit users (plus the HF that also played along). Tricky..
**Stefan Jansen**
Hong-Ngoc Emily Tran - who should read the book: it’s useful if you’re an analyst who wants to go beyond spreadsheet, a PM who wants to use ML, either to run things herself or by directing specialists. The book is fairly hands-on with 150+ notebooks; you should be pretty familiar with Python and the standard data science/ML libraries so you can focus on the domain-specific aspects.
**Stefan Jansen**
Rica the book mostly uses equities simply because the data is most easily available. Free data is rare, but we do have examples that use minute-data (somewhat ‘high-frequency’) as well as intl equities and pairs trading with ETF. I would say though, that the most important part is demonstrating how the ML algorithms can be used to inform a strategy and then backtest/evaluate the strategy. I think if you’re proficient with these applications you’re in a very good position to come up with your own ideas for other markets if you prefer.
**Rica**
Thanks for your response Stefan Jansen, I will explore and learn from your models 🙂
**Stefan Jansen**
Denis Lepchev backtesting is tricky as you’ve probably hear. I’m sure you’ve come across Marcos Lopez de Prado who is one of the leading practitioners in the field and has discussed this in great detail (now runs quant at Abu Dhabi’s SWF, before at Guggenheim and AQR [https://www.quantresearch.org/](https://www.quantresearch.org/)
). My book summarizes his points which are conceptually simple: given limited historical data, if you just try long enough you’re bound to find something that looks really good in hindsight. Just like ML model overfitting. There are ways to protect against this, being aware of the risk being the first and most important step.
Synthetic data would be (part of ) a solution and and there are certainly promising early research results. In the book, I reproduce the TimeGAN paper presented recentat NeurIPS [https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf](https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf)
. I haven’t had the time to experiment with this at scale but I’m sure most of the larger funds are exploring this as an option. It may take a few years until we know if this can produce sufficiently useful data; GANs have the mode collapse problem that delivers very realistic but not very diverse results, which is one of the issues that still needs to be overcome.
**nate8020**
Marco’s work is high-caliber, this fragment of his book (Advances in Financial Machine Learning / Marcos Lopez de Prado) really grabbed my attention when his book came out.
**Stefan Jansen**
Wendy Mak good and fair question. I would agree that the financial sector has overall taken on proportions as a % of GDP that is quite a bit larger than its contribution to overall welfare. There are some economic arguments how sophisticated investors make markets more efficient because they help ‘discover the right prices’ that are signals for other activity, but I’m not sure that justifies the outsized gains by some. Applies to other areas as well, though; I’m not sure the winner-take-all results in some industries are the best outcome overall.
**Stefan Jansen**
Vladimir Finkelshtein right, financial data is very noisy, precisely because markets are a competitive arena to the extent that quite a few say it should be next to impossible to predict anything about price movements at all. The best way out is of course access to (at least somewhat) exclusive data. Even then, and even more so if you rely mostly on market data, denoising via feature engineering (we cover e.g. Kalman filter etc) and regularlzation are key. In fact, complex models like deep NN are often at a disadvantage since they cannot benefit from the abundance of data that we have in other domains where they excel. In other words, success is unlikely going to rely on the biggest model but rather on a good overall setup from good data to robust risk and transaction cost management.
**Stefan Jansen**
Tino ML models perform best when there’s a systematic relationship between the input data and the target. They are least likely to do well when things fundamentally change. Fortunately, the Gamestop & Musk stories are not what dominates markets, just the news. So there’s plenty of room for models to add value (I think..).
**Stefan Jansen**
Arni Westh fair point, the book on Simons/Rentek in fact has some of the key figures of the fund on record that the one thing that never worked was trying to replicate academic papers. I work with clients and there are some examples in the wild that are working. It’s often based on a strategy that uses some specific data and has been developed manually before (and already created profits). ML is very useful to then automate and scale such a strategy. There are many use cases, just as there are many profitable traders that operate at limited scale, ‘under the radar’.
**nate8020**
Stefan Jansen in your view, where is the greatest opportunity at the intersection of ML and Algo-Trading right now?
**Vinoth Kumar**
Hi Stefan Jansen .
1. From your experience in developing ML systems for trading, what are the characteristics of a simple system and when does it become too complex. And do simple systems do Better than complex systems.
2. Retail investors have certain advantages over institutional investors in terms of size. How can retailers develop systems that play to their advantage over the big funds houses.
3. trading Systems also need to be aligned with the over market- bull, bear and side ways market. How can investors predict the future state of market or when they should fine tune their allocation to various systems.
4. What are some of the features that retailers usually miss in their trading systems design.
**Amr Alaa**
Hey Stefan Jansen, thanks for your answers here,
Can python developer or machine learning engineer benefit from your awesome book, or donwe need to review some of the basics first?
**Amr Alaa**
2nd question
Why it is so hard to get excellent results with ML in trading systems
We can develop reinforcement learning to deal with the market using paper money for 6 months, it will had most of the parameters needed to do good profitting trades after using real accounts, right?
I know it looks pretty silly, but why it is not working like this using modern learning techniques?
**Tino**
Hey Stefan Jansen :) thanks for the great answer :)
A follow-up question: depending on volatility and risk which varies from market to market, how would you rate the options for risk distribution over ones portfolio by using ML? And is this e.g. a more suitable/accurate use case?
**Vladimir Finkelshtein**
From your answers it sounds like that the industry prefers to rely on the advanced features engineered by domain experts and to apply less complex and reliable algorithms. Is there a hope that ML can find insights unknown to the experts? It seems like there are many regulatory and trust-related problems to use the latent features that ML offers in other domains.
**Stefan Jansen**
Vladimir Finkelshtein when it comes to financial data, esp market data (prices, volumes, returns), there is a lot of noise and complex models tend to overfit plus some domain expertise has developed over the last decades when this was the only data (plus fundamentals). What’s new is that now much more data is available - from new to credit card payments, mobile phone location or supplier activity. In this sense, finance is catching up to other industries and the evaluation of what works and what doesn’t is still underway. Clearly there are regulatory issues around privacy and also fair competition, which are also still playing out.
**Stefan Jansen**
nate8020 not sure there is one single greatest opportunity. There are several that range from -
* coming up with ingenuous use cases of novel datasets to
* NLP applications (given advances in the area and the limited use of text data thus far) and
* using ML/DL in low-latency context
**Stefan Jansen**
Vinoth Kumar
* The system should match the kind of data and scale you’re operating at. If you properly test your models you should notice pretty quickly when they over- or underfit. I don’t think there’s a general rule, it really depends on what you are trying you accomplish.
* Clearly, the most profitable hedge funds that use ML also operate at scale in terms of AUM, staff, data and range of different strategies. Your best chance as a retail investor is to find a niche that you know or are learning something about that has not attracted too many sophisticated players yet. I’m not sure it’s ‘the system’ as more the area you choose to play.
* Overall market trends is a macro prediction that retail investors can do as well (or poorly) as anyone else. It’s hard! But gotta keep trying 🙂
* Retail investors may want neglect to have a nimble system that permits them to constantly come up with new ideas and migrate them from backtest to paper- and then livetrading, while closely monitoring the latter. It’s always evolving.
**Stefan Jansen**
Tino there are attempts to use hiearchical clustering and others to handle asset/risk allocation, see Lopez de Prado 2016: [https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=2708678](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2708678)
Some results are promising, others not so much. Certainly some research in this direction.
**Tino**
Stefan Jansen thanks for the answer :)
**Stefan Jansen**
Amr Alaa if you’re familiar with python & ML using pandas/sklearn/tensorflow you should be fine and off to the races.
* It’s hard because markets are very competitive. See previous answers.
* Think about how trading works - for me to make money, someone else tends to lose some. There’s a lot of smart people at works because there’s a lot of money to be made. That’s why it’s hard. Market players adapt very quickly to changes in the market, news etc.
**Wendy Mak**
Hi Stefan, if I am not particularly interested in apply the algos to real trading, do you think the concepts introduced in your book still quite useful for other ML applications? e.g. I think you said there are some interesting timeseries algos you introduced in the book?
**Amr Alaa**
Great question by the way
**Stefan Jansen**
Wendy Mak I try to give a fairly broad overview of ML with plenty of background on how things work. Some have found it useful as a more general reference but really depends on your interests and background. Check out the website [https://www.ml4trading.io/](https://www.ml4trading.io/)
and the repo [https://github.com/stefan-jansen/machine-learning-for-trading](https://github.com/stefan-jansen/machine-learning-for-trading)
where you can see in detail what to expect.
**nate8020**
Stefan Jansen Say you set aside $100 (of real USD) of budget to build your own trading bot, what would be a good resource to put together such home-built service (APIs)?
**Stefan Jansen**
nate8020 not sure I understand, what would you spend the $100 on? Data, services, investment capital, learning reources?
**nate8020**
Say I want to create my own trading bot with real money ($100). Is there a public API I can use to actually execute trades and pull balances? You know, like the pros 🙂
**nate8020**
Stefan Jansen this is what I was looking for: [https://algotrading101.com/learn/robinhood-api-guide/](https://algotrading101.com/learn/robinhood-api-guide/)
**Stefan Jansen**
Oh sorry didn’t see your clarification. There’s also Alpaca; IB also has a popular API.
**Stefan Jansen**
The plan for the 3rd edition is to include the trading/execution part. If time permits..
**nate8020**
You’ve got your first sale ready! 😁
**Dan**
As someone who has read Stefan Jansen’s book thoroughly ([https://www.linkedin.com/feed/update/urn:li:activity:6712046823361536000/](https://www.linkedin.com/feed/update/urn:li:activity:6712046823361536000/)
) I can tell you that this book is fantastic.
Not only for trading and finance but ML and data science in general. It went brilliantly in tandem with my MS in Financial Math. It’s brilliant how it explains really complicated concept in easy to grasp chunks.
People like to compare this book to Marcos Lopez de Prado’s book but they’re not really a competition, I’d say they work in tandem. Furthermore this is the more technical one and with better applications.
MLDP’s is more general concepts and most of the code examples are from python 2. Not taking anything away from his books, they’re brilliant, but Stefan Jansen’s is definitely more up-to-date
Could not possibly recommend this book enough.
**Vladimir Finkelshtein**
What is the current hype in algorithmic trading and is it justified or not in your opinion? I heard people got excited about LSTMs but that was a while ago.
**Stefan Jansen**
Thanks Dan! Absolutely, I would not in my dreams attempt to compete with Marcos Lopez de Prado; I think we have very different goals. Adv. in Fin ML is a collection of research and practical advice from a leading practitioner and first and foremost delivers some very pointed insights straight from the frontier. I’m trying to provide a much broader introduction and think it is best read as a domain-specific intro to ML. I’ve done both trading and data science for while and when I was teaching DS in NYC I noticed that there was a lot of interest from folks in investment but hardly any specific material available. Personally I think it’s helpful to have a good grasp of how algos work I have included quite a bit of background and that’s possibly why I’ve heard a few times that folks have got something out of it about using ML more generally, esp in a time series context.
**Stefan Jansen**
Vladimir Finkelshtein I guess hype is almost by definition not justified? I think it’s a case of overestimated in the short run, underestimated in the long run. It takes time to figure out how to incorporate ML into trading processes since it’s not about switching over from human to machine in most cases; finding the right balance is a bit of work that takes many years. I’m sure in 10 years ML and the use of a very broad range of data is going to be common place. May look different than we imagine today but I think it’s hard to see that crunching excel is going to grow and python & co will disappear instead.
**tony hung**
Hey Stefan Jansen Im currently working on a forecasting model and I’m curious if the book goes into detail about ARIMA and othe r deep Learning models. I don’t know much about finance ,and more on the deep Learning side. Is the book geared towards finance people or ml people? I’m hoping both
**Stefan Jansen**
hey tony hung the book tries to sort of bridge the gap. there’s a chapter on arima, arch/garch and cointegration, and another one on RNN. and of course much more, check out the website and repo linked a bit above that should give you an idea
**tony hung**
Thanks, I’m going through the site now
**Stefan Jansen**
Great, pls let me know what you think!
**Vladimir Finkelshtein**
What are some other use cases of ML in algorithmic trading beyond traders making profit? (I am thinking of regulators here, but maybe something else?)
**Doink**
Stefan Jansen how to detect pump and dump characteristics in a stock? And to spot parties which create FUD? Do you think Renaissance Finance or Two Sigma handle this? I had seen a kaggle competition of 2 sigma where they tried to do sentiment analysis of stocks via news but they removed that data.
**Roman G**
I’ve heard a story about a system which detects insider trading at NASDAQ, which tries to estimate via ML how lucky you are. If you are too lucky, then a person in SEC takes a detailed look on your trading activity 🙂
**Doink**
Interesting stuff but how can you build such a system?
**Vladimir Finkelshtein**
That almost sounds like rule based approach: if you are making higher margins than others (in some metric), you will be looked at closely.
**nate8020**
Roman G Vladimir Finkelshtein do you know of an API in which I can place but/sell trades? I wanna play with the algos of the book with an actual (tiny) budget
**Vladimir Finkelshtein**
Not really.
**Roman G**
it depends on your scale. HFT traders usually use FIX apis to directly inject orders into broker system, but these are usually quite pricey: expect somewhere around 1-2k$/month + comissions.
**Roman G**
a low-cost solution can be using brokers public apis like this: [https://www.interactivebrokers.com/en/index.php?f=5041](https://www.interactivebrokers.com/en/index.php?f=5041)
**Roman G**
these APIs won’t give you ultra-low-latency, but it’s cheap enough to play with it.
**Roman G**
main pro of FIX is that you’re not tied to a specific broker and migrate freely. main con is that it’s quite complicated (market data tier2 stream for NASDAQ has so huge data rate so it’s non-trivial even to receive it)
**Chris Chia**
Hi Stefan, thanks for writing the book! Are there any recent papers or new ML/DL techniques that are still emerging, but you think could be particularly promising towards algorithmic trading?
**Stefan Jansen**
Vladimir Finkelshtein regulators would be interested in market manipulation through collusion, cornering or things like insider trading . methods like anomaly detection come to mind. here’s the SEC on the topic: [https://www.sec.gov/news/speech/bauguess-big-data-ai](https://www.sec.gov/news/speech/bauguess-big-data-ai)
**Stefan Jansen**
Doink for traders, detecting such schemes would be useful if they can profit from it. as such they would be looking for profitable momentum situations based on volume velocity and who is participating. regulators might be more interested in detecting such a scheme as and end to itself rather than a means to profit. they are certainly trying.
**Stefan Jansen**
Roman G see above - not familiar with this story but clearly if your trades turn a profit too many times you’ll raise flags just like in a casino.
**Stefan Jansen**
Chris Chia the book has tons of references, many of a recent nature. look up bryan kelly at yale/aqr, for instance who has been pretty active, and the work he and co-authors cite.
**Tino**
Stefan Jansen Is explainabilty a Big topic in trading? In credit risk for e.g. it is super important due to reportings for the regulator. Is it similar in trading?
**Stefan Jansen**
Tino the datasets themselves are certainly subjects to compliance re privacy etc. On the modeling side, I think explainability is more of an issue because decision makers (depending on the organization) want to understand if a model aligns with their priors than for the reasons you mention for credit decisions, for instance. It’s certainly a hot topic, but getting the models to work well is probably a higher priority.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Machine Learning Engineering in Action – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Machine Learning Engineering in Action
--------------------------------------
#### by [Ben Wilson](https://datatalks.club/people/benwilson.html)
##### The book of the week from 01 Mar 2021 to 05 Mar 2021
Machine Learning Engineering in Action is a roadmap to delivering successful machine learning projects. It teaches you to adopt an efficient, sustainable, and goal-driven approach that author Ben Wilson has developed over a decade of data science experience. Every method in this book has been used to solve a breakdown in a real-world project, and is illustrated with production-ready source code and easily reproducible examples.
You’ll learn how to plan and scope your project, manage cross-team logistics that avoid fatal communication failures, and design your code’s architecture for improved resilience. You’ll even discover when not to use machine learning—and the alternative approaches that might be cheaper and more effective. When you’re done working through this toolbox guide, you’ll be able to reliably deliver cost-effective solutions for organizations big and small alike.
* [Book's page on Manning](https://www.manning.com/books/machine-learning-engineering-in-action)
* [Book's GitHub repository](https://github.com/BenWilson2/ML-Engineering)
Questions and Answers
---------------------
**Akshaya Natarajan**
Ben Wilson Hi Ben! Can you share some common examples that you have come across where one might assume an ML solution to the problem but alternate approaches are way cheaper and efficient?
**Ben Wilson**
Hi Akshaya Natarajan, great question!
I’d have to say that, in broadly general terms, this is ‘the hammer’ problem.
“When all you have is a hammer, everything looks like a nail”.
I think it’s a default behavior for most modern ML practitioners that, when faced with a seemingly complex problem, they frequently default to supervised or unsupervised learning.
To answer your actual question, for examples… I might as well talk about dumb things that I’ve done and later fixed with a much more simplistic (and cheaper, easier to maintain, etc.) solution.
* Spending 2 months building a custom customer segmentation and similarity ensemble using KMeans, MinhashLSH, and a determinism-enforcing cluster labeling approach. It was about 3000 lines of modular and abstracted code. The final implementation was quantile bucketing on 3 variables to identify cohorts. With SQL. In about 25 lines of script.
* Transfer learning on ResNet50 to build a classifier that identified the primary color present in clothing on a person in an image. Training that on 118M images, adding some layers, and struggling with improperly labeled data took about 4 months of work. It got it right about 97% of the time at a cost of ~100k. Over a weekend, just for fun, I tried to use old-skool computer vision and some color space inversions (RGB-> HSV, filter skin tones -> RGB) and basic bucketing aggregations on the pixel data. It was basically 100% accurate and could run on my laptop. I had the MVP done in a weekend.
* Building an XGBoost + shap implementation for a business problem that identified as ‘key important features’ elements that were just general knowledge that the business operations team already knew and were part of their current daily job. (In this case, just walking over to the other side of the office building and having a quick 5 minute conversation could have saved me 3 weeks of work).
I have countless other stories like these that I’ve seen customers do (recent trends seem to favor a LOT of DL approaches to otherwise ‘already solved’ methodologies). It’s one of the reasons that I chose the example implementation in the book (Chapters 6 - 8 ) for time series modeling. Could an LSTM, properly tuned, beat the prediction accuracy? Sure, probably. Could you build it and have it ready for production in the same amount of time? Heck no.
**Akshaya Natarajan**
Wow amazing examples!! Thank you for sharing them. This is something to really think about! I completely get your computer vision example as I did something similar during one of my coursework. A simple MAP approach worked much better the ‘state-of-art-models’.
**Doink**
Ben Wilson so for time series which approach would you take?
**Ben Wilson**
Whichever approach is best for the problem being solved 😜.
Chapters 5,6, and 7 actually go through my exact process of prototyping (including all of the comical failures) for a seasonality dominated, non stationary forecasting use case.
I don’t get into a bakeoff with DL, but I have used RNN’s and LSTM’s in the past to solve forecasting problems, but I only go to DL implementations if the arguably simpler ARIMA-based methods don’t work well.
I always choose fast and cheap over complex and expensive if I can.
**Sejal Vaidya**
Hi Ben, nice to have you here.
Your book looks really interesting from an e2e ML project aspect.
My question is: Do you have a framework or a set of ways to understanding/approaching a problem and defining the business requirements and scope of an ML project?
**Ben Wilson**
A framework? Eh, not really.
I like to listen. I ask the business in a casual discussion about what they want to solve. Then I listen.
If the conversation seems like magical thinking (they’re talking about something that is so far beyond the scope of current capabilities of humanity that it’s doomed from an implementation perspective), then I gently steer the conversation to something that is more practical.
I never have just a single discussion at the outset either. After an initial ‘brainstorming’ session, I try to have some one-on-one discussions with the subject matter experts. If I’m building something that is intended to augment their job functions, then I want to watch them work. I’ll sit with them and quietly observe what it is that they’re struggling with, how they do that task, and try to gain as much insight into what they find frustrating or annoying about doing it manually.
If it’s something that is wholly new to the company (and myself!), then I like to have space between initial planning meetings and followup sessions where I can share options based on research and testing.
The TL&DR answer, though, is: ask a lot of questions, listen to the answers, and interview people 1:1. Then go and do some research and thinking about how to solve it, and discuss what the options are with the business in non-technical terms.
Just stick with that quote from Einstein and it won’t fail you:
“If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.”
You’ll get into trouble if, immediately upon hearing the problem, start thinking about feature data to collect, which algorithms to use, and instantly running out to find a blog that someone did on the subject to follow along with.
**Sejal Vaidya**
Thank you for answering! Of course, I agree that there can’t be a standardized way of doing this. I was actually thinking more around the lines of what you’ve stated in chapter 3 (planning & scoping) of your book: [https://livebook.manning.com/book/machine-learning-engineering/chapter-3/v-5/](https://livebook.manning.com/book/machine-learning-engineering/chapter-3/v-5/)
to get an idea on your approach of understanding the problem and gathering requirements based on that.
Your answer was certainly helpful. I’m presently looking at taking new responsibilities in Management along with my current Engineering tasks at work, and will try this out.
Thank you again. :)
**Ben Wilson**
So you’re going to be the gatekeeper and the rosetta stone for the ML team?
I wish you good luck and hope you enjoy the new role. It’s arguably the most critical position in a DS team.
My best advice on this? Make friends. Get to know people in the different business units (not just management folks, but more the ‘boots on the ground’ people who actually are doing the work in their departments or divisions). With a healthy working relationship, they can help to explain what a project needs to accomplish in the terms of how it applies to that group.
Ask a LOT of questions as well. Probe in as many ways as you can to figure out the hidden ‘gotchas’ of things that you may need to consider when building a solution (these elements are usually obvious elements to the business unit team, but are completely not-obvious to an outsider).
“Why do you not ever do ‘x’ to group ‘y’?”
“Oh, because they’re completely different in how they behave. We have different rules for how to interact with them.” <- these insights can help to define what they actually want out of the project. They’re also ways for you to identify if an ML solution can even handle those ‘special case’ elements.
As for scope, that’s all about collecting the total wishlist (with accompanying details that you garner from those interview-style interactions that you have with SMEs) and making a solid estimate based on your previous history solving similar tasks.
Then you lay out the timeline with options. Let the business sponsor determine what’s palpable. Associate each feature with a dollar and time value, which can really put the elements of features into terms that they can grok.
**Sejal Vaidya**
Wow, that’s very insightful! Thank you so much for sharing this. 😃 A lot of this is still quite new to me. I’m certainly not a gatekeeper, but I’m only gradually taking some of these responsibilities. So yes, nervousness with excitement but good support from my managers. Let’s see how it goes. 😅
**Alexey Grigorev**
Hi Ben! You have a chapter “moving from prototype to MVP”. What’s the main difference between a prototype and an MVP in your opinion? I see these terms are used interchangeably
**Ben Wilson**
I’ve always seen prototypes as the equivalent of ‘script complete’; basically, the model functions well enough (perhaps not fine-tuned, but it meets the requirements for the most part).
MVP phase is where everything else is done. The script has been transformed into maintainable abstracted code, unit tests are written, integration tests are finalized, monitoring / logging / registration capabilities are in place, and, perhaps most importantly, user acceptance testing (QA) is done. When all of those components are finished up with some form of hyperparameter tuning in place to automate away the infinitely frustrating task of guessing at good values, then you’re ready to do a production release with the MVP.
Basically when the MVP is delivered, it’s not the DS team that marks the project as a success. Rather, it’s the business that gives the thumbs-up, confident in the solution’s effectiveness to solve their problem.
The reason that I use the term ‘MVP’ is that all ML projects are, by virtue of how mutable they end up being over time (if they’re useful), constantly evolving to new versions. Features added or removed, more sophisticated approaches (or less!) are developed to solve the problem, or just minute drift-driven tweaks being done to the code base for ongoing maintenance.
**Alexey Grigorev**
Thanks! So there’s a vast difference between a prototype and an MVP!
**Alexey Grigorev**
Another one: the purpose of a prototype is to validate things as quickly as possible. How do we maintain code quality while still cutting corners and moving fast? Do you have any suggestions about that?
**Ben Wilson**
I never use my prototype code for building an MVP. It’s usually embarrassingly sloppy and difficult to read script. I’m trying out perhaps dozens of approaches, multiple different packages, have some unintelligible to anyone but me shorthand notes, and sometimes some less-than-professional variable naming conventions (depending on how frustrated I am in the build process of testing things).
Once I have something that seems like it works well enough, I use my terrible scripted nonsense to build out functions, abstracting away complexity and defining reusable fragments of code.
It’s from this point that I will do a presentation of the prototype to the team (and SME’s). If everyone agrees that it’s worthwhile to build out properly, I’ll start using those functions to design a maintainable code architecture, building classes, objects, et al. to create a ‘proper code base’. Every piece of work from that point on is done through branch commits, PR’s, and proper merges.
I’m also a fan of creating abstract utility modules to help me in my day to day work of prototyping and MVP development. I’m a super lazy developer and it pains me to have to re-implement the same logical block in different projects, so I’ll just put that functionality into its own repository with other similar ‘tooling’. Over time, as that common code base grows, the amount of ‘terrible code’ that needs to be written during the fast prototyping phase reduces significantly.
**Alexey Grigorev**
Great, thanks!
**Alexey Grigorev**
Just curious, how many of your prototypes don’t survive and don’t become an MVP?
**Ben Wilson**
My prototypes work 100% of the time, 5% of the time.
My general rule of thumb is that for any production-grade, truly useful ML solution that I’ve built, I typically throw away 80-90% of all code written during the journey to production.
There are always crazy ideas that I thought would be clever (and usually aren’t), inefficient code, computationally or space complexity issues that need refactoring, or just a load of trials of ideas that don’t pan out. I think that having a mindset when approaching ML work that you are expecting to throw away a lot of things (which is good - it means you approached the problem like a scientist, testing hypotheses) and also time-boxing idea trials is one of the most successful ways of going about it.
**Alexey Grigorev**
That’s a good idea to have that mindset - thanks for sharing it !
**Wendy Mak**
Hi Ben, what do you think are some of the biggest mistakes people/companies make when trying to move ML from experimental prototypes to fully functional products?
**Ben Wilson**
Hi Wendy, that’s a great question. It’s actually pretty much the synopsis for the entire book.
If I were to rank mistakes for the top 5 things that I see ‘blow up a project’….
1. Not having business buy-in on the solution / not knowing what the actual ‘thing’ is that you’re working to build. (No matter how fancy the algorithm utilized, how cool the code is, how production-ready it is… if you’re not solving the problem for the business, the project is D.O.A.)
2. Not having attribution monitoring set up (the “how well is this solution solving our problem” question). If you can’t measure your solutions, the business is just guessing at its effectiveness. You don’t want the project to get blamed for being either successful or detrimental with the nigh infinite realm of exogenous latent factors that drive a business. With data, statistical tests, and (perhaps) causality modeling to go along with the the hypothesis testing and attribution modeling, you’re armed with the munitions needed to defend the solution.
3. Terrible code standards. Unreadable code, fragile code, and dangerous code. The number of times that I’ve seen blind exception eating cause production down-time that erodes the business’ faith in the DS team is too many to count.
4. Decision paralysis. No one can decide on what direction to go in from the prototype phase because everyone has spent so much time on their solution. Having no process about time-blocking experimentation, research, and testing of theories can cause an emotional attachment to prototype solutions (I think there’s something in there about behavioral sciences and psychology… I’m not an expert about that, I’m just reporting what I’ve seen and experienced).
5. The business expects magic and no one on the DS team has told the business that they are not, in actuality, wizards, sorcerers, conjurors, or mythical beasts capable of snapping their fingers and just ‘making it happen’. The ‘it like magic’ perception at some companies and the hubris of very junior DS practitioners can be a recipe for disaster.
There’s about 600 other pages that’ll be in the final book that touch on many of the other things that I see (and have lived through) cause projects to blow up in people’s faces. 🙂
**Denis Lepchev**
Hi Ben!
How do you think a notebook based environment will evolve to make it more suitable for production-ready development? Do you envision a convergence between notebooks and full-fledged IDEs? Do you think the industry got it right at the moment?
**Ben Wilson**
Hi Denis!
Oooooh… is this a question for Ben the author of this book, or Ben the Practice Lead at Databricks? 😉
There is a great deal of active development in this area - basically to bridge the gap between the benefits of an IDE (depending on language utilized, this can be substantial - Java or Scala ML development outside of an IDE is a nightmare) and that of a notebook (embedded visualizations, widgets, live data-focused results in a friendly environment). A lot of work to support the bridging of these different development practices is in abstraction of code in notebooks, utilizing caller and callee notebooks that can be wrapped up in a project structure that can be properly version controlled (branching, merging, et al.).
Part 2: Do I think people have this figured out?
No. Emphatically, no. There is a massive gap between how DS’s work (primarily in notebooks), focused on a lot of feature work (ELT + cleverness) and algorithm utilization and how production code is written.
I don’t think an easy solution exists to make the transition between prototype script in notebooks (even if you’re using functions) to production-grade abstracted FP/OO based testable code. Every time I do a production-grade ML project, I’m doing my prototyping and hacking away in notebooks, only to have to port over functionality into OO modules in a package. It’s time consuming, frustrating to test during development and annoying to debug in an IDE.
I think a big part of why a lot of this hasn’t reached a critical mass until recently is that large-scale production-hardened ML code hasn’t been adopted by enough companies yet. Many are running notebooks in production, struggling with issues that crop up, wasting countless hours updating scripts to keep the lights on, and generally hating life when they ship something to ‘production’. There are, of course, plenty of teams out there that are packaging their ML solutions into .jar’s, .egg’s, .whl files, etc. They just sort of ‘deal with it’ like I do. That’s not to say that anyone enjoys the process, though.
**Denis Lepchev**
\> is this a question for Ben the author of this book, or Ben the Practice Lead at Databricks?
It was a question for both (I am a Databricks user) 🙂
Sometimes I feel it is like a big elephant in the room that nobody notices - but good to hear about the progress. You are right - probably critical mass hasn’t been reached yet.
**Ben Wilson**
You’ll really like what’s coming late summer / early fall this year 😉
**Alexey Grigorev**
Hi Ben!
What are the key skills that a machine learning engineer should have? And what are their core responsibilities? How is this role different from data engineering, in your opinion?
**Ben Wilson**
I know what I look for in people who I interview 😉 .
DE skill set…
* SQL knowledge = pro
* Python and/or Java or Scala = intermediate data-focused skills of those languages
* Deep understanding of space complexity
* Deep knowledge of real-time streaming architectures
* Relational data modeling theory
* Capable of implementing architectures in RDBMS, NoSQL, GraphDB’s, and how to interface with Queues (streaming).
* OO knowledge = intermediate (should know how to use abstract classes, how to implement factory patterns, and how to design with proper encapsulation, how not to make a mess with polymorphism, how to write data generators for unit tests…)
* FP knowledge is a plus (I guess? some people really like it. I think it’s pretty cool.)
* How to speak to Analysts properly.
* How to be an agreeable and pleasant human.
ML Engineer: I really think this is just the natural progression of a DS. It’s a continuing education broadening of skills that I’m a firm believer that any ML practitioner should be working towards.
* Deep specialization in some ‘flavor’ of ML (NLP, Timeseries, Computer Vision, traditional supervised learning, unsupervised learning, bayesian… whatever you work with the most, you should be VERY knowledgeable about not just how to use some API, but know how the algorithms work and be able to read their source code implementation and explain how it was coded to work the way it does).
* General knowledge of 1 or 2 other fields of DS work from above.
* Be able to construct standard DL architectures from scratch.
* Deep understanding of statistics, probability, and linear algebra.
* Highly proficient at OO.
* Skilled at creating visualizations.
* Capable of talking to other humans in a way that doesn’t give them a headache (translate the geekspeak to layperson’s terms)
* Highly proficient at distilling vague (or overly complex) requirements and designs into simple, explainable, and story-level work tasks
* Capable of public speaking
* Proficient in at least 2 software languages (non-declarative; OO / FP-based languages)
Maybe that was a bit too much. But, in essence, Data Engineers specialize in different things than ML Engineers do. The one thing that is common between them is that they’re both working with data, manipulating it to make it useful for purposes, and they’re both writing professional code (and can do so in several languages). To be successful at either requires a great deal of time, effort, and the ability to be humble and pleasant to work with.
**Alexey Grigorev**
Wow, that’s very detailed, thank you!
**Alexey Grigorev**
Just curious, why public speaking is in the list? is it a must-have or nice-to-have?
**Ben Wilson**
If a DS can’t communicate to a crowd, they’re going to be hampered when having to explain how their solution works to the business and will be at a disadvantage in discussions with internal customers.
I’ve sent DS employees that I managed in previous companies (I’m an IC now, no direct reports) to public speaking classes to help them be more successful in this critical aspect of their careers. When they can ‘find their voice’ and learn to communicate in a complexity-free manner with non-technical coworkers… that’s when they really shine.
**Alexey Grigorev**
That’s nice of you to send them to classes! Makes sense, thank you
**Alexey Grigorev**
Also, what is the breakdown between ML and engineering for ML engineers - in terms of time spent working on these things?
**Ben Wilson**
In a typical project that I work on…
30% research, reading, testing things, more reading, asking questions, talking to people about the problem.
30% Getting the data into a state that is useable for the project, acquiring the data (writing scheduled ETL), fixing issues with the data.
10% messing about with models, tuning them, having an algorithm tune them for me, figuring out how to improve the mess that I’ve created.
20% Writing maintainable, extensible, abstracted, and testable code. CI/CD sometimes (usually if a large team is going to be messing with the code base frequently). Attribution, logging, and AB framework construction.
10% Having meetings to win over the hearts and minds of the business. Listening to their feedback. Involving SME’s from the business to contribute their thoughts, perspectives, and deep knowledge to make the project better. Talking about it in presentations, convincing executives why it’s important. Apologizing to the Engineering Budget VP for why I blew through their yearly budget in the first 2 months.
**Alexey Grigorev**
That’s quite a broad set of responsibilities! Like a full-stack data scientist
**Ben Wilson**
I’d say that’s where I see most DS people end up going after 10 years experience. If they’re a solo person at a startup handling the full DS responsibilities (with a lot of DE responsibilities baked in as well), it’s basically a hard requirement to have all of those skills (and more 😉 ) to be successful.
**Alexey Grigorev**
\> 10% Having meetings to win over the hearts and minds of the business.
I see why you want the engineers to be good at public speaking!
**Matthew Emerick**
Hey, Ben Wilson! Thanks for doing this. What ratio do you see for data engineers versus data scientists? I know that more companies are suddenly hiring data engineers more than scientists, but where do you think it’ll end up?
**Ben Wilson**
Oh, that’s a good question.
I’d say it really depends on the industry, the age of the company, and the general skill of the workforce at the company.
At startups, I often see “DaVinci’s” - women and men who have to assume the role of both DS and DE to get their projects working. At those small companies everyone sort of has to do everything. You learn a lot, but burn out quick.
At non-tech-focused ‘smallish’ companies (200-1000 employees, total tech workforce in engineering / analytics / ds might be 20 - 100? depending on industry)…
If they were founded in the last 10 years, they’ve probably already got a modern-ish stack and have a lot of quality of life tooling to help them with DE tasks. So, typically not super DE-heavy. They also don’t have a lot of legacy systems loaded down with decades of techdebt to deal with. In these companies (if their data is cleanish), I see a big demand for DS’s who actually know what they’re doing (they’ve put 10+ projects into production at previous companies, have been around long enough to know how to generally do things right).
If they’re an older company, the DE demand is MUCH higher. Loads of techdebt, weird decision made that make moving data around a completely excruciating process, and just need a lot more talented DE’s than they do DS’s. A DS in this environment won’t have too much to do until most of the techdebt is erased.
Enterprise companies…
hrmmmm…
Which is more in demand?
YES.
Both. But skilled of both types. Typically you’re dealing with massive amounts of data (I’ve been working with a client recently who has a single data feed ingest through Kafka into Structured Streaming -> Delta of ~ 160k records / sec, each payload is around 2MB uncompressed JSON). Some of the DE workloads for these customers are not for the faint of heart.
ML for Enterprise companies is usually in pretty high demand and some execute better than others. The projects are typically very large-scale and complex so a large, well-managed team is critical to be successful there.
**Alexey Grigorev**
As I understood, the target reader of the book is a “data scientist familiar with supervised machine learning and the basics of object-oriented programming”.
Will software engineers or data engineers also find your book useful? In case they want to transition into ML Engineering
**Ben Wilson**
For the process around building projects, certainly. I intentionally don’t go into the nuts and bolts of algorithm work or feature engineering work (that’s what your book’s for 😉 ).
Hand in hand with some books on algorithms and applied ML / applied analytics and statistics, I think it completes the intro breadth of topics that people would need to be successful at the start of moving into this profession.
FWIW, I do hear a LOT from DE’s who want to ‘make the transition’. I always tell them the same thing when they ask how much they need to learn… “Just read up on stats. You already know how to code (ed.: most can code better than 90% of DS folks) and you already understand how to handle data. Algos and stats is all you’re missing.”
**Alexey Grigorev**
Makes sense, thank you!
**Alexey Grigorev**
And in general, what would you recommend doing for data engineers and software engineers if they want to get into ML engineering? What’s the best way to make the transition, in your opinion?
**Ben Wilson**
“Algorithms, stats, and get Alexey’s book on applied ML.”
**Alexey Grigorev**
Nice sugguestion!
**Neal Fultz**
Hi Ben - Im working with a company that has stacked their model in to another company’s (essentially a black box), and trying to control / monitor performance. How does ensembling / stacking / banditing affect the ways to monitor and troubleshoot models effectively?
**Ben Wilson**
Depending on the model, that could be a black box inside another black box (model inception? pimp my ride, ML-edition? 😉 )
Jokes aside, I feel your pain. Attribution modeling is rarely a super fun thing to build when you don’t have direct control of the model and can only kick off a retraining and ‘hope for the best’.
In the scope of ensembles in general, you’re compounding the issue of traceability to begin with. When you’re looking at performance, you’re evaluating the correlation of features to a target (I assume we’re talking about supervised learning here) and scoring some loss or error function.
The ‘easy way’ that most people build ensembles (that’s a joke. Ensembles are ML on hard mode for production stability and maintainability considerations) is to just score the final stage for stacking (and ignoring the interim stacked states or not logging them anywhere). For pooled ensembles, where you’re applying a custom function to the results of many individual models… well, hopefully each one is getting logged.
I can’t say that I see many ensembles in the wild; if I had to guess the percentages that are being used (referring to ‘this runs in production, is useful, and has been running for > 6 months’) as an overall percentage share of supervised modeling efforts, I’d say it’s less than 5%. My advice is to ensure that each model’s tuning histories are logged as children to a parent entry (the parent being the project as a whole or the final model in a stacked implementation). MLflow works great for this and gives you that provenance of all of the components.
As for attribution (how does this crazy-complex implementation impact the problem that we’re trying to solve?), that’ll be the same as any other model implementation; just treat the ensemble as a single entity and score its merits through proper hypothesis testing.
Stay tuned for Chapter 12 where I delve into this very problem in (potentially overwhelming) detail - specifically attribution modeling, AB testing, and drift detection.
I mention drift detection here because you’re going to have a LOT of it with an ensemble. It’s not going to be pleasant to diagnose through analytics either.
Best of luck!
**Alexey Grigorev**
What the process should look like from identifying a user problem to solving it with ML in production? What are the steps in the process and which tools should we use at each step?
**Ben Wilson**
The process that I typically use is probably a little controversial ;)
1. Make friends. Get to know the SME’s that you’re working with. Figure out how they best learn / understand / communicate. Once you can REALLY talk with them, they will tell you all the ‘gotchas’ about the domain’s problem space. If there is mutual respect and friendliness between the business unit experts and the DS team, you will help each other out and work as a single unit.
2. Always look for a way to solve the problem WITHOUT USING ML. If you can solve a problem with analytics or ETL, do that. We, as a profession, work to solve problems, not use algorithms. The simplest approach is always the best approach.
3. Do your homework. This doesn’t mean read 2 or 3 blog posts, see that everyone is talking about XGBoost or Tensorflow and assume that those libraries are the panacea to all of your woes. Read some books, read some white papers, take your time to understand the problem space and then…
4. Test a lot of alternatives. But do it like you’re competing in a Hackathon. Give each attempt a fixed period of time. Remember that if it’s super hard to get a proof of concept working, it’s going to be a nightmare to maintain in production.
5. Pick the most promising result from rapid prototyping. Don’t pick the ‘new hotness’. Some of the best models I’ve built that have lasted the longest in production have been generalized linear regression and Bayesian models. The ‘old stuff’ might not be talked about much these days, but IT WORKS and it works very well.
6. Validate your data. Seriously, just please do formal EDA. Correlation, collinearity analysis, distribution fits, etc..
7. Use the right tool for the job. Don’t use Spark for a 20MB training set and don’t use scikit for a 200GB training set.
8. Learn more than one language. We do a lot of ETL as DS’s. Scala is a great language.
9. Don’t be afraid to let the business know early that it’s an unsolvable problem. Pivot to something easier.
10. Watch your costs. Electrons are not free.
**Alexey Grigorev**
That’s a nice one! Love #1. Thank you!
**Andy Petrella**
the last 20 years I have been working on that subject told me that it very vast. There are some important paper on it also like the one I love from D. Sculley on hidden tech debt in ML.
My take on this is that it requires a certain level of maturity of the team and even organization, because as Ben Wilson says, in a way, the responsibility is spread as pipelines have dependencies. Therefore the data culture is important
The maturity can be achieved IMHO by introducing a specific management twist, which I called “data intelligence management” (I recently wrote an article on that matter) - the focus is on the data applications, not on the data.
Also, I have a forthcoming O’Reilly training on how to monitor ML in production with Python (the first session is the 28th of April) that I’ll invite you and the community to join — when the page and announcement will be online ahaha.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Designing Data-Intensive Applications – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Designing Data-Intensive Applications
-------------------------------------
#### by [Martin Kleppmann](https://datatalks.club/people/martinkleppmann.html)
##### The book of the week from 08 Mar 2021 to 12 Mar 2021
NoSQL… Big Data… Scalability… CAP Theorem… Eventual Consistency… Sharding…
Nice buzzwords, but how does the stuff actually work?
As software engineers, we need to build applications that are reliable, scalable and maintainable in the long run. We need to understand the range of available tools and their trade-offs. For that, we have to dig deeper than buzzwords.
This book will help you navigate the diverse and fast-changing landscape of technologies for storing and processing data. We compare a broad variety of tools and approaches, so that you can see the strengths and weaknesses of each, and decide what’s best for your application.
* [Book's page on O'Reilly](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/)
* [Book's website](https://dataintensive.net/)
Questions and Answers
---------------------
**Tino**
Hey Martin Kleppmann :) Thanks for taking the time! How important would you consider planning this process in different stages of a company? Should a start up immediately plan applications for data-intense use cases or should the focus be elswhere?
**Martin Kleppmann**
Hey :) I would say it depends on what the company does. If you’re in the business of processing large amounts of data (e.g. you’re crawling large numbers of websites), it’ll be worth thinking about architecture up-front.
But most startups (e.g. typical SaaS apps) don’t initially have a lot of data. For these early stage companies, I would specifically advise _against_ investing in a scalable architecture until you actually need it. The priority of an early-stage business is to be flexible and quickly adapt to the needs of customers as they are discovered. A complicated architecture is actually harmful here, because it reduces flexibility. Personally I’d just stick with PostgreSQL in this situation, and keep everything as simple as possible.
**pie\_f11235**
Hi Martin could you elaborate this (chapter 3) a bit more?
Counterintuitively, the performance advantage of in-memory databases is not due to the fact that they don’t need to read from disk. Even a disk-based storage engine may never need to read from disk if you have enough memory, because the operating system caches recently used disk blocks in memory anyway. Rather, they can be faster because they can avoid the overheads of encoding in-memory data structures in a form that can be written to disk
**Martin Kleppmann**
What exactly do you find unclear or do you want to know more about?
**Martin Kleppmann**
Data in memory typically has a different representation from data on disk. For example, in memory you can use pointers, but pointers to memory addresses don’t make sense on disk, since after you restart a computer all the data will be at different memory addresses.
**Martin Kleppmann**
Therefore, you need to convert between the disk-oriented representation and the in-memory representation. This conversion incurs a cost. A database that doesn’t maintain on-disk data structures can save the conversion, because it can work with the in-memory representation only.
**pie\_f11235**
when reading this from the book it gave me the impression is disk-based db + cache would be just as equivalent good and therefore we do not need to have in-memory db at all
**pie\_f11235**
or if we do then in-memory dbs essentially shall be used as cache.
I wonder if I miss any point if I think in that extreme?
**Martin Kleppmann**
For a more detailed argument I suggest reading the papers referenced in this section:
[https://hstore.cs.brown.edu/papers/hstore-lookingglass.pdf](https://hstore.cs.brown.edu/papers/hstore-lookingglass.pdf)
[http://www.vldb.org/pvldb/vol6/p1942-debrabant.pdf](http://www.vldb.org/pvldb/vol6/p1942-debrabant.pdf)
**pie\_f11235**
ok thanks Martin. let me check them out. I read the lookingglass one but not yet the other one. I probably misunderstood the whole thing
**Vladimir Finkelshtein**
In different data teams (small startup, midsize, large internet company) which role is usually responsible for the design?
**Martin Kleppmann**
I’m afraid I don’t have enough first-hand experience of different companies to give a good answer.
**Vladimir Finkelshtein**
It seems like you cover how different tools fit your tasks and other tradeoffs, and that is supposed to help you make your tooling choices. I wonder if in reality the choices are made based mainly on unrelated criteria. For example, how the pricing scheme of these tools matches your usage/budget or whether your team is already familiar with the tool.
**Martin Kleppmann**
I think whether a team is already familiar with a tool is an important factor. The fewer new tools you have to learn, the better! But it shouldn’t be the only factor: if you try to bend a single tool to all possible applications, you may run into difficulties too. As always, it’s a judgement call.
**Martin Kleppmann**
In practice, I suspect a lot of decisions are also made based on whether a technology is currently fashionable or not (e.g. whether the team members recently heard a conference talk about it, or read a tweet about it). That is poor practice in my opinion, but I can’t deny that it happens!
**Martin Mihal**
I just check book quickly (chapters, topics etc) - just question before invest time into full reading . Are those topics (eg. Scalabality, Maintanance, Replication ..) in world of cloud (eg. ..Kinesis, S3 as Data Lake, Athena, Snowflake..) still worth to go deep into? All of those tools were created to handle those topic “in price”… (I mean it’s important to undrstand those topics, question is how deep in this complex world we want/need to go)
**Martin Kleppmann**
Cloud services reduce the amount of investment you need to make into operations, because someone else is providing the operations team for you. But in order to know how to use a cloud service effectively, you still need a good understanding of what it can and cannot do. And you often don’t get that understanding from reading the vendor’s documentation alone, because a vendor will always pretend that their product is great for all purposes, even if it has strengths and weaknesses.
The purpose of my book is to teach you the fundamentals so that you can figure out the strengths and weaknesses of different technologies. That applies equally, regardless of whether you’re self-hosting or using a cloud service.
**Slim**
Hello, thanks for writing a great book like designing data intensive applications. My question is; do you think that relational database is the suitable solutions for banking systems. According to some experiences I had with finance companies they prefer relational database because of the utility of transactions and it’s straightforward to use. But sometimes I feel like it needs a graph database due to huge number of join and relation between tables which makes things complex. What do you think about that?
**Martin Kleppmann**
Hi Slim! I have never worked at a bank, so I don’t have personal experience. But my impression is that it doesn’t make sense to talk about the suitability of a technology for banking as a whole, since banking is a complex business where different parts of the business may have very different requirements. The needs of the people doing fraud prevention or anti-money-laundering will likely be very different from the needs of the people processing mortgage applications. The needs of those dealing with investments or trading will be different again.
**Martin Kleppmann**
I understand if there is a preference of relational databases since they are quite versatile and can satisfy the needs of a fairly wide range of applications (though not everything), and sometimes it’s better to stick with a tool you are deeply familiar with (“the devil you know”), even if it’s not quite optimal for the use case, than to incur the cost of learning how to use and reliably operate a different tool.
**Martin Kleppmann**
Keep in mind that choosing to adopt a certain database is not just a matter of the developers choosing one API or query language over another. There is also the need to set up backups, disaster recovery, to have ETL to bring the data from operational systems into a data warehouse, maybe audit logs, etc. etc. — adopting a new data storage technology also requires figuring out all this infrastructure.
**Martin Kleppmann**
So I have a lot of sympathy for organisations that are hesitant to adopt new technologies, even if they are better suited to the problem than whatever they are currently using.
**Slim**
thanks for this detailed answer :relaxed::relaxed:
**A McCauley**
Hey Martin :wave::skin-tone-3: great book! What would you say from your experience, is one of the most overlooked principles when designing data-intensive applications?
**Martin Kleppmann**
I’m afraid I don’t have a good pithy answer for this. Different companies/situations have different needs and different problems. Part of the ethos of the book is that there isn’t one single technology or principle that is universally applicable; rather, there are trade-offs, and knowing what they are will let you choose the right tool for the job.
**A McCauley**
And what is the most apparent challenge for teams/companies, etc. when designing data-intensive applications?
**Martin Kleppmann**
I think the biggest challenge is perhaps that there is such a bewilderingly large number of tools out there, and it can be really difficult to figure out which is best for a given task. Hence a primary goal of the book is to help you figure out what questions you need to ask in order to choose an appropriate technology.
**Slim**
what’s the approach that you recommend in microservices in order to guarantee the ACID properties in a transactions. I think it is not possible to guarantee ACID in distributed systems especially for rollback or when there is an error during the transaction.
**Martin Kleppmann**
most microservices architecture use transactions only within the bounds of a single service, and don’t have distributed transactions across service boundaries. this can sometimes make it really difficult to keep data consistent. there have been attempts to allow cross-service transactions (such as WS-AtomicTransaction in the SOAP world) but I fear this is not a good answer, since it introduces tight coupling between services.
**Martin Kleppmann**
one approach I’ve seen recommended is to draw the boundaries between services such that cross-service transactions simply are not necessary. this means that if you have two services that do need atomic commit, you need to merge them into a single service — i.e. don’t make the services too fine-grained.
**Martin Kleppmann**
another idea I’ve seen is to use a persistent log such as Kafka for communicating between services (instead of just ephemeral HTTP/RPC requests). this doesn’t quite give you atomic transactions, but it does give you pretty strong assurance that if one consumer reads a message off the log, all of the consumers will read that message. you can build some pretty strong guarantees based on that.
**Martin Kleppmann**
I elaborate on that second idea in this article on Online Event Processing: [https://martin.kleppmann.com/papers/olep-cacm.pdf](https://martin.kleppmann.com/papers/olep-cacm.pdf)
and Ben Stopford has also written about using Kafka for inter-service communication: [https://www.confluent.io/designing-event-driven-systems/](https://www.confluent.io/designing-event-driven-systems/)
**Slim**
thanks Martin. I have seen a similair approach mentioned in Microservices patterns book written by Chris Richardson.
**André Duarte**
Hey Martin! What would be the top 3 things you would add, remove or change from the book if you were to rewrite it?
Thanks for your time
**Martin Kleppmann**
Big question! I have been collecting ideas for a second edition but have not yet started on it. Some ideas for improvement are:
* talk more about hosted cloud DB-as-a-service infrastructure, which has gained popularity rapidly (but which still requires a lot of technical knowledge to use well)
* there are tons of new tools in areas such as time series, workflow engines, new transaction processing engines (“NewSQL”)
**Martin Kleppmann**
* maybe a bit more on decentralised and [local-first](https://www.inkandswitch.com/local-first.html)
technologies, since interest in those has been growing
* I was thinking it might be cool to interview people at a couple of major internet companies and get them to share examples of their architectures, as case studies to discuss in the book
**Sal DiStefano**
If you have not checked out Martin Kleppmann [blog](https://martinkl.substack.com/)
, you should. He is also on [Patreon](https://www.patreon.com/martinkl)
.
**Uriah Stephenson-Ward**
Hi Martin Kleppmann do you plan on updating or making any revisions soon? Is there any specific tech/concepts/etc coming out that you think is exciting or will likely become the dominant way of working?
**Martin Kleppmann**
I am collecting ideas for a second edition, but not planning to start work on it for at least another year, since the current content is still pretty up-to-date.
I replied about ideas for a second edition in [this thread](https://datatalks-club.slack.com/archives/C01H403LKG8/p1615222541049600)
**Elias**
Hello Martin, it’s a great honor, and thank you for your book! In the last chapter, you are writing about the future of data systems. And many of your ideas and approaches from this chapter are already common nowadays (derived state, unifying batch and stream, designing apps around dataflows, etc.). But from your point of view, in recent years in data systems:
* what are the things you expected to happen that didn’t happen (or you wish happened, or should’ve happened faster/at a bigger scale)
* and what are the things that happened but you were not thinking about them when writing the book?
Thank you!
**Martin Kleppmann**
some of this has happened, but my sense is that the big idea — redesigning interfaces and APIs around dataflow rather than request/response interaction — is still a long way away. there are plenty of REST APIs for online services out there, but not many of them let you subscribe to a log of changes. the best you can usually hope for is a webhook, and this is really just a notification mechanism — you can’t use it to reliably maintain derived views onto the data in someone else’s API.
**Elias**
Thank you for the answer!
But do you still believe this can/will happen, or “application” and “data” worlds are doomed to live apart :) (similar to DBMS and OS evolved separately)?
**Martin Kleppmann**
hard to say without a crystal ball. there is a huge amount of existing investment in REST API, and hence a huge inertia preventing a move to anything else. on the other hand, change data capture in databases seems to be becoming mainstream, and when you combine that with stream processing and event subscriptions you’re not too far off.
**Nikolay**
Hello Martin. I read your article “Please stop calling databases CP or AP”
Could you please provide some clarification? Does it mean that when we are talking about a distributed queue we can not talk about strong consistency ( linearizability) at all. I mean that even if value X is written to queue it will not be immediately available for reading even for that client(read your own writes consistency model). So distributed queue can not provide strong consistency, can it? if so… what the strongest level of consistency can we have in distributed queue.
**Martin Kleppmann**
Linearizability is a very useful property and it makes sense to talk about it. However, it’s a property of a particular _operation_ or set of operations; a system may well provide some operations that are linearizable and others that are not. For this reason it doesn’t really make sense to label the system as “consistent” or not in the sense of CAP. Rather, we should say, for example: the enqueue and dequeue operations are linearisable.
**Martin Kleppmann**
Defining the “strongest level of consistency we can have” is a bit tricky because there is no good formal definition of what “stronger” means. But I do think that in practice, linearizable enqueues/dequeues are probably the strongest useful consistency model you will find for a queue.
**Nikolay**
One more question from my side. if we have 2 nodes - A and B. A is a leader. B is a follower. client issue write(X,1) to a leader. if the leader uses 2PC in order to implement atomic commit. is it possible to have a case when A and B have different observable values before A sends ACK to client?
**Martin Kleppmann**
observable to whom? to another client that is querying A and B? atomic commit/2PC does not guarantee anything about concurrency (it’s about handling crashes cleanly, not about concurrency). it is entirely possible for a client to query A and B while the 2PC protocol is in progress, and to get different responses from the two nodes.
**Nikolay**
I have a question that I guess is related to chapter 12 of the DDIA book. Most databases have their own cache. For example, oracle, postgress has its own buffer cache ( its cache of blocks). Cassandra has a row cache. Oracle also has a row cache. On another side, we have solutions like Redis, Memcached. As far as I can understand database is moving in the direction to have a separate computation engine and separate storage engine. Why not have a separate cache engine? I mean it would be nice to have the ability to have cache “inside” database(integrated cache) but so that we would be able to scale this cache. as far as I can see it’s not a way of modern databases. So there is probably something wrong with this approach. I mean we can not have an integrated scalable cache, can we?
**Martin Kleppmann**
This would make for an interesting research project! I don’t think it’s possible to say for certain whether this approach makes sense without actually trying it and seeing whether it helps and where it breaks down.
**Martin Kleppmann**
There is a long-standing debate between implementors of database engines and implementors of operating systems about caching of disk pages in memory. The OS automatically uses memory that would otherwise be unused to cache recently accessed disk blocks. But then a DB may need its own buffer cache (e.g. because it needs to control when a dirty page is written back to disk, because it has to go to the WAL before the data page is written), so data ends up being cached twice, which wastes memory.
**Martin Kleppmann**
there is a difference between memcache/redis and these block-level caches, which is that application-level caches such as memcache/redis don’t just store disk blocks, but they store entire precomputed responses to complex queries, potentially including some business logic. serving that data from a cache doesn’t only save on I/O, but also on computation time, which might be significant.
**Nikolay**
Cool. Thank you very much for your reply. Just want to add that Oracle has its own ROW cache. “When a query executes, the database searches the cache memory to determine whether the result exists in the result cache. If the result exists, then the database retrieves the result from memory instead of executing the query. If the result is not cached, then the database executes the query, returns the result as output, and stores the result in the result cache.
When users execute queries and functions repeatedly, the database retrieves rows from the cache, decreasing response time. Cached results become invalid when data in dependent database objects is modified.”
**Nikolay**
Regarding Chapter 3 of the DDIA book. As I understand LSM is optimized for write compare to B+Tree because LST writes less to disks.
If I understand correctly in the case of B+Tree each block is mapped to a disk file and when we update even 1 byte in B+Tree we have to update the whole block. as I know in oracle we will not update each block when we change 1 byte. we will write the whole block to disk just in case of checkpoint (every 3 seconds in the background) and during update statement, we will only write to WAL. So when I do update idx\_value = 1 where id = ? database will write only 1 byte to WAL and the whole block will be flushed to disk only during checkpoint ( for simplicity I skipped part related to oracle undo segment). So it looks like for a simple update we will have to write to disk the same amount of bytes in the case of LSM and B+Tree, does not it?
**Martin Kleppmann**
I wouldn’t necessarily say that “LST writes less to disk” (a statement about write amplification), but rather that LST writes tend to be a more sequential access pattern compared to B-tree writes, and so there can be a performance advantage on disks where random-access writes are slower than sequential ones.
**Martin Kleppmann**
as far as I know, a disk always writes data an entire block at a time, even if you only change 1 byte in a file. (that’s why linux calls a disk a “block device”.)
**Martin Kleppmann**
a relational DB will tend to have its own concept of pages that doesn’t map exactly to physical disk blocks, but is roughly related.
**Martin Kleppmann**
it’s not the case that every database block/page maps to a separate file (file systems would not cope well with such a large number of small files). I think DBs usually allocate one big file containing many blocks, and use offsets into that file to identify blocks.
**Martin Kleppmann**
if you update just one row, you’re right that the WAL write will happen first, but the page containing the row will also have to be written sooner or later, even if the transaction is allowed to commit before it is written.
**Nikolay**
In chapter 5 of the DDIA book, you described multi-leader replication. Does it mean that in the case of multi-leader replication particular client can connect only to 1 leader? i mean that if we have 2 DC and lots of clients (say 10K) … each of our clients connects to a particular leader and if that leader is crashed … half of our clients can not connect to another leader? This part is not clear to me. as an example, it works like that in app calendar .. because i can connect only to 1 leader( which is on my own machine) . I can not connect to other clients :-). but what about 2 Data Center ( 3 DC)?
**Martin Kleppmann**
it will depend on the specific system, but in general I don’t see any reason why you can’t have one client to connect to multiple leaders. one potential setup would be for a client to connect to a leader in the local DC by default, and fall back to connecting to a different DC in the case of problems.
**Alexey Grigorev**
I have two questions from ankush khanna who’s travelling now and can’t ask the questions himself.
So the first one:
What do you think about the future of Serializable Snapshot Isolation?
**Martin Kleppmann**
I think it’s excellent, and more systems besides PostgreSQL should use it!
**Alexey Grigorev**
Second:
Your book covers a lot of ground regarding Streaming, but with advancement and popularity in streaming will you write more material on topics like Kafka or Pulsar?
**Martin Kleppmann**
Perhaps, although other authors have already covered streaming systems in great detail, so I’m not sure there is much more for me to add.
**Rishabh Bhargava**
Just to follow-up on this: which authors would you recommend reading to dive deeper into streaming systems?
**Martin Kleppmann**
“Streaming Systems” by Akidau et al.; “Kafka - The definitive guide” by Narkhede et al.
**Martin Kleppmann**
(I think a second edition of the Kafka book is in the works)
**Rishabh Bhargava**
Thank you!
**Alexey Grigorev**
Hi Martin Kleppmann! You were working as a software engineer, but then went to academia and started working as a researcher. What motivated you to focus on research?
**Martin Kleppmann**
I got tired of the short-termism in industry, especially in startups, where everything has always got to happen “right now”. I wanted a setup where I would have the space to think, to take the time to really understand things, and to work to improve the foundations of how we write software. In research I can work on things that may not be practical for another 5–10 years, and that’s fine. In a company you can’t normally work with such a long time horizon.
**Alexey Grigorev**
I can totally relate to that. Thank you for your answer!
**Jonathan Diaz**
Hi Martin Kleppmann. It’s awesome to e-meet you and I actually just finished reading DDIA a few days ago! One quote that stuck out to me was in the last chapter from Maciej Ceglowski “_Machine learning is like money laundering for bias_” which led me down a rabbit hole of finding the source [here](https://idlewords.com/talks/sase_panel.htm)
. What, in your opinion, can we do further so that ML algorithms/technologies don’t strengthen existing biases? Both for data folks who develop these algorithms and those who use them.
**Martin Kleppmann**
I’m afraid this is not my area; others are working very actively on this, but it’s fast-moving and I am not up-to-date. I suggest looking up work by folks such as Timnit Gebru and Cathy O’Neil.
**Manoj Agarwal**
Hi Martin Kleppmann I watched your amazing video series on [Distributed Systems](https://www.youtube.com/watch?v=UEAMfLPZZhE&list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB)
. Are videos of any other of your courses available publicly?
**Martin Kleppmann**
This is the only multi-lecture course available so far. I also have this playlist of conference talks I have done: [https://www.youtube.com/watch?v=fU9hR3kiOK0](https://www.youtube.com/watch?v=fU9hR3kiOK0&list=PLeKd45zvjcDHJxge6VtYUAbYnvd_VNQCx)
**Martin Kleppmann**
it includes some course-like material, such as this 2-hour lecture on formally verifying distributed algorithms: [https://www.youtube.com/watch?v=Uav5jWHNghY](https://www.youtube.com/watch?v=Uav5jWHNghY&list=PLeKd45zvjcDHJxge6VtYUAbYnvd_VNQCx&index=26)
**RH**
Hi Martin,
Background: I work in a startup with a small engineering team; I mostly work on building NLP based microservices (data analysis to devops) in Python to serve other parts of our product. We have a few models in production which are working quiet well, and en route to building a few more.
_Question 1_: When I joined, we did not have a lot of data (few Gigabytes), but data is starting to build up (Terabyte). We currently store most of our data in Postgres. Most of the data is large blobs of semi structured texts that are uploaded by our customers to be processed and saved (the data format is similar to resumes). I believe that Postgres was the right DB to start with because it was very versatile and was good for making this happen “right now”, but not sure what the right thing to do is next? The queries are becoming slower and slower. I have thought of saving all the larger blobs of texts in a S3 bucket, and saving a retrieval link for that object in Postgres, but that makes retrieval slower. We retrieve the data very often.
_Question 2:_ When the data is sent to us by the customer, we do not process it in anyway. We just save it the way it is sent by the customer. Do you think it is a good idea to process the data, and save “features” from the data in Postgres instead of reprocessing the data over and over again. That is, we should save the data the way the customer sent it (maybe in a S3 bucket) but we should process and save a version of the data that is relevant to our business case (this is something the engineering lead does not see value in, and prefer to beef up the servers).
_Question 3_ What is a good way to structure a data engineering / systems engineering career i.e. what makes a really good data/system engineer?
**Martin Kleppmann**
q1: it’s difficult to make a concrete recommendation without knowing much more about the characteristics of your data, the access patterns and queries you use, etc. You might be able to continue using Postgres by splitting the database across multiple machines, perhaps using something like Citus. If you don’t do many joins in queries, you could consider using a key-value store/document DB such as Cassandra or MongoDB, but that would be a much bigger change.
**Martin Kleppmann**
a distributed filesystem or an object store like S3 would make sense if the data items are large. this would scale well, but would make it much harder to do any indexing (i.e. finding documents based on some value that occurs within a document).
**Martin Kleppmann**
q2: Saving data in raw form, and separately storing data derived from it, is a great pattern. If you want to change your processing logic, you can then re-run it on the raw data. The datastore to use for the derived data will again depend on the data format and your access patterns.
**Martin Kleppmann**
q3: my suggestion would be knowing how to reason about trade-offs. there is never one true answer, only various options that each have their pros and cons. being able to figure out those strengths and weaknesses, and communicating them to the rest of the team, seems very valuable to me.
**RH**
Thank you so much for responding to my questions.
**Amr Alaa**
hey Martin Kleppmann
thanks for having this week with us, I am really enjoying reading these excellent questions and your great answers here, I do not think I have an exact question actually,
just if you have a plan for another book, more specific or more advanced
I believe that in this DATA era that we live in, you should consider a series of books or even a bundle of courses to help data engineers acquire more advanced skills in theory and in practical also
thanks again
**Martin Kleppmann**
Thanks! I am considering ideas for a book that would go into more details of algorithms used in distributed systems. But work has not yet started, so there is no timeline for when this might happen.
**Alexey Shvets**
Martin Kleppmann first of all thank you so much for you amazing work and legacy. In you video course you mentioned Leslie Lamport as a legend in the field of distributed system, but for many people you became a legend who made the field structured and accessible.
My question. What do you think, which programming language will dominate distribute systems development in the future? Now it is mostly Java, will it move to Golang/Rust? What is your personal preference?
**Martin Kleppmann**
I think programming languages are a question on which people have very strong opinions because each language is like a tribe, and people couple their identity to the tribe they belong to. In my opinion, the language in which a system is written rarely makes a big difference; more important is the system architecture. Most mainstream languages are probably okay for building many types of systems.
**Martin Kleppmann**
In particular, I think questions of expressiveness and details of language features do not have a big effect when it comes to system operations. What does make a difference is a language’s runtime characteristics, such as whether it supports threads, and whether it uses a garbage collection runtime. No GC means no GC pauses, which can be important for low-latency systems.
**Martin Kleppmann**
Rust is interesting to me because it’s memory-safe, supports threads, has no GC runtime, and is very portable (you can use it to build mobile apps or servers or web apps by compiling to wasm). But in the end, it’s the needs of a particular system that matter. There is no one language that is perfect for all systems programming.
**Pavel Bukhmatov**
Hey Martin Kleppmann! Thanks for a book and all the awesome research you do!
What is your opinion on implementing distributed storages using conflict-free replicated data types in future? The ideas behind CRDTs seems to be really compelling for distributed systems but the sheer complexity might overwhelm practical implementations as far as my limited understanding goes. What research / papers / books could you suggest on the topic?
**Martin Kleppmann**
I am a big believer in CRDTs, and have been doing research on them for the last 6 years! I maintain a community website [https://crdt.tech/](https://crdt.tech/)
that has a lot of resources on CRDTs, and links to all the latest research (including my own).
**Alisher**
hi, Martin Kleppmann, thanks for the book and all your educational activity!
I’d like to ask you a bit broad question - what do you think are the most promising and perspective topics in distributed systems research in next 5 years? which topics are underestimated and require more
researchers there?
Thank you.
**Martin Kleppmann**
Well, this is going to be very subjective, since every person has their own priorities! Personally I am excited about the potential of [local-first software](https://www.inkandswitch.com/local-first.html)
, and moving away some of the current cloud-centric view of distributed systems. That’s what I’m going to be working on for the next 5 years, anyway!
**Nikolay**
Hello Martin Kleppmann. Could you please help to build intuition about [Version Vectors](http://en.wikipedia.org/wiki/Version_vector)
and [Vector Clocks](http://en.wikipedia.org/wiki/Vector_clock)
. i read about them in chapter 5 but can not understand the difference. i am looking for some examples in order to build intuition.I know that it’s a long story and i read also some articles from Riak founders. But i’s not still clear for me ). Maybe some good example will let to understand it.
**Martin Kleppmann**
Ah yes, I have struggled with this as well. I’m afraid a proper explanation goes beyond what I can provide here in a few sentences. I should do a blog post on this at some point.
**Martin Kleppmann**
at a high level, the difference is in purpose. a vector clock is used to compare _events_ in a distributed system, and figure out which happened before which, and which are concurrent. a version vector is used to compare _states_ of replicas, and figure out whether one state supersedes the other. the mechanism in both cases is similar: a vector of numbers that are incremented.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Database Internals – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Database Internals
------------------
#### by [Alex Petrov](https://datatalks.club/people/alexpetrov.html)
##### The book of the week from 15 Mar 2021 to 19 Mar 2021
Have you ever wanted to learn more about Databases but did not know where to start? This is a book just for you.
We can treat databases and other infrastructure components as black boxes, but it doesn’t have to be that way. Sometimes we have to take a closer look at what’s going on because of performance issues. Sometimes databases misbehave, and we need to find out what exactly is going on. Some of us want to work in infrastructure and develop databases. This book’s main intention is to introduce you to the cornerstone concepts and help you understand how databases work.
The book consists of two parts: Storage Engines and Distributed Systems since that’s where most of the differences between the vast majority of databases is coming from.
* [Book's page on O'Reilly](https://www.oreilly.com/library/view/database-internals/9781492040330/)
* [Book's website](https://www.databass.dev/)
* [Slack community to talk about databases and the book](http://bit.ly/joindatabass)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Street Coder – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Street Coder
------------
#### by [Sedat Kapanoglu](https://datatalks.club/people/sedatkapanoglu.html)
##### The book of the week from 22 Mar 2021 to 26 Mar 2021
Software development isn’t an “ivory tower” exercise. Street coders get the job done by prioritizing tasks, making quick decisions, and knowing which rules to break.
Street Coder: Rules to break and how to break them is a programmer’s survival guide, full of tips, tricks, and hacks that will make you a more efficient programmer. This book’s rebel mindset challenges status quo thinking and exposes the important skills you need on the job. You’ll learn the crucial importance of algorithms and data structures, turn programming chores into programming pleasures, and shatter dogmatic principles keeping you from your full potential.
* [Book's page](https://www.manning.com/books/street-coder)
* [Book's GitHub repository](https://github.com/ssg/streetcoder)
Questions and Answers
---------------------
**Alexey Grigorev**
Hi Sedat Kapanoglu, thanks a lot for finding time to take our questions!
I prepared a few, so let me start with the first one.
I know there are some “rules” in software development. Like you need to have X% test coverage, your code needs to follow the SOLID principles, etc. So what are these rules? Do you have a list of them?
**Sedat Kapanoglu**
Thanks Alexey Grigorev for the opportunity! The rules in software development are mostly folkloric tales that come from curriculum, code reviews, or HackerNews. Because of their viral nature, it’s hard to come up with a complete list. But, the common narrative among those rules are that they are treated like immutable laws that can’t be questioned. For example, when talking about SOLID, you should follow SOLID all the way without questioning the reasoning behind them. That kind of approach usually makes you spend too much time on stuff that doesn’t really matter in a professional setting where I call “the streets” in the book. In the streets, the rules are vague, priorities are fluid. Similarly with X% test coverage, you might spend a disproportionate time developing the last 1% because you might feel the urge to at least achieve X%, rather than X-1%. So, I go over some of these rules and either explain the nonsense behind the rule, or explain the actual justification behind it so, the next time it comes up in code review, you can defend yourself. 🙂
**Alexey Grigorev**
Thanks! Are these other similar rules and best practices that we usually try to follow - and they are sold as silver bullets?
Like pair programming, following TDD and these sorts of things
**Sedat Kapanoglu**
Alexey Grigorev Exactly. I’m all for silver bullets, but like literal silver bullets, they are rare and they work worse than described in the books. They usually create more problems than they solve when treated as immutable laws. Since you mentioned it, I have a section in the book called “Don’t use TDD”. Unit testing is invaluable, but TDD itself is unnecessarily constraining. I think rapid prototyping and retrofitting tests to an existing code is a much faster and practical approach. In “the streets”, requirements change all the time and TDD assumes a spec is ready beforehand and won’t be updated a lot. I find it disconnected from the reality.
I don’t go into pair programming in the book, (perhaps I should). Pair programming is very effective in getting the code quality up, adding to the productivity of the developer. The problem is that pair programming achieves this with the expense of another developer, completely blocking their work. It’s inefficient in that sense. I find code reviews a fair compromise.
**Alper Demirel**
Hi Sedat Kapanoglu,
* What are the most important gains we will achieve when the book is finished?
* Also, what is your motivation to write the book?
I’m a big follower on Twitter, thank you this opportunity 👋
**Sedat Kapanoglu**
Hi Alper Demirel! I think the one key takeaway would be that the rules and paradigms have some reasoning, some logic behind them, and knowing their logic would help you decide to make better decisions in a constrained setting, usually the case with professional career. I usually try to explain the details of certain concepts (like how a class is laid out in memory), so you exactly know when a class or a struct would be preferable, and why, rather than “hey stick to classes and you’re fine”. When you internalize the reasoning behind these concepts, your decision making process in a professional setting becomes both fast and accurate. More importantly, you don’t dig yourself into a hole 🙂
About motivation, I started to keep notes years ago for a potential book that might help the software developers I work with, so I don’t have to repeat myself on every topic. A new developer is usually idealistic and dogmatic as neither have they an understanding of the inner workings of the rule set they’re working with, nor they have been in a professional setting enough to prioritize the problems. I wanted to fill in the knowledge gap from a perspective of a self-taught developer like me, and infuse the habit of questioning. We’re getting bombarded with best practices, the newest greatest framework every day, and I wanted to create a book that would give a clean perspective on approaching those.
**Alper Demirel**
Thank you very much for your detailed answer. I hope I read your book and benefit from your experience as a new developer.
**Alexey Grigorev**
There’s this saying (from Picasso I think) - “Learn the rules like a pro, so you can break them”.
Which software engineering rules, in your opinion, we should learn before we can break them? Do we need to know SOLID by heart before we can say “Okay I don’t care about O here, but I have to make sure D is there”?
Which rules would fall into this category?
**Sedat Kapanoglu**
I think many letters in SOLID are there just to make it look like a cool word. I’m not saying they don’t matter, but they don’t matter equally. If I had written my own SOLID, it would just be called “S”, because I find “God class” problem the most prevalent and causing the most problems in a professional setting.
I think new developers should incorporate the process of understanding the reasoning behind a rule before accepting it by heart, and make this a habit. That can be hard in a class setting because students can feel pressured to keep going at a certain tempo and you don’t want to anger other fellow students by insisting on these seemingly pedantic questions. But, you can make it a pastime to retroactively work on these concepts and find exactly why they are useful. I say, don’t internalize a rule before making sure it’s useful. Don’t start applying a rule before being convinced of its benefits.
Embracing immutable data structures, or at least avoiding mutating state after creating a class can help you prevent a whole set of bugs from surfacing entirely. I go into details of that topic in the book.
**Alexey Grigorev**
I’m also curious how did you come up with the name? Did you come up with it or the folks from Manning’s marketing? 🙂
**Sedat Kapanoglu**
I’m a self-taught programmer, and I had to learn everything taught in software engineering by myself, “in the streets” if you will. 🙂 I came up with the name Street Coder as I see myself as one. You don’t learn some of these stuff at school, and you don’t know value of some topics you learn at school before you experience them in the streets first hand. For example, the notion of algorithmic complexity can just be a boring lesson at the university, but it’s a mandatory tool to have on your toolbelt in a professional setting. I explain Big-O notation and algorithmic complexity in the book in simple and practical terms and show scenarios where such knowledge can make a significant difference.
**Evren Unal**
Sedat Kapanoglu i think the name of the book suits the book very well. 👍
**Sedat Kapanoglu**
Thanks! I’m glad to hear that 🙂
**Neal Lathia**
❔ what is the biggest dogmatic principle in coding that you’ve come across & that you want the world to be rid of?
**Sedat Kapanoglu**
That’s a great question Neal. I think I’d choose TDD as one of the top contenders. It impedes a rapid prototyping cycle (aka not fun and unrewarding), and it constrains you from making changes too early in the development. I love unit tests, but TDD is like converting such a fun activity like writing tests into a military drill.
Inheritance as a code reusability model comes probably the second. Usually, any problem with inheritance can be solved with composition, and in return you get great decoupling. It’s usually easier to start with inheritance at the beginning in some languages, so I’m not completely against it, but I suggest anyone who’s getting slightly complicated class hierarchies to move onto composition. Languages like Rust and Go have better default reusability models based on traits/interfaces.
**Alexey Grigorev**
I really like that you put TDD here as the number one thing. I sooo feel the same way.
I once tweeted about it and many people unfollowed me 😅
**Neal Lathia**
Thanks! Great answer. Am a big fan of Go too 🙂
**Anton Helm**
I think I fully agree with Sedat Kapanoglu. However, I think TDD and inheritance have their use-cases, and I think you should never use a principle purely. I think you should try to understand the idea behind each principle and ask yourself if it applies to my use-case or my project. (Same applies for programming languages but not for editors - VIM always wins!!! 🙂 ). I would say if someone applies a coding principle dogmatically, they should remove it. But automatic and repeatable tests in the codebase are required. Moreover, the code should be formatted consistently and have “some guidelines.”
**Heeren Sharma**
Sedat Kapanoglu Thank you for providing such elaborate answers. I wanted to ask your POV or argument that I often hear for “Street Coders” that they can’t thrive in larger codebases as they don’t embrace certain concepts fully but some parts here and some parts there. I remember a personal experience, 7-8 years back, I interviewed for a company and interviewer asked me if I do TDD. And my answer was something similar as explained in your “dogmatic principle” answer. And at the end, I didn’t get the job and interviewers really defended the fort by saying that without this, one will not go far in professional setting. Since then, I have seen two world of software developers. First is quite strict on coding principles while others is go with the problem statement, come up with working elegant solution without caring too much about strictness of certain practices (e.g. taking S from SOLID). But later one if I may say are also referred as “Code Hustlers” 😄 and I have an inherent feeling that they are looked down upon. I am interested if you had similar experiences and How did you answer that “Learning like a Street Coder is equally if not less powerful approach”?
**Sedat Kapanoglu**
Hi Heeren! Cargo cult is a real thing and I think it was a fortunate outcome that you weren’t hired. You wouldn’t be happy there. It’s also possible that the company was using strict workflows in software development that a spec would be set in stone first and the code would be written later. In that case, TDD can be a beneficial practice, similar to how PostgreSQL team writes documentation for a feature first, and develops the feature after. But generally, such strict development processes are never fully applied, and design is usually subject to change even during the development process. TDD even contradicts Agile principle of embracing evolving requirements in that aspect.
It’s natural that companies would want to hire people fitting to their own culture. I’ve also known my share of people who’d look down upon certain stereotypes, be it a “code hustlers”, “college dropouts”, “youngsters”, or even “new graduates”. Such people actually put themselves at a disadvantage by depriving themselves of great talent. Microsoft hired me as a software engineer to Windows team as a high school graduate because they optimized their hiring process to vacuum as many talents as possible regardless of stereotypes. I think the sector will eventually evolve to a point that hiring practices would be optimized to identify talent objectively. Anything else would be inefficient.
**Heeren Sharma**
Thank you so much for your answers.
**Mert Bozkır**
Welcome Sedat Kapanoglu, I am really happy to see you in here.
Here is my question:
I am freshman student in university and I am trying to improve myself in Machine Learning Engineer or Data Science areas (don’t focus title). I was following courses in platforms such as Coursera, edx, cognitiveclass. But I realized that if I follow the courses I am stuck with theoretical concepts and I was assumed if I want to learn C, I need to learn A and B too.
This is difficult situation and everything coming to your face. My question is how I need to change my learning style. I know theory is so important but I require focus practice, doing something even basic concepts.
What is your suggestions to Freshman or Sophomore students to practice ? How we approach this working style as students?(Your style 😉)
**Sedat Kapanoglu**
Hi Mert! Great question. If we were in the early 2000’s I would say “hey why don’t you develop a new project?”. I can’t say that now because the list of requirements to bring a software project to life can get really long (frontend, backend, multiple platforms, multiple programming languages, frameworks, libraries) .
Instead, I can recommend contributing to open source projects. Project owners have already handled the hard part of creating it. You just need to read the code, understand it, and write small bugfixes, tests, and even features to it. Not only would this count as a practice, you can also expand your social network with new developers this way. Developing good relationships is very critical in the streets, and doing that early would be the second bird you’d hit with that one stone. 🙂 You can proudly list the open source projects you’ve contributed to in your resume too.
You can also practice your coding skills with sites like Leetcode, Project Euler, Codekata, and those would prepare you for interviews too as the problems asked would be similar, but it wouldn’t be as fulfilling as contributing to a real project.
**ankush khanna**
Hi Sedat Kapanoglu, love your comment on TDD and honestly I would have benefited a lot from such a perspective early in my career (100s of hour wasted in fixing TDDs).
What are your thoughts around tech debt? When is the right time to tackle it and how can we as developers take actions on it, rather than waiting for the PM or PO to prioritise it?
(Considering the tech debt is huge and cannot be sneaked into a improvement ticket 🙂 )
**Sedat Kapanoglu**
Hi Ankush! There is a section in the book titled “Leave no debt behind”. What I argue in the book is that the constantly changing code is more adaptable to change than untouched code for a long time. That sounds unintuitive because any code change is a source of potential regressions. But, I find benefits of activities like refactoring, dependency upgrades, small code improvements, paying technical debts overwhelm the risks in most projects, especially at the initial stages of a project. You will experience breaks, but those breaks will teach you where your code is weak, and you’ll strengthen that part. The code will remain malleable, easily modifiable. Otherwise it turns into “`// DON'T TOUCH - NOBODY KNOWS HOW THIS WORKS`”. Dealing with such changes keeps you involved with the code, your knowledge stays up to date. And, I even claim in the book that after making all those changes, you can throw the new code you just wrote in the trash if you find it too significant or too risky. You get to retain the value of practice, experience, and even motivation. It heightens the sense of code ownership in team members too.
I think addressing tech debt should be a day to day practice like formatting code, as long as it’s aligned with business goals. I also talk about techniques to how to make it part of your daily work in the book.
**ankush khanna**
Thank you for the great explanation, I believe I never looked at it that way, although following it on many micro-services. Definitely helpful 🙂
**Eric Sims**
Sedat Kapanoglu - Your book is very engaging! I love your intro and story about doing your project in Pascal.
Figure 3.8 caught my eye because I am constantly battling with high quality variable and function naming. Your rule of thumb about avoiding needing to use “and” or “or” is great!
**Eric Sims**
Okay, I’m back. Chapter 3.9 “Don’t write code comments” is so helpful! I am definitely guilty of obvious commenting. `df = pd.read_csv() # read CSV` 😅 Also, the comparison between Listing 3.18 and 3.19 is so helpful for seeing how to pull small bits of code into functions.
Which brings me to a question…
I have a project I am working on right now that is essentially one giant function. The client is not super technical, so the function basically uses loops and returns a single final output. I am worried that if I put things into functions it will make the code harder for them to read/troubleshoot. Do you think it is better to leave the code in a more “linear” format for legibility, or is there something I am missing that would make it more legible by putting things into smaller functions?
**mucio**
What I got from a senior colleague years ago was: a function should do a single thing. Keep the function shorts and with meaningful names (so people will remember what that function does - no need to check again and again what it does).
I prefer to keep thing simple and use the power of the language.
For example if you want to log before and after every method you call (I saw that recently), use a decorator for that :)
**Sedat Kapanoglu**
Hi Eric! I’m so glad to hear that you liked the book! About your question, I think you’re already on the right track to address the problem. If you think what you end up with is less readable, then you shouldn’t be doing it. It might be the case where you are extracting too much logic from the function into other functions that the function itself fails to convey what it’s doing. Try to focus on extracting details irrelevant at the function’s scope.
For example, a function like `computeCsvAverage(filename, column)` could be opening a CSV file, parse its contents and compute average on a given column based on the parsed contents and return the result. since the main promise of this function seems to be “computing average”, the CSV parts can be abstracted away and the function can look like this:
`float computeAverage(string filename, string columnName) {
var csv = readCsv(filename);
var numbers = parseColumn(csv, columnName);
return numbers.Average();
}`
This tells what the function does without hiding any logic at the scope of the function itself. Bringing in any of the logic in one of those functions doesn’t help explaining what the function does. It might even hurt readability as it clutters the function body. If you follow a similar principle, readability shouldn’t be hurt.
(By the way, you can compute while reading the CSV without needing to read everything in memory beforehand, but this one was just for the sake of the example :))
That was a great question! I’ll consider adding a note about this detail in the book.
**Eric Sims**
Thanks for the helpful and detailed response! I’ll definitely take a look at my “giganto-function” and see where I can create useful functions without obscuring logic. I originally wrote the code in chunks in a notebook, so it’s fairly modular, but then I stuck it all together so I could have it in a single script file for deploying in a web app.
I am still learning how to properly use global and local scope, so this should be a good learning experience!
**Alexey Grigorev**
Hi Sedat Kapanoglu!
I’m a fan of the “make it work, make it right, make it fast” approach - do a quick-and-dirty PoC, prove value and then iterate. However, I often get a lot of resistance from people who want to get it right from the start.
What do you think about it? If you also agree with this approach, how would you convince others that it’s okay to cut corners sometimes?
**Sedat Kapanoglu**
Silicon Valley was built upon that principle. 🙂 I still think having even the barebones of a design or a roadmap helps a lot in setting the course for the development team. Some key technical decisions can be important for the initial launch too. But, I agree that pedantry should be left to post-launch.
My greatest example for that (after Facebook having been written in PHP) is Eksi Sozluk, which is now one of the most popular Turkish web sites in the world. I created Eksi back in 1999 over the course of three hours without knowing anything about web development. I even used a single plain text file as a database, probably the worst possible choice, just to get the product up and running as soon as possible. Today, it runs on a small server farm with multiple web, cache, load balancer and DB servers, handling almost 40 million unique visitors per month. Remnants of the original code can be seen here: [https://github.com/ssg/sozluk-cgi](https://github.com/ssg/sozluk-cgi)
Having a tangible, working product at hand that you can later shape like clay today can be more productive than working on a fictional idea with no results for weeks or even months. It affects developer psychology positively; it sets the tempo for rapid iteration, and user feedback starts flowing immediately. Good user feedback can easily be the most critical asset of a successful project.
Besides, you’ll encounter cases in the streets where you’ll need a product in such a short timeline that you don’t even have time for any planning or elegant design and you just have to get something up and running. It’s at least good to get yourself acquainted in the practice.
That said, remember that the idea of code is way easier to refactor than the code itself.
**Alexey Grigorev**
Well said, thank you!
What about the cases when you can a POC, it works, but inside it’s ugly and super not optimal. But management is excited and wants to start adding more features instead of “making it right”?
How would you handle their expectations?
**silverstone**
it’s really nice to see ssg in a data driven community. who knows maybe one day we can find an official api for eksisozluk 🤭
**Alexey Grigorev**
I’m checking the source code - wow, it’s been a while since I saw Delphi code!
**Ricky McMaster**
`I even used a single plain text file as a database, probably the worst possible choice, just to get the product up and running as soon as possible. Today, it runs on a small server farm with multiple web, cache, load balancer and DB servers`
Hi Sedat Kapanoglu - I’m really interested in this particular topic, and the evolution of standards in a company as it matures regarding data. A problem I come across again and again is where start-ups initially operate on the ‘move fast and break things’ mentality, and backend developers hurriedly throw together a database without worrying too much about data governance and maintenance. Technical/data debt accrues, and fast forward 5-10 years later to the point where it is difficult to confirm basic, strategic information e.g. historical product pricing information (I could mention many other examples).
I am not talking so much here about granular customer event-based data that a company _receives_, but rather how it organises and structures its _own_ product/operational standards (like pricing, product descriptions and taxonomies, campaign naming conventions etc.), particularly when this requires input and maintenance from business stakeholders in marketing, sales and finance, to name but three.
Do you have any words of advice here? Would you say there is still plenty of value in an ERP system, or do you see promise in a fresher approach such as the so-called postmodern ERP? Or do you favour in-house solutions?
**Sedat Kapanoglu**
Alexey Grigorev That’s a good question. First, even when writing your dirtiest code, never write code that you can’t bear to get stuck with. 🙂 Second, it’s usually hard to argue about fictional future costs to management, so usually, you can’t really bargain about this with them. But, you can improve code quality over the course of development with small steps of refactors. As I explain it in the book, there are multiple benefits of improving the code continuously. You don’t usually need to write everything from scratch. Sometimes, a minor change can improve the quality of life in orders of magnitude.
**Sedat Kapanoglu**
Ricky McMaster I haven’t had the chance to work on ERP systems, but you’re right in the sense that technical debt eventually becomes business debt. It can reach to a certain point where it cannot be ignored, and it actively hurts business. With a data sensitive area like ERP, it must be doubly so. Eksi Sozluk today suffers from technical decisions from two decades ago like using VARCHAR for text fields, which doesn’t allow features like emojis or other Unicode characters today, and switching to a new charset can be very expensive, it would at least require a long downtime for the upgrades and would increase the demand for storage and I/O. But, I’d say that any technical debt that can survive for at least a decade is a good tradeoff decision. I can’t really comment on which ERP solution would be ideal though as I’m a stranger to the subject.
**Ricky McMaster**
Thanks a lot for your response Sedat Kapanoglu
**Alexey Grigorev**
Which chapter are you working on right now? What’s the main idea of this chapter?
**Sedat Kapanoglu**
I’m currently working on the chapter about scalability, which is the penultimate chapter of the book. I’ll be clarifying how scalability differs from performance, and how we can be way more productive with monolith architectures than microservices. I’ll also be expanding upon asynchronous I/O concepts that can be instrumental in scalability scenarios, which were introduced in the previous chapter about optimization.
I’m really excited to be getting close to the end of the book, but I still have too many notes from the feedback from readers. I really love Manning’s MEAP process for that; it’s the rapid iteration cycle for books. The book will be v1.6 or so when it’s done. 🙂
**Alexey Grigorev**
Hehe, prototyping and iterating =) yes, MEAP is great!
**Alexey Grigorev**
In your opinion, what are the most important concepts in scalability?
**Alexey Grigorev**
Is it architecture, autoscaling, possibility to work on codebase by multiple people? What else could be there?
**Sedat Kapanoglu**
I think the dependency graph of a project is the most important aspect in scalability, be it the code itself or even in the database. For example, foreign keys are great for ensuring data integrity, yet they can be obstacles when you try to split up the database in multiple shards. They can even become performance bottlenecks. I’ll be focusing on “poor man’s scalability” in the chapter that will let you get as much scalability from your code on a single piece of hardware. That can be thought of as a performance-oriented chapter but there are ways to make code more scalable without improving the performance too.
**Alexey Grigorev**
Clear, thank you!
**Alexey Grigorev**
\> Street coders get the job done by prioritizing tasks, making quick decisions, and knowing which rules to break.
We already talked about which rules to break - but what about task prioritization and making quick decisions? If you were to give tweet-sized advice about each of these topics, what that would be?
**Sedat Kapanoglu**
I use two distinct concepts for prioritization: priority and severity. Priority is the business-related urgency of the matter while severity is the level of technical screw-up, aka developer’s urgency to fix it. A web page crashing can be very severe in the eyes of the developer, but if it’s a page that only 0.1% of users visit, it may not get a higher priority. Similarly, using wrong logo on the homepage isn’t technically severe at all, and the developer may not care much, yet it can have the highest priority due to business goals. So, sort your tasks on priority first, severity second.
About making quick decisions: Knowledge can be a curse in the decision-making process because you know all the possibilities of how something can go wrong, and that can cause disproportional delays. Budgetize your decision-making process itself and go with the best option at the end of your time budget. If you’re still undecided at the end of your time budget, you might as well roll a dice, because none of the options have stood out enough for you to have a decision anyway.
**Alexey Grigorev**
How do we determine the priority? By talking to the stakeholders? Or maybe there’s a way to develop this “sense of priority” in ourselves?
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Learning Tensorflow.js – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Learning Tensorflow.js
----------------------
#### by [Gant Laborde](https://datatalks.club/people/gantlaborde.html)
##### The book of the week from 29 Mar 2021 to 02 Apr 2021
Combining the demand for AI with the ubiquity of JavaScript was inevitable. With Google’s TensorFlow.js framework, seasoned AI veterans and web developers alike can help propel the future of AI-driven websites. In this guide, author Gant Laborde–Google Developer Expert in machine learning and the web–provides a hands-on, end-to-end approach to TensorFlow.js fundamentals for a broad technical audience that includes data scientists, engineers, web developers, students, and researchers.
You’ll begin by working through some basic examples in TensorFlow.js before diving deeper into neural network architectures, DataFrames, TensorFlow Hub, model conversion, transfer learning, and more. Once you finish this book, you’ll know how to build and deploy production-ready deep learning systems with TensorFlow.js.
* [Book's page](https://www.oreilly.com/library/view/learning-tensorflowjs/9781492090786/)
* [Amazon](https://www.amazon.com/gp/product/1492090794)
* [Book's GitHub repository](https://github.com/GantMan/learn-tfjs)
Questions and Answers
---------------------
**Ricky McMaster**
Hi Gant, thanks for doing this! At the beginning of your book you outline several benefits of using JavaScript for ML and I definitely understand that, especially the no-install aspect. However, for companies with a substantial ML implementation using Python, for example, are there compelling reasons to migrate regardless in your view, or would it only make sense to implement Tensorflow.js given specific business cases or regulatory reasons (you also mention privacy)?
**Gant**
In my opinion, if your AI is not your primary business then AI on edge makes a lot of sense. You’re trading freedom of your model for security as the model maintainer.
Example: Your AI does style transfer for makeup. Then it’s amazing to run this on the browser on the client and get more people to click purchase for their makeup or even hair dye packages.
However, if your model is proprietary and your work is to apply the makeup/hairdye it’s harder to protect your model, as it must be downloaded to the client.
**Gant**
I believe more and more people will lean towards scalability and even depend on cool things like federated learning, and to do that we’ll all need to have a JavaScript arm of our MLOps
**Ricky McMaster**
Got it, nice example Gant. So depending on the use case it can certainly make sense to have client- and server-based deployments in parallel, at least in the short- to medium-term.
**Gant**
absolutely. And you can easily convert TF to TFJS models. Most of them convert just fine.
**Ricky McMaster**
That would have been my follow-up question 🙂
**Ricky McMaster**
Thanks a lot for your response
**Mert Bozkır**
Hello Gant, I know you from Merve Noyan’s podcast 🙂
Welcome here.
A lot of people who aspring data scientist know python and libraries for this area. Machine Learning and Deep learning so important of course but I think python isn’t enough for ML Engineer or Data Scientist. This titles require Back-end of course. What do you think about back-end? there are 3 ways in back. You would deploy your model Web server, mobile application and of course Devops(docker,kubernetes, you can dockerize your model.)
My question is which one will populer in future ?
I want to add one more skill to myself but I don’t know which one I have to add. Actually this question is a little personal question because, You don’t know which one I will love or hate, but I wanted to ask.
**Gant**
If I understand correctly, you’re interested in my preferred backend solution for a web-hosted model.
I think microservices/cloud functions are perfect for models in the backend, because they are very “pay as you go” with infinite scalability.
So I would prefer setting up an AWS lambda that handles a result and reports back the results.
**Gant**
These are not limited by GPUs either. The whole problem of dynamically allocating resources for your server model is handled by the service, and faster and more scalable than either of us could hope.
**Gant**
The largest benefit, is that when the model is not in use, the costs are quite minimal. I’ve got several free AWS files/services I have released in the wild from past conference talks. They’ve never gone over $10 USD a month.
**Mert Bozkır**
Okey, I got it. Thankss :heart:
**Alexey Grigorev**
With TF.js, how does a training pipeline look like? Do you do training in python and then export the model to use in JS, or you do the entire thing in JS?
**Alexey Grigorev**
In Python, there are many nice libraries for data augmentation and different image manipulations. I’m wondering what happens when we move to the browser. Do we need to re-implement all this functionality in JS?
**Gant**
So if you’re needing the power of python libs, I generally use Python to train for my advanced networks and then convert it down to JS.
In the book, I show you object localization and detection. Both of those I decided to do in Python for ease and then convert over.
But you can definitely do it in JS, and I show how.
In the book the Tic Tac Toe model was trained in JS, and in chapters 8, 9, 10, 11, 12 all models are trained in JS.
**Gant**
I think Python wins in having a history of lots of libs. But the speed of training in Node is no joke!
**Alexey Grigorev**
What are some good image manipulation libraries in node?
**Gant**
Similar to imgaug is this: [https://github.com/piercus/image-augment](https://github.com/piercus/image-augment)
**Gant**
I think you’ll see more and more JS level tools coming in the next 2 years
**Alexey Grigorev**
Oh nice, thank you!
**Neal Lathia**
\> seasoned AI veterans and web developers alike can help propel the future of AI-driven websites
Which type of folks do you find using `Tensorflow.js` more - seasoned AI veterans, or web developers? (Am assuming that the intersection of these two camps may be somewhat smaller that the union 🙂 )
**Gant**
using more: web developers are playing with all kinds of amazing tech and will continue to grow. I think in quantity the current winner is web devs.
However, in quality, I think the best projects are coming from seasoned AI veterans who are leveraging AI in web. The most profitable and impressive projects are people who know how to make this new avenue of AI sing.
**Alexey Grigorev**
Let’s say I’m a python developer who wants to learn Tensorflow.js to use in my projects. How would you suggest to get started? Would your book will be enough or I already need to know some JS?
**Gant**
I expect a basic level of JavaScript skills. But all the code is available. If you’ve been programming for a while, you can probably learn a lot about JavaScript using the book. You might need to take a moment to go look up a function, but if you know Python you’ll probably be just fine.
**Alexey Grigorev**
Is there a good learn-JS-in-15-minutes tutorial for those who already know how to program?
**Wendy Mak**
I think if you just read/tinker with some js you can pick it up in 15mins (just remember those curly brackets ;)) (and also async stuff can trip you up if you’re not careful)
**Gant**
Glancing through this, here’s a 1hr jumpstart that seems decent
[https://www.youtube.com/watch?v=IEf1KAcK6A8](https://www.youtube.com/watch?v=IEf1KAcK6A8)
**Dmitry Yemelyanov**
Hello, Gant! I have tried porting some my TensorFlow2.0 models to TF.js and have been quite disappointed by the fact that lots of cool new features are poorly supported (for example, feature columns). Would you agree with my opinion that feature-wise TensorFlow.js is cursed to always lag behind TensorFlow? Or maybe one day it will take the lead?
**Gant**
Yes. TFJS will always lag behind. However, in a few ways it is ahead. TFJS 1.0 was using TF 2 syntax before TF 2 was released.
I think you’ll continue to see TFJS parity be a priority, but yes, lagging behind.
**Dmitry Yemelyanov**
Thanks for your answer. I hope this project will get more attention thus voluntary contribution from the open source community.
**Gant**
Would you crazy kids be interested in seeing some internal graphics and process stuff?
**Amr Alaa**
That will be awesome for sure
**Alexey Grigorev**
Yes!
**Gant**
I had to do my own graphics. I did OK
**Gant**
Some of my examples were just plain silly
**Gant**
The editors were AMAZING though, they loved everything I had
**Gant**
I’m quite proud of this diagram… but I know what they make will be better for print
**Gant**
Also, I used coworkers in examples. This will let me know if they actually read the book
**Gant**
I was worried if some of my graphics would work in B&W, but they gave them the thumbs up!!!!
**Amr Alaa**
Wow, looks interesting already
**Amr Alaa**
Hey Gant
Thanks for your time with us here, waiting forbthe interanl graphics..
Here’s my question
does TensorFlow.js have access to the file system in the browser host environment?
if it does not, that means available data resources are limited and can put restrictions on file sizes needed for training and verifying, right?
**Gant**
TensorFlow.js can run on client or server.
So if you’re running your training on a machine using Node.js you have direct access to the filesystem via Node.js. Then you’re just managing stream and RAM.
If you’re running on the browser, you are sandboxed. This means you cannot access the filesystem and you’ll need to load your data via compatible URIs. Browsers have DBs in local storage, RAM, cookies, and all kinds of fun places you can gather data.
Because of the sandbox, I would recommend only minor amounts of training to happen in the browser if you have a significant data source. The browser is much better at utilizing trained models rather than training models.
This does break down when you want customization for your client, however. So if you’re going to do transfer learning or federated learning, you can get real benefits from training in the browser, but yes, it’s constrained.
My suggestion, do base training in Node.js, then ship the model. Do transfer/federated learning in the browser as needed.
**Amr Alaa**
Super, looks clearer now
**Alexey Grigorev**
Which chapter did you enjoy the most? And which one was the most challenging for you?
**Gant**
That’s a tough one!!!
I guess the capstone project was pretty hard for me. Bc I had to come up with something that tied all the lessons from the book together into a cohesive unit. It was a fantastic end for the book though! But tough!
Second hardest chapter is chapter 1… I rewrote that damn thing 10xs
My favorite chapter was probably Chapter 10, which was simple image training. It felt fun and rewarding.
Chapter 6 was a secondmost favorite, because we hook a model up to a webcam!
**Alexey Grigorev**
Sounds like a lot of fun!
**Alexey Grigorev**
Also, as an author myself, I’m really impressed by how fast you finished your book. Can you tell us a bit about your process, and how you organized your time to work on the book?
**Gant**
I have to say the pandemic helped. I was able to get into a writing frenzy. My editor told me, “Ride that wave!” and so I did. The more I wrote the easier it became and I hit a traction point where I realized I was running out of chapters. At that point, I had to assess what makes it in the book and what didn’t.
After that strange hiccup, the finish line was in sight and I used the rest of my frenzy to finish the book.
Since I finished the book, I’ve done a whole lot of nothing! I think I’m recovering. I don’t even read cool projects right now, I binge watched a whole season of a Netflix show.
**Alexey Grigorev**
How did get into this flow mode? Or it just happened itself?
**Gant**
To be honest, I didn’t get into it at first. I had so much trouble writing chapter 1. However, like gradient decent, each time I took a step that felt right, I leaned into it. Next thing I knew I was running.
**Gant**
One thing that reallllly helped, was having people review chapters. That helped me see large leaps of logic I made, and when I went back to fill in the missing gaps, my mindset carried me to how they should see the next thing and the next. Sort of like reading the book from an outside perspective, I had a conversation with a hypothetical audience.
**Daniel Wexler**
Hi Gant can you compare the back-end features of tfjs-node against Python? What’s missing from tfjs-node that would encourage more use of JS on the back-end? Does tfjs-node have any advantages over Python on the back-end?
**Gant**
I feel like web frameworks that use Python are generally treated as second-class in the web world. Node for cloud etc is easy.
Node has even outperformed Python in quite a few benchmarks.
IMO, if you’re using backend, it needs to match the team maintaining it. Focus on dev preference.
**Gant**
The big advantage is that the code is portable to client, and IoT
**Daniel Wexler**
Similar to portability, you can provide the same computations in the client, on customer hardware, and at scale on the server.
**Daniel Wexler**
I’m just trying to understand why I don’t see more discussion of TFJS on the backend? I don’t have stats, but I see much more traffic about TF+Python than TFJS in Node. Like you said, I’d expect the web backend work to be more JS/Java, but I just don’t see much about that in the wild.
**Daniel Wexler**
Wendy Mak’s comment about async/await makes me think that Data Scientists might just be more comfortable with Python’s less asynchronous nature?
**Gant**
In my opinion it’s a momentum thing. Same reason JavaScript is everywhere.
Python was the spoken language for primary data culture, so it will take time for new solutions to root.
**Wendy Mak**
hi Gant do you find the async nature of javascript a bit annoying to work with when you are building models/training? Also, do you use tfjs node with the GPU? the browser can obviously make use of webgl but this is less clear for node? there’s [https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-nodegl](https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-nodegl)
but the doc says it’s under heavy development
**Daniel Wexler**
FYI, tfjs-node-gpu works just fine in node and uses GPUs when available using the main TF Cuda backend. Interesting question about async JS. Perhaps many folks feel more comfortable with the more traditional Python TF binding?
**Gant**
I’m find with async await. It’s common in JS, so it feels natural.
As for the second part, tfjs-node-gpu works great for accesing the GPU on node.
The browser uses WebGL, and they have an experiment with WebGPU which is promising.
Lastly, they are doing some amazing stuff in WASM with threads! I know this is a bit off-topic, but it’s cool.
CUDA now has JavaScript API, so I imagine tfjs-node-gpu will just keep getting better and better. It worked great for me.
**Wendy Mak**
\> I’m find with async await. It’s common in JS, so it feels natural.
ah, cool, although, I think I like synchronus better lol – the async things gave me no end of headaches when I was a js dev 😂 (well, I was also trying to munge together something that takes about 10 steps each of which has a callback…)
**Daniel Wexler**
Thanks for the GPU.js tip!
**Gant**
Oh, GPU.js is a different library. Brain.js uses GPU.js
WebGL and WebGPU are being supported as backends for TFJS
**Gant**
we need more acronyms ^ 😂
**Daniel Wexler**
I know! I hadn’t heard there was a (limited) JS Cuda binding until you mentioned it.
**Hironori Sakai**
Hi, Gant Thanks for participating book of the week.
I have a question about TF.js and federated learning. My question can be stupid, because I do not understand federated learning much. But if I understand correctly, then
* Federated learning proposes a model training on a distributed system without collecting the data in one place.
* 3rd-party cookie will be replaced with FLoC (federated learning of cohorts).
Because FL(oC) involves a distributed system, especially web clients/browsers for FLoC, it is natural to use TF.js to implement FL so that FLoC is one of the use cases. So my question is: Do you think TF.js can be mainstream of FL(oC)?
(Recently Google released a new version of chrome on which FLoC works, but I do not know its implementation.)
**Gant**
Federated learning is pretty complicated, and there’s the idea and implementation details.
The idea (at least in my head).
1. Create a model
2. Send copies to people
3. Use transfer learning to personalize and improve the model
4. Send back the improved models (NOT the user’s data)
5. Improve the core model that is distributed.
I’m unaware of the FL(oC) specifics, but I significantly believe Google has a lot of benefit in getting federated learning going. I think you can expect TFJS to be a core contributor to this.
I know Google is seeking lots of adjustments to W3C standards based on TFJS, but it’s hard to know which pieces are prioritized and why.
**Hironori Sakai**
Thanks for your valuable answer. When I googled FL with TF.js quickly, I could find only one TF.js project and it is not maintained. But I found it quite strange, because JavaScript is the default option if lots of different browsers are/must be involved. This was the motivation of my question. I am wishing that a good FL(oC) Framework on TF.js will appear soon.
**Gant**
I have no doubt that this will emerge. I think there’s a lot of green opportunities in TFJS for sure.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# AI and Machine Learning for Coders – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
AI and Machine Learning for Coders
----------------------------------
#### by [Laurence Moroney](https://datatalks.club/people/laurencemoroney.html)
##### The book of the week from 12 Apr 2021 to 16 Apr 2021
If you’re looking to make a career move from programmer to AI specialist, this is the ideal place to start. Based on Laurence Moroney’s extremely successful AI courses, this introductory book provides a hands-on, code-first approach to help you build confidence while you learn key topics.
You’ll understand how to implement the most common scenarios in machine learning, such as computer vision, natural language processing (NLP), and sequence modeling for web, mobile, cloud, and embedded runtimes. Most books on machine learning begin with a daunting amount of advanced math. This guide is built on practical lessons that let you work directly with the code.
* [Book's page](https://www.oreilly.com/library/view/ai-and-machine/9781492078180/)
Questions and Answers
---------------------
**Krzysztof Ograbek**
Laurence Moroney Is this a book for all coders, regardless of their years of experience or programming languages they use?
**Darya Petrashka**
Hi, Laurence Moroney thanks a lot for this wonderful opportunity! My question is: after studying properly the NLP section would it be possible to create a meaning-extract NLP algorithm that accepts a natural language phrase like “Show me sales by the company N” and ‘translates’ it to a query for the database?
**Matthew Emerick**
Hey, Laurence Moroney! Thanks for doing this.
Question: how much math do you go into in your book? How much math do you expect your audience to know before reading it?
**Matthew Emerick**
How much theory do you cover in your book?
**Matthew Emerick**
What resources do you recommend after reading your book to go deeper into AI and ML?
**Matthew Emerick**
Are there any supplemental books you recommend that go well with your writings?
**David Cox**
Laurence Moroney Looks like a fun read! In particular, I’d be curious to hear more about how/why you chose the deployment options you did later in the book?
**Laurence Moroney**
Krzysztof Ograbek – The first half of the book is about learning how to train models, and it will require you to know a bit of python. Not expert-level by any means, but it would be helpful. The second half of the book deals with deploying models to Android (Kotlin), iOS (Swift) and the browser (JavaScript), so will need you to understand a little of them.
**Laurence Moroney**
Darya Petrashka – The NLP section is primarily focussed on the underlying basics of NLP, leading to two outcomes – classifying text for sentiment, and generating new text, so not NLP->SQL type querying, sorry. I think there are a number of _products_ that can do that, but I teach generally the underlying principles in ML
**Laurence Moroney**
Matthew Emerick – Almost no math! The point of the book is so that the usual math/calculus barrier that prevents people from learning ML is removed.
How much theory? – Only enough to help you get to the point of being able to code. For example, I’ll teach (lightly) what a convolutional filter is, but from a code perspective (i.e. multiply these values by these) and not a math perspective (no greek letters!)
Resources to go deeper: Aurelien Geron’s book and/or Andrew Ng’s Coursera courses
Supplemental books: Anything that will help you with the language parts (basic Python, basic Android/Kotlin, basic iOS/Swift etc)
**Laurence Moroney**
David Cox – Deployment is what generally differentiates TF from other ML frameworks. It’s ecosystem allows you to deploy optimized ML to servers (via TF-Serving), iOS/Android (via TF-Lite), Browser/Node (via TF.js) and microcontrollers (via TF-Lite Micro). The goal was to give the developer a broad understanding and a grounding in what they need to know to get their models deployed to these runtimes.
**Laurence Moroney**
David Cox – I ended up not being able to put TF-Lite Micro into the book for microcontrollers, as that would have required many chapters to cover properly, and Pete Warden / Daniel Situnayake’s book covers that brilliantly
**David Cox**
Got it! This was very helpful. The know-how to deploy ML models is often a critical skill I see many data science students lacking when they join our team. Excited to see you talking about that in this book and to check out your implementation!
**Laura Uzcátegui**
Welcome Laurence Moroney, 👋
**Lamjed Debbich**
Laurence Moroney, thank you for presenting your book. I wonder what is the specificity of this book compared to the number of books currently on the market, and is it intended for people with previous data science practice or new learner? Thank you for clarification.
**Laurence Moroney**
Lamjed Debbich – Hi! The goal of this book was primarily to bring ML into the hands of the traditional software developer. As such half of it is an introduction to building common types of models for general, vision, nlp and sequence, and the other half is in getting those models into people’s hands with different deployment technologies. So, someone who previously has data science might find it useful.
**Laurence Moroney**
I think it’s really useful for new learners – as it assumes no prior knowledge, particularly of data science.
**Amr Alaa**
Hey Laurence Moroney
Thanks for your time with us here
First question is about the title itself
“coders”
What do you mean by coders exactly?
How is it differ than developers?
**Rohan**
Hello Laurence Moroney,
I had a question about fit\_generator method in Tensorflow. What does steps\_per\_epoch argument do?
I have tried to look about it online but couldn’t understand them.
And secondly, are we only supposed to ask questions about the book ?
**Laurence Moroney**
Amr Alaa – Haha , good question! I don’t think there’s a deeper meaning in it, other than I know many people who are dabbling in code, but don’t have a job title calling them a ‘developer’ yet, so I wanted to focus on the book being for people who code, as opposed to somebody with a particular job title 🙂
**Laurence Moroney**
Rohan - I’m not sure if that argument is fully used any more, I should check. But the idea behind it was to specify the number of steps to take in an epoch based on the data size and the batch size. So, for, example, if you have 60,000 training records, and a batch size of 1,000, then you’d have 60 steps in each epoch to get through the whole data. AFAIK it’s automatically counted now, (and fit\_generator has been deprecated in favor of just fit)
**Rohan**
Ah! Got it.
Thank you for the clarification. 😄
**Laurence Moroney**
Rohan – Feel free to AMA. I may not be able to answer _everything_, but I’ll try 🙂
**Rohan**
Thanks a lot! 😄
**Seed Badran**
Hi Laurence Moroney. Thank you for this great opportunity. I got certified as a _Tensorflow Developer_ few months ago and I wonder where would you put your book (knowledge dependency) in comparison to your “Tensorflow Developer Professional Certificate” specialization on Coursera? Which one would you recommend going through first?
**Laurence Moroney**
Seed Badran – They are complementary, and both the book and teh course were developed from the same material.
**Seed Badran**
Thanks. If the book is following the same great approach as in the Specialization (and expect it to be) I would definitely enjoy reading such book. Thanks for answering.
**Seed Badran**
I am always looking for resources where I can practice building an End-to-end Machine Learning project(s), does the book cover such scope?
**Caíque Coelho**
Laurence Moroney is an honor to have this conversation directly with you! My million-dollar question is: is this book a good way for those looking to get their first job as data science? What are the fundamental points to be learned in the book to help me search for my first job in data science? I am currently looking for a migration from the software testing area to data science
**Bayram Kapti**
Laurence Moroney since you mentioned the book focuses on Android, IOS and web deployments for ML algorithms, does this mean only Client Side ML Deployment is included in your book vs Server Side ML?
**Laurence Moroney**
Seed Badran – I guess it depends on the project, but in general, yes.
**Laurence Moroney**
Caíque Coelho - It really depends on the requirements for the job. If the job requires hands-on coding of models, as well as the usual data science stuff, then I think this book would be useful, but in and of itself, it’s probably not enough. A survey of employers looking for ML Programmers had many wanting people who knew Computer Vision and/or NLP, and this book digs into those.
**Laurence Moroney**
Bayram Kapti They’re not really the “focus” of the book. The first half of the book is on building models. The second half is on deployments, including Android, iOS and Browser. It also has a chapter on how to use TF-Serving for server side models accessible via REST or GPRC
**Bayram Kapti**
Got it! Thank you!
A follow up question on deployment, what would be your recommendation for a startup to try ML algorithms for the first time for some optimization purposes?
Client Side or Server Side? I think I’m more interested in the ease of deployment and maintainance for now.
**Laurence Moroney**
Bayram Kapti – I think client side for sure, just because of the install base, and because it’s easier to wrap an interface around a model in Android/iOS/JavaScript than dealing with the REST/GRPC interface.
**Bayram Kapti**
Amazing! thank you Laurence Moroney ! I think your book will be an amazing resource for me.
**Laurence Moroney**
🙂 Thanks!
**Gant**
Hey hey Laurence Moroney!!! You know I already have a copy of the book 🙂 However, I’m interested in what was your favorite dataset from the book, and why?
**Laurence Moroney**
Gant – Hey hey! We’ll have to have you do an AMA too! 🙂 I think my favorite dataset, being completely biased, is my horse-or-human one. It’s a _really_ tough one to build a good model from, but when it works it’s really cool. The images are all CGI, I synthesised them myself, and they greatly prove the concept of feature detection in CNNs because the computer can use features learned from a CGI image (like a horse’s ear or a human hand) to classify real, non-synthesised images!
**Kenny**
Laurence Moroney For a developer who wants to get into machine learning and AI, what change in mindset is required to do well? Is there a different fundamental approach to solving problems? Thanks!
**Eric Sims**
Laurence Moroney Your book looks very interesting! I’m particularly interested in learning how to convert Python models to JavaScript. I want to get back to JavaScript. I tried learning it a few years ago before I knew any other languages, and it was just too over my head. I think I could handle it much better now.
The other thing I’m keen to understand is federated learning. How would you recommend even starting with it? Could I use a couple of local devices to create a tiny “federation” and experiment at my kitchen table? Or am I totally misunderstanding what I have read on Wikipedia and elsewhere? 😅
**Laurence Moroney**
Kenny – I think so – It’s as much about good data selection to train a model, as good code selection for definining one. It’s definitely a new mindset, but IMHO, a good one!
**Laurence Moroney**
Eric Sims Thanks! I do have a couple of chapters on TensorFlow.js in my book, but all of Gant’s book is about JS! 🙂 Right now TF-Federated is a bit difficult to get started with as it is very experimental and doesn’t yet have a mobile runtime (you simulate the devices using Python)
**Rohan**
Hello again, Laurence Moroney
While working with Natural Language Processing problems with Tensorflow, I have been getting stuck at choosing the correct model architecture for the project.
I have used,
Bidirectional LSTM layer ,
Or A GRU layer,
Or a Convolution network,
But none of them have reached a good accuracy.
Am I missing out something?
In your perspective what model architecture would you go with ?
**Ricky McMaster**
Hi Laurence Moroney, thanks for doing this!
I’m definitely in favour of making this subject more explicable to a wider audience - however, I wonder if you could comment on potential risks. I work in data, and have often encountered legacy problems that originated from someone having insufficient understanding of creating/maintaining databases. Do you think there are similar dangers with AI/ML, and if so do you have any advice on safeguards/preventative measures, other than (obviously) a strong organisational culture?
**Amr Alaa**
Laurence Moroney I really liked your decision to make the book easy to be read by coders from different backgrounds, including me I think 🤔
Here’s my second question
In your book I see from the content table, i see that there are 4 chapters for TensorFlow.js and 3 for TensorFlow lite
Which one do you think will dominate the next phase of developing satisfying needs of rapid development and growth?
**Rohan**
Laurence Moroney I wonder if the book covers contents from your course on Coursera “ TensorFlow Advanced Techniques”. 🤔
And does it also covers topics such as TensorFlow serving?
**Ramit Surana**
Laurence Moroney Does the book explore any ideas on building large scale NLP models like GPT3 using Tensorflow? What’s your take on it? Is it possible for small scale companies/groups to build and serve models with billions of parameters by scraping the internet for data in future. Thanks.
**Chirag Aggarwal**
Laurence Moroney I wonder if book covers the deployment of DL model over a web based tool developed using tensorflow-python and tensorflow.js both. So that someone can better understand stand how to use them in production environment and when to use which one.
Thanks
Chirag
**A McCauley**
Hi Laurence Moroney - what are your favourite types of projects to work on with AI? E.g. such as working with images/ video etc . Which do you have the most fun with or feel most passionate about? 😊
**Laurence Moroney**
A McCauley – I can’t say I have a particular favorite data type with AI projects, to me I’m more excited about particular difficult projects to solve, like medical diagnosis (be it based on image or NLP), but recently I’ve been exploring more about how models come up with their decisions, with a view to turning the black box into a glass box, and stuff around images here is fascinating. For example, when doing a ‘Class Activation Map’ for the famous Cats v Dogs dataset, I found the main thing my model was making a decision on was the eyes. I had hundreds (maybe thousands) of feature extractor filters, but in the end, the eyes have it (sic!), and being able to show and prove that was awesome.
**Laurence Moroney**
Chirag Aggarwal - Yes. There’s a chapter on TF-Serving with a basic scenario of putting a model on a server and using TF-Serving to wrap it in a REST/gRPC interface. There are several chapters on JavaScript, making models, using models, and transfer learning with models.
**Laurence Moroney**
Ramit Surana – No, I don’t focus on large scale NLP models, more on the fundamentals of how NLP works, using smaller datasets for classification. I do a text generation algorithm though, trained on Traditional Irish songs, so you can write your own! 🙂
**Laurence Moroney**
Rohan – It predates ‘Advanced Techniqes’, so it doesn’t cover that. There is a chapter on TF-Serving though
**Laurence Moroney**
Amr Alaa The 3 on TFLite are – 1 on the tech, 1 on an Android scenario, 1 on an iOS scenario. For TFJS, I needed some extra time to discuss it, because we don’t just do inference in JS, we also do training, and there are some special considerations you need to take into account when training in the browser. I think both can be huge for the future, but only time will tell.
**Laurence Moroney**
Ricky McMaster – In some ways AI/ML doesn’t really change the ethics of responsibility with data and/or with how the work we create is used. I think the fact that AI can lead to more powerful solutions has brought the conversation to the forefront, and that’s a good thing, but issues with ethics, bias, responsibility etc have always been with us. Now’s the time to double down on them and take them more seriously.
**Ricky McMaster**
Good to know. That’s kind of reassuring, in a way - the need for oversight, actively maintained by responsible management, is enduring.
**Laurence Moroney**
Rohan – For NLP layers, it really depends. I did find, however, that for NLP solutions, good vocabulary management, good, clean, datasets etc had much more of an impact on my final results than a neural architecture search for the best layer types and hyperparameters. That’s one of the things I discuss in the book. 🙂
**Rohan**
Laurence Moroney I think that building a model and working with data is an art. But data is present in different format and working with them can be very daunting sometimes. Transforming the given data into an insightful information is not everyones cup of tea.
What would you suggest if a person has difficulties working on a dataset? 😅
(The person cannot leave this field. 😂)
**Ricky McMaster**
Hi again Laurence Moroney, on the subject of text generation… the stuff I’ve read from GPT-3 is fun, and impressive in its own way, but do you foresee a point where it could be used in a genuinely creative context? At least one use might be in literary works with elements of pastiche, but are there other contexts you can think of, or are aware of?
**Laurence Moroney**
Rohan – I think as tooling evolves, where one can see how tuning input data affects output performance, that this part of the job will get a bit easier. Often there’s a lot of burden now where you’ll have a thesis that a particular format, slice, function, on data can impact a model positively, but then you still have to go through the process of building the model, adjusting the architecture to how the data is formatted, training, testing etc, before you can get any results to prove/disprove your thesis. As that process gets faster and easier, and then becomes automated and scriptable (think AutoML for data management too), then I think that part of the process will get much easier to do, and you can focus more on solutions and less on massaging data.
**Laurence Moroney**
Ricky McMaster – I’ll say ‘never say never’, but when you do text generation, or indeed any type of generation, it rapidly descends into gibberish, when you realize that it’s a prediction on a prediction on a prediction on a prediction etc. That’s a hard problem to solve for it to be really useful. Large scale models like the one you mention attempt to solve that by being trained on more data, but I think there’s a ceiling to what’s possible with that approach. That being said, I’m sure somebody will someday have a breakthrough that makes text (or any other type) generation more realistic.
**Ricky McMaster**
Thanks Laurence Moroney, that’s interesting to hear. Of course it’s debatable whether it’s worth the effort… in music production (so for example, mastering algorithms) I am definitely aware of creative or practical uses for AI, but with large-scale text generation, I guess the entire thrust of a narrative or argument could be upset by a single word.
**Laurence Moroney**
Music is a great point…and there are often repeating rhythms in music so that a long piece of information is repeated sections of smaller pieces. Text is harder because that’s not usually the case.
**Ricky McMaster**
That’s exactly it! Especially with electronic music, but even with more complex forms there are always repetitive elements, and whilst the creative uses might be more limited you can still train a model to take care of mastering.
Whereas text (or more particularly literature) is an endless series of forking paths…
**Rohan**
Laurence Moroney often while training a model, we encounter a case of stalling. The network stops learning at all. We do use early stopping technique to stop the training but, what do you think the person should keep in mind while building the model?
What are your observations while tweaking those hyper- parameters 🤓
**Laurence Moroney**
Rohan – Stalls can be a result of too small a learning rate, so maybe it’s good to adjust that. Or, it could be because of sparse or too little data. Have you tried Keras Tuner to do a hyperparamter search?
**Krzysztof Ograbek**
Laurence Moroney you used Python to train models but other languages to deploy them. Why don’t you use Python to deploy? Is it because of performance? Are there more reasons?
**Krzysztof Ograbek**
Laurence Moroney I’m just curious about your predictions, what’s the next 3-5 years will look like. Is there any language that may soon take over training models from Python? Is Python going anywhere or its position is strong enough?
**Laurence Moroney**
Krzysztof Ograbek – Because of _where_ they are deployed. For example. the primary languages to build Android apps are Java and Kotlin. Or for iOS it’s Objective-C or Swift. Thus models, created with TF in Python need to be executed elsewhere, and we need to be able to interface with them.
**Laurence Moroney**
Krzysztof Ograbek – I think Python will be the primary one, but keep an eye on languages like Go or Kotlin.
**Krzysztof Ograbek**
Thank you so much for answering my questions, Laurence Moroney. And thank you for doing this AMA. I never used any of those 2 languages
**Krzysztof Ograbek**
Laurence Moroney I have a few more questions that are out of the scope of your book. It would be awesome if you share your thoughts 😉
How do you explain to people that are completely non technical, what AI is?
Can Software Developer’s job be automated by AI? Are Developers gonna be replaced with AI? What are the skills that Developers should improve in order to future proof their careers?
**Rohan**
Laurence Moroney hadn’t heard about Keras Tuner. Will try it out! The problem which I often face is of Vanishing Gradient. I tried SGD with momentum to tackle this. But that too didn’t have much of an effect.
The second thing which I wanted to know was, how do you decide the dimensions of a picture while feeding it into the network, given that the dataset has pictures of different dimensions which are very high!
**Laurence Moroney**
Krzysztof Ograbek – I generally like to explain that AI is the result of using programming techniques that have computers act on data the way the people would. For example, Computer Vision, a person would look at a picture of a cat and say it’s a cat. When a computer can do the same, you have the beginnings of AI. Are developers going to be replaced by AI – absolutely not, in the same way as developers weren’t replaced by compilers, IDEs, intellisense, emulators etc. 🙂 Skills that they should improve on: ML, Data Science, Python etc.
**Krzysztof Ograbek**
Just a quick follow up question here, Laurence Moroney. By improving ML here, do you mean learn also math behind it or simply learn how to train models?
**Laurence Moroney**
Ideally both, but if you’re a developer, it’s best to start with the code.
**Laurence Moroney**
Rohan – Definitely give it a try. KT might be able to help find a model that doesn’t have VG. For deciding dimensions of a picture, there’s a lot of trial and error of training time against accuracy…or, I’d do something like a Class Activation Model to determine what the features are that determine classificaiton of the image. For example, with the classic Cats v Dogs model, a good classifier, when passed through a CAM ended up showing me that the eyes were the single most important feature to distinguish…and if that was the case, maybe the eyes could have sufficient fidelity in tiny images so as not to make a difference, so why train on larger ones? 🙂
**Laurence Moroney**
By the way, small factoid for everybody reading – one of the best things you can do for any book, to support it, is to give a review on amazon. A star rating is good, but a review is best. I’ve had books where sales were boosted even after a poor review! There’s clearly something in their ranking/recommendation engine that values written reviews. So, while I’m not asking for you to review mine (although I would love it), I would ask you to remember this to support authors. Margins on books are razor thin, and many writers simply give up because it’s not financially worth it – I’ve observed an almost inverse Moore’s law, where royalties halve every 18 months. I didn’t write this book for profit (all proceeds go to charity per Google employment agreement), but there are authors out there who need the income, so please remember this for any books you buy and/or read! Thanks 🙂
**Krzysztof Ograbek**
Laurence Moroney When doing an ML Project, how do you know, which algorithm to use? What are the factors? Is Deep Learning always a better choice than the traditional ML algorithms?
**Laurence Moroney**
Krzysztof Ograbek – Not always…it really depends on the problem. Things like CNNs for images (mostly), LSTM/RNN/GRU for text etc. Generally, I would need to see what the problem I’m trying to solve is, and explore solutions for similar ones to use as a starting point.
**Saurav Maheshkar**
Laurence Moroney it’s amazing to have the opportunity to talk to you. I have taken all of your courses with [deeplearning.ai](http://deeplearning.ai/)
and I love the way you taught and broke down the API into small chunks.
My question is: What according to you is the ideal “_Full-Stack_” pipeline for building deep learning models ? How do we incorporate methodologies like _CI/CD_, _Testing_ and _Deployment_ with the Tensorflow Ecosystem. Does Post-Training Optimization methods such as _Pruning_ and _Quantisation_ come under “_Full-Stack_” and what according to you is the future of the `tfmot` package ?
**Rohan**
Laurence Moroney I have asked several technical questions. But I was curious about journey in this field of AI. What excites you the most?
You are an inspiration to students like us. Any suggestions you can provide to people starting their career in the vast universe of AI.
Thank you. 😄
**Krzysztof Ograbek**
Laurence Moroney thanks again for this amazing opportunity. I didn’t know you don’t get any profit for your book. This leaves me speechless
**Laurence Moroney**
Rohan – My inspiration. I guess there’s a number of places. First – I graduated college in 1991, in the UK, in the middle of the biggest economic recession since WW2, and matched only by the current one, I think. I was unemployed, and it was really hard to find work. I spent some time as a clerk for London Underground (and got laid off on Christmas Eve, very Dickensian), and doing odd-jobs here and there. I remember working in a recycling plant, being the guy that collects the tin cans that people come in off the street to hand us, expecting payment, weighing them, and seeing people’s disappointment at the few pennies they got. Then, the UK government started a scheme where they wanted to find 20 people, who are currently unemployed, to train them in AI to be a cohort of future consultants in industry. It was 1992 at this point, and I had been out of work for about a year, and really struggling. I went to the testing center, along with thousands of others, and was the first person selected to enter the program, with the highest scores. It was a massive boost to my flagging confidence! Unfortunately the program failed miserably after about 3 months, but the contract I had signed had entitled me to 2 years of education, so it got parlayed into a Masters degree, fully paid, which I graduated from in 1993, as the economy was emerging, and my career got on track. Then…Fast forward to 2017, I’m now at Google, and we had an initiative to train all engineers in ML and AI. I remember sitting in a conference room in Kirkland (near Seattle), with a few hundred other folks, excited as anything….and….after 3 hours of calculus, and most of the room bored out of their minds, I realized something was wrong with how we teach ML as an industry. I approached the TF team showing how a code-first approach would work better, and how the material is aimed at the 300K current AI practitioners globally (measured by academic publications) instead of the 30M+ software developers.
So they hired me, and asked me to fix it.
And here I am :)
**Rohan**
Wow!
It’s amazing. A true example of never giving up no matter what! 😃
**Seed Badran**
I can relate to this. It helps to know (for real) how others overcome their life challenges… Thanks for sharing 🙏
**Laurence Moroney**
Saurav Maheshkar – Thanks! 🙂 As for ‘Full Stack’ – I don’t think it’s well defined yet, but I will take a stab at definining it similarly to how we define full stack web developers – and that is someone who is involved, at least in part, in every part of the overall stack to solve a problem. A full stack web developer, for example, is involved in coding the part of the solution that lives on the server, but they’re not necessarily an expert in infosec. They’ll interface with the people who are. They’ll also be involved in coding the front end, and the UX, but they’re not necessarily a designer or a UXE. So, similarly, when it comes to ML, if you explore the life cycle from data->feature engineering->model training->deployment->management, I think a full stack ML Engineer will be involved in all of them to some extent, but the closer you get to the center of that chart, the more they do.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# The Practitioner's Guide to Graph Data – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
The Practitioner's Guide to Graph Data
--------------------------------------
#### by [Denise Gosnell](https://datatalks.club/people/denisegosnell.html)
##### The book of the week from 05 Apr 2021 to 09 Apr 2021
Graph data closes the gap between the way humans and computers view the world. While computers rely on static rows and columns of data, people navigate and reason about life through relationships. This practical guide demonstrates how graph data brings these two approaches together. By working with concepts from graph theory, database schema, distributed systems, and data analysis, you’ll arrive at a unique intersection known as graph thinking.
* [Book's page](https://www.oreilly.com/library/view/the-practitioners-guide/9781492044062/)
* [Book's GitHub repository](https://github.com/datastax/graph-book)
Questions and Answers
---------------------
**Alexey Grigorev**
Hello, everyone!
The book of this week is [The Practitioner’s Guide to Graph Data](https://datatalks.club/books/20210405-the-practitioners-guide-to-graph-data.html)
by Denise Gosnell
\> Graph data closes the gap between the way humans and computers view the world. While computers rely on static rows and columns of data, people navigate and reason about life through relationships. This practical guide demonstrates how graph data brings these two approaches together. By working with concepts from graph theory, database schema, distributed systems, and data analysis, you’ll arrive at a unique intersection known as graph thinking.
* Ask as many questions as you’d like (one question - one thread, please)
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
**Alexey Grigorev**
Hi Denise Gosnell!
When we connected on LinkedIn, you mentioned that you can tell us how graph databases can be distributed.
So. How graph databases can be distributed? what are possible approaches?
**Denise Gosnell**
Alexey Grigorev great question.
My answer is based on knowledge of the three types of graph data structures:
1. Adjacency matrices
2. Edge lists
3. Adjacency List
Irrespective of distributed or single instance, each data structure has its pros / cons.
Adjacency matrices are too large to write on disk, but they are the fastest for looking up graph data.
Edge lists are the most compact to distribute, but they require scanning the entire edge set to look up relationships.
Adjacency Lists are the middle ground. They allow for constant time lookup of vertices and then reduce the edge scans to those local to the vertices.
Therefore, when working with distributed graph data, I recommend finding a way to implement adjacency lists to distribute your graph across your cluster.
**Denise Gosnell**
Cassandra is the distributed system with which I am most familiar (from the early Titan DB days in 2013).
In the book we are reading this week, we are working with adjacency lists across Cassandra partitions.
Chapter 5 goes into detail into how we do it.
LucidChart Images 5-7 through 5-12 illustrate how we do this.
**Denise Gosnell**
Vertices and edges map to partitions in cassandra.
The key to understanding how to distribute a graph comes from understanding how we model the partition and clustering keys under the hood.
(to use graphs in Cassandra, you don’t have to know the details)
The vertex tables work just like regular C\* tables - nothing special there. You have partition keys and clustering columns like normal.
The fun parts come into to play with the edge tables!
The partition key of an edge table is:
* the outgoing vertex’s entire primary key
The clustering key of an edge table is:
* the incoming vertex’s entire primary key
* (maybe a property)
The images below are from Chapter 5 where we discuss this in detail.
**Alexey Grigorev**
What’s a C\* table? something Cassandra related?
**Alexey Grigorev**
So, let’s say we have an adjacency list. From my understanding, a graph in this representation looks something like that
0 => \[1, 2, 5, 10\]
2 => \[3, 4, 5\]
3 => \[0, 3, 10\]
…
(these numbers are ids of nodes)
If now we want to distribute it to multiple partitions, would we use something like consistency hashing to map each node to a partition? and keep the entire list of outcoming nodes within that node partition? (+some extra properties of the node)
**Denise Gosnell**
Alexey Grigorev I apologize for using shortcuts that are apart of our vernacular at DataStax – switching between Slack universes can be confusing and I forgot where I was 😜
C\* == Cassandra.
When I said, “C\* table”, I was referring to a virtual table in Cassandra. A table contains many partitions.
**Alexey Grigorev**
Hehe, okay =) got it, thank you!
**Denise Gosnell**
And - the answer to your second question Alexey Grigorev is yes - you got it!
However, Cassandra’s use of consistency hashing may be a bit different than expected.
Each table has a partition key that is hashed using murmur3 algorithm. The whole hash space forms a continuous ring from lowest possible hash to the highest. After that this ring is divided into chunks (vnodes, 256 by default) and these chunks are fairly distributed among multiple nodes. Each node hosts its own part of the ring and a replicated copy of other vnodes (according to replication factor.)
A visual example from Chapter 5 in the book is shown for how we distribute graph data using Cassandra’s hashing algorithms. The images below are images 5-11 and 5-2, respectively.
I provided a link to the LucidCharts in the channel if you would like to explore them (and some that were cut from the book!)
In Cassandra, we use
**Toxicafunk**
Like Janusgraph with a Cassandra backend for instance?
**Alexey Grigorev**
Not sure if it’s a clarifying question for me or not. But I meant something more general - what are possible approaches and how it’s implemented in modern graph databases
**Toxicafunk**
I know that Janusgraph, for instance, can have a Cassandra or an Hbase backend, so you get the same distributed approach they have
**Toxicafunk**
Generally speaking, a graph database is a nosql database with a network (path-based) querying model
**Toxicafunk**
But this is a very good question, I’d love to know what else is out there 😄
**Denise Gosnell**
Toxicafunk yes!
I worked with Matthias Broecheler on this book.
Matthias invented Titan. Janusgraph is a fork of Titan.
The book we wrote details how we mapped Adjacency Lists onto Cassandra. This is the same way they do it in Titan (Janusgraph) 🙂
I shared details on how we mapped adjacency lists onto C\* partitions in the thread above. I included images from Chapter 5 to show how we translate C\* schema as an adjacency list.
Please let me know if you have any further questions!
**Toxicafunk**
wow! this is exactly what I’ve been thinking about for the past m,onth, thx!
**Denise Gosnell**
Toxicafunk awesome!
**Toxicafunk**
Or do you mean something else?
**Vladimir Finkelshtein**
When you mention graph data, do you mainly refer to graph databases or to ML algorithms taking graph structure as an input (I am thinking of clustering as the simplest example)
**Denise Gosnell**
Vladimir Finkelshtein we are mainly referring to graph databases for this week’s discussion.
In chapters 10, 11, and 12, we go into detail into how Netflix uses graph structured data to do recommendations for the ML algorithms behind the “Recommendation pane” on the app. I hope this helps!
Please let me know if you have any more questions. 🙂
**Matthew Emerick**
Hello, Denise Gosnell! Thank you for doing this!
How useful do you see graph data and graph algorithms in the field of artificial intelligence?
**Denise Gosnell**
Matthew Emerick You are welcome! Glad to be here 🙂
I am biased (ha!).
I am seeing graph data used more and more as a new type of feature within ML algorithms.
WRT to graph data - properties like “path distance” between two points or a boolean of “is x in the 3rd neighborhood of y” are being used as features in ML systems that do things like:
* Calculate the likelihood of fraud in an insurance claim
* Quantify identity matches between two profiles in social networks
* Provide “realtime” content personalization when viewing a webpage
Researchers and engineers are converging to using features like the 3 I shared above after they spent time running graph algorithms across their data.
I hope that helps! I’m going to move to your next ones :)
**Matthew Emerick**
After working through your book, what is you recommended next best step in working with graph data?
**Denise Gosnell**
Great question!
As I see it, the way to apply this to a new problem is to:
1. Model your domain data as a graph
2. Check if the known patterns are relevant to your business problem. (Known patterns are neighborhoods \[chs 3,4,5\], hierarchies \[chs 6,7\], paths \[chs 8,9\], and recommendations \[chs 10, 11, 12\])
3. Use graph algorithms to find new correlations in your data.
**Matthew Emerick**
Before picking up the book, what prerequisites do you recommend people have to get the most out of what you teach?
**Denise Gosnell**
Matthew Emerick that depends on which part of the industry you are coming from 🙂
The preface as a longer version of this.
The tl;dr - I recommend that you
* Have worked with databases before
* Understand or have experience in doing data ETL
* Have an open mind to trying new things 🙂
**Neal Lathia**
❔ What are the most impactful use cases for graph data?
**Denise Gosnell**
Neal Lathia I am going to pick up with this question on Tuesday morning! Thank you!
**Denise Gosnell**
Neal Lathia Thank you for the question.
I have a bias behind my answer. My bias is that “impactful use cases” are those which are deployed into production settings. And, the majority of my experience from production deployments is in the data management space and less in production data science.
From that perspective, the most popular ways that companies around the world are using graph data in production is:
1. Customer 360 style apps
Chapters 3, 4, and 5 go into detail on how a bank (and insurance companies, governments, etc) I’ve worked with use “neighborhoods” to create a “360” view of important entities of the business.
Specifically, the most popular question that I have seen graph data used to answer is: “what are all the pieces of information we know about this entity (usually a person)?”
Those queries typically inform one of 2 systems in my experience:
* Customer Service interfaces (showing all recent interactions across different channels like social, web, and/or in-store)
* Analyst Investigation interfaces (like fraud or marketing analysts)
There are other popular use cases, but customer 360 is the most common deployment. The other 3 super popular ways graph data is use in a production system are:
2. Recommendations (eg: netflix or shopping cart recommendations)
3. Hierarchies (eg: employee trees)
4. Path Finding (eg: how are these 2 pieces of information related?)
I hope this helps. Please let me know if you have any additional questions. Thank you!
**Alexey Grigorev**
What’s customer 360? This?
[https://globalz.com/customer-360-single-customer-view/](https://globalz.com/customer-360-single-customer-view/)
**Alexey Grigorev**
I guess wikidata could be something similar, right?
**Denise Gosnell**
Alexey Grigorev you got it!
**Nick McClure**
I actually own this book! I’m on chptr 3. In chptrs 1 & 2, the book talks about the importance of recognizing when a problem needs graph-DBs/graph-data. I’m worried about being over zealous about using new technologies sometimes.
Can anyone give an example of an instance or case where they considered graph data / DBs and decided not to?
**Denise Gosnell**
Nick McClure great! Thanks!
I am going to pick up with this question on Tuesday morning
**Denise Gosnell**
Nick McClure great question!
And – I would love for everyone in here to jump on and add ways they have used graph databases in production.
(my response in this thread)
**Denise Gosnell**
One example I personally built was the United States Health Graph
* [Co-occurance graph of digital services](https://github.com/denisekgosnell/denisekgosnell.github.io/blob/master/data-day-2017/health/providers_100k_v1_small.jpeg)
* [Referral Network of US Physicians](https://github.com/denisekgosnell/denisekgosnell.github.io/blob/master/data-day-2017/health/top_grossing_referral_physicians_small.jpeg)
There are many others, but these two are my favorite because they each generated business outcomes that saved millions of dollars.
First - the co-occurance graph of digital services was constructed and analyzed in 2015. This graph had the largest connected cluster around “online therapy services”. In 2015, this demonstrated the upcoming trend of traction for using online digital interactions for therapeutical services, like talk therapy. It is very heartwarming to see how many people in 2021 are benefiting from the US Healthcare industry’s investment in digital care for mental health professional services.
The second graph, the referral network, unveiled a billion dollar fraud ring in Florida! This fraud ring is observable in the visualization linked above. We noticed 2 aspects about a specific doctor in florida.
#1) the doctor received and outlier amount of reimbursements from medicare (data available from the CMS)
#2) the patient network was distributed atypically across the entire united states
[Here is a screen shot of the article from the Miami Herald](https://github.com/denisekgosnell/denisekgosnell.github.io/blob/master/data-day-2017/health/1BFraudCase.png)
I would love to hear from others on how they have used graph dbs!
**Nick McClure**
Thank you for the graph examples! I am excited to use the technology- but were these examples were you ended up _not_ using graph technology? Specifically I am asking about over applying it where it was not needed.
**Jessie Yaros**
I’ve read that if you’re not so interested in the relationships among items, or if you’re data doesn’t lend itself to being modeled as a network of connections, then you may not want to try for graphs over rdbms. Total noob here though- that’s just what I’ve heard.
**Denise Gosnell**
Nick McClure thank you for the clarification!
Here is a list of projects that we didn’t use a graph db for:
1. Business metrics dashboard (eg: viewing week over week trends of company metrics)
2. Searching for products by name, or any other filter like color, size, etc
3. Inventory management of a product catalogue
The common theme in the examples above is that the data fits into one table.
**Denise Gosnell**
Hello everyone! I am jumping in here for a bit to answer questions. 🙂
To start - we open sourced two parts of our book.
1. [All our images are here in LucidCharts](https://lucid.app/folder/invitations/accept/0b959629-50f6-4b1e-96ce-98df831ea9c0)
2. [Our code examples in notebooks are here](https://github.com/datastax/graph-book)
Thank you so much Alexey Grigorev for inviting me into this week’s session. See you in the threads!
**Alexey Grigorev**
Thank you for agreeing!
**Denise Gosnell**
Oh! If you would like to get chapters 3 through 5, [you can download them for free from DataStax](https://www.datastax.com/resources/ebook/oreilly-graph-guide)
**Vladimir Finkelshtein**
I am wondering if recent theoretical solution of graph isomorphism problem had any impact on the practical side. I imagine it is relevant for pattern searches in the graph, but then I guess that for most common patterns, there were already decent algorithms…
**Denise Gosnell**
Vladimir Finkelshtein that is a great question.
can you post a link to the recent solution you are referring to?
in production systems, I haven’t seen pattern matching extend beyond triangle predictions. There are many creative ways to apply graph isomorphisms, but none that I have seen be deployed in a company’s data architecture.
**Vladimir Finkelshtein**
[https://en.wikipedia.org/wiki/Graph\_isomorphism\_problem](https://en.wikipedia.org/wiki/Graph_isomorphism_problem)
I am not sure if it is relevant for graph databases. I just remember when it was announced, quasipolynomial bound was a very big deal for people in CS, but I still don’t know what are the applications of it 🙂
**Vladimir Finkelshtein**
I was imagining that if you query the database for certain relationships, it means that you need to find subgraphs isomorphic to the graph.
**Toxicafunk**
Hi Denise Gosnell, thank you for doing this!
What are your thoughts on Apache Tinkerpop and particularly on the Gremlin graph traversal language to be used on mutliple backends as constrasted against somtehing like Neo4j’s Cypher which is, presumably, optimized for a single backend? (I understand Cypher is used on other backends besides neo4j but I am assuming it works best on neo4j)
**Denise Gosnell**
Toxicafunk great question, thank you for asking it!
apache tinkerpop is the most widely adopted graph traversal language as it is used by most graph DBs. There are even popular gremlin to cypher compilers.
Yes, cypher works best on Neo4J.
Irrespective of the optimization of the query language, the graph industry is in need of an _easier_ to use language than both Cypher or TinkerPop. I love gremlin, but there is a super steep learning curve (as evident in a 400+ page book on how to use it! 😆 )
I am currently working with Apollo to see how we can potentially look at the GraphQL language as an easier way to express graph traversals and graph shaped problems. However, under the hood, the functional programming approach to gremlin for server side optimizations is brilliant and has yet to be matched with another tool. Marko, Daniel, and Stephen (and others!!) created a beautiful and brilliant language for solving graph problems. We just need an easier interface for using it, IMO.
I am most excited about the innovation in the space of graph query languages. I am excited to see what others invent!
**Vladimir Finkelshtein**
I am sure we will hear about many use-cases of graph databases. When should we not use them? Are there simple examples of data, where relationships are better stored and queried in a relational database?
**Denise Gosnell**
Vladimir Finkelshtein thank you for asking this one.
tl;dr: if you only care about properties of your data (like, what is the age distribution of my customers?) than a graph database is overkill.
Use a graph only when the _relationships_ between your primary entities are central to your question.
For a longer explanation – we go into detail on this in chapter 1. After working with 100s of teams around the world, I created the image below as a way to think around this question.
This is image 1-5 from Chapter 1. The link to the live LucidChart is in this channel.
**Denise Gosnell**
Please let me know if you have any further questions, thanks!
**Jessie Yaros**
Omygosh, I just joined this community this week, and I am so amazed that this is the first book of the week I’m exposed to! Thanks to Denise Gosnell and Alexey Grigorev. I actually added your book to my amazon wishlist a few weeks ago, so this is a wonderful coincidence. I’ve been using graph theory and algorithms in my phd research, but am really hoping to transition and be able to employ ‘graph thinking’ and tools in the real world. I’m curious if you have advice/ideas on the types of industries or even job titles that are most likely to value background in working with graphs. I’m finding that its rare to find job postings that require, or even mention wanting candidates who have a graph theoretical background or experience with graph algorithms. With how excited people seem to be about graphs, do you envision these types of tools and skillsets will become ever more popular/ in demand? Or am I totally looking in the wrong places, potentially using the wrong keywords?!
**Denise Gosnell**
Jessie Yaros – i _love_ these questions - thank you!
I will pick up with yours first thing on Wednesday, if not sooner.
**Denise Gosnell**
Jessie Yaros great question!
There are two places to look:
1. Jobs for now
2. Jobs for the future
Jobs to look for now will be: data scientists, data strategists, data analyst, data architects, etc
In the future, the concept of a “data mesh” is likely to become a prominent aspect of every company’s data architecture. A “data mesh” graphs the movement of data around a company from one place to another. This is higher up than instances of graph data, but the entire flow of data is a graph. I recommend following Martin Fowler’s work on this topic in the coming 5 years to see how it evolves.
I also recommend the following industries for interesting graph data: logistics, entertainment, and healthcare.
If you are looking for a job, happy to help connect you to people looking for graph folks. Just lmk!
**Jessie Yaros**
Denise Gosnell Awesome - just followed his blog - thanks for the rec - haven’t heard of data mesh yet!
And uhhhhhhhhhm yeah I’d love some connections/introductions! I’m based in LA so I’ve been thinking entertainment may be a good option but in my heart of hearts, I think healthcare might be a better fit with my neurosci background and health interests. I have a few ideas for clinical graph projects that I’d really love to tackle, once i wrap up writing my dissertation haha. My advisor says I have to stop creating more projects for myself! I’ll connect with you on LI, and would love to hear more of your thoughts!
**Alexey Grigorev**
Denise Gosnell I know almost nothing about data mesh, but it seems it’s more an organizational structure than some technology. So I’m wondering where do graph databases come into play when it comes to datamesh?
**Denise Gosnell**
Alexey Grigorev great question! I think of a data mesh at a higher level up from the actual data base.
In a data mesh, you still have “edges” and “vertices”.
The edges represent streaming or batch jobs that move data through one flow to another. The vertices are data stores. These data stores are anything from:
\> custom views (dashboards), to
\> databases (like relational, or graph databases, or s3 buckets), to
\> tools (like marketo, salesforce, workday, etc).
The data architecture of an entire company is essentially a graph - and therefore folks are starting to recognize that we are really building a massive data mesh.
**Alexey Grigorev**
okay, that makes sense now, thank you for clarification!
**Denise Gosnell**
You’re very welcome Alexey Grigorev 🙂
**Jessie Yaros**
Do you have strong opinions on the best graph databases. graph query languages to start working with? I’ve dabbled with neo4j (and cypher), but am curious whether other options like TinkerPop, might have larger appeal in industry. I’m noticing that more platforms seem to use gremlin than cypher, so I’ve been wondering which would be better to invest in learning.
Edit: I see now you are with Datastax but still would love your thoughts 🙂 . Another question this brings up, is whether there are major differences in working with a db marketed as NoSQL vs Graph. Are NoSQL just more versatile? Could that cause steeper learning curves?
**Jessie Yaros**
2nd Edit- I just read your other thread that goes into gremlin/neo4j !
**Denise Gosnell**
Jessie Yaros I recommend knowing gremlin, GraphQL, and SQL. Those 3 languages will teach you everything you need to know to query databases and use graph technology.
Cypher is only used by Neo4J and was designed off of SQL.
**Jessie Yaros**
Awwww well that makes sense, why Cypher seems intuitive to me –since I know SQL. So at least that’s one down! Thanks for the tips!
**Jessie Yaros**
Do you think that GQL will change the game once its released? Would you expect most (property) graph databases to try and implement its usage?
**Denise Gosnell**
Jessie Yaros I am biased with my response here. But! I have been following GQL for 5+ years and haven’t seen any results. As with any new tech, expect the language to take at least 3-5 years from being released to being relevant with adopted use.
**Jessie Yaros**
Denise Gosnell Do you believe that there will always be different/ideal use cases for property graphs and RDFs/knowledge graphs? Or do you think that one type might become more dominant? LPG’s seem so much more intuitive to me, but knowledge graphs have become such a buzz word!
**Denise Gosnell**
Jessie Yaros personally, I see each of the technologies as being useful for different problems. So I expect both will stick around
LPGs are much more popular today than RDFs.
“knowledge graphs” are a catch all term with lack of specificity in the two domains.
**Jessie Yaros**
Denise Gosnell Yeahhh I do see some people using the term knowledge graph interchangeably with RDF, which was confusing to me since I assumed you could build a KG with both! And I’m glad to hear that LPGs are more popular. When I started working with graphs for brain connectivity, I wasn’t aware yet that they were of the LPG type. So when I went to a few talks on knowledge graphs, expecting the LPG format, i was like, waiiiit. What are these triplets. 😅
**Denise Gosnell**
Jessie Yaros I can relate!!
**Jessie Yaros**
🤣
**Vladimir Finkelshtein**
What are some public graph datasets you can recommend for practice with graph databases?
**Denise Gosnell**
Vladimir Finkelshtein the GitHub with my book comes with datasets :relaxed:
I get my graph data from:
1. Snap: [http://snap.stanford.edu/](http://snap.stanford.edu/)
2. Kaggle: [https://www.kaggle.com/datasets](https://www.kaggle.com/datasets)
3. Or, this GitHub
[https://github.com/awesomedata/awesome-public-datasets](https://github.com/awesomedata/awesome-public-datasets)
**Vladimir Finkelshtein**
Thanks for the links. The datasets in kaggle (and most public datasets I have seen) usually come as single tables, so I am not sure how many interesting relationships between entities I can explore in the majority of them. You mentioned that you used graph database to build referral network of physicians. I am pretty sure, this is not public, but I was wondering if there is anything of this type where people can explore how it works.
**Jessie Yaros**
At least for me its been an adjustment reinterpreting data in terms of how we can represent it in graphs, since we are so used to thinking about data in a relational table like manner. But even data in one table can be thought of in terms of relations, (rather than thinking of relations as being only joins between tables). For instance, you might have a simple table with just one column for customers and one column for favorite movies. You could then re-represent that information in graph format by having nodes for each customer, and nodes for each movie, where you link customers to the movies they like. Then you could do fun stuff like looking for customers that have similar movie preferences, and could make educated movie recommendations based on that!
**Denise Gosnell**
Vladimir Finkelshtein and Jessie Yaros if you want “graph ready, no etl” datasets, head straight to SNAP from Stanford
The US Referral network data is hosted by the CMS:
[https://www.cms.gov/Regulations-and-Guidance/Legislation/FOIA/Referral-Data-FAQs](https://www.cms.gov/Regulations-and-Guidance/Legislation/FOIA/Referral-Data-FAQs)
**Vladimir Finkelshtein**
Thanks a lot
**Vladimir Finkelshtein**
Jessie Yaros indeed, you can represent movie ratings datasets as a graph, but I never thought about movie recommendations as a graph problem (I realize now it’s 3 chapters in the book of the week 😃). For example, matrix factorization techniques for recommendation engines - they do indeed factorize the adjacency matrix of a graph you described, but the latent features that they discover don’t clearly have much to do with the graph, as far as I know. These latent features are more naturally seen as vectors. Some other techniques, like cosine similarity between users can be expressed in the graph via paths, but it will be more cumbersome then just using vectors.
I was thinking of older approaches in e-commerce like finding association rules as graph based, but my impression was that they are not widely used any more, because they are too slow.
But I guess, part of the purpose of this book is to change this mindset. I am wondering if any interesting things can be said about movies by inspecting properties of neighborhoods in this graph. Anyway, just some thoughts…
**Jessie Yaros**
Cool to know! I’ve heard of matrix factorization but don’t more more than the name. I know some people are doing graph embeddings to vectorize graphs pre ML pipelines… wondering if that is similar?
Im not sure how often graph rec engines are using metrics of overall similarity vs just traversing paths based on shared movie interests to make those recommendations…
**Denise Gosnell**
Vladimir Finkelshtein I like using graphs to do recommendations because the algorithm essentially comes down to a group count! so much easier to think about (at least in my head) than matrix factorizations and cosine similarities 😄
**Denise Gosnell**
Jessie Yaros the graph rec engines are just traversing paths and counting - and, you can use weights on the edges if you wanna get fancy.
TBH, I like counting instead graph paths instead of matrix and vector spaces that are hard to visualize. :woman-shrugging:
**Jessie Yaros**
Yeah! It’s nice how intuitive it is.
**Alexey Grigorev**
A slightly unrelated question:
What do you do as a chief data officer? And, in your opinion, is it different from other companies with a similar role?
**Bayram Kapti**
I am interested in this question as well!
**Denise Gosnell**
Great question Alexey Grigorev and Bayram Kapti
As a CDO, the top 2-3 outcomes are:
1. Ensure continual delivery of business critical data, metrics, and feeds throughout the enterprise to support decision making. (this looks like establishing CI/CD around data sources, tools, and custom built software)
2. Minimize cost and risk within the tech stack and supporting resources. (here you run into compliance, budget, knowledge management, etc)
3. Maximize efficiency and access to tools and information (eg: process and operations, data sprawl wrangling, etc)
**Denise Gosnell**
This creates more of a data mesh architecture at the business level than the granular use of graph data.
so, graph knowledge not required - but it does make it easier to wrap your head around an entire company’s data processing.
**Bayram Kapti**
Thanks for the insights Denise! Much appreciated!
**Denise Gosnell**
Bayram Kapti anytime! happy to help 😊
**Alexey Grigorev**
What kind of skills does a CDO need? Apart from good knowledge of graphs 😃
**Denise Gosnell**
Alexey Grigorev A CDO needs to be able to:
* Manage Budget
* Optimize investments
* Minimize risk / expenditures
Graph knowledge not required 😞
**Denise Gosnell**
Being a CDO essentially feels like the master builder of a company wide marble run set. :)
**Alexey Grigorev**
That’s a nice marble run set! Thank you for your answer
**Denise Gosnell**
Alexey Grigorev of course, you are welcome!
**Alexey Grigorev**
Denise Gosnell you mentioned data mesh in one of the threads. Recently it became quite popular. Do you have some ideas why?
**Denise Gosnell**
Alexey Grigorev yes!
I hypothesize that data mesh architectures became very popular because:
* Data is sprawled out all over a company and siloed by business function
* Data is heavy to move
* Data is more valuable when connected
Those facts all-together are driving more companies to create streaming highways from one silo of data to another instead of lifting and shifting all data into one big new system.
**Alexey Grigorev**
So I guess the main downside of the central lake/platform is that you need to move the data across the entire org, and that’s often difficult and expensive.
If I understood you (and what I heard about data mesh previously), the main idea is now each team/department can have their own lake.
These lakes have to be connected somehow - in such a way that an analyst knows how to pull these datasets together to do their analysis.
And graphs help to connect the dots - to connect different “lakes” to each other and have this big picture.
Am I close?
**Denise Gosnell**
Alexey Grigorev you got it!
There is a data mesh slack channel if you wanna hang out there and see how folks are discussing the idea: [https://join.slack.com/t/data-mesh-learning/shared\_invite/zt-nrh42jd1-B~YAplAKzHl3hyP039UQSw](https://join.slack.com/t/data-mesh-learning/shared_invite/zt-nrh42jd1-B~YAplAKzHl3hyP039UQSw)
**Alexey Grigorev**
Thanks! I’m actually there already, but the amount of information there is a bit overwhelming 😃 So thanks for making it clear to me, and kudos to Scott Hirleman (he/him) for organizing the community!
**Denise Gosnell**
I didn’t know Scott Hirleman (he/him) was in this slack universe! he is _everywhere_ 😁
**Vladimir Finkelshtein**
NLP related question (sorry if completely off topic): I have heard some time ago of an approach to embed the vocabulary of a language in a graph: words as vertices and (weighted) edges representing semantic (or any other) similarity between two words. So it would be logical to use such graphs for NLP problems. Do you know of any such applications?
**Denise Gosnell**
hey hey Vladimir Finkelshtein - i love the conversation, keep it coming!
I am not well versed in this style of graph for NLP.
How would you calculate similarity between two words? What is the cut-off for storing an edge to represent similarity? And, how would you query or use this structure once you created it?
Thanks!
**Vladimir Finkelshtein**
I am not well versed in NLP either. There are many notions of similarity, and many complicated ways to compute them. NLP libraries like NLTK have them. One can also train word embedding and calculate distances between words there. I don’t know if this is actually used, but a naive way of thinking (just to get the idea) is to take a big text corpus, and define similarity between two words as the number of times they appear together in the same paragraph divided by total number of paragraphs each of the words appears in. This way, bus will be more similar to driver than to elephant. Of course, this is too naive. Wikipedia entry on semantic similarity has many references to different ways to compute it.
Setting thresholds which edges to include is a matter of fine-tuning.
As for uses, I don’t have a good idea, that’s why I asked. One example off the top of my head: say you have a news article, and you look at all the words that appear in the news article, and look at the subgraph in your database that these words induce (I hope it’s clear what I mean here). My intuition tells me that if one finds a big clique in this subgraph, these would be the words that represent the main topic of the article. The less connected words are the ones that are not good for specifying the topic. So something like this could be used for topic classification or keyword extraction.
Sorry if the explanation is too messy, it’s just an idea. It just seems very natural to use this structure in NLP context.
**Vladimir Finkelshtein**
Basically what I describe above, is a version of kmeans clustering, but done not in Euclidean space but on a graph. Cliques are the dense clusters.
**Denise Gosnell**
Vladimir Finkelshtein I know NLP well as I implemented semi-discrete matrix decomposition for my PhD.
As we are outlining here, the crux of the problem is how modeling word similarity in a graph would be a better (faster? more accurate? more expressive?) way to determine topics within a corpus.
I agree with how you outline the problem.
And, that clique detection would be the algorithm to use once the data is in the graph.
However, every word has a measurement of similarity to any other word. Thus, the whole graph is _technically_ a clique. So, the fine-tuning of edge weights for inclusion in the graph, or algorithm if we take the approach to store everything and filter on processing.
You are outlining a really interesting problem. And, it would be very valuable to understand if the graph based approaches are measurably better than other matrix based representations. Could be a good topic for a PhD! 😅
**Vladimir Finkelshtein**
If I had to bet, it will be slower for sure, but it has a chance to be more expressive. All the word embeddings that we have now are in Euclidean space, and it is very restrictive in what kind of distances it allows between points in space. For example, there is no good way to draw a world map in a plane without totally distorting a lot of the distances. Graphs give you complete freedom in this sense.
As for concern of everything being a clique, I agree that choosing a good threshold might be hard and depend on very good notion of similarity. One could also try to replace finding a clique by finding dense regions (in terms of sum of edge weights).
**Jessie Yaros**
Another one for you Denise Gosnell ! How do people model temporal or timeseries data in graphs? Most of the graph stuff I’m familiar with is more static in nature, but I know in reality many networks are more dynamic… Like, for instance are edges assigned time point values, or could whole graphs be made for individual time points and then be stitched together? Or something fancier??
**Denise Gosnell**
Jessie Yaros I love the questions, keep ‘em coming!
tl;dr: time in graphs is usually represented as a property on an edge. And, an edge is stored every time there is data to capture.
We go though this in detail in chapters 2 and 7. Throughout ch 6 and 7, we walk through what we built for one of the US energy companies who care about the time of communication between two sensors in the energy grid.
The image below is 7-1, showing how we store multiple edges between two sensors in Seattle’s energy network: one edge per transmission between two sensors. The property on the edge is the time when the communication occured.
**Jessie Yaros**
Ooooh so coooool, thank you!!
**Rich**
Hi Denise Gosnell, Thanks so much for hosting this channel on Graph Data. I just recently discovered your book and I’m reading through it now. I’ve been exploring this topic from the Semantic Web aspect where products such as AnzoGraph (multi-model) use the standards based RDF and SPARQL along with the use of Ontologies and Knowledge Graphs. How do these technologies fit into or extend the utilization of Graph Data as presented in your book?
While reading your book I also ran across a reference to the Unified Modeling Language (UML, fig 2.1). I’ve been a Software Engineer for many years and built many object models using UML so it extends well beyond relational data models. At the time we went onto persist our object model into an Object Database such as Objectivity (and their follow-on graph database InfiniteGraph). In many ways I view the uprising of Graph Data as a new spin on an object model and Object Databases (adding the graph related algorithms such as Centrality, etc.). What is your take on this view?
Lastly, I’ve been a user of Python NetworkX which has an abundance of different graph algorithms. To your knowledge, can their implementation sit on top of an existing graph database for persistence? As a Data Scientist I’ve been looking at ways to incorporate ML into this Graph Data world and I’m still getting my arms around it. Do you have an recommendations beyond your excellent book?
Thanks for your time……Rich
**Denise Gosnell**
Hey Rich - thank you for the questions!
\> RDF / SPARQL – How do these technologies fit into or extend the utilization of Graph Data as presented in your book?
The world of RDF & semantic graphs is a set of adjacent technologies to property graphs. [I recommend this excellent discussion on the topic.](https://www.youtube.com/watch?v=OTPTq5_ifvY)
TBH, I wouldn’t do the topic of RDFs justice and refer to the work directly written by Jans Aasman or Juan Sequeda on the topic 🙂
\> In many ways I view the uprising of Graph Data as a new spin on an object model and Object Databases. What is your take on this view?
At the end of chapter 2, we detail our view 😊 We present the “GSL - graph schema language” that is very close to UML. The GSL is a way to visually communicate the details of your graph schema, just like UML or ERDs outline relational database.
The tricky parts of the GSL (or communicating about graph schema in general) are representing the multiplicity of edges in your graph; something my co-author and I spent a long time working through for our book (longer than we care to admit 😆 ).
I would love to know you thoughts / criticisms / improvements to the GSL. It sounds like we both see the importance in this type of innovation for the industry.
\> NetworkX - To your knowledge, can their implementation sit on top of an existing graph database for persistence? Do you have an recommendations beyond your excellent book?
YES! I love python and also use NetworkX! 😄
However, in production data architectures, the common pattern is to connect to a graph database via spark and use spark’s library of graph algorithms. Spark supports [graphFrames and graphX](https://spark.apache.org/graphx/)
and is much more widely adopted within data science teams. Therefore, I have more experience working with those libraries.
I recommend becoming familiar with [Spark’s GraphX libs](https://spark.apache.org/docs/latest/graphx-programming-guide.html)
as they are supported by multiple graph databases.
**Rich**
Hi Denise Gosnell, Thanks for the great references. I did watch the video you suggested and it did clear up the difference between Property Graph and RDF implementations. If you don’t mind here is what I learned:
1. Property Graph appear to be more project based given they don’t use a Taxonomy or Ontology that would be defined for the Enterprise. The RDF approach is geared more toward a coherent method for the entire Enterprise as opposed to perhaps a siloed approach.
2. I also thought I hear that RDF allowed for an embedded Object definition much like an Object-Oriented language would support.
3. RDF were more complicated to use and implement but I think this was based on the broader scope of what these usually entail (building a Taxonomy and an Ontology).
Jans also said that Gartner was thinking that Graph databases would become the predominate platform within ~ 10 years. What are you thoughts on that prediction?
On that same theme not sure if you’ve heard of the “Data-Centric” architecture ([http://www.datacentricmanifesto.org/](http://www.datacentricmanifesto.org/)
) but this is what Dave McComb is pushing a move to Ontologies and Graph databases.
One last question. Could you provide any additional resources on “Entity Resolution”? I’m reading though this in your book now.
Thanks again for your thoughtful comments and insights.
Rich
**Denise Gosnell**
Rich thank you for the follow up!
wrt to Gartner’s predictions - my passion and life’s work centers on graph data and databases. So, I hope they are right! I am seeing graphs used more and more as the next secret weapon for advancing feature development and signal investigations at large companies. Thus, I agree with this prediction. However, I am a source of bias here as this topic is what people come to ask me about.
I had not heard of data-centric architectures with this specific label. After reading the manifesto - this sounds very near and close to the same momentum behind the data mesh community. Looks like a lot of overlap! 🙂
For entity resolution - what kind of resources are you looking for - algorithms, applications, latest papers, etc? My PhD is on this topic – specifically the application of it to trace identity throughout social networks, or social fingerprinting – so I would love to share more.
**Rich**
Thanks for the quick reply Denise Gosnell. The more I look at the Graph data and the related semantic model the value proposition makes sense to me. The ‘schema on read’ seems like a very important feature. I do wonder about the scalability.
I came across the Data-Centric movement when doing some work for the DoD last year. While I think the idea is a very good one, I believe many large organizations will struggle to migrate away from their stove piped relational world. Dave has published a book on the topic here ([https://www.semanticarts.com/publications/](https://www.semanticarts.com/publications/)
) which also seems to fit into an architecture I’ve seen called Data Fabric (looks like an earlier cousin of Data Mesh). The “schema on read” featured of Graph databases controlled via an Ontology fits well within his focus on pairing down the many Enterprise schemas but having an extendable core model.
On the Entity Resolution problem, I’m looking at a new opportunity that involves this task. I’ve used NER before to identify entities within a body of text but this area of study is new to me. Any resources you think would be helpful would be much appreciated. Of course, you have a great example I’m working through in your book.
Many thanks again…..Rich
**Denise Gosnell**
Rich do you know when the “data centric” manifesto started? I would love to learn more.
And yes - looks like “data fabric” is the earlier cousin of data mesh ✅
wrt to NER and Entity Resolution – what data sources are you using in your new task? If you are looking at a large corpus of text, the latest to consider might be [how to do so with Spark.](https://towardsdatascience.com/named-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77)
I have had the luxury to work with more structured data like telecommunication records (or I choose to boil the problem down to entities). Thus, like I outline in chapter 11, my experience in Entity Resolution is closer to matching and ranking algorithms. I find [Dr. Amy Langville’s book](https://g.co/kgs/W3CLXR)
to be very helpful in understanding the math and science of ranking, which comes into play when you are deciding if your algorithms are correct (or not).
**Rich**
Hi Denise Gosnell, My apologizes for the belated response. I looked back through Dave McComb’s blog post and see the oldest entry is in 2015 ([https://tdan.com/the-data-centric-revolution/18780](https://tdan.com/the-data-centric-revolution/18780)
). Based on my conservations with him he started down this Data-Centric path after seeing some of these large ERP systems being built (SAP, etc.).
If you have a moment I wanted to get your thoughts on one additional thing. I was reflecting back on some of my graph related work and wondered if “Similarity-Based Networks” could be a good framework for Entity Resolution?
Thanks again for the book and great dialogue here. It has been very beneficial. Rich
**Denise Gosnell**
I am loving all of the questions this week! Thank you all so much for inviting me in here and making me feel so welcome :heart:
You are welcome to ask-me-anything! I am on the east coast of the USA and will be around until the end of the day on Friday for Q&A.
**Vladimir Finkelshtein**
You defined social fingerprint in your research. Does it mean that graph data should be treated like dataset of fingerprints from the point of view of privacy? Is there a way to anonymize graph data?
**Denise Gosnell**
Vladimir Finkelshtein yes! The interactions we generate on social media are uniquely identifying for any individual in the world. Your words of “graph data is a dataset of fingerprints” is a really great way to describe it.
The way to anonymize graph data is very similar to how advancements in blockchain are progressing - by uniquely changing the ID of your vertex for every network interaction.
**Toxicafunk**
that’s what monero does I think
**Denise Gosnell**
Toxicafunk yes! monero is it; I couldn’t remember the name. You are correct, thank you!
**Jessie Yaros**
We have neural fingerprints in fmri research!
**Toxicafunk**
How does that work Jessie Yaros?
**Denise Gosnell**
coo cool Jessie! I would love to hear more
**Jessie Yaros**
Sure! There is not a specific method for neural fingerprinting, but seems to be a catchall term used in studies that use brain connectivity data to identify individual participants. For fun I’ll link two papers that do it differently. One uses similarity between connectivity/association matrices/graphs, and another uses a ML classifier I think. I may give it a try with graph embeddings and some sort of clustering technique with my own dataset, we see we see
[https://www.nature.com/articles/nn.4135](https://www.nature.com/articles/nn.4135)
[https://www.sciencedirect.com/science/article/pii/S2589004219305474](https://www.sciencedirect.com/science/article/pii/S2589004219305474)
**Dr Abdulrahman**
Thank you Denise Gosnell for writing such a needed book especially from practitioner’s perspective.
Few questions I have:
1) Do you think graph data and graph algorithm will dominate and becomes more popular? Are not they complex and have a high learning curve?
**Denise Gosnell**
Dr Abdulrahman great questions, thank you!
I am seeing that utilizing the connectedness of data is a valuable trend that will become a regular part of the ML and data science workflow for feature detection.
However, you are correct. There is still a steep learning curve wrt to using graph technologies like algorithms, query languages, and databases.
**Dr Abdulrahman**
2) What is the hidden technical dept that organizations might face adopting graph data/algorithms? Technical issues, skill shortage, difficult to scale, hungry for computation power..etc
**Denise Gosnell**
Dr Abdulrahman The largest hurdles to overcome, today, are:
\> 1. data ETL into a graph shape
\> 2. graph query languages
I have found that each #1 and #2 have a root in skills shortages, which is why I wrote this book. I poured my heart and brain onto the pages to hopefully help out the community to feel better prepared to address them. :heart:
I am not finding that “computation power” is an issue because getting bigger machines seems to be somewhat easy compared to the other aspects of the problem.
The VC circles are currently investing in both sides of that adoption issue. Thus, I hope to see significant improvement in those areas in the next 5 years.
**Dr Abdulrahman**
3) If I understand correctly, forr graph algorithms to work, I must have all the graph at once to see the whole relation? It can not be used on batched mode. Am I Right? Otherwise, relations might not appear.
**Shankar Somayajula**
One can load/build up the graph incrementally in a batched fashion.
Rather than think of the graph in terms of complete/incremental/batch sense, one can analyze the graph in its full/complete form or in a custom/scaled down/localized form as applicable to the analysis.
You can issue some commands on the graph and reduce it as you please (reduce in terms of edges and/or vertices) … The resulting graph becomes a sub-graph. Graph Algorithms can be issued on the sub-graph. Think of this as a custom/localized application of the Graph Algorithm.
Global and Local versions of the same algorithm can co-exist and lead to novel usages of graph.
E.g: Find the go-between in terms of complete Graph and see who the intermediaries are at a global level using “Betweenness” algorithm/resultant KPI ranking. Now subset the graph to a region and/or only keep certain types of transactions and then apply the same algo … see new intermediaries ranked in terms of their “Betweenness” role in the custom scenario. Some of these local intermediaries may not have a big enough footprint to be visible/noticeable on the global scale but still be important from a fraud or compliance perspective when viewed locally.
HTH
**Denise Gosnell**
exactly what Shankar Somayajula said, thanks!
I have nothing to add :partying\_face:
**Dr Abdulrahman**
Thank you Shankar Somayajula. and Denise Gosnell
**Vladimir Finkelshtein**
As was mentioned somewhere, you can work with adjacency matrix instead of a graph, they have the same information. And many graph operations like counting paths and examining neighborhoods are then done by matrix multiplication. And there is no problem to do parallel/batched computation with big matrices, by splitting it into blocks. This is the formal version of what Shankar Somayajula said, I think.
**Alexey Grigorev**
Good morning/day everyone!
As Denise mentioned yesterday, she’ll still be taking questions today, so we’ll announce the winners of her book tomorrow.
Have a great day!
**Mansi Parikh**
thanks, Alexey, for asking all of those questions today as they surely helped beginners like me!
**Alexey Grigorev**
I think we still have some time for more questions.
So, why do you think graph databases are not getting a wide-spread adoption? There are many use cases that they can solve, but people tend to go with traditional relational DBs.
(Or maybe I’m wrong and they are getting a lot of adoption, but I just don’t see it?)
**Denise Gosnell**
Alexey Grigorev great question!
I think the adoption of graphs is blocked by the mental model which is ingrained in our heads:
\> data maps to a table.
We think and perceive the world in a network. I am friends with Casey. Casey’s family lives in Chicago. Chicago is my mom’s hometown… etc.
We perceive our world in relationships and our thoughts flow through them, similarily.
But, the earliest adoption of relational technologies enforced a mental shape of data which we are having a hard time navigating away from today: data goes in a table.
OH! Bonus fact – the CODASYL group in the 1960s was the first formal group to organize database technology. They started with hierarchies first (or graphs) - but we didn’t have the computational power to make them easy. So, they died and went for relational instead (enter Codd’s work in the 1970s!)
**Jessie Yaros**
WOAH that is sooo interesting. And a sure testament to how –at least at first– there was an attempt at organizing data more like we mentally do it. And how it can be hard to now rewire both at the individual and institutional level!
**Jessie Yaros**
So its kind of like how neural nets came back when we actally had the processing power to support them?! Bodes well for graphs….
**Alexey Grigorev**
What are the most useful/practical graph algorithm that data scientists should know about?
**Denise Gosnell**
Alexey Grigorev honestly? bi-directional search for shortest path algorithms.
That one isn’t solved well (yet) in OSS graph algorithm libraries, and is known to be the fastest way we can do path finding.
Then, #2 is connected componenets.
I make these two recommendations, bi-directional search for pathfinding and connected components, because when they are used together that is the fastest way you can answer the question:
\> “How is a connected to b”?
Which is the main thing people want to use graphs to answer.
The pseudo code works like this:
Q: Is `a` connected to `b`?
Pseudo Code:
\> 1. Run Connected Components on the graph: this assigns a label to a vertex. All vertices that are connected together by any edge get the same label. Two vertices have different labels if there is no possible path between them.
\>
\> 2. Is the label of `a` the same as `b`? If no – return “not possible”. If yes, run bidirectional search
\>
\> 3. Bi-directional search: starting at `a` run one iteration of breadth-first-search. Store the results in a set: `a_neighbors`. Then, go to `b` and run one iteration of breadth-first-search. Store the results in a set: `b_neighbors` . Run the intersection of `a_neighbors` and `b_neighbors` . If the intersection is empty, go back to `a` and continue down BFS, and compare to the next iteration of BFS to `b` . Continue until you find a common vertex in the intersection of `a_neighbors` and `b_neighbors`
\>
**Denise Gosnell**
I would love to see this implemented in any OSS library!
I hope that description helps!
**Alexey Grigorev**
maybe the last one - what are the most popular graph databases and which are the most typical use cases for each?
**Denise Gosnell**
Alexey Grigorev since I work for a graph database vendor, folks might not trust my opinion. 🙂
There are many things to consider when looking at graph databases - I would recommend starting with [this excellent article in TDS](https://towardsdatascience.com/comparing-graph-databases-5475bdb2e65f)
on the topic :)
**Denise Gosnell**
Thank you all very much for letting me join this world and talk about graphs this week. The pleasure is all mine as I thoroughly love and enjoy this topic. Feel free to ping me in here anytime, or drop me an email: [denisekgosnell@gmail.com|denisekgosnell@gmail.com](mailto:denisekgosnell@gmail.com|denisekgosnell@gmail.com)
Let’s stay connected! 😜
**Shankar Somayajula**
Denise Gosnell You’ve been very sporting and generous with your knowledge in answering all the questions. Cheers, Great Advocacy :-)
**Alexey Grigorev**
Thank you for answering our questions - and also for one extra day of your time!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Transformers for Natural Language Processing – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Transformers for Natural Language Processing
--------------------------------------------
#### by [Denis Rothman](https://datatalks.club/people/denisrothman.html)
##### The book of the week from 19 Apr 2021 to 23 Apr 2021
The transformer architecture has proved to be revolutionary in outperforming the classical RNN and CNN models in use today. With an apply-as-you-learn approach, Transformers for Natural Language Processing investigates in vast detail the deep learning for machine translations, speech-to-text, text-to-speech, language modeling, question answering, and many more NLP domains with transformers.
The book takes you through NLP with Python and examines various eminent models and datasets within the transformer architecture created by pioneers such as Google, Facebook, Microsoft, OpenAI, and Hugging Face.
* [Book's page](https://www.packtpub.com/product/transformers-for-natural-language-processing/9781800565791)
* [Amazon](https://www.amazon.com/Transformers-Natural-Language-Processing-architectures-ebook/dp/B08S977X8K)
Questions and Answers
---------------------
**Krzysztof Ograbek**
Denis Rothman thank you for doing this!! Maybe I’ll start with this question: Do transformers outperform RNNs in any kind of NLP tasks? What makes transformers so great?
**Rodney Silva**
Denis Rothman Hi! How should a beginner implement Transformers? Using Hugging Face,Google Trax,MS Azure or implement from scratch to control maintenance?
**Mert Bozkır**
Denis Rothman What do you think about future of Natural Language Processing? What will the location of the Transformers in future ?
**Lalit Pagaria**
Denis Rothman Isn’t Transformers constraints by compute resources? What is way forward for startups with less resources?
**Vladimir Finkelshtein**
When do you think we will see transformers for non-NLP tasks? There seem to be papers and attempts with transformers for vision, but no breakthrough, or at least no mainstream adoption of it.
**Denis Rothman**
I’m live so you can ask any question you wish.
**Denis Rothman**
Krzysztof Ograbek There are to reasons transformers outperform RNNs:
**Denis Rothman**
Krzysztof Ograbek I mean there are “two” 🙂 main reasons: 1.optimal transport= RNNs carry all of the information from word(or token) to word and pile the information up in a big backback. 2/ The architecture of RNNs is obsolete with variable sized layers. Transformers have fixed sized industrialized layers. I explain this in chapter 1 of my book but also in this video:
**Denis Rothman**
Krzysztof Ograbek This is the link to the video [https://youtu.be/tQTpCvZ1-0w](https://youtu.be/tQTpCvZ1-0w)
**Denis Rothman**
Krzysztof Ograbek Proof? Transformers have wiped RNNs off the top ranks of the SuperGlue Leaderboard:
**Denis Rothman**
Krzysztof Ograbek Here is the link to SuperGlue:[https://super.gluebenchmark.com/leaderboard/](https://super.gluebenchmark.com/leaderboard/)
**Denis Rothman**
Rodney Silva A beginner should begin by understanding the original Transfomer model explained in Chapter 1 of my book. Even with a piece of paper and a pencil! Or a spreadsheet!
**Denis Rothman**
Rodney Silva Implementing in a real-life project is something else entirely. If it’s a low level non-strategic project and isn’t really necessary and the project has only a small budget because of that, you can implement almost anything you find since the interest of the solution will be limited! Now if you are implementing Transformers in a critical project, that’s something else…
**Denis Rothman**
Rodney Silva The first question is what is the goal of the project. To really understand it. Then to go hunting for the right data with standard software approaches. This can take anywhere from 1 month to 6 months, a year, and maybe two years! In the meantime, you need to find a reason for your customer or employer to keep you on the project so it stays alive! If you understand the project, you can begin with a useful user interface that enables the users to visualize, understand their data and run short but productive tasks. …
**Denis Rothman**
Rodney Silva If you can get your project in cruise mode as described above then you have plenty of time to build a transformer prototype of the project with the model you find best suited for the project in terms of SLA (Service Level Agreement) with the customer. SLA is standard IT practice that is well documented and contractual.
**Denis Rothman**
Mert Bozkır Your question : What do you think about future of Natural Language Processing? What will the location of the Transformers in future ? My answer: I see transformer-driven NLP (until a new model comes up) on Cloud platforms such as AWS, Googe Cloud or IBM Cloud or another giant such as Microsoft Azure. Why? They have reliable scalable servers! I doubt anybody can match without the power to back up the sales. The smaller solutions will survive but not be critical unless they are partners of the tech giants. NLP will spread out to every area of human activity.
**Denis Rothman**
Vladimir Finkelshtein Your question: When do you think we will see transformers for non-NLP tasks? including vision. My answers: First of all,l let’s take vision out of the picture because there are more legends than actual reality here. Computer Vision (CV) relies heavily on massive non-AI algorithms. Then the CNNs do a pretty good job. Transformers are kicking in for image captioning. Ok that’s for vision because vision is ok right now. There are a lot of solid AI and non-AI tools. Now in non NLP, you have transformer driven-Recommenders! They can be applied to e-commerce behavior prediction and also can be used in manufacturing. See my video here:[https://www.youtube.com/watch?v=tQTpCvZ1-0w&list=PL9uLp9IOO56Gv0YEnRWdrIZOPKwRBHnJe&index=10&t=1s](https://www.youtube.com/watch?v=tQTpCvZ1-0w&list=PL9uLp9IOO56Gv0YEnRWdrIZOPKwRBHnJe&index=10&t=1s)
**Vladimir Finkelshtein**
Denis Rothman Thanks, awesome examples. If I understand correctly, the tasks are just naturally reframed as NLP tasks (e.g. in recommendation systems: sentences = list of products that one likes, recommendation = fill the mask).
As for the optimal transport solution, it wasn’t quite clear how the rewards are used. Do you just pick a threshold, for which you keep a sequence of actions if its reward is above the threshold, and throw away bad sequences? Also, I didn’t quite understand how initial and final distributions are encoded, but I guess there are more details in the book…
**Vladimir Finkelshtein**
As for vision, for example this paper ([https://arxiv.org/abs/2010.11929](https://arxiv.org/abs/2010.11929)
) claims that transformers achieve almost SOTA or SOTA performance with significantly reduced training resources. So perhaps scalability can play a role in adopting transformers there.
**Denis Rothman**
Vladimir Finkelshtein Now, let’s take your question a bit further for non-NLP transformers. I see powerful solutions being kept secret by tech giants. For example, in terms of recommenders, China has access to far more data in terms of volumes and information than anybody in the world. They are advancing toward a digital society and are many years ahead of the West. For example, they have a national digital currency that is expanding with information (like VISA with its tens of thousands of transactions per minute but China is more like 300 000 transactions per minute). You can easily perceive the patterns you can draw from that with sequences of customer behavior.
**Denis Rothman**
Lalit Pagaria Your question:Isn’t Transformers constraints by compute resources? What is way forward for startups with less resources? My answer: Let’s go beyond the hype and into the real topic! If you use a nice simple robust model BUT have excellent datasets you will even beat GPT-3 which used a supercomputer!!! Proof? The Pattern-Extracting Training (PET) method I describe in the introduction of Chapter 6 of my book. I describe an average sized transformer model that trained on an optimized tiny dataset and…beat GPT-3 on the SuperGlue Leaderboard!!! More proof? Look at line 13 of the SuperGLue Leaderboard today and you will see PET (Tom Schick) with a nice and good accessible computer and then look down at line 16. What do you see today? GPT-3 is 3 ranks behind! Conclusion: Humans that have imagination can beat super-rich humans and machines any time!
**Lalit Pagaria**
Thank you Denis Rothman for clarification 🙏
**Denis Rothman**
You are more than welcome! 😊👍🙏
**Rodney Silva**
Denis Rothman Do I need to build a Transformer from scratch to fully understand it?
**Denis Rothman**
Rodney Silva No. But you do need to understand the architecture of the Original Transformer described in Chapter 1. You can have a look at the chapter and ask me questions this week in our thread.
**Vladimir Finkelshtein**
What is your position on positional encodings? They seem like an afterthought and it is not so clear when to use them. I saw suggestions in the forums to try to train models with and without them, because a priori it is not clear in which type of problems they are helpful.
**Denis Rothman**
Vladimir Finkelshtein Positional encoding is a fundamental part of the architecture of the Original Transformer. RNNs contained a separate cumbersome positional vector.
Transformers ADD positional encoding to word(token) embedding. This is a powerful feature : the word’s “meaning” will take its position into account which is critical for grammatical and semantic structures.
That being said à deBERT model trains positional encoding separately.
In any case, positional encoding is a very effective tool.
**Vladimir Finkelshtein**
Are there other metrics outside of the GLUE benchmarks on which NLP models are compared? For example, if I compared regression with decision trees, I could compare sensitivity to outliers, tendency to overfit, reduction of performance due to imbalanced data, robustness, explainability (I guess this one is model agnostic now) etc.
Could we compare which NLP model is more susceptible to biases, because they are overrepresented in the data?
**Denis Rothman**
Vladimir Finkelshtein Interesting question.
First, some context. SuperGLUE was mostly created because the recent arrival of transformers blew the GLUE human baselines out of the sky! So SuperGlue was a solution. Now transformers are again exceeding the human baselines of SuperGlue.
If you use decision trees or any other ML/DL algorithm just:
1.Download the SuperGLUE datasets and train them with any algorithm you wish to explore and see what happens. You might even end up in the leaderboard. Who knows? 🤔
**Zhi Men Lau**
Hi Denis Rothman what is your take on interpretability of Transformer? There is a lot of buzz around explainable ML and interpretability, how does Transformer address these issues compare to other technologies.
**Denis Rothman**
Zhi Men Lau The best Explainable (Interpretable) AI methods are MODEL AGNOSTIC. This means they use the input data to explain very precisely why a result was obtained.
I explain this in my book on [Hands-On Explainable Artificial Intelligence(XAI)](https://www.amazon.com/Hands-Explainable-XAI-Python-trustworthy/dp/1800208138/ref=mp_s_a_1_3?dchild=1&keywords=Denis+Rothman&qid=1618930655&sr=8-3)
For example, for an input such as “the coach(bus or sports coach?) was unhappy”,
**Denis Rothman**
If you have any questions please feel free to ask me.
**Zhi Men Lau**
Thanks Denis, didn’t know about your other book on XAI. Going to check it out.
**Rodney Silva**
Denis Rothman Can you guess what will be the evolution of transformers in the next years? Less complexity,efficency,less training time,etc…
**Denis Rothman**
Rodney Silva Transformers will expand to every single field involving sequences including recommenders.
Then a new model, one day, will take over. Mastering transformers will make the transition to the future much faster for developers and designers.
**Krzysztof Ograbek**
Denis Rothman Thank you for yesterday’s answers. Correct me if I’m wrong, but I believe that training transformers doesn’t require any knowledge about the language itself. I mean to train a transformer we just need a pre-trained word embedding, feed it to the transformer and wait for the results. So the question here. Nowadays, what is the point of learning things like: Linguistic syntax, dependency parser, Part-of-Speech, Named Entities, etc? How does this “language knowledge” make one a better NLP Specialist?
**Denis Rothman**
Krzysztof Ograbek Good question! In real life implementations the models take weeks if not months to train. Mastering the Linguistics of the Superglue tasks, e.g.,and translations will save you an incredible amount of time! Otherwise, understanding errors and correcting them can take exponentially longer.
😉
**RH**
Denis Rothman: If someone is starting their journey into NLP, what is the pathway you would recommend them to take? What knowledge/skills makes a strong industry ready NLP practitioner?
**Denis Rothman**
I recommend two axes:
1. Mastering basic calculus, matrix and vector multiplications and good basic statistics.
2. Linguistics. Knowing what a lexical field is, what semantics is about, phonology(intonation, for example)
Then everything in NLP becomes easy to understand!
**Doink**
Denis Rothman Is learning about Bag of words, tf-idf, w2vec, rnn and lstm a waste of time? How much do we need to know in NLP it’s an ever growing field.
**Denis Rothman**
Nothing you learn is a waste of time. Learn everything you can and spend all the time you have.
You will then quickly discover two fantastic things about yourself :
1.The more you learn the faster you learn something new. It’s exponential!
2.The more you know, the faster you will implement NLP projects. On top of that, you will be able to explain your work much better to others in a team. 👍
**Doink**
what is the proof of 1. ?
**VK**
Denis Rothman What type of research was implemented for writing this book?
**Denis Rothman**
1.First my background helped: mathematics and linguistics. You can find out more in [this article](https://www.amazon.com/Hands-Explainable-XAI-Python-trustworthy/dp/1800208138/ref=mp_s_a_1_3?dchild=1&keywords=Denis+Rothman&qid=1618930655&sr=8-3)
2.My previous research, corporate experience and writing books. You can find out more [here](https://www.amazon.com/Denis-Rothman/e/B07DFRCJG8/ref=dp_byline_cont_book_1)
3.Reading hundreds of pages of papers on Transformers and exploring all of the source code available in 24/7 mode. It was quite a fantastic experience! 😎😜😀
**VK**
Thank you Denis Rothman
**Akshat**
Denis Rothman Hi Denis, can these transformer models be used in predicting physical quantities in car crash simulations or other numerical simulations?
**Denis Rothman**
Let’s be careful here! Transformers are excellent in predicting sequences. They can be used to predict if a car crash is probable.
However, for precise numerical series, I would rely more on standard math.
**Akshat**
Thanks Denis Rothman! 🙂
**Rodney Silva**
Denis Rothman What’s the most common difficulty for people trying to learn the architecture of Transformers?
**Denis Rothman**
The most common thing I have seen is a lack of patience and underestimating the work it takes to understand the Original Transformer as described in Chapter 1 of my book. If someone takes the time to understand this chapter, then the rest becomes easier if not easy with some basic knowledge of Linguistics that really helps.
**Lalit Pagaria**
Denis Rothman this not a question but appreciation message. Tomorrow I received your book still in first chapter. So far so good. Really nice book. 🙏
**Denis Rothman**
Thank you very much for your encouraging message! Please ask me any questions you may have. 😊🙏
**Lalit Pagaria**
Thank you Denis 🙏
**Denis Rothman**
Your more than welcome. 😊
**Rodney Silva**
Denis Rothman How much time did it take to write this book, from the research till the final version?
**Denis Rothman**
The time it took requires context :
1.I have been doing AI research for the past 35+ years on a daily basis from expert systems to ML/DL. In that same period I implemented my research on key corporate sites.
For more you can peek into my LinkedIn profile starting here.
2.I studied linguistics in college and at the time taught computer science as well. I’ve been studying cognitive sciences all my life.
3.I’m used to writing papers and books on AI.
That being clarified 😊, I did the research, (papers and source code), tested code/wrote code, and wrote the book in 10 weeks.
One last point. I’m a workaholic when I write. I ‘m in 24/7 mode, can get up at any of the night because something just popped up, develop at the dinner table or anywhere.
As long as the book isn’t finished, I just keep on pounding on the laptop!😊
**Denis Rothman**
P. S. Here is the link mentioned in the message :
My 1982 Word2vector-Word Piece model patent led to an AI NLP Cognitive Chabot and a Turing-APS model used in Major Corporations to this day!
[https://www.linkedin.com/pulse/did-you-miss-ai-parsing-train-denis-rothman](https://www.linkedin.com/pulse/did-you-miss-ai-parsing-train-denis-rothman)
**Alexey Grigorev**
10 weeks! Wow! That’s super impressive!
**Krzysztof Ograbek**
Alexey Grigorev, agreed. Denis Rothman, this is just amazing. Your background also
**Denis Rothman**
Thanks. It’s hard work. 😊
**Krzysztof Ograbek**
Denis Rothman I don’t know how to properly phrase my question, but I hope you’ll know what I mean. How to take advantage of speaking multiple languages? How to use it in NLP field?
**Denis Rothman**
Mastering more than one language is naturally an asset in NLP. First, it develops better knowledge of language structures and meaning.
When testing translation models, it’s a good asset.
However, we cannot master all of the language s we will face, so there is a limit to that asset!
**Krzysztof Ograbek**
Denis Rothman Could you recommend some good resources for learning linguistic, that can be useful in NLP?
**Denis Rothman**
A good place to start with linguistics is a good elementary or high school book of grammar!
Then an easy book on semantics and phonology /phonetics.
In short, the easier the better to build a solid approach.
Then once that is done, look for a nice book or course you like. 😊
**Krzysztof Ograbek**
Denis Rothman, thank you again. I found [this tutorial](https://www.youtube.com/playlist?list=PLoROMvodv4rOFZnDyrlW3-nI7tMLtmiJZ)
on YT, it has 100 videos. Would you be so kind to skim through the titles and say if the content makes sense for beginners? I already learned plenty from this. I just wonder if the direction is right 🙂
**Denis Rothman**
I see that these videos were designed by Stanford professors. If you find them interesting, then of course you can follow them.
Just keep transformers in mind as well.
**Rodney Silva**
Denis Rothman What was your reaction the first time you realised how transformers worked? Do you think that you could have invented Transformers?
**Denis Rothman**
When I first explored the Original Transformer model, I was in awe. The small Google team that was working on this new model were experimenting all sorts of ways to solve the limitations of RNNs.
It was low level trial and error. Then they decided to drop the 30+year old concept of recurrence and replaced it with attention!
They also industrialized the layers that are all the same size, spilt the layers into parallel processing, and went on like that for everything in it.
They surprised themselves by outrank ing everybody on the NLP leaderboards.
I was admirative of this little team within a large corporation. 😊
**Rodney Silva**
And how about the second question? Denis Rothman
**Denis Rothman**
Right. Could I have invented transformers?
No. Why?
I don’t think of NLP only in terms of statistics. I think transformers are a fantastic evolution but will also, like RNNs, be prehistory as IoT sensors develop and concepts are added.
In my book in AI by Example, I teach a CNN how to learn concepts in Chapter 10 CRL. In chapter 6 on translations, I introduce symbols in the Google translate API:
[link](https://www.amazon.com/Artificial-Intelligence-Example-advanced-learning/dp/1839211539/ref=mp_s_a_1_3?dchild=1&keywords=Denis+Rothman&qid=1619109365&sr=8-3)
Now why do I think this?
I often use the simple word “hello”.
There are billions of interpretations of that word.
Let me give you some.
1. Person A is sitting in an office working. Person B comes in and says “hello” in a neutral tone.
Context : B has never said hello in 10 years to A, even when in the same elevator.
Interpretation by A: Am I going to be fired? What’s going on?etc.
2.B walks in and mumbles a gloomy hello.
Context : B is always chirpy, smiling and happy.
Interpretation by A: Did I do something wrong? Is something wrong with B? What’s Going on.
3.B comes in and gets very very close to A and says “helloooo” like a hungry wolf.
Interpretation by A: This happens every day. Oops! Is this some kind of harassment? What should I do.
4 to infinity!
Now add more words, situations and body language, cultural habits and emotions!
Humans are very complex machines faced with an infinity of complex situations.
Statistics can help. But more is to come!
**Rodney Silva**
Denis Rothman What do you think is the most brilliant thing in Transformers: multi head attention or sinusoidal positional encoding?
**Denis Rothman**
Using attention with simple matmul operations was baffling. Then adding cos/sin positional encoding to the vectors/matrices instead of adding more vectors was brilliant. I enjoy trigonometry so I liked that.
Both ideas are beautiful. Some recent models have separately trained positional encoding(deBERT, for example).
Everything is brilliant. It’s like building a new motor engine in your garage at home. That’s how they did it.
**Lalit Pagaria**
Denis Rothman what is faster ways to test which encoding would be great for a task instead of going into whole train - test life cycle. At least any fast intuitive way to try first in order to get fast feedback.
**Denis Rothman**
The fastest way to choose a model in production is to have learned transformers from A to Z in general. And during that process select the one you like the best.
Once that is done, there are many ways to train and fine-tune models. Have a look at the PET approach, for example, in the beginning of chapter 6 of the book.
PET is pattern-extracting training which processes the data BEFORE training a model. It like a good teacher that well prepares a course instead of throwing raw information at the students!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tiny Python Projects – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Tiny Python Projects
--------------------
#### by [Ken Youens-Clark](https://datatalks.club/people/kenyouens-clark.html)
##### The book of the week from 26 Apr 2021 to 30 Apr 2021
A long journey is really a lot of little steps. The same is true when you’re learning Python, so you may as well have some fun along the way! Written in a lighthearted style with entertaining exercises that build powerful skills, Tiny Python Projects takes you from amateur to Pythonista as you create 22 bitesize programs. Each tiny project teaches you a new programming concept, from the basics of lists and strings right through to regular expressions and randomness. Along the way you’ll also discover how testing can make you a better programmer in any language.
* [Book's page](https://www.manning.com/books/tiny-python-projects)
* [Book's GitHub repository](https://github.com/kyclark/tiny_python_projects)
Questions and Answers
---------------------
**Dustin Coates**
Hi Ken Youens-Clark, great to see a fellow Manning author! Would you say your book is a good fit for someone coming from other languages? I’ve a lot of experience with JS and Ruby, for example, but most of my Python is written with a JS accent. Would your book be good to learn the Python way of doing things?
**Ken Youens-Clark**
Hey, Dustin. Yes, I specifically hope that someone is coming from a previous language, but I also try to cover everything just in case this is a person’s first Python book. I try to show Pythonic ways to do things you might know from other languages, but I also try to show that Python is quite flexible and often has many ways to accomplish the same task. For instance, the list comprehension is very Pythonic (it’s similar to Haskell in this way). C-type languages such as JavaScript might initialize an empty list and use a `for` loop to build it up
`>>> powers_of_two = [] >>> for i in range(4): ... powers_of_two.append(2**i) ... >>> powers_of_two [1, 2, 4, 8]`
But a list comprehension can do this in one step:
`>>> powers_of_two = [2**i for i in range(4)] >>> powers_of_two [1, 2, 4, 8]`
I think these kinds of idioms make Python easier to read and share.
**Ken Youens-Clark**
I should say that I quite flagrantly violate the Python Zen of “there should be one obvious way to do something.” I specifically show many ways to accomplish the same ideas, and I really like to get the reader used to purely functional ideas. So, if you can see how a `for` loop (which is a _statement_ that produces no result) can be written as a list comprehension (which is an _expression_ that returns a value), then you might be comfortable writing this with `map`:
`>>> list(map(lambda i: 2**i, range(4))) [1, 2, 4, 8]`
Similarly, if you are comfortable using a list comprehension with a guard to conditionally accept some values:
`>>> evens = [n for n in range(10) if n % 2 == 0] >>> evens [0, 2, 4, 6, 8]`
Then you might be good with learning to use `filter`:
`>>> list(filter(lambda n: n % 2 == 0, range(10))) [0, 2, 4, 6, 8]`
JavaScript really uses functions in a similar way, in my experience. That is, functions are first-class objects that can be passed as values and overwritten, used as callbacks, etc. I think teaching these kinds of ideas to novice programmers will make it easier for them to launch into other languages, too.
**Eric Sims**
Wow, those examples are really helpful. I am getting better at using list comprehensions, but that `filter` with the `lambda` function to do the same thing is cool!
**Ken Youens-Clark**
When you start thinking in terms of functions and how they fit together, you can see how to fit code together like Lego pieces. Working in a strongly/statically typed, purely functional language (e.g., Haskell or Elm) is one way to burn your boats and force yourself to think this way. My experience in Elm totally changed my approach to using imperative, dynamically typed languages.
**Ken Youens-Clark**
Note that you don’t have to write `lambda` defs. You can also use regular functions:
`>>> def p2(n): return 2**n ... >>> list(map(p2, range(4))) [1, 2, 4, 8]`
**Brian Fleming**
Hi Ken Youens-Clark
Can you recommend whether you think a Marketing Analyst (GA/SQL based) that’s looking to upgrade their skills would be better starting with something like your book and Python or concentrate on a new platform and deprioritise coding/Python?
**Ken Youens-Clark**
SQL databases have been an integral part of everything I’ve written over the last 25 years. For the longest time, I relied on Perl for this, but Python’s integration with SQL platforms is really great. For instance, I now mostly use Postgres and rely on the `peewee` module to create an object-relational module (ORM) that minimizes the amount of raw SQL I write. When I combine that with something like FastAPI and type hints, I can bang out a web back-end API in relatively.
For instance, I’m currently working on [ct.c-path.org](http://ct.c-path.org/)
for my employer (The Critical Path Institute). This a mirror of [clinicaltrials.gov](http://clinicaltrials.gov/)
with a custom query interface. All my code is in [https://github.com/criticalpathinstitute/ctweb](https://github.com/criticalpathinstitute/ctweb)
, and you can look in the “fastapi” directory to see how I leverage these technologies to create what is, IMHO, a readable codebase for interacting with Postgres.
So, TL;DR: Yes, I would definitely recommend learning Python and how to integrate with databases. If you combine types and tests, then I think you’ll get even more out of this, and I discuss these at length in my book(s).
**Brian Fleming**
Thanks Ken, I’ll have a look at the repo 🙂
**Ken Youens-Clark**
I have plans to pitch a book about Python + ORMs + SQL databases (SQLite, Postgres) one day. My current project is _Hands-on Systems Programming with Rust_ for O’Reilly, and I might pitch a book on purely functional programming with Elm next. Not sure when I’ll get to Python/SQL.
**Matthew Emerick**
Hey, Ken Youens-Clark! Thanks for doing this.
Do your tiny projects make for a great portfolio strictly as they are in the book? Are there recommendations to expand them?
**Ken Youens-Clark**
If I were a potential employer and saw that someone had managed to create a repository of programs that included documentation, sample data, and tested code, I’d be fairly impressed. If a person did nothing other than type out the code I present and run the tests, I think that would be a fairly impressive first step in learning how to organize, run, and test Python. If they had managed to write their own solutions before consulting my versions, that would be even more impressive. If they went on to expand the programs by adding features and tests, that would be . Each chapter has a “Going Further” list of suggested expansions, but one’s imagination is the limit.
**Matthew Emerick**
Do the projects put together teach a wide enough variety of skills to give a deeper understanding of what Python is capable of?
**Ken Youens-Clark**
Python can do an incredible amount, but I can only move the user a couple clicks along. That is, when I first started talking with Manning, they suggested I imagine a scale 1-10 where 1 is a beginner and 10 is an expert. I can only hope to move, say, a 2-3 level user to maybe a 5-6. Trying to move someone from a 2 to a 9 in one book makes for material that moves too quickly. So I try very hard to make a novice programmer feel comfortable with testing which in and of itself is usually considered some sort of super-skill. Testing is not actually all that difficult, but I hardly find anyone trying to teach this vitally important skill. While I try to show just how flexible Python is as a language, I would say my secret goal is to get people to think about how to write small, testable, composable functions. I want my readers to be able to take these skills and apply them to learning and writing any other languages they encounter.
**Matthew Emerick**
Is there a project in your book that shows you how to call C code?
**Ken Youens-Clark**
No, I would consider that a very advanced skill. To be honest, I’ve never written a C program beyond “Hello, world!” I never have need for C, and I have no idea how to call it from Python. My next book is a similar beginner’s guide to Rust, which I would consider a much better language for trying to interface with external libraries.
**Sara Garcia**
Hi Ken Youens-Clark
I’m reading the list of the Python projects of your book and I wonder if the reader should have a previous knowledge of python, or any other programming language or concepts. I also want to know what concepts do you recommend to expand, and if you have another book to recommend for that.
**Ken Youens-Clark**
Hi, Sara. I honestly think _Tiny Python Projects_ is a great 2nd or 3rd Python book. If you don’t know the language at all, you might find _The Quick Python Book_ or something like that a good introduction. If you know any other language at all, my book is probably a fast way to get familiar with the language. Just reading about concepts is not at all the same as trying to write your own programs. Writing forces you to really interact with the language, and learning how to test your programs will really focus your attention.
I think my book explains a lot of basic programming concepts reasonably well, things like loops and variables and functions. I completely avoid object-oriented programming for various reasons I could explain if you care. If you don’t known OOP, then you definitely don’t need that concept. I subtly push my own biases that include a penchant for purely functional program, immutable data structures (e.g., tuples), the use of testing, and the introduction of type hints which comes in the last chapter. If you like these ideas, you might be interested in my second book, _Mastering Python for Bioinformatics_ (O’Reilly) which will be in print next month. That book goes much deeper into testing, types, and functions, even if you are necessarily interested in biology + computer science. The material is actually fascinating, and the solutions can get quite complex.
**Anna Parfenova**
Hi Ken Youens-Clark!
Despite I don’t belong to Z-generation, it’s hard for me to start learning (programming) from books, when there are plenty of video-courses and other web-resources available.
I see one big _pro_ of book:
* it will make me re-type things instead of copy-pasting.
What are other pros (and maybe cons)?
**Alexey Grigorev**
Actually, you can copy-paste from pdf…
**Ken Youens-Clark**
The reader will gain the biggest benefit by writing the programs. Even if you do copy-and-paste from the PDF, the act of getting the code, altering the programs, and running the tests will give you a categorically different experience from simply reading the book. I strongly encourage the reader to type in all the examples, both the examples in the REPL and the actual programs. In the language of pedagogy, this is an “active learning” approach where the student _does_ stuff rather than just listens or reads. I really enjoyed my times in the classroom using these kinds of exercises to get people writing and testing programs. Those classes were so different from times when I would just lecture about Python lists and blah blah blah. If you want to learn to write programs, you have to write programs!
**Ken Youens-Clark**
Oh, also all the code/tests/solutions are directly available from my GitHub repo at [https://github.com/kyclark/tiny\_python\_projects](https://github.com/kyclark/tiny_python_projects)
.
**Ken Youens-Clark**
And I’ve also created videos to walk a person through the solutions, step-by-step. You can find them all from [http://tinypythonprojects.com/](http://tinypythonprojects.com/)
. This is another important skill I’m trying to teach–that is, how should one start a program, what’s the next logical step. For instance, I suggest using the `bin/new.py` program in the repo, then defining the parameters using `argparse` and testing the interface with the first couple tests. Then move to maybe printing something, maybe reading a file, etc. The process of writing and testing is much more important than just learning Python syntax.
**Vladimir Finkelshtein**
In my personal experience, online videos tend to be on the shorter side, so they will usually cover each topic in much less detail than a corresponding chapter in a decent book. Many of these videos are made by reading a chapter from a good book and trying to squeeze it into a determined time interval.
**Anna Parfenova**
Thank you, Ken Youens-Clark 🙏
Good point, Vladimir Finkelshtein 🤔
**Eric Sims**
Not so much a question at the moment, but a comment from the GitHub repo: I really like your philosophy of “test-driven development” and writing tests before writing code. Taking the time to define success before jumping into a project would probably save me a fair bit of time and certainly help manage scope creep! 😄
**Ken Youens-Clark**
“Weeks of coding can save you hours of planning.” – Anonymous
**Ken Youens-Clark**
I really feel that TDD is an under-appreciated teaching tool. I found that giving my students a test suite makes it crystal clear what the expectations are for a program. I would actually see students get excited when they would go from all failing tests to passing one or two tests and finally passing all tests. The motivation of see all those passes should not be dismissed.
**Eric Sims**
I hadn’t thought of it that way, but you are definitely right. It feels so rewarding to solve lots of small problems along the way and celebrate the small wins.
**Ken Youens-Clark**
Tests really prove their value when you try to add a new feature to your program and find you’ve accidentally broken something that used to work.
**Simon Steinkamp**
I had a quick look over the projects and the Github Repro and it sure looks interesting 🙂
As someone who has done quite some coding in Python so far (including a little package with tests) but never took the time to get a formal / solid Python foundation - I was wondering what someone like me could get out of your book and whether it could help to get more structure in the hacky coding style? ( I hope the phrasing isn’t to confusing)
**Ken Youens-Clark**
If you are a more experienced developer, the first few chapters may feel a little simple for you. I’d still encourage you to work through even the most basic programs to get a feel for using tests to verify your programs. TDD encourages us to first write tests and _watch them fail_. Then fix the code until the tests pass. If you’ve never done this before, you may be surprised at just how easy testing with a framework like `pytest` can be. I really feel like “testing” is made out to be some really advanced topic, beyond the comprehension of students and novices, and is something you’ll learn about on the job. My experience is that testing is easier than not, and that not enough people on the job use testing or ever teach it. I may also surprise you with how differently Python can be written when you rely mostly on functions and tests. For instance, if you’ve mostly done OO-style programming with Python, you’ll see much simpler code in my examples. So, yes, I think you’d get a good bit from going through my examples.
**Simon Steinkamp**
Great! Thanks 🙂
**Gant**
Hi Ken Youens-Clark what is the most fun project in the book?
**Ken Youens-Clark**
T’ehres stenhoimg aoubt The Srebalcmr taht I find rellay fun. I’ts amnizag taht your biarn can raed this text, and I ejnoy wtrinig pgmraros taht eplmoy rodnamsnes. ([https://github.com/kyclark/tiny\_python\_projects/tree/master/16\_scrambler](https://github.com/kyclark/tiny_python_projects/tree/master/16_scrambler)
)
**Ken Youens-Clark**
I also love ideas about encryption, and [Gematria](https://github.com/kyclark/tiny_python_projects/tree/master/18_gematria)
is interesting to me. I got that idea while reading _The Chosen_ by Chaim Potok.
**Ken Youens-Clark**
Although this is not a reversible encryption. My latest book using Rust will likely include a simple ROT13 (rotate 13 characters) encryptor.
**Ken Youens-Clark**
Each chapter is intended to teach some few skills. Even though they are playful, trivial programs like “Apples and Bananas” can lead you to explore many interesting ways to write the same idea ([https://github.com/kyclark/tiny\_python\_projects/tree/master/08\_apples\_and\_bananas](https://github.com/kyclark/tiny_python_projects/tree/master/08_apples_and_bananas)
) while testing helps you learn how to _refactor_ programs while still ensuring they work.
**Glenn**
Hi Ken, what’s something about Python that you learned while writing this book that you didn’t know about beforehand? And possibly the most important question, in terms of “RegEx”, are you of the camp that it’s pronounced with a hard g (like garden) or soft g (like generator)?
**Ken Youens-Clark**
Hi, Glenn. Your first question is really interesting. I actually find it remarkable that I got a chance to write a book on Python when I’d only been working in the language a couple of years. I spent most of my career using Perl, which I find to be extremely similar in all the important ways (dynamically typed, c-like syntax, modules, variables). Still, there were lots of nuances to learn, and I’d say that I got much more familiar with testing/`pytest` and the use of randomness which I’d never really used much before. E.g., while writing one of the exercises I discovered that the reader might make some random choices in a different order than I did and so would end up with a different answer. Not a _wrong_ answer, just different because the RNG (random number generator) would be called differently. I learned much more about testing writing this second book (_Mastering Python for Bioinformatics_, O’Reilly, 2021) such as how to better organize tests and data as well as how to integrate linting and type checking.
As to your second question, I guess I’m completely inconsistent because I pronounce “regex” with a soft _g_ like “rej-ex” but “regular” has a hard _g_. Completely random anecdote: A couple of years ago I was walking across the Univ of AZ campus to get a coffee with a friend and we bumped into Noam Chomsky (who’s done tons of work in linguistics and “regular” languages and such). We were star-struck.
**Glenn**
Thanks for the response, and a very cool story about Noam Chomsky! I remember seeing Perl listed on a Stack Overflow survey as the “most hated language” by programmers. What are your thoughts on that?
**Ken Youens-Clark**
It’s easy to write bad code in any language. Perl’s penchant for sigils like `@{ $hash{ $key }}` can be really hard to read. I understand that fine, but I also appreciate that it’s hard to read. Perl was ascendent in the late 90s and early aughts when I became a web developer and fell into bioinformatics. Perl was king of text processing and regular expressions, and it worked really well in those domains. I loved it very much and only used Perl for about 15 years. I released a few modules ([https://metacpan.org/author/KCLARK](https://metacpan.org/author/KCLARK)
), SQL::Translator was perhaps one of my best works, and I built and maintained a cool genomic map viewer ([https://pubmed.ncbi.nlm.nih.gov/19648141/](https://pubmed.ncbi.nlm.nih.gov/19648141/)
).
Sometime in the last 10 year, Python simply took over scientific computing. Modules like Numpy, Pandas, and Scikit-learn are just too good and important to ignore. In web development, I found Perl modules I’d used for so long couldn’t compare with the features I found in JavaScript and Python. It was teaching that pushed me over the edge, though. I asked my boss at UA to move our intro programming from Perl to Python because I thought it was much easier to teach and we were doing a disservice by teaching students an older language when Python was clearly more in demand. Perl still has a special place in my heart, but I’ve moved on. I thought Perl 6/Raku was interesting, but I’m not sure it’s ever going to draw me in. My biggest push lately has been into using Elm for web front-ends and Rust for systems programs, and, in fact, my next book with O’Reilly is an intro to Rust.
**Glenn**
Thanks for the detailed response, Ken! My first language was JavaScript and have worked on teams that have used Go, Swift, and Python. I found that Python was the quickest to learn and be productive in (but could also be because I started working with it after years of programming experience). I’m still fairly new to it, but blown away by pandas and scikit-learn. Sounds like you’re busy writing a lot of books!!
**Jeffrey Jex**
Hi Ken, with the idea of working on fun projects and puzzles, what minimum age range would this book be appropriate for? I saw you mentioned it’s a great 2nd or 3rd python book in an earlier comment. In other words, would this work for early teenagers still learning to code?
**Ken Youens-Clark**
I wrote this material when I was teaching college-aged beginners while I was working at the Univ of AZ, but I had in mind at least high-school aged readers and perhaps even younger. I illustrated it with my crude cartoons and tried to interject enough levity so it doesn’t feel like a textbook. The material is drawn from simple games that kids would likely know like “Telephone” and “Mad Libs” so I can focus on learning syntax and _testing_ more than anything else. I would love to think that teenagers would find this approachable, but I don’t have direct experience with students younger than 18 or so.
**Jeffrey Jex**
👍 thanks for the info!
**Tatyjana Ankudo**
Hi Ken! Thanks for your work. How is this book different from another Python books? What particular did you pay the most attention to writing the book?
**Ken Youens-Clark**
I think it took me writing the book to finally realize that the biggest difference is that I’m trying to teach people how and why to test their code. There are so many beginner Python books that just snippets of code like how to implement something like a Caesar cipher, which is cool, but they don’t show how to integrate this into a _complete program_ that, for instance, takes a text file as an argument, validates said file, reads it, encrypts it, and writes the output perhaps to STDOUT or maybe to another file. Along the way, how do you test the program to ensure it does all those things correctly? And how does one go about writing such a complicated program? I think it’s not actually too difficult to teach all these concepts to novice programmers. It’s actually better to set the bar a little higher and let new programmers know that we’re expected to write programs that have documentation and tests, that it’s incumbent on the author to provide good code to users. But I try to make all this subtext. The foreground is solving little puzzles and these other things are techniques that help one do that. When the reader has finished the book, though, they’ve seen dozens of unit and integration tests and have used their own interfaces to run their programs. I think this is the most important thing my readers could learn and definitely these are skills I don’t see other authors attempting to teach.
**Ken Youens-Clark**
Sorry to go on and on, but I love this quote about testing:
More than the act of testing, the act of designing tests is one of the best bug preventers
known. The thinking that must be done to create a useful test can discover and eliminate
bugs before they are coded—indeed, test-design thinking can discover and eliminate
bugs at every stage in the creation of software, from conception to specification, to
design, coding, and the rest.
—Boris Beizer, Software Testing Techniques
**Tatyjana Ankudo**
I am glad you wrote this book, because I absolutely agree to your point of view. I was lucky to get very good mentor who make me love testing. Your book is in my list “must read”
**Alexey Grigorev**
Were there some tiny projects that you wanted to include, but couldn’t?
**Ken Youens-Clark**
I was only able to include about 1/3-1/2 of the exercises I originally sketched out. That being my first book, I had no idea of the exposition I would end up writing. For some reason, I originally thought I would spend just a few pages per exercise and let the code speak for itself. I came to realize this wasn’t really teaching, and so I had to create much more prose. I put a bunch of other ideas into [https://github.com/kyclark/more\_tiny\_python\_projects](https://github.com/kyclark/more_tiny_python_projects)
some of which might one day find their way into a more advanced Python programming book, one that would advocate more for the use of types which I do in my bioinformatics book. I’m surprised that types seem to be rejected by a lot of very accomplished Python programmers, so this might be a somewhat controversial stance. I’m also very interested to see how the latest language features of 3.10 and beyond evolve, e.g., pattern matching (which is not the same as regexes) and switch/case statements could really improve Python code. If I write another Python book, I will definitely push to show how these features would look.
**Alexey Grigorev**
That’s a very familiar story. thanks for sharing!
**Alexey Grigorev**
Also, for someone who’s learning some other programming language (say Rust), would you recommend the same set of tiny projects or something else?
**Ken Youens-Clark**
Absolutely! In fact, to become a better programmer in general I would strongly advocate to try writing these ideas in any other language you know. Try JavaScript or C. Then go make yourself learn something completely different like Haskell or Lisp. Certainly Rust is an interesting choice, too. My next book project is “beginner” programs in Rust, but the level of that beginner is expected to be a bit further along in their career than for TPP, so the programs are more complex.
**Ken Youens-Clark**
For example, the first Rust project in my book is an implementation of `echo` which repeats back anything it is given. This is essentially “Hello, world” and is very close to the Crow’s Nest exercise in TPP. I have Tic-Tac-Toe in TPP, and I might include Hangman in my Rust book (depending on space). Both are simple games that iterate through play and maintain state. TPP has text transformations like The Scrambler, Gematria, and Jump The Five, and my Rust book will likely include encoders for ROT13 (rotate characters 13 so A becomes M, etc.) and a Pig Latin (“ig-pay atlin-lay”). The TPP Workout-of-the-Day (WOD) shows how to read a CSV file, and in Rust I’m writing a version of `cut` that also handles delimited text.
**Ken Youens-Clark**
The goal is to get the reader to see patterns. I mostly see the world now in terms of primitives like strings and numbers and then data structures like lists, sets, and dictionaries. Functions transform these in various ways, so it’s mostly a matter of hooking up the plumbing correctly, no matter the language. Some languages are stricter in the compile phase so they help you find places where you didn’t get it right. Rust won’t even budge until it’s satisfied that everything is perfect, whereas Python will just blow up at runtime so it’s important to include types and tests.
**Alexey Grigorev**
Thanks!
Just curious, why would you recommend learning lisp or haskell? Just because they are quite different from the mainsteam imperative languages?
**Alexey Grigorev**
I also know that you’re quite a prolific author. How do you manage to finish one book and find the energy to jump on another one immediately?
**Ken Youens-Clark**
I’m not sure I’d call myself prolific just yet. My second book, _Mastering Python for Bioinformatics_ (O’Reilly) is heading to press next week, and I have started my third on Rust. Maybe if I get to five books I’ll accept that description! I’ve been programming for about 25 years and spent a good bit of time teaching and mentoring and reviewing code. I finally found enough to say, and now I feel like I just need to write and write to get all these ideas down on paper.
**Ken Youens-Clark**
I recall when I started my undergraduate degree at the Univ of North Texas back in 1990. I signed on to be a Jazz Studies major because I wanted to study the drum set. I’d be playing since age 9, and the choices were either to study music ed to become an instructor, music performance to become an orchestral player, or Jazz to study the kit. Anyway, it seems ridiculous now that I was studying Jazz because I’d never listened to it in my life.
**Ken Youens-Clark**
I could barely read music, couldn’t hear the difference b/w a major and minor scale, couldn’t vocalize, didn’t know jack about music history or theory. Mostly I could bang out some rock beats on a kit, and yet I was going to study music! Anyway, I come to quickly realize that I’m in way over my head. I’m trying to learn how to play swing on the kit when I’ve mostly just listened to hard rock. It was a radical change. I had no vocabulary.
**Ken Youens-Clark**
Every member of a jazz group is expected to be a virtuoso and take a solo. You have to know the form of a tune, play in the correct style, interact in real time with other players. When it came time to take a solo, I would just freeze. I had no idea what to “say” on my instrument.
**Ken Youens-Clark**
One of the instructors patiently said one day that jazz is something of an old person’s music. That it takes a really long time, lots of studying and playing and listening, to find something to say on your instrument, particularly something new.
**Ken Youens-Clark**
Until then, we should just study what others have said, mimic them, write down ideas, synthesize until we found our own voices. This would ultimately take decades, so don’t be in too much of a hurry.
**Ken Youens-Clark**
After a couple of years, I switched majors (a few times), but I kept studying music on my own. I got deeper into literature and reading and thought about writing fiction, but, again, this was so beyond me. I still harbor delusions of writing stories. The words of my jazz prof stuck with me, though. It would take a really long time for me to find something to say, so I’ve just kept trying over the years to work out what I have to bring.
**Ken Youens-Clark**
I also really thought about teaching as a career, but obviously programming pays way better. While working at the Univ of AZ, I finally got a chance to teach in a classroom by helping my boss, Dr. Bonnie Hurwitz. At first we split classes, her teaching science (metagenomics) and me programming so our student could learn how to do computational research.
**Ken Youens-Clark**
Teaching was a huge challenge, and I really sucked at it at first. I would type out my lectures entirely long-hand, that is, as a prose document to work out what I wanted to say in an hour. That’s pretty much the length of a decent chapter. I would devise examples and sequence things to show, for instance, what is a list, why it’s useful, how to manipulate it, where you’ll find one in the wild.
**Ken Youens-Clark**
After a couple of years, I finally designed an entire course just on the programming material and got to teach for entire semesters. That was really life-changing. I enjoyed working with students and seeing their struggles. Since I learned to program so long ago, it was easy to forget what was hard to learn. I kept refining my examples and prose and lectures and moved entirely towards live-coding in class.
**Ken Youens-Clark**
“Active learning” is a great way to keep people from falling asleep in class and really helps people retain information. So I have adopted this active approach to writing. I describe a challenge, give the reader some information and hints on how to solve it along with the data and tests to know when things are correct, and push the reader to work it out. Then I present some different ways to solve the problem.
**Ken Youens-Clark**
I don’t know if it’s just a formula I’m following for now. I’m not trying to be derivative, but this is the format for TPP, the bioinformatics book, and now the Rust book. There are plenty of other books that describe everything you want to know about a list or a dictionary for a given language, but not enough that show you when and how to use one in an actual program.
**Ken Youens-Clark**
As for the energy, I’ve got issues. I feel really OCD about writing at this point in my life. Like I’ve been trying to learn to speak and finally figured it out. My father died young, and I think I’m haunted by ideas of mortality. I recently watched (and rewatched and rewatched many times) “Hamilton” with my daughter, so I guess “I am not throwing away my shot.”
**Ken Youens-Clark**
Sorry if that took a weird turn, but you hit a button, I guess.
**Alexey Grigorev**
What I can see is you definitely love writing 😅
**Alexey Grigorev**
I was really curious where the jazz story is leading!
**Alexey Grigorev**
So, if I can attempt to summarize it, you did a lot of work prior to writing the book, and already had experience with structuring your thoughts on paper. Am I close?
**Alexey Grigorev**
I also like the idea of active learning! That’s really great! I haven’t heard this term (in this context) previously
**Ken Youens-Clark**
Yes, you summarized it fine. I started to scratch the surface of learning formal pedagogy when I was working on my MS at UA. I may have an opportunity to teach there this fall, and I’d love to continue learning more about the science and art of teaching. As interesting as I think my own lectures are, I’ve literally seen my students struggling to stay awake. That doesn’t happen when everyone is trying to write code together. Sometime I have them do pair programming in class, too. Another great teaching tool.
**Alexey Grigorev**
Okay, I have a few more questions!
You just finished a book about bioinformatics. Can you tell us a bit what bioinformatics is and what kind of problems it solves?
**Ken Youens-Clark**
Yes, my latest book is with O’Reilly and is called _Mastering Python for Bioinformatics_ ([https://learning.oreilly.com/library/view/mastering-python-for/9781098100872/](https://learning.oreilly.com/library/view/mastering-python-for/9781098100872/)
). Bioinformatics is the application of computer science to biology. There was a time up until the 1990s when most biological data sets would easily fit into small Excel files. With the advent of genome sequencing, that is no longer the case. Sequencers have been able to create more and more data for cheaper and cheaper prices at a rate that outpaces Moore’s law. Biology is swimming in data, and so it’s imperative to use computer science to find patterns.
**Ken Youens-Clark**
In French, “informatics” is computer science, so it makes a little more sense. Anyway, I’ve been working in and around bioinformatics since 2001 when I got a job as a web developer in the lab of Dr. Lincoln Stein at Cold Spring Harbor Labs. He was a big Perl name, author of core modules and several books. Hanging around him at conferences was like being around a rock star back then. It was weird.
**Ken Youens-Clark**
I knew basically nothing about biology. My undergrad degree was English lit (minor in music) I was hired to build databases and code Perl web interfaces. I also wrote a comparative map viewer that many people used for a while ([https://pubmed.ncbi.nlm.nih.gov/19648141/](https://pubmed.ncbi.nlm.nih.gov/19648141/)
) but I never felt I had my _bona fides_ in the field. I jumped at the chance to work at the Univ of AZ as I wanted to earn my MS which I finished in 2019. That gave the me chance to fill in a lot of missing knowledge esp in stats and machine learning.
**Ken Youens-Clark**
When I was writing material to teach basic programming at UA, I started with the ideas in Tiny Python Projects, and then showed how those simple ideas had cognates in biology. I had dozens of ideas for chapters for a book on bioinformatics, but I realized that the [Rosalind.info](http://rosalind.info/)
problems just couldn’t be topped.
**Ken Youens-Clark**
So I chose 14 of those challenges and showed how to solve them in Python, ordering them in such a way to build on the ideas as in TPP. Then I threw in 5 more of my own ideas to show some more advanced programs.
**Ken Youens-Clark**
I haven’t quite answered your question – What does bioinformatics solve? Well, pretty much all the big advances in biology including things like the COVID vaccine are due to using computational approaches to solving data problems. For instance, protein folding is a wicked hard problem, but all the information to figure out the 3D structure of a protein is necessarily contained in the raw DNA sequence. We just have to figure out how to go from one to the other.
**Ken Youens-Clark**
I worked for 13 years on a plant genomics project where we looked at things like orthologous genes, that is genes that basically encode the same proteins/function in different plants. For instance, some species of rice are more drought tolerant than others. Why? Is it one gene or a set of genes? Can those be transferred to another species? Humans have been doing gene transfers for centuries in plants and animals through selective breeding. Now we are able to select individual genes.
**Ken Youens-Clark**
Then I worked in metagenomics at UA which is the study of uncultured genomes found in the wild. Think about taking a sample of ocean water and trying to figure out all the microbial species living in it by extracting the DNA and sequencing everything. We only know a fraction of the species existing on earth, so a lot of the DNA you find will be unidentifiable. Still, you can find genic regions and compare them to known genes/structures to infer what’s going on the organisms that are present. Extend that to a sample from a person with an unknown infection. Take a sample maybe from the gut. Do they have C. difficile or something else like a virus? Antibiotics don’t work on viruses, so antibiotics could actually make the patient sicker. If someone is septic, you give the worst antibiotic possible but you need to de-escalate quickly so you need to identify what organisms are present to choose the right antibiotic (or something else entirely).
**Ken Youens-Clark**
The field of bioinformatics is huge and growing, e.g., biotech. It’s an exciting field.
**Alexey Grigorev**
And why did you decide to write a book about it?
**Alexey Grigorev**
Is it something you do at work?
**Ken Youens-Clark**
Oh, I should have answered that here, but yes, most of what I’ve done over the last 20 years.
**Wendy Mak**
to add to the above questions above bioinformatics– I find it quite an interesting topic, but as someone who has no biology background beyond high school, what’s a good way to learn about it/get involved in projects etc? would you say your new book a good starting point?
**Ken Youens-Clark**
I had no formal training when I started, but I was also not hired to do bioinformatics per se but web development. If you are truly interested, a formal training would be pretty necessary. Many people in the field have PhDs or at least an MS. There are some degree programs in bioinformatics, but usually people train in something like molecular biology or biochemistry and do lots of coding. Some people formally train in both biology and CS, and those people usually have it really going on.
**Ken Youens-Clark**
If you wanted to test the waters, you might try working through the exercises in my new book ([Rosalind.info](https://learning.oreilly.com/library/view/mastering-python-for/9781098100872/%3E)
and then continue on with others from logistic regression -> xgboost -> e2e deep learning. This allows you to understand your problem domain better and prevents wasted engineering/science effort.
Are there similar pathways for RL where we can get the majority of the benefit out of e.g. a very simple optimisation or bandits before having to go full deep RL. What are the tools/methodologies you’d recommend getting projects started with?
**Miguel Morales**
Hey, John Savage. This is an excellent question. I think the same idea applies, but a bit differently. In RL, there is an environment and an agent. Usually, the environment is given, and researchers concentrate on the agent. However, in an engineering sense, you likely don’t care about Atari games. You care about your problem. So, one of the main challenges in RL is finding an environment to train your agent, or better yet, creating one yourself. What simulation engine are you using? Will you use the real world for training? Will the agent be able to collect enough data? Will you use a sample efficient algorithm for training? Etc.
My recommendation here is to spend time implementing an environment, use off-the-shelf algorithms first, and make your environment look like the environment those algorithms are tune for solving. Once you iterate a few times over your environment, then start changing the agents, models, and training schemes, then go back to the environment and polish once again. Lots of folks want to solve everything at once and trust me, RL is challenging. So, isolating the problem is the way to go.
**John Savage**
That’s a great insight, thanks a lot, makes a bunch of sense.
**Tim Becker**
Hi Miguel,
I started looking into RL and your books looks like a great asset! I would like to ask you couple of questions concerning the book and the general topic:
* Do you have any recommendations for projects to work on alongside your book? I implemented some algorithms for the cart-pole environment from the OpenAIGym package and everything seemed to work, but when I moved to more complex environments my agents did not perform. I was wondering if you have recommendations that are more complex than the cart-pole problem, but still do-able for starters.
* In the introductory chapter of your book, you mention that the book focuses on algorithms and not on environments or modeling problems. In my attempts, I found it very difficult to find a good description of the current state and to assign useful rewards. I assume this is part of the modeling problem. Do you have any recommendation on how to best do this?
* To compare different policies, would you compare estimates of the value function?
* In your experience, where and how would you start to debug if your model is not learning anything?
* I read in another book that DRL is currently sill very hard, especially to get consistent high-quality results and that you usually have to tune the hyperparameters quite a lot and often you also need a little bit of luck. Would you agree to this statement?
**Miguel Morales**
Hi, Tim Becker. Thanks for the questions!
* Yeah. The LunarLander environment is a bit more complex (8 variables for observation, 4 for action). You also have a Lunar Lander continuous with continuous variables for the action space. I recommend staying away from image-based environments until you have solved a couple of other environments with the same implementation. The problem with image-based environments is the feature extraction process can take quite a bit of time without the proper equipment (and even with the right equipment, it takes much longer than non-image-based environments.)
* Unfortunately, this is a challenging problem, usually left out because it is not as “researchy” but more of an engineering problem. However, for most folks, this is really where the money goes. You may not be able to invent a new RL algorithm, and maybe you’re not even interested in that. Instead, you may want to use one of the available algorithms and train them to solve your problem. Modeling problems (as you say: “assigning useful rewards”) is essential, and someone should spend some time creating a book or some content that explains how to do that. Sadly, I don’t have any recommendations at the moment, only to study how MDPs work (chapter 2) and try to replicate, add complexity. Feel free to explore how others do it. Look for `gym <search term> environment` and dig into codebases (e.g. atari-py, mujuco-py, hfo-py, etc.)
* To compare policies in practice, you would evaluate them in environments under similar conditions with several random seeds. Capture the mean total return (sum of rewards from the initial state to the end of the episode, averaged x times, over n seeds), and go from there. You can monitor the accuracy of a value function by comparing the estimates to the actual returns. But on a policy-to-policy comparison, I commonly use the sum of rewards.
* This is sooo challenging. What I always try to do is to simplify the problem. As opposed to debugging a complex system, simplify the solution. Train against a simpler environment. Use standard hyperparameters for the type of environment that you’re using. RL is very challenging to debug because a single typo can break things. And, what is probably worse in my opining, an implementation with a bug can “work” under certain conditions. That’s cruel, let me tell ya! :)
* I somewhat agree. Many folks attribute their incompetence to RL and use the phrase: “RL is hard” to excuse themselves. Yes, I believe RL is challenging, but more often than not, it is not working due to user error. When you look at implementations available online many out there are flat-out wrong (including in books, BTW–sadly). But, if you start small and build up, then it is not that bad. You need tenacity and focus, but you’ll have so much fun! Hyperparameter tuning is not that bad once you learn what parameters work for certain kinds of environments. I recommend using a hyperparameter tuning framework and trying a random search. RLlib and tune are excellent starting points.
Thanks for the great questions. I didn’t do justice to them, but it hopefully helps!
**Tim Becker**
Thank you Miguel! This helps a lot! As you expected, I was looking at image-based environments that gave me headaches. The LunarLander sound like fun 🙂 I will give it a try.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Governance: The Definitive Guide – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Governance: The Definitive Guide
-------------------------------------
#### by Evren Eryurek, [Uri Gilad](https://datatalks.club/people/urigilad.html)
, Valliappa Lakshmanan, Anita Kibunguchy-Grant, [Jessi Ashdown](https://datatalks.club/people/jessiashdown.html)
##### The book of the week from 24 May 2021 to 28 May 2021
As you move data to the cloud, you need to consider a comprehensive approach to data governance, along with well-defined and agreed-upon policies to ensure your organization meets compliance requirements. Data governance incorporates the ways people, processes, and technology work together to ensure data is trustworthy and can be used effectively. This practical guide shows you how to effectively implement and scale data governance throughout your organization.
* [Book's page](https://www.oreilly.com/library/view/data-governance-the/9781492063483/)
* [Amazon](https://www.amazon.com/_/dp/1492063495)
Questions and Answers
---------------------
**Phil Winder**
Hi there. Given that I haven’t read the book yet - I will - and I totally understand that this is out of scope, but I’m really interested in how governance can be applied to layers further up the stack. I think there’s a lot of lessons that could be applied to other areas of the stack, which is common in certain industries like finance, but non existent everywhere else.
Q: Given what you’ve learnt writing and experiencing this, where can these ideas best be applied next? Which part of the ML stack could benefit most?
**Uri Gilad**
I am not quite certain I am clear on what “layers up the stack” mean, but the topic of ML models is interesting. An ML model is both an executable (meaning it needs access to machine resources - CPU/RAM) and operates on data. The combination of Data Governance and Resource Governance becomes complex and interesting, but generally speaking, we tried to stick to broad concepts in this book. Great topic for a follow-up though!
**Phil Winder**
Sorry Uri Gilad I should have clarified but yes I think you get my drift. You’ve done data governance.
What about governance for:
* Features
* Training
* ETL
* Models
* Serving/inference
* Monitoring
* Etc.
This is what I mean by stack. MLOps terminology. Sorry.
So the question remains, from your experience writing this book, what do you think about governance all-the-things?
**Matthew Emerick**
Thanks for doing this! Is there really a major difference between top-down and bottom-up data governance? Which tends to be more effective?
**Uri Gilad**
I am taking “Top down governance” to mean pushing down policies from above. While there are benefits to centralized management care should be taken not to overwhelm individual analysts with rules, because they will find ways to “do their job faster”. I would prefer a “carrot” (no stick) approach in which you get access to MORE data under centrally managed policies
* AI Governance includes several topics discussed in the book such as lineage of where the data came from and how it is used in a model. There is a highly related topic concerning bias in AI which we did not approach in the book which focuses on how to make sure data used in AI does not introduce certain biases (many examples )
* Where do we see data governance going in the next five, ten years? - I believe it will become tablestakes for an organization to care not only about access to data but also about how it is used, and for what purpose. The book really talks about basic processes you can set, but its up to “corporate culture” to adopt the approaches
* “How effective is data governance? Does it really protect us individuals against maleficent usage of our data by corporations and governments?”
* data governance is not a “solve all” - it is a set of principles (and processes) you can put in place to make sure the data an analysts accesses is appropriate for their permissions and their task at hand. At the end of the day, you are trusting people and there are very few ways to prevent individuals from willful misuse (for example, you can always put a video camera in front of your monitor and scroll tables in front of it)
**David Cox**
Very excited to see this book come through! For individuals working in locations where they’re building this all from scratch, the totality of everything that needs to get done can be daunting. How do you recommend assessing and prioritizing tasks to build out mature data governance systems?
**Uri Gilad**
My recommendation would be to prioritize. Start with the big rocks, and focus on where “most data is” and proceed from there. While you do that, keep scaling up in mind so that you will be able to encompass more and more areas. There are multiple tools at hand (discussed in the book ) Some of those are already included with your Bigdata infrstructure
**Jessi Ashdown**
I would also add to begin with a few things and then fan out from there.
**Jessi Ashdown**
One thing we’ve seen work pretty well is to begin with selecting the most important data that you have to govern: remember it’s not only the “scary stuff” (like PII) but could also be data that is used to make business decisions
**Jessi Ashdown**
once you have identified a few of these “top categories” you can begin focusing on classifying/tagging/labeling and treating this data first
**Jessi Ashdown**
This is especially useful when you don’t have a lot of headcount to be mining through data, enriching it, curating it, etc.
**David Cox**
These are all very helpful comments! Thanks Uri Gilad and Jessi Ashdown!
**Kyle Shannon**
❓ In your experience what are some of the ways you were able to convince executive level to understand the business value of governance and feel the need to invest?
**Uri Gilad**
The low hanging fruits data governance solves are
“having analysts spend less time on finding data, and more time on extracting value from data” and in addition, “being able to tell who has access to what”
If those two are not already a problem n your org, you are in a very good place already!
**Kyle Shannon**
Thanks, as a follow up to the second low hanging 🍎 . If there isn’t concern or understanding about the value of who has access to what, how would you get engagement in that conversation?
**Uri Gilad**
many organizations are liable to regulation such as GDPR which mandates (not explicitly) understanding of this sort.
Smaller organizations which are not liable, or alternatively have a “everyone has access to anything” should still set the ground rules in a way which eventually, once they scale up - complying with such regulation becomes easier
**Kyle Shannon**
Thanks Uri, I’m 💯 with you, I’m curious if you have any tips or pointers on how to show the business value to setup now for the future or does it typically come down to when they need the regulations it gets called into action
**Neal Lathia**
❔ What are the first things that companies should think about when starting out with data governance?
**Jessi Ashdown**
One of the first places to start is to identify what data you should start to govern first and what kind of headcount you have that can work on this effort. You cannot govern what you don’t know, so you must begin with identifying (at least) two classes of data: sensitive stuff and then the “valuable stuff” (data that help the business make better decisions). Once you’ve identified these classes you can begin to categorize and enrich the data that falls into these classes. If you have the headcount to do more than this - great - but when you’re starting out you likely don’t. You likely have a few people who are having to do this work ON TOP of their other daily tasks.
**Jessi Ashdown**
The second thing is to identify who is doing what. This is often a piece that gets overlooked and when it does it becomes a free-for-all with tasks not being done well, or worse, not at all. Even if you only have a few people who are taking on governance tasks make it clear who is responsible for what part of the process.
**Neal Lathia**
Thank you!
**Jessi Ashdown**
you are so welcome!!
**Bayram Kapti**
Hi Uri Gilad & Jessi Ashdown! Thanks fortaking the time to answer our questions.
Do data lineage, data cataloguing (data dictionary), data security & user access management go under Data Governance? If you’re just starting to take these steps in an organization, what’s the recommended order to start things up? And do you cover these in your book?
**Uri Gilad**
Hi Bayram!
yes, we do cover these topics in the book. Generally speaking lineage/catalog/security/access managements are aspects of Data Governance.
I would recommend actually starting from the Data view - what data does your org manage, what are the requirements for said data (e.g. protect PII) and later finding the tools to help with that.
**Xavier Gumara Rigol**
Hi! Looking forward to reading the book in the next few weeks! 📖
I am curious to see your view on Data Governance initiatives implemented in small companies versus big organisations. Do you think it is more of a challenge in big corporations and _easier_ small companies?
**Jessi Ashdown**
Great question! I hate to say it but it depends 🙂 I’ll unpack that a little though - there are different factors that make it easier/more difficult and company size is only one of the them.
**Jessi Ashdown**
The kind of data and amount of it that’s sensitive effects the difficulty of governance regardless of if an organization is large vs. small. A small company may PRIMARILY deal with sensitive data which can make governance a lot more challenging vs a large company that only has a small percentage of sensitive data.
**Jessi Ashdown**
Also, on the one had smaller companies often have less employees so there is less complication with granting access controls, however, smaller headcount can also mean that less governance tasks overall are completed. Chapter 3 of the book goes into greater detail on these specific factors!
**Xavier Gumara Rigol**
Thanks! Good point on smaller headcount can also mean less governance tasks are completed 🤯
**Xavier Gumara Rigol**
When implementing a data governance initiative, we know it is usually hard to convince management to fund it properly.
In my experience, it is also hard to “convince” Data Engineers/Analysts/Scientists to invest in data governance initiatives (good documentation, working agreements between teams,…). How would you suggest to tackle investment in data governance initiatives in Individual Contributors that are not motivated so spend time on this topics?
**Jessi Ashdown**
Excellent point and insight (chapter 9 of the book goes into this as well!) One of the things we mention in the book is creating a data culture. There are many aspects of a successful culture of protection but you’re exactly right - motivation is key. One way we’ve seen this work well is in being really intentional about how you divide governance tasks. Often Data Engineers/Scientists/Analysts are not motivated because too many tasks fall onto them. When you define a process and identify who is responsible for what it makes the governance strategy less ambiguous. Another thing we’ve also seen work is to have ongoing “training” that is _relevant_ to folks. So, perhaps there is a Governance Week that includes daily 1hr trainings that apply to these roles - such as how gov tools you currently have can help them do their job better, or a training around something that is very relevant to governance that’s been in the news. These sorts of things tap into the psychology of folks connecting these tasks to themselves - and when they can see the benefit and aren’t overwhelmed they’re far more likely to follow through.
**Xavier Gumara Rigol**
What are the most important things to formalise first when starting a Data Governance initiative?
**Jessi Ashdown**
First and foremost it’s really important to identify where your company is in its governance journey to begin with: do you know what all your data is? Do you know _where_ all of it is? Do you know who has access to what? This is the first thing to assess. Most organizations will answer “no” or “sort of” to the above and in that case they must first begin with what data they need to govern _now_. In general this will be the “scary stuff” (PII, etc.), and the “business critical stuff” (the key data that helps the company to make data driven biz decisions). This is where to start as trying to govern everything all at once is incredibly overwhelming! The next step is to then think about the process: you’ve identified what needs to be governed, now you need to identify _how_ you’re going to do that. Will you use tools? Tagging? Labeling? Just moving data into different storage areas? You need to define the process and then the _tasks_ that involved in this process (note: if you have minimal budget and/or headcount try to identify the bare minimum here). Then (and this is super important) you need to define _who_ is going to do those tasks. And to your point above, it can help to divide these tasks up. But the key here is to make it VERY clear who is responsible for what tasks - and have a way to check on this; make sure they get done.
**Xavier Gumara Rigol**
Thanks a lot for your answers! Looking forward to reading the book!
**Jessi Ashdown**
You’re so welcome! 🙂
**Alexey Grigorev**
Maybe it’s a silly question, but what is actually data governance? why should we care about it?
**Jessi Ashdown**
Actually a great question! The way we’ve defined gov in the book is really looking at the policies and procedures that maximize the utilization of data while also ensuring data quality, security, and regulatory compliance. So - it’s more than just security or how to conduct access control - which is often how governance is defined. So in this way we should care about it because it’s the process by which you can make your data _useful -_ in terms of being able to derive insights from it while also protecting it.
**Alexey Grigorev**
Do you have an example of such policy from the top of your mind?
**Alexey Grigorev**
Something like “All data must go to this database and must be documented in this spreadsheet”?
**Jessi Ashdown**
Yes, that’s def an example of one such policy. Another could be: all data from 3rd party vendors go into XX storage. Or credit card numbers are only retained for 30 days… or even, sales, income, and roi all are labeled “revenue”
**Alexey Grigorev**
That’s clear now! Thank you!
**Alexey Grigorev**
If I work as a data scientist, what are the main advantages of implementing it for me?
What about analysts and data engineers? Why should they care about it?
**Jessi Ashdown**
We have a chapter that goes into depth both about the people/process as well as the culture, but in short, governance needs to be a well thought out program with strategic process. That doesn’t mean that it has to be super complicated and require an inordinate amount of headcount, but it DOES need to be intentional. And part of being intentional is to define _who_ does _what._ Not only does this help to ensure tasks are done, but it also gives shared responsibility and ownership. When someone has ownership of a task they are far more likely to see how they personally benefit from it. And it should probably go without saying, but as stated above: a gov program is not just about securing data - it’s also about making it useful - and any user of data (analyst, scientist, engineer..) wants to have data that is useful!
**Alexey Grigorev**
And another one - who should actually be driving it? Analysts, data engineers, or data product managers?
**Jessi Ashdown**
oof! Loaded question 😉 we go into more depth in the book but I will say that it shouldn’t be one group - it should be bottom up, top down, and part of a company culture.
**Alexey Grigorev**
I have an even more loaded follow up - how to make it a part of company culture? 🙂
**Jessi Ashdown**
More loaded indeed! Well, one of the things we’ve seen work well is an actual intention to _make_ it part of company culture. So things like setting up an intentional strategy, defining tasks, designating roles…all that is really important. But the special sauce is really embedding it from the top down and engaging from the bottom up. So what I mean by top down is for there to be intention and focus around doing _ongoing_ training. Now - most will define a gov strategy, do a training, and call it a day. But there needs to be continuous training. And this doesn’t have to be super intense or time consuming - could even be a 1hr training around “new gov tools” or a guest speaker, or even “YIKES! this happened in the news - how do we make sure it doesn’t happen to us”. The point, really, is that it’s even thought of at all and given at least a little bit of focus. And from bottom up I mean that this is where employees need to be engaged; do a training on a topic related to gov they care about or that actually helps them do their jobs better. Don’t just send out the annual click thru and check the box. When you engage the employees they help to facilitate the culture and it becomes symbiotic .
**Alexey Grigorev**
Maybe the last one. Let’s say we implemented it in our organization. How can we measure the effectiveness of it?
**Jessi Ashdown**
Chapter 8 goes into monitoring, specifically, but I think of the key points is that you cannot measure what you haven’t pre-defined. So, a successful gov program needs to be intentional, with specific processes and tasks, and people who will do them (note: this doesn’t mean complicated) - when you have mapped out your strategy you can then track it over time. So, you defined these tasks - are they getting done? Let’s say part of your strategy was to only keep PII data in one particular storage location - is it? In short, the way that you define your strategy will help to inform what you’ll need to measure in order to know how it’s going.
**Ricky McMaster**
Not sure if you’re still taking questions, but if so: do you think data governance has generally deteriorated in recent years? Or perhaps is taken less seriously?
**Jessi Ashdown**
I certainly don’t get that sense. For the past 3yrs I have talked to many, many different organizations about their data management and data governance strategies/programs/needs and it seems that it’s only becoming MORE top of mind as not only there are increasingly stricter regulations to comply with but also the amount of data companies now collect is higher. And the reason to collect that data is to derive insights from it which organizations know cannot be done (or at least done well) without some sort of governance process in place.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Building Machine Learning Pipelines – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Building Machine Learning Pipelines
-----------------------------------
#### by [Hannes Hapke](https://datatalks.club/people/hanneshapke.html)
, Catherine Nelson
##### The book of the week from 07 Jun 2021 to 11 Jun 2021
Companies are spending billions on machine learning projects, but it’s money wasted if the models can’t be deployed effectively. In this book, Hannes Hapke and Catherine Nelson walk you through the steps of automating a machine learning pipeline using the TensorFlow ecosystem. You’ll learn the techniques and tools that will cut deployment time from days to minutes, so that you can focus on developing new models rather than maintaining legacy systems.
* [Book's page](https://www.oreilly.com/library/view/building-machine-learning/9781492053187/)
* [Book's website](https://www.buildingmlpipelines.com/)
Questions and Answers
---------------------
**Sricharan**
Thanks for doing this Hannes Hapke.
Are CPUs predominant for powering inference services today?
**Hannes Hapke**
Hi Sricharan, I think the answer depends on the model architecture. But I think it is true in most cases. At [digits.com](http://digits.com/)
we have deployed a number of transformer models, and we have optimized them to such an extended that the latency is decent on a CPU instance.
**Sricharan**
What are your thoughts on inference chips like Google Coral or Jetson Nano?
**Hannes Hapke**
I haven’t really seen consumer facing applications since the users always need the device. That changes in an instance as soon as a phone contains a coral chip.
**Sricharan**
What are your thoughts on building and deploying continuous training models at scale?
**Hannes Hapke**
A must for scalable ML projects. 🙂
**Sricharan**
What inspired you to write this book? Do you see a future where ML deployment/training a part of “full-stack” software engineer portfolio?
**Hannes Hapke**
When I wrote the initial proposal, I was missing an overarching toolchain for MLOps. Luckily that developed during the course of the book writing with the most promising contender TFX.
ML part of “full-stack” software engineer? I don’t think so. The term “full-stack” software engineer is already overloaded, I doubt we can squeeze in another definition. However, I would like to see that more domain experts (regardless of the field) apply more ML. The standardization of ML will help with the adoption.
**Alper Demirel**
Thank you so much for being here Hannes Hapke.
What do you think are the biggest challenges a junior machine learning engineer or intern will face when building a machine learning pipeline? What should they pay attention to?
**Hannes Hapke**
biggest challenge: Seeing an ML project end-to-end.
**Hannes Hapke**
From the data investigation, model architecture selection, validation, deployment, feedback loop (last one often missed).
**Alper Demirel**
thank you for your answers, sir
**Hannes Hapke**
My pleasure!
**Lalit Pagaria**
Thank you Hannes Hapke for being here.
I see very tight coupling between training and serving pipeline. Most of ML pipeline concentrate more towards training side in comparison to serving side. But real user of Model are served by serving pipeline. What is your opinion about completely delinking these two pipelines? and what do suggest towards building scalable serving pipeline?
**Hannes Hapke**
Hi Lalit Pagaria I think the processes are already delinked. At last at Digits where I work. We produce model version candidates which come out of our pipelines. After a human review, those model get deployed via a 2nd automated process.
**Hannes Hapke**
To keep your pipelines scalable, I think the combination between Apache Beam and Kubernetes is key.
**Hannes Hapke**
Beam lets you scale all data heavy tasks. You can start off with a direct runner (runs within a k8s container) and you can later export the data heavy tasks to its own cluster (with Beam using Apache Flink or GCP Dataflow)
**Lalit Pagaria**
Thank you Hannes Hapke
I will explore this idea of using Beam’s Pipeline IO for my project [Obsei](https://github.com/lalitpagaria/obsei)
.
**Hannes Hapke**
Cool Project! Thank you for sharing.
**Shankar Somayajula**
Hi Hannes Hapke Thanks a lot for taking questions.
Most DS work seems to occur in python but what role can SQL play in the execution/orchestration of ML Pipelines especially within Tensorflow ecosystem?
Also cant we use SQL in Model Analysis … A lot of the Fairness/Checks can be performed using sql. Analyzing the performance of a model is akin to performing BI tool actions to compare actual and prediction data sets. Its similar to comparing Sales or Marketing Campaign performance for 2 applicable products.
**Hannes Hapke**
Hi Shankar Somayajula
SQL is very useful for building the data ingestion in ML pipelines.
SQL for model validation? I think this only works if the model is deployed in some way. The results would be stored in the db and then analyzed with SQL. But the model validation should happen before a model deployment, therefore SQL isn’t a great tool here. If we do it in Python, we don’t have to store the validation results in a db and can keep it memory.
**Shankar Somayajula**
Hannes Hapke Some follow up questions. Hope you dont mind :)
Sorry, I meant sql for purpose of Model Monitoring post deployment to catch things like data drift, concept drift, model internal patterns becoming stale etc, not Model Validation (during model build). Yes, i agree this would require that all scoring details have to be stored in the database.
Regd sql usage in ML Pipelines, i guess you mean that sql part of pipeline exists prior to Tensorflow coming into the pic. Do you know of any cases of Tensorflow doing its magic (so to say :D) and sql doing some analytics post TF? Is that a rarity, python usually does the post processing, if any.
Background/Explainer: We use In-Database ML in our company, so sql is the go to tool even for ML tasks.
**Hannes Hapke**
In-Database ML seems to be great if the models are smallish. I haven’t seen such a setup for larger language models (might be a minority anyway).
**Hannes Hapke**
We use BigQuery very heavily at Digits and therefore use SQL for analytics of the deployed models
**Shankar Somayajula**
Thanks for the info. Personally I dont know much about NLP… Our In-Database usecases are more traditional ML usecases like classification, regression, anomaly detection etc. I work for Oracle so the technology is based on Oracle Db and Oracle Cloud Tech.
**Rona Ainslie**
Hi, what do you see as the main differences between deep learning and machine learning? I’ve never used tensorFlow, but I have used a little sklearn to create neural networks (eg MLPClassifier) is that still machine learning or does it cross into deep learning?
**Hannes Hapke**
Hi Rona Ainslie Deep learning is subset of ML where model are “deeper” meaning they contain more layers to capture more complex relationships. I think the boudary between traditional ML and deep learning are fuzzy. If you are interested in MLOps with Scikit learn, take a look at this example below on how to use TensorFlow Extended for Scikit Learn models: [penguin\_pipeline\_sklearn\_local.py](https://github.com/tensorflow/tfx/blob/master/tfx/examples/penguin/experimental/penguin_pipeline_sklearn_local.py)
**Rona Ainslie**
Brilliant, that looks very interesting. Thanks
**Bayram Kapti**
Hi Hannes Hapke,
What are the roles in a team that supports building ML pipeline?
How do MLEs, DEs & DSs work together to build a ML pipeline?
Who manages this process? DS Lead, ML Engineer Lead, DE Lead?
**Hannes Hapke**
The roles are often defined differently from company to company. In the context at [Digits.com](http://digits.com/)
, ML Engineers work hand in hand with Data engineers. For example, data engineers are the experts to provide BigQuery queries for data ingestion component. ML Engineers then take it from there and build the down stream components.
**Hannes Hapke**
Q: Who manages this process? DS Lead, ML Engineer Lead, DE Lead?
Honestly, I haven’t seen a place where all three roles come together.
**Bayram Kapti**
And as a follow up, do you discuss these in your book? Thanks!
**Hannes Hapke**
We discuss the aspects of the different roles in Ch 1
**Dr Abdulrahman Baqais**
Hi Hannes Hapke. Thanks for your effort in putting your experience in this book. Very interesting book and very valuable for team leads. Couple of questions:
1) Do you think having an automated process is a necessary step or a luxurious one for data teams?
2) Shall an automated pipeline be only enforced in mature teams at big organizations or even startup should consider it?
3) Automated pipeline usually is discussed at modeling and deployment….should not that be extended to project scoping , business analysis and data gathering and customer satisfaction. So then we havean overall operating model for the whole process?
4) Can we have the same pipeline for all types of analytics: Insights, ML, DL, RL? Or DL & RL might demand different requirements?
5) Can automated ML tools from Datarobot, Google and Amazon helps in setting up the right pipeline for new teams? Or an expert ML must be there to design and operate?.
**Hannes Hapke**
1. For long term projects a necessity
2. Startups with a solid product market fit will tremendously benefit from the standardization around the ML practices. Our data team at Digits doesn’t have the size of Amazon’s 🙂
**Hannes Hapke**
3) MLOps is focusing on the continuous “production” of ML model versions for a given model. I think the tasks you mentioned are important, but I think they are part of the job of a ML product manager or data scientist. The project scoping wouldn’t change with every new model version. It can be (and should be) set before the project starts.
**Hannes Hapke**
4) I don’t think that insights require ML pipelines, but a proper data engineering work flow,. Here, we aren’t training a model which needs to be validated and deployed.
ML Pipelines generally focus on ML and DL models.
I haven’t seen a pipeline for RL models. But that doesn’t mean that a pipeline couldn’t be used. I just haven’t seen a proper production RL model so far (except for hand crafted, very specific solutions).
**Hannes Hapke**
5) I think products like Datarobot provide the same thinking behind their products. Last time I checked Datarobot was only focused on ML models. Whoever operates Datarobot, Google, etc. on your team needs to understand the implications of the ML model and investigate it before deploying it.
**Dr Abdulrahman Baqais**
Thank you much for this elaborated answers. Indeed ML pipeline is necessary for data teams.
**Saskia Kutz**
Hi Hannes Hapke. The description of your book sounds very interesting. How well do I need to know Tensorflow (and further prior knowledge) to understand your book?
**Hannes Hapke**
Hi Saskia Kutz,
We have focused on the TF ecosystem because it provides the most holistic toolset for ML Engineering. Some chapters are framework agnostic (e.g. TensorFlow Data Validation can be used with Scikit learn or PyTorch models). Since we published the book, TFX improved further to be used with scikit, JAX, or PyTorch models. See please [penguin\_utils\_sklearn.py](https://github.com/tensorflow/tfx/blob/master/tfx/examples/penguin/experimental/penguin_utils_sklearn.py)
for an example. Furthermore, the book introduces a variety of concepts (e.g. model feedback loops, ML Privacy) which transfer well to other frameworks.
Viele Gruesse,
Hannes
**Ricky McMaster**
Thanks a lot for doing this Hannes Hapke
Given the greater scope/need for scalability in contemporary ML pipelines, do you see there being increased chances for bias/prejudices in model development, as a result of the need for greater automation?
Or does the fact that there is increased role specification (e.g. ML engineers were fairly rare a few years ago) help to mitigate this, so that this issue could be more formally addressed in the development lifecycle?
Do you discuss this in your book?
**Hannes Hapke**
Hi Ricky McMaster,
Re: Bias due to automation - Yes and no. Let me explain. if the training is just automated and “forgotten”, it will (most likely) lead to a bias in the models. But if the pipelines are well set up (e.g. TFMA is cont. tuned), the feedback loops continuously analyzed and the model reviewed by a data scientist before the final deployment, then I think it can contribute to less bias in a model. Why? All model versions are stepping through the exact same steps - basically being treated equally and a human can’t cut corners.
**Ricky McMaster**
Thanks Hannes Hapke. That’s a really good point about TFMA, I’ll bear that in mind. But in essence I guess you’re saying it’s the same as it ever was - you need to have proper processes and standards in place to counteract the risks.
**Hannes Hapke**
Yes, it is important to look for the Unknown unknowns.
**Hannes Hapke**
as much as possible
**Ricky McMaster**
🤝
**Ricky McMaster**
I really wish it wasn’t Donald Rumsfeld that coined this phrase… since it’s a good one
**Hannes Hapke**
I totally hear you. Was he the first one using the terminology?
**Ricky McMaster**
I thought so! But it seems (thankfully) that he got it from NASA: [https://en.wikipedia.org/wiki/There\_are\_known\_knowns](https://en.wikipedia.org/wiki/There_are_known_knowns)
**Kyle Shannon**
❓ Hey Hannes Hapke, thanks for contributing and sharing you knowledge with the data community. What does a healthy evolution of an ML product look like when trying to POC and iterate? What would be some key things learned to make sure to take care of earlier or wait until later to do?
**Hannes Hapke**
Hi Kyle Shannon ,
**Hannes Hapke**
Great question! We are treating an ML project in two phases. First, solve a real business with a model proof of concept. This could be based on a limited dataset, in a Jupyter notebook, and with experimental code. Only when the project stakeholders see the value in the ML model and you can show that it is solving the business problem, then focus on the pipeline development. From that model on try to keep the model architecture constant, increase the datasets and automate as much as you can. It will be helpful in the long run (e.g. in 6 months when you want to update the model architecture or update the model validation). Pipeline code is generally cleaner than an experimental Jupyter notebook.
**Kyle Shannon**
Thanks Hannes Hapke that makes a ton of sense. As a follow up, how do you typically monitor and maintain performance for these models as they evolve once pipelined into production?
**Hannes Hapke**
We have built a ton of custom code at Digits since I haven’t found a managed tool for that purpose which checked all boxes. It all ties back to model feedback loops to check if the model predictions performed as the users expected.
**Kyle Shannon**
What kinds of challenges do you think prevent a tool like this from existing? or is it just a matter of time?
**Matthew Emerick**
When do you think ML pipelines should be learned in relation to learning ML itself?
**Hannes Hapke**
Hi Matthew Emerick I think it depends where the person is coming from. For a DevOps person, it could be their entry point to the ML world, esp. if they have data engineering experience. Data scientists would probably first learn ML and then focus on the productization of their models.
**Matthew Emerick**
Do you any ideas for another book?
**Hannes Hapke**
A variety of good books was released recently:
* Machine Learning Design Patterns (really love this book)
* Kubeflow Operations Guide
* Kubeflow for Machine Learning
All publications provide a good ML Eng overview.
**Matthew Emerick**
I meant that you would write. 🙂
**Hannes Hapke**
Oh, haha. Yes, TBA :)
**Matthew Emerick**
What do you of the use of pipelines for AI in general?
**Hannes Hapke**
I am not sure what you mean. Do you mind rephrasing your question? Thank you.
**Matthew Emerick**
Do you see a use for pipeline for other AI techniques such as genetic algorithms, decision trees, expert systems, cognitive architectures, multiagent systems, etc?
**Hannes Hapke**
I think the concepts apply to all trained models.
**Shankar Somayajula**
Hannes Hapke What do you think of this potential advantage of implementing pipelines via sql? Needless to add, i’m a sql addict :D
If we’re able to consolidate the entire pipeline processing into a (complicated, multi-stage) sql then it’s possible to inject some flexibility into the process by deferring config/ETL (or ELT) decisions to runtime and allowing the users freedom to experiment with what suits them. I understand that this wont scale well but many big data use cases start big but do devolve into small/normal data by the time it gets into the Data Scientists ambit once the scope of the analysis is defined and implemented.
E.g: One may want to see the profile of Customers who buy or responded to a mail in campaign dealing with two products sold as part of a promotion in a region/country/state like Aus or NZ (say)… once we establish the necessary scoping filters of analysis this could eminently be in the realm of sql based analysis. Now if we are able to do this entire pipeline of analysis via sql (say, db views for data prep to cover the etl ot elt part) then via a BI tool we can give freedom to the analyst to decide the time period between which the two products could have been bought (perhaps same customer bought the two products on different days 8 days apart, i.e. not in same transaction … this can be a hit or a miss depending on analyst/business user discretion/decision to consider 8 days separation between events as ok or not).
An interactive UI can allow user to leverage parameterized sql by having a sliding bar from 1 (same day) to 10 days and see the effect of the setting on key Campaign Metrics. If we did this in the ETL or ELT cycle or in the pipeline then that decision is baked in at say 7 days and needs a new instance of the pipeline to modify the same from 7 to 10. Business users need to use Notebooks or similar tools to modify the setting. Does the what-if tool allow for such flexibility? I know it gives flexibility in the analysis part, the data processing part which is post load. Can it reach through to give some flxibility/benefit pertaining to the load/transform part of the pipeline too?
**Hannes Hapke**
Hi Shankar Somayajula. The What if tool is designed for post training model analysis. It helps you perform a model sensitivity analysis. I think you can filter your data to test your model with, but only in a limit fashion. I don’t think the WIT can use the feature engineering from TFTransform. Having said this, TFMA wasn’t able to use it until recently too, so maybe the tool is already able to do so. But I think you are looking for a different tool, IMO.
**Shankar Somayajula**
Thanks for the response. Hannes Hapke
**ankush khanna**
Hi Hannes Hapke
How is building ml pipeline different from building data pipelines? And what can a data engineer focused on building ETL pipeline move to building ML pipelines?
**Hannes Hapke**
I think they share a variety of properties, for example scalability, task chaining, DAG, etc.
At the same time, they are also very different. For example, in a data pipeline, your main concern is the final outcome (e.g. a transformed dataset). You probably don’t want to store snapshots of the artifacts from the individual steps. But this is critical for ML pipelines to repeat and reproduce ML models.
**Hannes Hapke**
Hi ankush khanna 🙂
**ankush khanna**
Thanks for the answer. What can be some focus points for data engineers to learn more about ML pipeline?
Any open source solutions, tools, languages, you suggest?
**Hannes Hapke**
I think TFX does a god job at introducing the necessary steps in ML pipelines. From there, you could take a look at the orchestration of such pipelines.
**Clara Matos**
Hi Hannes Hapke 👋
Thank you for taking the time to answer questions!
During the data ingestion stage where can we draw the line between what is a data pipeline and what is feature engineering? When using tabular data from different sources (such as different tables in a data warehouse or relational database) is combining the data part of the data pipeline? If so from your experience is it commonly performed by data engineers? And then the machine learning engineers perform feature engineering on top of the final dataset?
When should the validation take place? On top of the final dataset or on top of each table used to build the final dataset?
From your perspective what is the best tool/approach to building a good (and scalable) data pipeline to feed a machine learning model? How do you address training-serving skew? (When serving, the incoming data should go through the same data pipeline as during training?)
**Hannes Hapke**
Hi Clara Matos, I am sorry for my belated reply.
**Hannes Hapke**
what is a data pipeline and what is feature engineering?
I would transform any data in a data pipeline which can be reused by other models. Model specific transformations belong in the feature engineering in my opinion.
**Hannes Hapke**
Q: If so from your experience is it commonly performed by data engineers? And then the machine learning engineers perform feature engineering on top of the final dataset?
A: I think so, however, my colleagues at Digits are going back and forth. Data engineerings transition to ML projects and vice versa.
**Hannes Hapke**
Q: When should the validation take place? On top of the final dataset or on top of each table used to build the final dataset?
A: I would perform it on the final dataset. Why? Because you want to capture all stats/schema changes in respect to the model.
Checking each table seems a bit more like a data quality task to me. What do you think?
**Hannes Hapke**
Q: How do you address training-serving skew?
A: TFTransform 🙂 I always try to build the feature engineering on top / inside of the actual model
**Hannes Hapke**
TFTransform is really amazing and a great help for our team.
**Hannes Hapke**
Q: From your perspective what is the best tool/approach to building a good (and scalable) data pipeline to feed a machine learning model?
A: We are using Beam for our data and ML pipelines
**Clara Matos**
_Checking each table seems a bit more like a data quality task to me. What do you think?_
I see what you mean. Do you think tfx data validation is also suited for data quality checks in data pipelines?
**Clara Matos**
Also, have you ever integrated scikit-learn models within a tfx pipeline?
**Clara Matos**
and thank you for taking the time to answer the questions 😃
**Hannes Hapke**
Anyone interested in the internals of TFX, join the special interest discussions here: [https://github.com/tensorflow/tfx-addons](https://github.com/tensorflow/tfx-addons)
**Hannes Hapke**
We have a mailing list and bi-weekly meetings to chat about TFX, and addon components. The discussion is open to anyone, beginners (very welcome) or experts of TFX
**Tim Becker**
Hi Hannes Hapke I just finished chapter 2 of your book and it seems to be very useful. As you discuss in the book, so far, I have been creating my own custom solutions. Using TFX seems to be much easier than I initially thought. I have been a little hesitant to look at libraries for pipelines. I thought, it is difficult to practice it on my own, because I need datasets that are continuously updated, and it might be much more work than the usual data science project. But I was probably wrong, and I would like to ask you some related question:
* Do you have ideas for toy projects that have a reasonable size with data that is regularly updated?
* If you already have a model, which would be the parts of the pipeline you would focus on first? In your experience, what is the most crucial part of the pipeline. Where does a data science team benefits the most?
* If you build, for example, a model for stock trading, which metrics would you use to monitor that your model is still performing? Do you monitor your return? Or the error? Maybe, the drift of the error?
Thank you very much!
**Hannes Hapke**
Hi Tim Becker
* Q: Do you have ideas for toy projects that have a reasonable size with data that is regularly updated?
A: I would check for some time series data, e.g. stock prices. There are probably free sources which are updated daily. This would be a great way of showing the full benefit of Ml Pipelines
**Hannes Hapke**
* Q: If you already have a model, which would be the parts of the pipeline you would focus on first? In your experience, what is the most crucial part of the pipeline. Where does a data science team benefits the most?
A: I would start with the data validation (e.g. with TFDV or Great expectations). It is easy to use, and immediately provides great value. Data scientists will probably find interesting snippets about their data sets with such tools. Then I would on the model analysis part to compare model versions. Once the data scientists see the value in such models, focus on the automation of the entire end-to-end process. Here you could use TFX for.
**Hannes Hapke**
* Q: If you build, for example, a model for stock trading, which metrics would you use to monitor that your model is still performing? Do you monitor your return? Or the error? Maybe, the drift of the error?
A: The stock trading problem sounds like a time series forecasting issue to me. In such a case, I would pay attention to the mean and std, calculate it for a window of 30 days. check if your std is changing drastically. For classification problems, I would pay attention to the distribution of the your labels. At Digits, we do monitor the predictions and the feedback very closely. TFDV can help you to calculate the L-0 norm as an option to detect if your data moved more than X%
**Tim Becker**
Hannes Hapke thank you very much! I will definitely give it a try 🙂
**Hannes Hapke**
Let me know how it goes.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Advanced Algorithms and Data Structures – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Advanced Algorithms and Data Structures
---------------------------------------
#### by [Marcello La Rocca](https://datatalks.club/people/marcellolarocca.html)
##### The book of the week from 31 May 2021 to 04 Jun 2021
As a software engineer, you’ll encounter countless programming challenges that initially seem confusing, difficult, or even impossible. Don’t despair! Many of these “new” problems already have well-established solutions. Advanced Algorithms and Data Structures teaches you powerful approaches to a wide range of tricky coding challenges that you can adapt and apply to your own applications. Providing a balanced blend of classic, advanced, and new algorithms, this practical guide upgrades your programming toolbox with new perspectives and hands-on techniques.
* [Book's page](https://www.manning.com/books/advanced-algorithms-and-data-structures)
* [Book's GitHub repository](https://github.com/mlarocca/AlgorithmsAndDataStructuresInAction)
Questions and Answers
---------------------
**Alex**
Hi there, Marcello La Rocca! Pleasure having you around here 😄
As an aspiring data scientist, I hear many different opinions regarding algos and data structures: some say they are really a must in order to succeed, and others say it is not fundamentally needed in order to carry out a decent job. What’s your opinion on this? Is it really that important to manage them well? Or can you decently make it without going too deep on the subject?
Thanks a lot!
**Rodney Silva**
I also had this doubt when I started to learn data science. It’s good to know the basics,but rarely someone will ask about algorithms during interviews.
**Marcello La Rocca**
Hi Alex, thanks, it’s a pleasure for me as well!
(Thanks a lot to Alexey Grigorev for inviting me!)
So, that’s a great question, thanks for asking!
The short version is, it depends on what you are working on and where.
**Marcello La Rocca**
I believe you strictly need some knowledge of the basis everywhere today, and then in some roles is naturally more likely than others to have to deal with algorithms in depth: anything related to research, any job where you have to deal with high volumes of data, there you are more likely to need some more-than-basic knowledge about algorithms and data structures; also in backend jobs it’s probably more likely than in frontend ones (although you should take this with a grain of salt, like al generalizations).
Orthogonally to those considerations: some companies, regardless of the position, have interview processes that are heavily based on questions on algorithms and data structures, so depending on what’s your target, you might want
to get to a certain level on the subject.
**Marcello La Rocca**
That said, I can also give you some reasons why (IMHO) studying about algorithms and data structures can can help any developer being “their best possible version”:
1. Performance: choosing the right algorithm can speed your application up dramatically: if we take something like search, on large volumes of data it makes a world of difference going from linear search to binary search to (possibly) Grover algorithm. Likewise, for a simpler task like adding elements to a list, if you do know that adding to the head or to the tail of the list changes the performance of the task, you are less likely to fall for mistakes that can slow down and crash you app (I’ve seen this several times in production, last time was not longer than a few months ago)
2. Security: if you choose the wrong algorithm, an attacker can use it to crash your server/node/application. Consider, for instance, the hash DoS attack, where the use of a hash table as a dictionary to store variables sent with POST requests was leveraged to overload it with a sequence causing a huge number of collisions, and this in turn would make a server unresponsive. Another interesting example was how flawed random number generators allowed hacking online poker sites.
3. Efficiency in designing code: if you already know there exist building blocks for whatever you’d like to accomplish, you will be faster in developing it, and the result will be cleaner (especially if you reuse code). For instance, if you know what caches are for and how to use them, you won’t likely need to write your own custom container, rather you’ll directly go and search for an existing library that does what you need.
**Alex**
awesome, thanks a lot for the reply! that being said, I understand that your book is focused on advanced algos and data structures. is there any book recommendation for non-technicals (I come from Economics) to get a good grasp on the subject first?
**Marcello La Rocca**
Morning Alex !
So, if you’d like to get started with algorithms, certainly _Grokking Algorithms_ can be a good starting point.
Besides that, it really depends on what you’d like to learn, and what kind of algorithms you’d be interested in… can you tell me a little more?
Like, for instance, would you be interested in machine learning, or algorithms to process data, or networks… you tell me 🙂
**Alex**
Awesome! Will add that one to my collection as well 😄
In regards to what kind of algorithms, I’d say pure CS algorithms related to data processing would be great given my job requirements.
Thanks again, Marcello La Rocca!
**Marcello La Rocca**
Hi Alex, sorry for the delay, I wanted to search out some links that I wanted to send you.
So, if you are interested in data-oriented algorithms, I’d suggest [https://www.manning.com/books/algorithms-and-data-structures-for-massive-datasets](https://www.manning.com/books/algorithms-and-data-structures-for-massive-datasets)
If you are also interested in the basics of Machine Learning, don’t miss [https://www.manning.com/books/grokking-machine-learning](https://www.manning.com/books/grokking-machine-learning)
Both are great books!
But if you also would like to have an overview focused on the non-technical angle, I wanted to suggest you might take a look at these MOOCs:
[https://www.edx.org/course/artificial-intelligence-for-everyone](https://www.edx.org/course/artificial-intelligence-for-everyone)
[https://mit-online.getsmarter.com/presentations/lp/mit-machine-learning-online-short-course](https://mit-online.getsmarter.com/presentations/lp/mit-machine-learning-online-short-course)
Is that something that might interest you? Meanwhile, I’ll try to think harder about other resources I could recommend to you!
**Alex**
Thanks a lot, Marcello! I really appreciate the reply. Will give them a look for sure :) have a lovely day!
**Marcello La Rocca**
Nice!
Thanks, you too!
**Tino**
Hey Marcello La Rocca 🙂 Do you see a trend in interview questions which refer to algos and data structures? I heard Google e.g. relies heavily on this whereas other companies less. So should advanced knowledge be interviewed by more companies? And does this help to understand pre-implemented classes/algorithms like in scitkit-learn, etc. easier?
**Marcello La Rocca**
Hi Tino, thanks for your question!
Yes, definitely, many large companies rely, for their interview process, on algorithms (and in general on challenge-like questions, the kind of questions you can find on codewars etc., just to be clear).
I myself had my fair share of algorithmic questions when I interviewed with the likes of Google and Twitter! There are also many other companies (including [tundra.com](http://tundra.com/)
, where I currently work) that tries different ways, balancing algorithmic questions with questions closer to the daily work that candidates would have to perform in case they joined the company.
**Marcello La Rocca**
In general it depends a lot on where you are applying, so my best advice is to target your preparation on the job/company you are applying
(incidentally, I’m writing a piece with some advice for interviews on [https://dev.to/mlarocca](https://dev.to/mlarocca)
, hopefully it will be out next weekend, if you’d like to take a look)
**Marcello La Rocca**
As for the second part of your question: knowing the inside out of algorithms in libraries it’s not strictly necessary, but it can help you to decide better which algorithm to apply, and to understand better what to expect in terms of final result, but also resources needed.
For instance, especially with stochastic algorithms and heuristics, it’s important to understand if you can expect to get always the best possible result, or even a correct result, and how much resources you’ll need (usually compared to other alternatives)
**Tino**
Hey Marcello La Rocca that is amazing! Thanks so much for the insights 🙂
**Marcello La Rocca**
You are welcome! 🙂
**Agrita Ga**
Hei Marcello La Rocca. 👏 Are there any algorithms or data structures - in your opinion - that are underused or underappreciated? Or, then as well, algorithms/data structures that are heavily misused or overused (due to buzz or just because of familiarity)?
**Marcello La Rocca**
Hi! This is a very good question, really interesting. Let me give it some proper thought, and I’ll try to answer it at my best at the end of the round🙂
**Marcello La Rocca**
So, here we are: this is probably one of the most challenging questions for today, so I left it for last! Thanks a lot Agrita Ga for asking!
Let’s see, the first part: _underused algos/DSs._
For sure, I’d argue that there are many of the simplest ones that could be used more and improve performance, while sometimes they are just overlooked.
For instance:
* The binary search algorithm can provide a huge speedup in search if the collection to search is stable and can be ordered once, but often this is just overlooked and linear search on an unordered set is used.
* Speaking of, adaptive sorting algorithms are massively underused and underknown! When you have a list/array that is changing slowly (low write/search ratio) and you need to keep it sorted, using an adaptive algorithm can outperform quicksort by several orders of magnitudes.
* I’d say maybe Disjoint sets are also underused (but also underappreciated, which I’ll tackle next)
**Marcello La Rocca**
So, part 2: _underappreciated_ algorithms:
1. Disjoint sets, because they are clever and fast, but since there are already slower but simpler alternatives which are still linear or sublinear, they tend to be implemented sparingly
2. Discrete Fourier Transform. I mean, it’s certainly well appreciated, but not many people appreciate how fundamental it is
**Marcello La Rocca**
part 3: overused and misused
That’s a tricky question 😄
This usually depends on the problem you re trying to solve, any algorithm can be an overkill for some problems.
Familiarity is of course one of the most likely reasons to overuse something (see the Maslow’s hammer rule 🙂 ), but that’s mainly subjective.
Related to hype, instead, today I’d say that some machine learning algorithms, or machine learning in general, are somehow more likely to be misused.
For instance very powerful algorithms (well, more like _categories_) like deep learning can be used because of buzz when the data volume would instead suggest a different approach less prone to overfitting, like logistic regression or random forests.
Or maybe sometimes a machine learning model is sought or developed, when some good old-fashioned statistics analysis could serve better.
It really depends on the situation, no algorithm or technique is intrinsically bad of course, but it’s important to run a proper requirement analysis upfront and do some research to understand what suits best the problem we are facing.
**Marcello La Rocca**
Thanks again Agrita Ga, this was a challenging and interesting question, it was fun to reason about it!
I’ll try to think about other examples, and maybe I’ll add to the thread.
Also if you have follow up questions, please, by all means!
**Agrita Ga**
Delighted and ecstatic to hear this was interesting for you!
And obviously great to pick up your brain on this. 🧠
I also feel that sometimes misuses or using something due to familiarity comes from some personal boilerplate function repos (I’m guilty of this), when you need to do quick and dirty coding and don’t think about efficiency that much - at least while it does not impact actual correctness of outputs. But then again, I guess, the keywords here are _quick and dirty._ 😅
And ahh, the great hype around machine learning. Quite often the hype and fanciness stays with the model in a notebook when one has to deploy (read: also maintain and debug) overly complex, super fancy machine learning model where the problem could have been solved in way easier manner.
**Agrita Ga**
Related question to this thread, so won’t create new one (but if you think this deserves separate thread, I’ll separate it out) - your thoughts on dataframes as data structures? Obviously it serves a purpose - data people think in dataframes, but maybe in general - overused?
**Marcello La Rocca**
Hmm… 🤔
Do you mean something like Panda’s or Spark’s dataframes?
Yeah, it’s a good take - I mean everything can be useful or overkill, depending on the task.
I’d say that whenever you need to manually run some analysis, inspect data, even prototype a model, then a dataframe is very convenient.
Using it in the middle of a data pipeline, instead, would feel a bit like overkill to me - but it might just be a personal preference, of course.
What do you think instead?
**Agrita Ga**
Yes, exactly - R’s, panda’s, spark’s, etc. dataframes.
As I originally started with R, I have to say that I’m very skewed into liking dataframes and often designing logic that is somewhat dataframe specific. However, I remember when Spark introduced dataframes, some developers had hard time adjusting from RDDs to DataFrames - as they explained, you have to manipulate data in a different sequence that did not come naturally to them.
This is why I asked your thoughts on overused, as personally I sometimes feel that when you have gotten really used to dataframes, it’s sometimes hard to switch back.
Additional note and this is really not related to data structures, rather quick food for thought - with data manipulations I feel that dataframes (with all the in-built functions) gives us cleaner code and greater visibility what’s actually happening. However not saying that this also comes with speed.
**Marcello La Rocca**
Absolutely, I 100% agree.
And it’s always tricky when you have to switch to a different way of doing things… I guess the important part is making sure one is switching for a good reason, not just to embrace the latest trend/news.
But for dataframes, I fully underwrite your comment, it does help writing cleaner and better structured code, and that’s why I was thinking that anything that’s highly interactive or anyway with a relevant human factor is good use case for it.
**Matthew Emerick**
Hello, Marcello La Rocca. Thank you very much for doing this.
How do you differentiate between an easy/fundamental/intermediate algorithm and an advanced algorithm?
**Marcello La Rocca**
Hi Matthew Emerick, it’s a pleasure! Thanks a lot for your questions.
Let me try to answer them in order (and inline).
**Marcello La Rocca**
The first one is very interesting: I suppose there isn’t a single answer to this, and that it’s a bit subjective.
There are, however, some criteria that can be used, for example
* If an algorithm/data structure is built upon other algos/DSs, or it’s a more advanced version of another DS: for instance:
◦ A priority queue is a more complex version of a queue
◦ _Dijkstra_’s algorithm use priority queues as part of it’s logic
◦ _Breadth First Search_ is a simplified version of _Dijkstra_’s, which in turn can be seen as a simplified version of _A\*_. And so on…
* If an algorithm requires previous knowledge, or math knowledge: take the discrete Fourier transform, just on top of my head
* How complex is an algorithm: at a glance, looking at the code or just its description, you can immediately see that _BTrees_ are more complicated than _Binary Search Trees_.
**Matthew Emerick**
What are the main problems these algorithms solve?
**Marcello La Rocca**
In general? Well I think there are several categories…
It can be storing and querying data (many data structures revolve around this, in the end), or it can be processing data (DFT, ransac, but also ML algorithms)
**Marcello La Rocca**
Of course feel free to follow up on this topic!
**Matthew Emerick**
What algorithms do you cover that most books do not?
**Marcello La Rocca**
Terrific question, thanks for asking!
First, I think the unique value is that we cover a set of advanced algorithms that can’t be found in any single other book: from space-aware search to machine learning to genetic algorithms.
If we look at the individual algorithms/DSs, I think
* no other book covers `SS+-trees` and `OPTICS` (a clustering algorithm), distributed clustering using `MapReduce`
* few or no books books cover `k-d trees` and `R-trees`, algorithms to draw graphs in the plane, `DBSCAN` (another clustering algorithm) , `Tries`, `Bloom filters`
But also I’d say that, last but not least, for the rest of the material, I tried to present it with a different angle, explaining the theory but also discussing how to use these algorithms in practice
**Matthew Emerick**
What do you recommend we read before reading your book?
**Marcello La Rocca**
_[Grokking algorithms](https://www.manning.com/books/grokking-algorithms)
_ is a nice introduction to algorithms for beginners and people just approaching the topic.
Another book that I loved and always recommend to have a quick bootstrap is Skiena’s _[The algorithm design manual](https://www.amazon.com/dp/B00B8139Z8/)
_
**Matthew Emerick**
What do you recommend we read after?
**Marcello La Rocca**
Depending on what you liked most on the book, there are many different topics that one could follow up. I tried to add links within each chapter to books and papers that can help the readers dig deeper into each topic.
Just for the sake of it, a few suggestions that I also mentioned at the end of my book:
* Grokking Artificial Intelligence Algorithms
* Algorithms and Data Structures for Massive Datasets
* Grokking Machine Learning
**Rodney Silva**
I know that data structures is a very difficult topic in computer science. What do you think is the best way to study with this book in order to absorb all the knowledge presented?
**Marcello La Rocca**
Hi Rodney Silva! Thanks for bringing this up, that’s a good question, I’m glad I get the chance to talk about this.
So, first of all, I’d say: what do we mean when we think about absorbing all the knowledge?
I know not everyone might share this view, but the way I see algorithms (and the way I was thought about it), the goal should not be remembering by heart every detail of each algorithm .
When I was in college and I followed algorithms 101, my lecturer was clear to us: “your goal is to learn that these algorithms do exist, and know what they do and when they should be applied. This way, whenever you will need them, you can pick up this textbook from your shelf and refresh your memory”.
**Marcello La Rocca**
So every time I do a class or a talk about algorithms, I tell the same to the audience: try to grasp the fundamentals of each topic, and then you can take it from there when you need it.
**Marcello La Rocca**
Complementary to that, we tried to provide the possibility of an incremental approach to the readers, organizing chapters so that you can stop at different points, or skip sections, depending on what you are really interested about:
* There is an introductory section for each (most) chapters explaining a problem, trying to solve it in different ways, and showing pros and cons of a few data structures/algorithms
* Then the chapter usually focuses on a single data structure, going deep into its logic, and then the implementation details
* In some chapters, there are clearly marked advanced sections where we explain the theory behind the algorithm just described
* In most chapters, there is a section about practical ways of using the algorithm or data structures just introduced.
**Marcello La Rocca**
So, for instance, you could skim first skim through chapters reading about the problems, then try to use the code (you can get implementations from our [repo](https://github.com/mlarocca/AlgorithmsAndDataStructuresInAction)
on github, or use a library implementing the same algo), then go deeper into into the theory and try to implement a DS/algo yourself (good exercise especially if you are preparing for interviews), and finally you can delve into the theory to understand why something works that way.
**Marcello La Rocca**
Or, alternatively, you could also use this book as a catalog matching problems to algorithms, skim through it to understand what you could reuse and in what situations, and then when you find yourself facing a (now) familiar problem, you can resort to the book and delve into the details.
I’d say we tried to write the book in a way that gives you some leeway and lets you decide how to learn best, given your needs
**luckylittle**
Hi Marcello La Rocca, I have a question about graphs and implementing them. Mathematical representation makes it clear on the paper, but what’s the best way to store them into a data structure and how should we store edges? What are the different approaches and potential pros and cons, please?
**Marcello La Rocca**
Hi luckylittle, thanks for your question and sorry for the delay! (I guess you’ll only find it tomorrow morning, sorry!)
Luckily enough, I spent some time writing an intro to graphs in chapter 14, and I go through the details of exactly these questions: you can read more about it here: [https://livebook.manning.com/book/algorithms-and-data-structures-in-action/chapter-14/v-14/30](https://livebook.manning.com/book/algorithms-and-data-structures-in-action/chapter-14/v-14/30)
To give you a short answer, yes indeed, that’s a very good point, there is a difference between the math representation and the actual implementation.
It is possible to implement graphs as algebraic data structures (so following the mathematical definition): these implementations, if correct, provides a few advantages (like guaranteeing you’ll have a consistent and valid graph), but usually have two main disadvantages:
1. Performance (they are slooooow)
2. Notation might be _a tad_ verbose and cumbersome (although in some languages, mostly functional languages, you can get around this)
**Marcello La Rocca**
More practical implementations of course do exist, and the main difference between them is exactly how you store edges (a critical point):
1. Adjacency lists: each vertex has an adjacency list where outgoing edges are stored
2. Adjacency matrix: rows and columns are labeled with vertices, and cell (v,u) contains the edge from v to u, or it’s empty if there is none
3. Incidence matrix, a matrix whose rows are vertices and columns are edges; each cell it’s either a 0/1 (0 if the edge is not incident on the vertex, for edge e=u->v, cells (u,e) and(v,e) would be 1), or can hold the edge’s weight.
Each representation has pros and cons:
4. adjacency list is used for sparse graphs, and better if the graph is dynamic (vertices-wise)
5. adjacency matrix always require quadratic space, but for dense graphs this is asymptotically irrelevant, and it can speed up some algorithm (Floyd-Warshall, transitive closure etc…)
6. incidence matrix is convenient for multigraphs, and also can speed up some algorithms
**Marcello La Rocca**
Then, beyond the representation, or once you choose it, there is another question connected to the implementation: for instance, in the adjacency list representation, how do you implement edges?
Do you need a class for them, and keep references to the vertices? And then how do you ensure consistency when you update the graph?
A talk a bit about this at the link I shared above, but you can also check out a practical implementation (and read about the design) here: [https://github.com/mlarocca/jsgraphs/blob/master/readme/tutorial.md](https://github.com/mlarocca/jsgraphs/blob/master/readme/tutorial.md)
**luckylittle**
Oh wow, this is very comprehensive answer. I had a look at chapter `14.1.2 Graphs as Algebraic Types` and it is fantastic - exactly what I need. Let me also have a look at your GitHub. Many thanks 🙏
**Marcello La Rocca**
Perfect! You are welcome, glad I could help!
**Matthew Emerick**
Just want to thank Marcello La Rocca again for doing this and especially for the detailed answers to my questions.
**Vladimir Finkelshtein**
What are some major recent advances in algorithms? I have read somewhere that multiplication of large numbers is now done in n log n, but I guess this only has importance as a trivia question.
**Marcello La Rocca**
Hi Vladimir Finkelshtein! Thanks for your question, this is a really interesting topic.
So, first things first. Well, actually, last things first, as I’d like to start from the second part of your question.
I suppose you refer to this method [https://hal.archives-ouvertes.fr/hal-02070778/document](https://hal.archives-ouvertes.fr/hal-02070778/document)
to multiply two integers, presented a few yeas ago.
I’m not sure it has been validated by peer reviews yet, but that would certainly be a breakthrough in their field.
And by the way, it would be a super-important one also in practice, because multiplying large integers is crucial in many fields, especially in cryptography - often used when encrypting streams of data or communications.
On the other end, while knowing its existence could be interesting for mere trivia questions, I doubt it would be relevant as an interview question: the method uses a _fast Fourier transform_ with 1729 dimensions, it would take more than an interview to explain it 😄
**Marcello La Rocca**
A few more of recent breakthroughs (well, for a given value of “recent”)
* Grover’s algorithm (1996) [https://arxiv.org/abs/quant-ph/9605043](https://arxiv.org/abs/quant-ph/9605043)
* Soft heaps (Chazelle, 2000) [https://dl.acm.org/doi/abs/10.1145/355541.355554](https://dl.acm.org/doi/abs/10.1145/355541.355554)
* Quantum Approximate Optimization Algorithm (QAOA) (2014) [https://arxiv.org/abs/1411.4028](https://arxiv.org/abs/1411.4028)
* Transformers (2017) [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)
* Disproof of Hedetniemi’s conjecture (2019) [https://arxiv.org/abs/1905.02167](https://arxiv.org/abs/1905.02167)
* Fully-dynamic planarity testing in polylogarithmic time (2020) [https://dl.acm.org/doi/10.1145/3357713.3384249](https://dl.acm.org/doi/10.1145/3357713.3384249)
**Vladimir Finkelshtein**
I meant that the previous multiplication algorithms already gave something close to n log n. And the implicit constants are not specified in either algorithm. In encryption people rarely use numbers with more than 1024 digits, so it is unclear if the asymptotic improvement will be relevant on this scale.
Thanks for the list. Surprised to see Shitov’s result here. It is not clear to me what is the application of it outside combinatorics though.
**Marcello La Rocca**
Oh I see, sorry for misunderstanding.
Well, as you said, if it is going to be relevant in practice, it’s going to depend on the constant multiplication factors, and on the actual implementations. Also consider that we might higher security in the future, so the scale could go up (if we have the hardware to support it).
About Shitov’s result: indeed, that’s mostly a theoretical result, but still, quite impressive (also, I’m partial to graphs theory 😄 )
**Rodney Silva**
What was the most difficult chapter to write in this book? Why?
**Marcello La Rocca**
Hey Rodney Silva, thanks for the question!
I think the hardest chapters to write for me were chapter 10 (similarity search trees) and chapter 13 (distribute clustering using MapReduce).
Both because the topics were more complex, because I had to do a lot of research on them to make sure I had a good understanding and I could explain all the topics in the best possible way, and because there was little or no material at all available about these topics.
For SS+trees and OPTICS, in particular, I had to dig research papers, with no implementation of these algorithms to compare to, or to be used to validate mines.
So it’s been challenging but also fun 😃
**Agrita Ga**
If you’d write vol II of the book or if you’d add more chapters, which algorithms and/or data structures you’d be most excited to cover?
**Marcello La Rocca**
Thanks for asking Agrita Ga !
Interesting, let me think about this…
I think it would be great to write another book focusing on graphs, like taking and expanding what I have from this book, going deeper into some topics I just touched in this book, like Bellman-Ford, Floyd-Warshall, the maximum flow algorithms.
But also having a part of the book about dynamic graphs, a hot topic today.
**Marcello La Rocca**
Perhaps what I’d enjoy even more would be writing a book about genetic algorithms, with a first part expanding chapter 18 of AAaDS (like, 1 chapter for crossover, one for natural selection, and so on), and then including many more in-depth examples of practical problems that one could solve using GAs
**Amruta**
Hey Marcello La Rocca!
Hope you are doing well!
I have a very basic question to start with, What was your major motivation/idea behind authoring this book?
**Amruta**
Also, there are many books being written on Data structures and Algorithms in recent times, how is your book different/unique from the others?
Last question, Which algorithms are your favorite/preferred ones? Do you have any list ?
**Marcello La Rocca**
Hi Amruta, I’m great thanks! How about you?
Thanks a lot for your questions! Let me try to answer them in order (in this thread).
So, the first question is a tricky one 😄
First, you need to know that I have been working on this book for 5 years!
It’s been long enough for me to forget why on Earth I had started… or arguably even to start regretting it 😂
I’m just kidding, of course… well, more or less. It’s kind of hard to remember exactly when the idea started taking form. I can say that algorithms (and in particular graph theory and evolutionary algorithms) are my passion beyond work; back then I was regularly writing on a blog platform, Sitepoint - mostly I was writing about _JavaScript_. So I thought I wanted to prove myself and write about something I really liked, because it could be fun, it would give me a chance to learn more about these topics, and last but not least, it might help me in my career.
**Marcello La Rocca**
`there are many books being written on Data structures and Algorithms in recent times, how is your book different/unique from the others?`
That’s an excellent question, thanks for giving me a chance to tackle it!
Some reasons I think makes this book different:
1. The unique set of topics it covers. There are some topics that can’t be found in any other books, like SS+trees and distributed clustering algorithms, but also I’m not aware of any book that starts from he basics and covers in a single place tries, k-d trees, cache, graphs, evolutionary algorithms, etc…
2. The approach used is some sort of middle ground between textbooks and practical guides. Or better, it tries to get the best from both worlds, explaining the theory but also giving a clear practical approach to the reader, describing the problems that can be solved with an algo/DS, and how you can employ them. We even go into profiling and all the considerations that should be made to decide what data structure best fit a given problem, and I believe that’s one of the greatest pros of this book.
3. The book is structured (or at least, we tried to structure it) such that you can read it at many different levels, depending on what you are interested in: an overview, practical approaches, or in-depth theoretical considerations
**Marcello La Rocca**
`Which algorithms are your favorite/preferred ones? Do you have any list ?`
That would be a long list!!! 😄
OK, let me think… the ones that I really really like the most…
* Genetic algorithm, that’s probably my favourite (also being a meta-algorithm, it comprises a lot of variety!)
* Dijkstra’s and A\* (know them by hearth by now)
* MapReduce (OK, that’s a _programming model_, to be _picky_) because it’s so simple and elegant but so powerful at the same time
* The fast Fourier transform (because when you actually get to understand it, it’s like 🤯 )
**Amruta**
Thankyou so much Marcello La Rocca for explaining my questions in such detail! Your answers are really helpful for beginners in this field (like me). Appreciate them :)
**Marcello La Rocca**
You are welcome Amruta! 🙂
I’m glad it helped, feel free to ask any time if you have follow ups or if you just could use some advice!
**Rodney Silva**
How much time did it take to research and write this book?
**Marcello La Rocca**
Good question Rodney Silva! 🙂
So, it took me 5 years to get from the initial idea to having it published.
I had finished writing it a little longer than a year ago, then it took some time to polish it, and make it production-ready (with a lot of help from the publisher and the hard work of a lot of great people).
In terms of work-hours, I think I spent somewhere between 2000 and 3000 hours working on it. Maybe something more.
For each chapter, I’d start with an idea for the topic, then come up with a few viable examples that would demonstrate the algorithm, and possibly discuss it with the editors.
Once I had an outline for the chapter, I’d start researching the topic in-depth (also to see if and how it was covered in other publications).
This could well take a week or two, depending on how much free time I had.
Then I’d write the code for the chapter, also to make sure I understood the topic in-depth, and test it through.
When I was confident enough, or in parallel, I would start to write the _story_, i.e. the narration of the chapter. Then we’d go through several reviews of the intermediate drafts
**Lalit Pagaria**
Does your book cover LSM tree, Bloom filter and Indexing (B+ tree, inverted index etc)? Even though they are closely associated with DBs but seeing their use for highly scalable system design.
**Marcello La Rocca**
Hi Lalit Pagaria, thanks for your question!
So:
* LSM tree, no
* B+ tree, a little (they are introduced when talking about R-trees)
* Bloom filters yes, there is a chapter on them
The whole part 2 is dedicated to multidimensional data and there is a chapter on k-d trees, also used for DB indexing. An another chapter, the one with an intro to B-trees and R-trees, focusing on SS+trees, which are also at scaling on high dimensional datasets (arguably better than k-d trees and R-trees)
**Lalit Pagaria**
Thanks Marcello La Rocca. I never knew about SS+ Tree. Looks like good reason to read you book 🙂
**Marcello La Rocca**
Thanks Lalit! You can take a peek of these similarity search trees here on Manning’s livebooks
[https://livebook.manning.com/book/algorithms-and-data-structures-in-action/chapter-10/v-14/107](https://livebook.manning.com/book/algorithms-and-data-structures-in-action/chapter-10/v-14/107)
I hope you enjoy it!
**Doink**
Marcello La Rocca I wanted to extend Agrita Ga question’s a bit more you mentioned that Deep Learning and in general Machine Learning algorithms are overused and misused so do you feel we stick to logistic regression or what alternatives do you suggest. Also what is your take on Federated Learning paradigm of doing things? Federated Learning tends to follow a somewhat Map Reduce type of pattern. Will we see you covering Federated Learning in your second volume?
Mainly which techniques would you prefer for distributed scalable algorithms which can protect privacy.
**Marcello La Rocca**
Hello Doink, thanks for following up on this.
So wait, no, I wouldn’t argue we should in general stick to logistic regression! 😄
What I’m saying is that each problem has different characteristics and it might be best solved with a different algorithm.
If your dataset is “small” (let’s say less than a million data points?) then probably a powerful model like deep learning might overfit it (well, you could try to mitigate using regularization, like dropout, but still, there will be a threshold for the size of the data below which some models will overfit)
Anyway, it’s hard to tell precisely in advance, and the best guideline to follow is “listening to the data”: trying out several models, training and testing them and comparing the results to see what model fits best and generalizes best your data.
**Marcello La Rocca**
Then there is another orthogonal question that is about transparency, i.e. interpretability, of these models.
To that extent, deep learning models are black boxes, and it’s hard to interpret them, which can lead to all kinds of unwanted biases and unexpected behaviors.
From that point of view, I think that today we should really make an effort to design and use interpretable models, especially when health or life-changing decisions are made with the help of a model.
A powerful interpretable alternative to neural networks are for instance GAMs, described for instance here: [https://www.facebook.com/watch/?v=477576069385721](https://www.facebook.com/watch/?v=477576069385721)
Recently Denis Rothman wrote a wonderful book about explainable AI, you can take a look here
[https://www.linkedin.com/feed/update/urn:li:activity:6802898159904276480/](https://www.linkedin.com/feed/update/urn:li:activity:6802898159904276480/)
[https://www.amazon.com/gp/product/1800208138](https://www.amazon.com/gp/product/1800208138)
A free resource on the topic
[https://christophm.github.io/interpretable-ml-book/](https://christophm.github.io/interpretable-ml-book/)
**Marcello La Rocca**
About _federated learning_, I think that’s an extremely promising technique; it’s also not applicable (or advisable) in all situations, but indeed whenever data sharing would not be possible (either because of privacy or because the data is too heterogeneous) it seems like the way to go.
Side benefit, as you note, distributing the load allows scaling out your system (at a price, of course, because the sum of the total resources needed might be higher than the ones of a centralized model - though not necessarily more expensive than that - and the final result might not be as good as the theoretical centralized equivalent).
Privacy-wise, as an alternative I could only think about anonymizing data… but one way or another, if you share it, you reveal something about your source (sort of a fingerprint).
But although harder, I suppose that even a model could be “reverse-engineered” to discover some statistics about the dataset on which it was trained. So it all depends on the level of security/privacy you need.
Anyway, yeah, if I wrote a follow up book, definitely both interpretable ML and federated learning could be among the topics I’d like to cover!
**Doink**
Marcello La Rocca I love the long answers. Regarding Federated Learning for what kind of applications do you feel it’s advisable and for which use-cases do you feel they are not advisable?
Also are you aware of the other approaches in the privacy preserving ML framework such as differential privacy, homomorphic encryption,secure multiparty computation and trusted execution environment?
Also Privacy and Fairness don’t go hand in hand.
**Marcello La Rocca**
😄
Federated Learning: I think it heavily depend on the context, as we discussed when privacy is a concern or data is too big to be processed in a centralized way.
Hard to say _a priori_ which industries or categories… just thinking out loud, I think something like a joint venture between several companies, where no single company wants to share its data with the others, could be the ideal case.
But also anything related to health, for instance something that requires collecting data from several hospitals across the Country: this could allow developing a much better model, without sharing PIIs outside of each medical center.
**Marcello La Rocca**
`Privacy and Fairness don't go hand in hand`
Well, I mean, true sometimes, but we could struggle to find ways to make it work. Sometimes it might feel like privacy might be used as an excuse to deny fairness, but that’s another story 🤔
**Marcello La Rocca**
`Also are you aware of the other approaches in the privacy preserving ML framework such as differential privacy, homomorphic encryption,secure multiparty computation and trusted execution environment`
I have to admit I’m no expert on this field, I’ll be sure to dig and read more about these approaches, they look fairly interesting.
Maybe Alexey Grigorev can answer better about this?
**Alexey Grigorev**
No, unfortunately I can’t! But if there’s a book about differential privacy, I can try to invite the authors 😃
**luckylittle**
Marcello La Rocca Sorry to ask about graphs again. I had a look at your lightweight library to model graphs - `JSGraphs` , specifically the _BFS_ ([https://github.com/mlarocca/jsgraphs/blob/master/readme/tutorial.md#bfs](https://github.com/mlarocca/jsgraphs/blob/master/readme/tutorial.md#bfs)
) and based on the examples there, you are showing examples of directed graphs (all the edges point in a direction), e.g. first example is _1_ -> _3_ -> _4_ -> _6_ -> _7_. Can these methods be theoretically used in scenarios with mixed graphs (some edges are bidirectional, few are not) - specifically to obtain connections between cities, e.g. some big cities are hubs connected to other big cities, but some villages only have one road in?
**Marcello La Rocca**
Hey luckylittle, no reason to be sorry! On the contrary - graph theory is one of my favorite topics!!! 😄
The scenario you describe is normally modeled with a directed graph, where one-way roads are modeled with a single directed edge, and two-way roads, instead, correspond to a pair of directed edges.
Directed graphs are more flexible, in this sense, because an undirected graph can be translated to a directed one using this trick (using a pair of directed edges for each undirected edge); the opposite instead is not true.
Then if we move to multigraphs, those can have multiple directed edges between each pair of vertices. To have a parallel, one could think about a situation where multiple edges can only be used under some conditions… like maybe the villages in your examples are connected by roads, but also through canals, or there are flights between them, or sideroads that can only be traveled with a dirt bike… you get the idea 🙂
The good news is that _BFS_ can work with all these graphs 🙂
Does that answer your question, and did I understand it correctly?
**luckylittle**
I’m glad that I tickled your fancy with your favourite topic 😆 It certainly does explain a few things which I was not entirely sure about.
**luckylittle**
On a different note, I had a look at this very subject in your book (via Manning’s 5 minutes free preview, so really just skimmed through it!) and in that short period of time I had a feeling of reading an “academic” journal. Would you say that your book is going to explain these advanced topics in a way that non-academic people will understand? Maybe I should have another look tomorrow…
**Marcello La Rocca**
That’s a good point. So, we tried to balance the “textbook” part with a more practical approach. So yeah, there will be a theoretical part that’s gonna cover the topic a bit more in-depth than average “hands-on” books.
But in each chapter there is also room for examples and a more practical approach.
For graphs in particular, since I was only writing a quick introduction, that feeling of “academicness” (I think I just made up a word, but bear with me) is probably stronger than in other chapters.
For comparison, I’d suggest you could take a look at _Grokking Algorithms [https://livebook.manning.com/book/grokking-algorithms/chapter-6/](https://livebook.manning.com/book/grokking-algorithms/chapter-6/)
_
which has a less formal approach, and you can pick whichever works best for you right now
(actually I always strongly recommend _Grokking Algorithms_ as a primer on algorithms)
**luckylittle**
Awesome, thanks for the tips and explanation. Keep doing a good job!
**Rodney Silva**
What is the most complex data structure to master in this book? Why?
**Marcello La Rocca**
Hey Rodney Silva, nice question 🙂
My vote would go to k-d tree and Ss-trees, because they work in multidimensional spaces and for our brains (well, at least for mine) it is hard to wrap around these concepts, as it is hard to visualize them.
Also, and especially for Ss-trees, the logic of the algorithms is quite complex, and it requires a fair amount of studying and writing down examples to get all the possible cases right.
**Wendy Mak**
Hi Marcello, in your book repo you used 3 different languages to illustrate the various algorithms– is there language(s) that you find easier to implement algorithms in?
**Marcello La Rocca**
Hi Wendy Mak, that’s an excellent question, thanks for giving me a chance to discuss this!
First thing I need to say, is that I consider myself language-agnostic: in my career I used many programming languages, and more or less I like them all - or hate them all, on bad days 😄
I strongly believe that programming languages are tools, and so you need to pick the right one for the task you are presented.
For instance, the three languages I currently used on the repo could hardly be more heterogeneous: Java, JavaScript and Python!
Each of them has different characteristics, so for instance Python is great to quickly write prototypes, because you don’t have to worry too much about the type system and it has a very rich and intuitive syntax for lambdas, generators, list comprehension etc.
On the other hand, though, a strongly typed language like Java take care of the static checking for you, which allows me to be more confident that my program is going to work (because types are checked by the compiler) and also to write less tests (because, say compared to JavaScript, if a function takes a int, I don’t need to test what happens when someone tries to pass it a string 🤦 )
**Marcello La Rocca**
OK, anyway, sorry for the digression. In a nutshell, which one is easier heavily depends on personal preference and on what one is more used to. And, needless to say, on what you are going to implement.
So for me, for instance, for _containers_ it was probably easier to write them in Java (mostly for the reasons in my previous comment) while for anything machine-learning-related, a Python notebook was much easier (existing libraries and frameworks are another very relevant factor here)
**Wendy Mak**
also, if you are practicising implementing algorithms from scratch, how do you build test cases to check that it’s doing the correct thing? (since an incorrect or suboptimal implementation might still give you the correct answer?)
**Marcello La Rocca**
Ah! Tricky one! 😊
So… well, it is tricky, it’s actually the trickiest part. Just yesterday I was saying exactly this, that for example Ss+trees were the hardest for me to implement, because there was no existing implementation that I could use as a cornerstone.
But in my experience this is the norm, in my daily work I’m often in the same situation, and actually writing (good) tests is often the most time-consuming part of my work.
So, what you can do in these cases?
Well, you can start by going through the logic of the algorithm, reason in terms of edge cases, see if your code works when you get empty inputs (or null), unexpected values, large inputs, check that it fails when it’s supposed to, and finally try the regular cases.
If you are using a suite/framework, running test coverage can help you understand if you tested all the possible cases (but also, beware of the myth of 100% coverage…)
Incidentally, this sanity check is a process that can help you a lot also in interviews (you should always try to test your whiteboard code, starting from edge cases).
**Marcello La Rocca**
Bringing this to the next level, you might want to embrace TDD (test driven development) or BDD (behavior driven development). This means that you’d start by writing your tests based on the domain knowledge and requirements analysis you have, and only after you lay down enough tests to express how things should go in your expectation, only then you start writing the code (well, more or less - no need to be inflexible on this kind of things, but you get the idea)
Honestly I highly encourage TDD. Last piece of advice: have your test fail. This works perfectly with TDD: it’s important that you check that your tests initially (or at some point, before you finish implementing a feature) fail - this makes you more confident that you wrote a good, proper test (otherwise sometimes, by mistake, you can find out that your test is not actually checking what you were expecting)
**Doink**
Marcello La Rocca How to balance understanding depths and breadths of internals of any algorithm?
**Marcello La Rocca**
🤔 I’m not sure I understood your question correctly: what to you mean by depths and breadths? Do you mean having a high level understanding (breadths) vs knowing the inside-outs, the smallest details (depths) of an algorithm?
**Doink**
Marcello La Rocca Yes
**Marcello La Rocca**
Thanks for clarifying!
Then I’d say the most important thing (IMHO) is to learn the “breadths” part, getting a high level idea of what an algorithm do, where you can apply it, what problems you can solve.
Given that our memory is limited, it’s unlikely one can remember the details of all algorithms.
Instead, if one understands it at a high level, when the time comes one will be able to pick up that algorithm and go in depth to understand the algorithms internals and implement or adapt it.
Of course there are exceptions, for instance if you are specializing in a field, or if you are preparing for interviews, then you might wanna go in depth. Or likewise, for the basics: it’s better to get an in-depth understandings of basic algorithms, before starting to study the ones built upon them.
**Rodney Silva**
What’s your educational background?
**Marcello La Rocca**
Hey Rodney Silva, thanks for asking
So I studied software engineering in high school, then got my master degree in _computer science_ (in my alma mater it was part of the Math department, so we were focusing more on math, algorithms, computability and complexity theory etc. than SE).
I’m also a PhD dropout (robotics and machine learning).
As for programming languages, I started with _Turbo Pascal_ in high school, then since college I studied/worked with:
* Java
* Haskell
* C
* C++
* Scheme
* Fortran
* JavaScript
* Php
* C#
* COBOL
* Python
* Scala
* Go
I might be missing some more 😄
But of course I’m not proficient in all of them, I daily work just with Java/JavaScript/Python/Scala
**Alexey Shvets**
Marcello La Rocca which programming language do you personally like the most?
**Marcello La Rocca**
Tricky question Alexey Shvets! 🤭
OK, let’s say that only considering the joy of programming and abstracting from any practical considerations, I’d pick `Scala`, because it’s elegant and expressive, it’s functional (I’m totally partial to functional programming) but it’s also concrete (and OO 😉 )
**Ajay kumar saini**
Hi Marcello La Rocca Thanks for doing this. I have few following questions (forgive me if you’ve already answered them, I am still going through your response 🙂 )
* Which data structures or algorithms you use most frequently in daily engineering work?
* Do you think using advanced data structure or algorithm could reduce code readability or code onboarding time?
* When one shouldn’t use a particular data structure or algorithm even though it fits their use case?
**Marcello La Rocca**
Hi Ajay kumar saini thanks to you for your excellent questions!
Let me try to answer them here in order
**Marcello La Rocca**
`Which data structures or algorithms you use most frequently in daily engineering work?`
Nice one 🙂
Of course the basic containers are ubiquitous, can’t spend a day without using hashing or hash maps, for instance.
If we are talking about the advances ones, and the ones in the book, I’d say right now machine learning and as such gradient descent.
Cache would be a close second.
**Marcello La Rocca**
`Do you think using advanced data structure or algorithm could reduce code readability or code onboarding time?`
I think using the right data structure, and advanced data structures too, could _improve_ code readability: especially if you reuse existing (well tested) libraries or anyway you do a good job encapsulating the data structure’s logic in its own class, it will make your daily code shorter, cleaner and clearer for who reads it.
As for the onboarding time, I think it can also speed it up: the most time when you are onboarding at a new position is spent, in my experience, in catching up with domain knowledge.
So if you don’t have to worry about the details of algorithms (be it sorting, queueing, running _ransac_ on your radar data or whatever 😄 ) you can focus on the domain knowledge and catch up more quickly.
(please feel free to let me know if I haven’t captured well the sense of your question)
**Marcello La Rocca**
`When one shouldn't use a particular data structure or algorithm even though it fits their use case?`
Excellent question! There could be many complementary questions you could and should ask when deciding if a given data structure/algorithm fits your problem; some are strictly technical, some are more business-related - let me enumerate some of them (just thinking on my feet here):
* Are you operating in a multithreaded environment, and is the DS thread safe?
* What’s the performance of this DS/algo? Does it scale with the size of the input I have?
* Is there a library (well tested and widely adopted) that implements this DS/algo
◦ If so, is it open source?
◦ Not open source:
▪︎ Can I trust it?
▪︎ Is there any privacy concern?
▪︎ Is there any safety treat in using it?
* If no existing implementation exists or can be used:
◦ Is there someone within the company that has the knowledge to write it?
◦ Is it worth using this DS/algo over a given alternative (which might be less ideal, but you wouldn’t need to implement from scratch)?
And so on… 😉
**Doink**
how much of data structures does one actually write? Isn’t everything sorted by a well tested library?
Also how often does one have to deal with thread safety?
**Marcello La Rocca**
`how much of data structures does one actually write?`
That heavily depends on the kind of position.
You could spend an entire career working in machine learning, without having to write a single ML algorithm. You can of course make great progress and killer apps that puts together NLP and object recognition or gesture recognition, and never write a transformer nor gradient descent: you can do it all by plugging together ML libraries.
Same goes for algorithms, every developer uses a lot of great algorithms every day without even realizing it (just think about the graph algorithms implemented by your compiler or by the garbage collector or, to stay more code-related, the containers in the standard libraries).
**Marcello La Rocca**
`Isn't everything sorted by a well tested library?`
Well… not so fast.
Yeah, it’s possible that you can find everything you need already implemented and tested. Even more, if you find such a library, you should rather use it and avoid re-inventing the wheel and rewriting something from scratch.
But, that’s not always the case. A few examples of situations where you might need to write your own implementation:
* You (or your company) are adopting a fairly recent language, for which there aren’t many libraries already available.
* You need to implement a niche algorithm (because it’s a bottleneck in your pipeline and you can use any improvement). This can happen also with more popular or even mainstream languages, for instance try to find implementations for OPTICS, DeLiClu or Ss+trees in Python or JavaScript
* You need to customize an algorithm for your peculiar needs. I’d say this is by far the most common case. You might still be able to use a generic implementation, but adapting it might be too slow or consume too much memory.
And I probably also miss other good cases
**Marcello La Rocca**
`Also how often does one have to deal with thread safety?`
That again depends on the position… I can tell you that I currently work on the backend of web applications, and I have to deal almost every day with writing thread-safe code.
And in my previous jobs on data infrastructure, I found it even more important.
_Remember that thread-safety is a precondition to scaling out._
**Ajay kumar saini**
Thanks Marcello La Rocca for your insightful answers 🙂.
**Ajay kumar saini**
Hi Marcello La Rocca I saw in previous threads that you are also learning quantum computing. Any good recommendation for practical resources for beginners?
**Marcello La Rocca**
Hey Ajay kumar saini, of course!
Some great books
[https://manning.com/books/learn-quantum-computing-with-python-and-q-sharp](https://manning.com/books/learn-quantum-computing-with-python-and-q-sharp)
[book 1](https://manning.com/books/quantum-computing-for-developers)
, [book 2](https://oreilly.com/library/view/programming-quantum-computers/9781492039679)
, [book 3](https://amazon.com/gp/product/0521879965)
, [book 4](https://amazon.com/Quantum-Computing-Introduction-Engineering-Computation/dp/0262526670)
, [book 5](https://amazon.com/Quantum-Computation-Information-10th-Anniversary/dp/1107002176)
. (here a bit more info on each of them [in this Twitter thread](https://twitter.com/mlarocca/status/1321421052367523841)
)
**Marcello La Rocca**
Free resources:
[https://qiskit.org/learn/](https://qiskit.org/learn/)
[https://www.youtube.com/c/qiskit/playlists](https://www.youtube.com/c/qiskit/playlists)
(in particular [https://www.youtube.com/playlist?list=PLOFEBzvs-VvrXTMy5Y2IqmSaUjfnhvBHR](https://www.youtube.com/playlist?list=PLOFEBzvs-VvrXTMy5Y2IqmSaUjfnhvBHR)
and [https://www.youtube.com/playlist?list=PLOFEBzvs-Vvp2xg9-POLJhQwtVktlYGbY](https://www.youtube.com/playlist?list=PLOFEBzvs-Vvp2xg9-POLJhQwtVktlYGbY)
)
[https://www.springer.com/journal/42484](https://www.springer.com/journal/42484)
[https://anchor.fm/quantumcomputingnow/](https://anchor.fm/quantumcomputingnow/)
[https://docs.microsoft.com/en-us/azure/quantum/tutorial-qdk-intro-to-katas](https://docs.microsoft.com/en-us/azure/quantum/tutorial-qdk-intro-to-katas)
**Marcello La Rocca**
I hope it helps!
**Ajay kumar saini**
Thanks for sharing. It does 🙂
**Jasper**
Thank you for adding me! 🙂
**Filmetto App**
Hello Marcello La Rocca, hope you fine, and thanks for doing this question/answers!
I would like to ask you:
For you, for who is this book? Let say, for example, I am a newbie data engineer, and want to go further in this career path, does your book will help me?
What is the knowledge the reader will need to have before approaching your book?
Thanks!
**Jasper**
Although I haven’t read the book yet, I did browse through the chapter contents on github. During my previous course at college on data structures we discussed various of the mentioned algorithms and learned about implementing them.
This book is a treasure of knowledge, in particular if you want to do high level design of systems and go beyond the basics. I can definitely recommend it for your career. I grew so excited about the book that I’ll likely end up buying it if I don’t win a copy. 😄
**Marcello La Rocca**
Thanks a lot Jasper for your kind words! I really appreciate them, and hearing comments like this is the best motivation to write a follow up!
Any chance you’d like to repeat that in a review on Amazon? 🙂 Just kidding! 😄 (although well, if you would like to, it would be a great review 🙂 )
**Marcello La Rocca**
Filmetto App thanks for your questions! Let’s see:
`For you, for who is this book?`
Ideally it’s for beginners with some experience in coding/computer science. I’d say it’s for anyone who would like to discover more about algorithms and good practices in software engineering.
**Marcello La Rocca**
`Let say, for example, I am a newbie data engineer, and want to go further in this career path, does your book will help me?`
What I tried to do in this book is, besides talking about algorithms, also showing good practices, how to approach a problem in a systematic and rational way, how to reason about requirements and through that choose what’s the best data structure or algorithm to apply, and how to profile your implementation to find bottlenecks and compare different implementations to pick the best one.
If I managed to do all that I was hoping for, then I think it might indeed help you in that situation.
**Marcello La Rocca**
`What is the knowledge the reader will need to have before approaching your book?`
The strictly required knowledge is just some basic math and basic programming (understanding conditionals, loops etc.).
There is no expectation of previous knowledge of any specific programming language, because the book uses pseudocode to explain each algorithm (so the emphasis is on the logic of the algorithm, not the implementation details).
And actual code is also provided on github in 3 (for now, more coming) different programming languages, so there will be great choice.
For what concerns CS knowledge, of course if you had taken a CS 101 course it would help, but the book also has appendices explaining algorithms basics, from big-O notation and asymptotic analysis to core data structures (arrays, lists, stack, queue etc.).
That should help even the beginners catch up - otherwise reading something like _Grokking Algorithms_ before this book would also be a great alternative)
**Jasper**
_[Marcello La Rocca](https://app.slack.com/team/U01U72KABBR)
, being an author myself (although fiction and not non-fiction) I know how important good reviews are. I’d gladly repeat my comments there for you, but unfortunately I don’t have a copy of the book. I would love to review it though and see if I can get more people interested in buying the book there from you._ 😀
**Filmetto App**
Thank you so much for taking the time to respond to my questions. It’s very clear and actually make me more willing to acquire your book 😄
**Marcello La Rocca**
Thank you both, I really appreciate it!
**Marcello La Rocca**
Jasper don’t worry about the review, they are not even open yet, it will be possible to write them only after the book is actually available (not just for pre-order). So, depending on the store, that might be end of June or beginning of July.
And anyway of course you should have a chance to read your copy first! BTW, good luck (to both of you) with the extraction!
**Rodney Silva**
Why do you think computer science bachelors suffer so much to pass in algos/ds interviews from Faang?
**Marcello La Rocca**
That’s a very interesting question.
Rodney Silva Just to understand better the scope of the question, do you mean bachelors suffered compared to master graduates? Or compared to other backgrounds?
**Rodney Silva**
They suffer in general, for other backgrounds I think it’s even more difficult
**Marcello La Rocca**
Rodney Silva sorry for the delay!
So yeah, this is a very interesting issue, to me it’s also a problem.
I’ve had my fair share of interviews at Faang, some of them I failed, some others I passed. And then I have also been on the other side, the interviewer side.
One thing I could see is that most of the time the interview process is focused on topics that are not part of the daily work of people.
So for many of these interviews, you get asked questions about coding challenges, and you have to write code on a whiteboard - which isn’t exactly the normal setup in our day-to-day activities, right?
If you ask me why it is that people struggle so much with these interviews, I think a few reasons might be:
* algo/dss are not part of the normal work for many positions
* b/c of that, people tend to forget them after a while
* In general algorithms are sometimes (maybe even often) neglected in people’s career, in favor of more practical topics
◦ for those who had college education, algorithms usually are studied early in the curriculum
◦ for those who went through other paths, often algorithms were not even part of their studies
* Moreover, these interviews ask you to face these challenges in a very short time - some people are good at thinking on their feet and coming up with quick answers, for other people this is more stressful and difficult.
* And so if one can’t come up with an answer quickly, is also more likely to panic and block, of course
So you see, in short, the only way to cope with the last couple of points above is practicing hard, but not everyone has the chance to spend so much time, nor people expect to need it, before being in such an interview.
**Tim Becker**
Hi Marcello La Rocca! Really cool book! And thank you for answering our questions. I appreciate it a lot. I also have a few:
* In your opinion, which are the algorithms and data structures a data scientist would benefit from knowing the most?
* What is your background and how did it lead you to writing this book? What was your motivation to learn more about algorithms?
* Why did you use Java and Javascript for most of the examples (github)?
* If you are working on a programming task, how to you approach the challenge of finding the best algorithm or data structure for the task?
**Marcello La Rocca**
Hi Tim Becker! Thanks a lot, I appreciate it 🙂
Let’s go with your question below:
**Marcello La Rocca**
`In your opinion, which are the algorithms and data structures a data scientist would benefit from knowing the most?`
Ah, tricky one… They would benefit from all of them? 😄
OK, kidding, I don’t expect you let me get off the hook so easily 😛
Let’s see… of course, all algorithms related to machine learning, because it would help them to understand better how to apply these techniques.
Probably MapReduce programming model is also going to be gold.
Then besides that I’d say it really depends on the area one is focusing on 🤔
**Marcello La Rocca**
`What is your background and how did it lead you to writing this book? What was your motivation to learn more about algorithms?`
I have a master degree in computer science, major in AI and algorithms.
Then I spent most of my career working on web applications and data processing, but I always tried to keep up to date and do some research on algorithms.
Evolutionary algorithms, graph theory, are my passion (within CS), and writing a book about them seemed a good way to combine business and pleasure
**Marcello La Rocca**
`Why did you use Java and Javascript for most of the examples (github)?`
And Python as well (OK, a bit less!)
So, JavaScript: because when I started working on this book, a long time ago, the idea was to focus on “Algorithms and DSs in JavaScript”, or in general for the frontend.
So I had written a lot of code in JavaScript, and moreover I was working on a library for graphs in JavaScript (lately rewritten into [https://github.com/mlarocca/jsgraphs](https://github.com/mlarocca/jsgraphs)
)
Then Manning accepted to publish the book and we pivoted to a more broad audience, so I wanted to add a typical backend language, and Java was the mainstream one with which I was the most comfortable
**Marcello La Rocca**
`If you are working on a programming task, how to you approach the challenge of finding the best algorithm or data structure for the task?`
Ah, that’s tricky! How much time do we have to talk about this? 😄
So there are many aspects to keep in mind, but one thing applies IMHO: while you don’t always need the absolute best possible solution, the most important thing is to avoid all the bad solutions.
Let me explain: it might change little if you use `mergesort` instead of `quicksort`, but it’s more important that you avoid `selectionsort`, which is definitely going to be slow.
Or even more, it can be fine using one of many heuristics that only slightly differ in their running time, but the important thing is to avoid brute-force search.
**Marcello La Rocca**
More specifically, I try this approach.
* First, I clarify the requirements, both in terms of goals and resources available
* Then, I start reasoning/researching about the possible solutions
◦ I look for existing libraries that I can use
◦ If nothing off-the-shelf works, I search inside the company who could have the knowledge to implement a custom solution
◦ Then weigh the options and how much time they would take
◦ I try to pick the best affordable option, say the one that could get me the most value within those requiring a reasonable time (depending on requirements) or anyway those that are below the median.
* Then once implemented I profile my code, and see if the algorithm is a bottleneck. If it is, and there is an available algorithm that (with some effort) can give an asymptotic improvement, I go for it. Of course this iterative approach only works if at the first step I choose a quick solution (either off-the-shelf or quickly implementable)
**Tim Becker**
Marcello La Rocca thank you 🙂 a lot to think about
**Marcello La Rocca**
You are very welcome! Please feel free to ping me any time if you have follow up questions 🙂
**Rodney Silva**
Do you think this book can have new editions in the future?
**Marcello La Rocca**
Well, it’s possible of course, it will depend on the publisher too.
But I think not necessarily, maybe it’s more likely to have a volume 2 🙂
The reason is that these concept age well, the book is (programming) language agnostic so it shouldn’t become outdated any time soon.
Of course there can be advancements in the field meanwhile, but those won’t likely invalidate or deprecate what’s discussed in the book (for instance, Ss+trees and R-trees didn’t made k-d trees obsolete, they are still relevant and preferable in some situations)
**Rodney Silva**
What kind of new advances would you like to see in the future for data structures?
**Marcello La Rocca**
Ah, good one! Rodney Silva
The one advance that I’d love to see is on the _P vs NP_ problem.
I know, it’s a theory-intensive topic… but hey, if someone proved that P=NP (which I doubt is true, but you never know!) then we would have at least an algorithm that could be used to solve all problems in NP in polynomial time - that would be have huge practical consequences!
More realistically, I’d be excited to see new progresses in quantum algorithms, I think that’s going to be a hot topic in the next years, and many game-changing discoveries could be made there.
Restricting to data-structures in particular… I’d be looking forward to seeing new natively-distributed data structures that leverages even more parallelism. And maybe for Graphs, I’d like to see some open problems solved, maybe Zarankiewicz’s conjecture proven, that would be nice (well, at least for me, since I worked on the topic 😄 )
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Graph Databases in Action – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Graph Databases in Action
-------------------------
#### by [Dave Bechberger](https://datatalks.club/people/davebechberger.html)
, Josh Perryman
##### The book of the week from 14 Jun 2021 to 18 Jun 2021
Relationships in data often look far more like a web than an orderly set of rows and columns. Graph databases shine when it comes to revealing valuable insights within complex, interconnected data such as demographics, financial records, or computer networks.
In Graph Databases in Action, experts Dave Bechberger and Josh Perryman illuminate the design and implementation of graph databases in real-world applications. You’ll learn how to choose the right database solutions for your tasks, and how to use your new knowledge to build agile, flexible, and high-performing graph-powered applications!
* [Book's page](https://www.manning.com/books/graph-databases-in-action)
* [Amazon](https://www.amazon.com/Graph-Databases-Action-Dave-Bechberger/dp/1617296376)
Questions and Answers
---------------------
**Kshitiz**
Hi bechbd and Josh Perryman - I am assuming that you guys would have covered the application of Graph databases in the book. I have a few questions about it.
Note - This is coming from someone who has little to no idea about the Graph databases.
1. Do you see any potential in the processes where Graph databases can replace the existing databases?
2. What areas do you think Graph databases can be useful, which have not yet adopted it?
**Josh Perryman**
I’ll start with state graph dbs have distinct advantages over other dbs in two ways: 1) developer productivity and 2) technical performance.
On developer productivity - graph databases, and the query languages used for them (Cypher, Apache TinkerPop’s Gremlin, etc…) make relationships first order citizens. This gives developers tools to reason over the relationships directly. In other query languages these constructs either don’t exist or are late additions. Developers with better language tools have better mental models, are able to express those models more clearly, which allows them to code faster, make fewer mistakes, and come up to speed on other’s code more quickly. Since developer time tends to be the most expensive, any gains here usually have a dramatic impact on the bottom line.
On technical performance, for certain types of queries, especially “multi-hop + variable numbers of hops” graph databases can have a dramatically better performance over other databases. In some cases, I have take queries in SQL, translated them to a graph and seen a performance increase up to 100 times faster. This is use-case specific. But there are a whole category of types of queries for which graph dbs excel. This includes recommendation engines and dependency analysis.
**Josh Perryman**
On 2, the more common problem is people choosing a graph db, then looking for use cases and choosing poorly. 😀
**Kshitiz**
Thanks Josh Perryman for the detailed reply here. This is pretty good explanation for anyone who is new to graph dbs. I have a follow up question though - Which use cases are good to go for graph dbs?
**bechbd**
So determining a good graph use case really comes down to knowing the answers to two questions:
* Does my domain lend itself to using connected data?
* Are the questions I’m asking leveraging those connections to provide the answer?
If the answer to both are Yes then you have a candidate use case for graph data. Since graphs are all about utilizing the connections within data. In general good graph use cases do one or more of the following:
* Navigate (variable) connected data structures
* Filter or computer the result on the basis of the strength, weight, or quality of the relationships in the data
* Require traversing an unknown number of connections
Since graphs/graph databases work fundamentally different than something like a RDBMS when it comes to connections in data they tend to perform better when traversing these connections is a crucial part of the application. Some examples of some good graph questions are:
* What does this person want to buy?
* How are these two people/entities connected?
* Why did X impact Y?
These sorts of questions are common across so many domains such as recommendation engines, fraud, advertising, life sciences, e-commerce, identity graphs, knowledge graphs, etc.
**Kshitiz**
bechbd - Thanks for answering this question. Definitely helps in getting a better understanding.
**Josh Perryman**
Thanks Alexey! I have timezone advantage over Dave, so I’ll try to grab the easier questions 😁
**Matthew Emerick**
Hey, bechbd and Josh Perryman! Thanks a lot for doing this.
**Matthew Emerick**
Would you recommend learning graph databases or relational databases first in today’s world?
**Josh Perryman**
I 😍 this question. I strongly recommend understanding relational databases first, and I think that Dave will agree with me on this.
We are big fans of relational databases. They have been around for decades. Every popular one out there is a very mature technology. The performance is astounding for most data use cases. The ecosystems (tools, docs) are expansive. Though ORMs tend to be mature and very helpful, it still helps to understand the data and be able interrogate it directly.
In nearly any job a developer will start in, there is almost certainly going to be a relational database. If you have to have a persistence engine (aka a db of any sort) you are almost certainly best off starting with a relational db _because you can find/hire developers with experience__._
**Josh Perryman**
In our book, we actually assume that the reader has experience with relational databases. Several examples in the start begin with a representation in a relational schema with example SQL, before “translating” it to a graph representation. We presume that SQL is the _lingua franca_ of the data world, and in our experience that’s pretty much been the case.
**bechbd**
Most of the world runs on relational databases so I agree with Josh that learning relational databases first would be what I would also recommend. I also suggest you then learn the other types of databases (Document/Key-Value/Column Oriented/Graph) and look at the strengths and weaknesses of each. These databases each perform some types of tasks better than others which is why they were created. Understanding these tradeoffs is really a key piece of information to have when trying to decide the best technology to use
**Matthew Emerick**
How important are graph algorithms in learning graph databases?
**Josh Perryman**
I see them as two separate concerns. One can do a lot of data engineering and architecture with graph databases with little to no need for understanding graph algorithms. In most cases, developers will use functions already in place rather than code the algorithms for themselves.
There are places where it is nice to know the relative benefits of one approach or algorithm over another, but this usually only comes with performance tuning, and only in the most exceptional of tuning use cases. I’ve worked with dozens of companies on graph db designs and builds, and just about the only time that talk of algorithms would come up would be over 🍻 after a day of writing code and making things work.
**Matthew Emerick**
How useful are graph databases in graph-based machine learning?
**Josh Perryman**
bechbd this one is all you
**bechbd**
From what I have seen graphs and machine learning tend to work together in three ways.:
* Graph algorithms - Graph algorithms, specifically ones around cluster/centrality/label propagation/random walks tend to provide some novel insights into data that are not easily achieved using other mechanisms. I have frequently seen where the output from these sorts of algorithms are used as one of the features input into ML models.
* Embeddings - Graph embeddings are like other embeddings in as much as you are looking to take a high dimensionality space and represent as a low dimensionality vector which maintaining the similarity differences. In graphs these tend to either be node embeddings where the node and its neighborhood/connections are represented or whole graph embeddings.
* Graph ML - Graph based machine learning is a hot topic in AI as some of the newer algorithms such as Graph Neural Networks allow you to take graphs as input to produce highly effective predictive models, especially when the domain/question are benefitted from the rich connections in the data.
I actually gave a talk on this last week which should be available on YouTube tomorrow [https://www.youtube.com/watch?v=SLKW5Q\_URq4](https://www.youtube.com/watch?v=SLKW5Q_URq4)
**Lalit Pagaria**
Thanks for answering it Dave. Really informative.
**Julian Stevens**
Can a graph database be implemented alongside my current relational database, i.e., can I create a pipeline between my relational database and a graph database?
**bechbd**
This is a very common pattern. Graph databases make a great compliment to relational technologies when the questions being asked leverage the weight, quality, or quantity of connections between entities. I sort of think about it as RDBMS systems are great at handling the “What” questions in an application and graph databases are great at handling the “Where/Why/How” questions. As a colleague once put it to me “RDBMS will tell you how much it costs to buy everyone in the room a beer, graph databases tell you why they chose the beer they are drinking”.
As far as the best way to setup the pipeline between the two that really depends on the tools that the specific vendor has in place. I would say that in most cases I have seen the RDBMS system tends to be the system of record for data (either because its a better fit for many questions or because it was there first) and any data/mutations are sent via either a CDC type mechanisms or by reading from something like a Kafka topic to the graph databases.
**David Cox**
Very interesting book! Admittedly, I have never really dove into graph databases. What are the advantages of graph databases for data scientists?
**bechbd**
I think the advantages for graphs for a data scientist really depend on the domain that they are working in. Graphs (and by extension graph databases) really allow full expressivity of data in domains (e.g. social networks, fraud, supply chain, identity resolution, supply chain optimization, etc.) where the connections between entities are as (or more) important than the entities themselves.
From a data science perspective these sorts of domains often require looking at interdependencies between entities, inferring meaning using connections, and then using these to try and predict behaviors. Because graphs treat these connections as first class citizens in the data they enable data scientists to perform these sorts of tasks faster/easier than before as well as enabling some new techniques/algorithms (e.g. centrality/clustering) that are not really capable of being performed on other database types.
**David Cox**
This is very helpful!! Thanks!!
**Jessie Yaros**
Thanks for being here bechbd and Josh Perryman ! I’ve been getting really into graph theory lately, so I love anything about graph databases! I’m wondering if you guys have any preferences in specific graph databases. Do the options out there tend to excel in specific areas that can guide people on how to choose one for specific use cases? I’ve only dabbled with neo4j, because that was the first one I was aware of. But now I know how many options are out there and its overwhelming.
**bechbd**
I currently work on the Amazon Neptune team so I am going to steer away from recommending specific databases but I can provide a high level overview on the process I use to evaluate a graph database.
The first thing to think about is if your use case is a good graph use case. I addressed some of the specifics here in the earlier comments so I’ll skip over the details for now.
The next thing I usually think about is if this needs to be a one time or batch type use case or if I need to run the use case many times. If you are only doing this one time or its a batch type process you probably need to take a look at if a graph database is even needed or if you can use something like GraphX/NetworkX to achieve your needs.
If you decide that a graph database is appropriate for your issue then you will want to next decide if you have an RDF or a Property Graph problem. RDF databases are based on triple patterns (subject/object/predicate) and use IRIs to uniquely identify entities in the graph. They are queried by SPARQL (almost exclusively) and are really great at solving issues such as MDM and knowledge graph use cases and are very popular in certain industries such as life sciences/finance. Property graphs consist of nodes/edges/properties and excel at pattern matching and path finding type problems.
Once you have decided on what type of database you are looking at you then need to decide take a look at a variety of factors:
* OSS vs Commercial - There are advantages/disadvantages on both sides of the fence here but understanding the true cost , skillset, and availability constraints of the team is important to keep in mind
* Hosted vs On-Premise
* Managed vs. Self managed
* Data size
* HA/Scalability
* etc.
**Jessie Yaros**
Thank you for this! A lot to think about.
**Jessie Yaros**
I also asked Denise Gosnell when she was here, and would love your takes as well - what would the top languages and or tech tools/ frameworks you recommend to become proficient and flexible across uses of graph databases? (I know there seems to be different graph query languages across many databases… so i wonder how transferrable these separate languages are…)
**bechbd**
Denise is just awesome.
I mean as far as top languages for development the ones I run across most frequently are Java/Python. As far as query languages/tools, unless you are doing RDF the tooling and languages tend to be very specific to a database. While there are a few open source query languages in the property graph world (TinkerPop Gremlin and openCypher) each database tends to implement their own sub or super sets of these languages (e.g. Neo4j Cypher != openCypher) so its hard to say which one to learn. Generally I think the bigger concept to stride for is to make the shift from thinking about data as tables/columns and start to view it as a network/mind map. This is transferable across any of the different tools and is much more impactful than learning X or Y framework.
**Jessie Yaros**
bechbd In my PhD work i use graph theory a fair bit, so I’ve gotten uses to thinking in terms of networks! But as you mentioned in the prior comment, my specific use cases have all allowed for the batch approach! So I’ve mostly used networkx and other python/matlab packages for network analysis, though I’m very interested in the database approach for the future. I’ve found there are a lot more advanced algorithms and implementations available in python/R packages/ github repos than I’ve seen in say, Neo4j’s graph data science library. Are there recommended ways of taking portions of networks from graph dbs and porting them into python for use with such packages? Are some dbs perhaps more compatible for this? I realize you may not be able to answer latter question because of Neptune affiliation, but thought I’d see!
**bechbd**
Yeah I have no specific knowledge of why Neo has chosen the algorithms that they have chosen but I suspect that its a combination of the computational complexity of running them at large scale and the frequency of use. Something like a graph coloring algorithm is rather computationally complex (O(c^n) or something like that) and is rarely used so putting it into a database would be prohibitively complex for little return on value. To get subgraphs into something like Network X is usually relatively easy in either cypher/openCypher or Gremlin by casting the results to a list of maps and then returning those.
**Jessie Yaros**
Thank you!
**Jessie Yaros**
And piggybacking on that, do you think GQL will ever become a thing that helps harmonize use across graph db choice? Or is it more pipedream?
**bechbd**
My hope is that GQL becomes a real standard but the adoption of it is yet to be seen. Adoption will depend on vendor support as well as what flexibility it provides for vendor specific extensions (i.e. SQL). I think the reality is that we are multiple years (5ish hopefully) away from this truly becoming a settled standard with wide ranging support
**Josh Perryman**
I’m part of a “Property Graph Schema Working Group” which is focused on schema part of this question. The group is composed of representatives from academia and industry, including prominent graph database vendors and folks like me: users of the technology.
From my experiences there seems to be a near-universal desire for standardization. This is understood as necessary step in growing the use of the technology. We are moving in the direction of having more generally accepted standards, but the timeline is best thought of in years like Dave indicates.
**Toxicafunk**
I’ve noticed data governance tools like Amundsen or Apache Atlas both have GraphDBs as their backend storage system but I’ve haven’t been able to find any deep dive into how our why GraphDBs are useful or desirable for governance. Yes, I kind of get why intuitively, but do you have any insight/specifics on the subject or some useful links?
**bechbd**
I think this is mainly due to the flexibility that graphs provide in terms of schema evolution and queryability(if thats even a word). The implicit schema of graphs lends itself to easy ingest of data with differing properties or differing data types for properties in ways that something like a RDBMS would not. The ability to query and trace paths, especially of unknown depth, through data to find its origins is also not something easily achieved in Document or RDBMS databases.
**Tino**
Hey all! How important would you think is graph db knowledge for data scientist? I know that a lot of company let rather their data engineers or others do this work 🙂
**bechbd**
I think understanding the fundamentals of graphs and graph databases is important for data scientist to understand how their work fits into the larger picture of productizing the models/work they have done. Additionally, depending on the graph/graphdb/data being worked on it may be easier to retrieve/load data for the projects you are working on from a graph database versus many CSV/Parquet files
**Ken Lee**
Totally new to Graph DB although have been hearing its name for quite some time. I am a machine learning engineer and just wondering what are the use cases of Graphs in machine learning context? TQ
**bechbd**
No problem, I actually put some answers to a similar questions [here](https://datatalks-club.slack.com/archives/C01H403LKG8/p1623683981273700?thread_ts=1623675871.268200&cid=C01H403LKG8)
**bechbd**
Additionally I gave a talk last week on this subject which you can view [here](https://www.youtube.com/watch?v=SLKW5Q_URq4)
**Ken Lee**
thanks and appreciate!
**Ajay kumar saini**
Thanks for being here bechbd and Josh Perryman , I’ve productionized multiple graph database (Agensgraph, Dgraph). One problem I faced in both graphs is the performance of graph traversal queries(connected component size, degree etc.) in highly connected data with large component size. How is it handled in other graph databases? Is the performance of these query degraded in other graph DBs as well given graph has to iterate lots of nodes?
**bechbd**
Well I guess there are two parts to this question, the performance of graph traversals on large data sizes and the performance of graph algorithms on large data sizes.
In general the performance of any graph traversal is directly related to the number of vertices and edges that it must touch. The earlier you can add the most selective filters in a query the better off the performance will almost always be. However this does still mean that starting at point a may take X ms while starting at point B may take 2X ms if you have to touch twice as many elements in the graph to get the answer. Many people don’t expect this but it is a very common quality across graph databases. The same is true with non-graph databases but the nature of traversing connected data and the corresponding branching factor at each level seems to amplify this in graph databases.
As for the time to run graph algorithms on large datasets, this tends to boil down to just the nature of the computational complexity of the algorithms themselves. Even the simplest one you mentioned there, degree, requires touching each vertex once and each edge twice. Something like connected components is O(V+E) complexity. In my experience many people don’t understand the complexity of the algorithms and therefore have expectations that no database can meet when it comes to scale and latency of those types of operations. Generally we see people end up with a sort of almost lambda type architecture where they use a batch process to calculate these sorts of statistics/attributes and then save them back into a node/vertex as a property. These properties are then used to serve some sort of real time/transactional use cases
**Tim Becker**
Hello 🙂 Thanks for doing this. In the introduction of your book you mention that graph databases are good for complex data. Do you have any way of quantifying how complex the data is and which database type to choose based on the complexity?
**Josh Perryman**
TL;DR: no, I don’t think there’s a way to quantify complexity based just on the data in and of itself.
But I think it is less about the data and more about the questions.
In my present work, and this has held for much of my past years of experience, we chose the data engine based on the question we are asking of the data at different parts of the application. Where we need full descriptions of the objects, we use a relational database. Where we need to work with the relationships between things we use a graph. Where the need is for performance and the data involved in light and slowly changing, we use a cache. Where we’re asking questions of large swaths of the data, some analytics database.
We get into this in the first chapter where we survey _\*types\*_ of questions and I think it hold here as well. We need to understand the types of questions that are important for users of the data.
**Tim Becker**
How much knowledge concerning traditional databases do you need to get the most of your book?
**Josh Perryman**
I think very little is needed, if the intent is to start building software using graph databases.
The main expectation we have is a basic capability in writing software. Most of the book will be opaque to those who don’t have some familiarity with writing and operating simple software programs.
On the data side, we use relational databases, with some SQL, as a connecting point to common developer experience. But this knowledge isn’t essential. If someone has complete an introduction to software development course in a popular language (Java, C#, Python, JavaScript) and has completed one or more non-trivial projects, projects where there’s more than one “section” or “class” or “module” or “function” in the code, then they should be able to follow along reasonably well.
Our target is developers with some level of professional experience, even if they aren’t a full-time software developer. If you have at least 3 months writing code mostly full time in a professional setting, then the book is ideal to quickly expose you to the techniques of working with connected data.
**Tim Becker**
Do you have any recommendation on how to get started with db in general? Something that would be good to read before starting with your book.
**Josh Perryman**
SQL, and relational databases, are the language and tools of the data world.
I think it good to be familiar with how relational databases function and there’s so much material for this that I hardly know where to start. I recommend looking around your world and note the following:
* _What databases are in use around me?_ Common ones are MySQL, Oracle, Microsoft SQL Server and PostgreSQL. These are the top 4 on [db-engines.com](http://db-engines.com/)
and have been there for many, many years.
* _What programming languages are in use around me?_ Common ones are Java, C# (.NET), Python and JavaScript. I rather like Python for someone just learning, or JavaScript for those who think they are interested in web development. But any of these four are broadly supported with vast communities.
* _What tools / frameworks are in use around me?_ The use of an “Object-Relational Mapper”, or ORM, is most common. Every major language has one or more.
Based on that I’d like for a tutorial that covers the database engine of choice, and uses the programming language of choice. Usually the framework or ORM tool will have a good tutorial as well. These types of learning aids can either be bound books, or online courses. Choose that based on your preferred learning style.
I think that the trick here isn’t to pick “the right one” but to pick the technical stack (engine + language + tools) which is common in your part of the world. This will give the best opportunity for applying your new skills quickly.
**Tim Becker**
How useful is a good understanding of graph theory to follow your book or is the brief introduction in chapter 1 sufficient?
**Josh Perryman**
I feel that Graph Theory is of minimal use in the actual day to day work of using graph databases. Graph Theory provided the primitives (vertices, edges) but working with connected data is more about understanding how to work with data than a mathematical theory. In my mind they are wholly separate from one another for practical work.
**Tim Becker**
btw, I read through chapter one to get an understanding of what the books is about and I already learnt a lot, so 👍
**Josh Perryman**
Tim, thank you for reading and joining us on this journey!
**Tim Becker**
Josh Perryman Thank you for answering all my question, I appreciate it!
**Amruta**
Does anyone know any good books on Graph Theory (especially for beginners)?
**Marcello La Rocca**
Hi Amruta!
* you can certainly find the basics in “Introduction to Algorithms” [https://www.amazon.com/Introduction-Algorithms-3rd-MIT-Press/dp/0262033844](https://www.amazon.com/Introduction-Algorithms-3rd-MIT-Press/dp/0262033844)
Which however covers much more than graphs, and from an algorithmic perspective.
* If you are more interested in a mathematical angle, I recommend Trudeau’s “Introduction to Graph Theory”, a classic: [https://www.amazon.com/Introduction-Graph-Theory-Dover-Mathematics/dp/0486678709](https://www.amazon.com/Introduction-Graph-Theory-Dover-Mathematics/dp/0486678709)
* About graph visualization, I liked this one: [https://www.amazon.com/Graph-Drawing-Algorithms-Visualization-published/dp/B00Y2UUWDO](https://www.amazon.com/Graph-Drawing-Algorithms-Visualization-published/dp/B00Y2UUWDO)
* I have another good one in my bookshelf, but I can’t remember the title 🤦 I’ll try to find it, but meanwhile here there is also a good source for more resources, if you are (or will be) interested in more advanced books [https://neo4j.com/blog/top-13-resources-graph-theory-algorithms/](https://neo4j.com/blog/top-13-resources-graph-theory-algorithms/)
**Amruta**
Thankyou so much Marcello La Rocca! Very helpful…Especially the second suggestion! :)
**Marcello La Rocca**
You are welcome! Glad it helped 🙂
**Shankar Somayajula**
Hi, Thanks for the opportunity bechbd and Josh Perryman to ask questions. Are there any simple/beginner tutorials in using graphs for Inference? Use graph (properties and processing) to tease out causal linkages between products or data elements?
Also can one relate this to sub-graph/directionality of the edges of the graph …
{ (customers with attributes) a, b } (buys) c => (buys) p … or (strong signal for c => p)
but not vice versa…
i.e. { (customers with attributes) a, b } (buys) p NOT => (buys) c …. or (weak signal for p => c)
**bechbd**
I do not know of any tutorials specifically around this but I bet if you looked for RDF graphs and OWL you will probably find some.
How this relates to the sub-graph/directionality is really a domain specific concept. Property graphs generally let you traverse edges in both directions but that may or may not make sense depending on the domain. For example in something like a social network you might have two people connected by a friends/follows edge. If the network is like facebook, the relationship is reciprocal and traversing the edge in both direction makes sense (i.e. Person A and Person B are friends). In the network is like Twitter, Person A might follow Person B but that does not mean Person B follows Person A
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Cleaning Data for Effective Data Science – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Cleaning Data for Effective Data Science
----------------------------------------
#### by [David Mertz](https://datatalks.club/people/davidmertz.html)
##### The book of the week from 21 Jun 2021 to 25 Jun 2021
It is something of a truism in data science, data analysis, or machine learning that most of the effort needed to achieve your actual purpose lies in cleaning your data. Written in David’s signature friendly and humorous style, this book discusses in detail the essential steps performed in every production data science or data analysis pipeline and prepares you for data visualization and modeling results.
The book dives into the practical application of tools and techniques needed for data ingestion, anomaly detection, value imputation, and feature engineering. It also offers long-form exercises at the end of each chapter to practice the skills acquired.
* [Book's page](https://www.packtpub.com/product/cleaning-data-for-effective-data-science/9781801071291)
* [Amazon](https://www.amazon.com/dp/1801071292/)
* [Book's GitHub repository](https://github.com/PacktPublishing/Cleaning-Data-for-Effective-Data-Science)
Questions and Answers
---------------------
**Alex**
Hi there, David Mertz! I’ve recently come across a realization that I, as data analyst, spend so much time extracting, cleaning and preprocessing data in order to analyze it as effectively as possible.
How would you frame the usefulness of this book for someone who is not directly related to data science?
**David Mertz**
An eminent colleague of mine (Alex Martelli) provided a very nice review of the book. He is very fond of it, but humorously scolded me for the phrase “data science” which he thinks is always sloganeering … He prefers “data engineering.”
While I wouldn’t go full in with that chastisement, this book definitely is on the engineering side of things. I only minimally touch on specific machine learning models, but entirely on the concrete effort needed to get data ready for being modelled, visualized, statistically analyzed, or otherwise put to work. I guess my disagreement with Alex might simply be that I believe that concrete science (like physics, biology, chemistry, ecology, etc) is always just as messy and requires just as much engineering. Hence, I think the word “science” is okay.
To your actual question, I try to provide as much as I can for how to make your data USEFUL, without great attachment to the admittedly buzzword-ish phrase “data science” per se.
**Lalit Pagaria**
David Mertz thank you for doing this.
1) We have so many end to end tools for ML pipeline. But I have observed pre processing part in it always semi manual. I also find very less tools solving this. Why is so? My main focus is towards Text besed NLP domain.
**David Mertz**
I am not really optimistic about something like _Autoclean_ coming to exist on the analogy of _AutoML_. To simplify greatly, AutoML is kinda just “throw a bunch of different models at a problem, and compare the metrics. Cleaning data requires a much more nuanced understanding of _why_ the data has gone wrong, and how to remediate it.
There’s an epigraph that I start an early chapter with, from Hadley Wickham (borrowing Tolstoy, of course): Tidy datasets are all alike, but every messy dataset is messy in its own way.
**Lalit Pagaria**
Thank you David. It is insightful
**Lalit Pagaria**
2) For text cleaning (specifically large documents and paragraphs in QA domain) what framework do you suggest?
**David Mertz**
I’m afraid I largely have to plead ignorance here. I use, and write about things like NLTK, but I also haven’t worked as much in NLP for a number of years. There could be more tailored tools for this domain that I am not aware of in my attention on mostly tabular, mostly numeric, data.
**Lalit Pagaria**
3) Fun question, is it possible to use BERT based model to clean text itself (using model for pre processing)? Of course with proper training using good dataset. 🙂
**David Mertz**
Maybe. A colleague who advised me during writing really felt I should have a chapter on using models for the data cleaning itself. I haven’t worked with BERT as of date, but in general I felt that this topic was _interesting_ but still at the stage of _promising_ rather than ready for produciton.
**Ken Lee**
David Mertz thanks in advance for answering this question of mine.
You mentioned the use of version tracking of data. It’s a good practice, definitely agreed. Any great tools that you would recommend for this purpose? Heard of DVC but never use it before.
**David Mertz**
I have myself mostly just used regular Git tracking of data, for example with GitHub LFS. That is relatively minimal change sets for textual data formats.
However, I know there is an important limit in this approach. While it is good for saving linear sequences of changes, it doesn’t completely scale to more complex branching patterns when data has shared provenance. I do not know if DVC handles that well, but I probably should.
**Ken Lee**
Thanks David! Will definitely check out about Github LFS
**David Cox**
David Mertz Love the idea for this book and excited to check it out! It looks like it has a lot of great recommendations for how to handle different file types and common errors. My questions are around tools and technologies for helping speed-up the cleaning process. For example, do you have recommendations or insights on the benefits and drawbacks of different tools such as Open refine, Trifacta, Cloudingo, etc.?
**David Mertz**
Much of what I talk about doing with Python, R, or other tools with a programmer focus, can be done in those UI front ends as well. In the book, while I give particular examples, I try to remain tool agnostic. So even though I might show some Pandas code to perform a certain step, I do not mean it to be a book about Pandas. Doing that step with Open Refine, or Trifacta, or Cloudingo, is just another option, much as would be doing it in the R tidyverse.
That said, I will remain more fond of the kinds of tools I write about personally. Part is simply my old-timey attachment to command line and programming language tools. The UIs like those you mention are targeted at folks somewhat less comfortable with programming.
But beyond mere personal preference, these “friendly” UI tools have some real drawbacks. On the one hand, many are commercial, which means both that they have extra costs per-seat, but even more importantly, means that they have some vendor lock-in. Once you’ve developed complex work-flows in one tool, it becomes more difficult to switch tools.
Maybe even more important than the lock-in concern is when you reach limitations of a tool. For 90% of what you need to do in data cleanup and preparation, those tools will probably do it. However, once you reach that other 10%, it becomes very difficult to add a new data manipulation, and usually means going back to the programming tools that they are meant to avoid.
In contrast, with Pandas, or scikit-learn, or the R tidyverse, you can also certainly hit limitations in the provided capabilities. But adding more—well, it still might be a pain in the ass, but it’s a PITA using the same toolset—it is always possible to write custom functions that plug into the hooks provided by those general libraries. The tools are designed around such extensibility in significant part.
**David Cox**
Thanks, David Mertz! Very helpful. I think one of the big challenges I run into with my team is simply the time it takes for proper data cleaning (says all data scientists everywhere 😂). Do you have any recommendations or best practice suggestions on building efficient and automated pipelines for these data cleaning tasks?
**David Mertz**
I do. But basically, it takes 300,000 words to express them :-)
… I should go do a word count on the book to fine tune that number.
**David Cox**
Hahahaha…fair enough!
**Asmita**
Hi David Mertz, I have recently dived into the Data Science field and realised obtaining an efficient set of input data consumes the maximum time. Like you said every messy dataset is messy in its own way, and this might result in the same process being performed on all datasets irrespective of it works or not. While understanding the data, and feature engineer, visualisation can help, how else can we figure out which technique of cleaning will give a better result?
**Asmita**
Also what advise would you give to someone who is new to the concept and is learning about the process of cleaning data ?
**David Mertz**
For better or worse, I think there really is just a lot of trial and error needed to find best practices, which are often specific to your domain and problem set.
At some end of the process, you do something like train a model and have metrics to judge quality. If those measures do not show success, ONE thing you should try is massaging the data preparation/cleaning process.
Visualizations can definitely often help. Finding the right angle in which to present data can many times make the problems really stand out. But that’s very much an iterative process as well. Many visualizations which are completely sensible in principle, even valuable, may not be the ones that actually expose the problems.
**Asmita**
So is it always trial and error, even with experience helping out one understand the data and insights that can be obtained?
**Neal Lathia**
❔ At what stage of trying to clean & make sense of data would you give up and get the upstream system to log better data?
**David Mertz**
This feels like a social question at least as much as a technical question. I might have a variety of kinds of relationships with the upstream provider, and the influence I might have over their processes varies hugely.
Unfortunately, we as data scientists or data engineers often have little influence on that upstream. This can be true within one organization, but that much more so when different companies, agencies, etc are at different ends of the data pipeline.
I don’t think a cut-off or threshold is the best way to think of it. Rather, universally, yes there are problems. But almost always there is _some_ utility in the current data. So it tends to be about incremental, and ongoing, improvements to the upstream data, inasmuch as we influence that.
**Vladimir Finkelshtein**
Apart from manual inspection or anomaly detection is there a way to know that the data needs more cleaning?
I have seen this idea in some content from Andrew Ng, in image classification they purposefully added some amount of mislabeled images and calculated the reduction in the performance of a model. This should give an estimate of how much performance could be gained from correcting certain amount of labels. I wonder if similar approach can work with tabular data by, say, randomly assigning extremal values in some columns…
**David Mertz**
I think some of these concepts from machine learning models don’t entirely apply to data engineering (cleaning) steps. We don’t really have straightforward metrics in the same way
Of course we can repeat statistical tests, or perform new ones, but doing so is informed by domain knowledge or task purpose, rather than by any independent measures of goodness/badness.
If you assign random values to data cells to validate your remediation, it’s completely predictable how much you’ll detect from the probability distributions you use for randomness. But you don’t really learn anything new thereby.
**David Cox**
Another question I’ll pop out here, in case others are interested. How do you go about making decisions on how much/what to clean for operational databases versus pushing to data lakes and cleaning if/when needed? Asked differently, the volume of data many orgs handle makes it difficult to clean everything. What is your general strategy for picking what data to clean to maximize operational and innovative efficiencies?
**David Mertz**
This is a great question! I’d be very interested to read how others in this thread weigh these concerns.
For myself, I find that data cleaning is sufficiently task driven that it often not possible—but more specifically, not even meaningful—to “clean the data” without that’s to some particular goal.
**David Cox**
Agreed! I’ll be interested to hear what others indicate, as well. Some of the artful balance we are trying to strike is figuring out what data might be meaningful/relevant to various business decisions/operations before they specifically request it so that we can more efficiently respond to higher probability data requests.
**Heeren Sharma**
I totally agree with this approach as well. I think cleaning is highly contextual and use case driven. Some teams take this decision based on the frequency with which the particular data is being used. However in my personal experience, devil lies in the detail. A financial data which needs to be used once a year but needs to be of correct quality and on the other hand, tweet/retweet stream where you are reading continuously but doesn’t care much about text encoding (or other elements) of each and every tweet. So all is about use case binding for me as well.
**Shankar Somayajula**
Hi David Mertz, Cleaning data in a BI context usually refers to assigning defaults or unassigned dimension tags/keys to records which fail regular lookups… i.e. Have a special dimension value for each dimensional attribute so that this gets assigned in case nothing else works e.g. case when then else end. This effectively ensures that important event/fact/transactional information is retained and not lost if/when one enforces the fact to lookup/dimension joins in downstream applications.
Coming to your book (from the ToC): Why impute missing values via some process rather than simply assign them a default/special category of “unassigned”? Can’t we view Data Science usage as yet another example of downstream usage of data (just like Business Intelligence)?
**David Mertz**
Use of sentinels for missing (or unreliable) data is definitely relevant and sensible. It’s downstream from the point where that is done that value imputation or discarding rows becomes an important decision.
For example, numeric values like 999.9 and -1 are often used to mark missing data in formats that require numeric columns. As long as those do not overlap with plausible measurement values, they are good sentinels.
However, when you want to perform visualization, or statistical summary, or machine learning modeling, on the data, those sentinels become problems… specifically _because_ they are implausible data values. At that point, doing things like imputing median values in their place is often more useful (when accompanied by appropriate footnotes or metadata that make clear what you have done).
**Shankar Somayajula**
I take your point regd numeric fields and defaults like 999.9 and -1 or -99 etc. Normally we should just have Nulls instead of these special values. Median calculations should ignore such null value records.
However, for categorical fields, i find the approach of imputing most prevalent value (mode) during Cleaning slightly worse than choosing sentinels (unassigned but valid, default value indicating missing data) as in my opinion that decision is not reversible (even if documented) and ought to be in the hands of the ultimate users of the data - The Analysts/Data Scientists etc.
I don’t see what difference having implausible data values has for categorical variables (or for Viz/Stats/ML activities) when they too have been documented and catered for via defaults for missing as has been the case with BI typically.
I would rather have 90 valid records with valid values and 10 records with missing categorized as “unassigned” letting me choose what to do with those 10 records than all valid values for this field across 100 records with metadata note: 10 missing have been replaced by median value .
**Heeren Sharma**
Hi David Mertz I was searching for a book like this for quite some time as I realised that discussions around data quality are still somehow centered around commercial tooling. This book is definitely a refreshing breeze. 🙂 As a data engineer (and on the top a consultant 🙈), I have come across various discussion which centres around data cleaning, a quick blame game jump over data quality and then somehow eureka moment style statement “We need a data catalogue 😄 “. Building on the top of David Cox’s last question, do you also see data cleaning and data quality in the similar light? And, may you provide some thoughts around data catalogues for primal use to boost data quality. I am trying to explore to capture the knowledge “why you are cleaning that data” into some sort of another structured and persistent format (JSON/ YAML config file) rather than code only.
**David Mertz**
Data catalogs are definitely valuable. Part of metadata can and absolutely should include known problems… as well, of course, as version information on datasets that have undergone various remediations (cleaning).
I avoided discussing closed source tools in the book. This is both a natural distrust of them (who knows if it will become unavailable or lose features), and because they create walled gardens. But some tools like Apache Atlas and Truedat do exist.
**Heeren Sharma**
Many thanks for your insights! Another quick follow up question, do you recommend a way/process in keeping data catalogues up to date? That’s also something that I have seen quite a bit in industry as a challenge. 🙂
**David Mertz**
I’m afraid I don’t have any specific insight about that currently.
**Tim Becker**
Hi David Mertz, thank you for doing this. Does your book provide recipes for how to best solve different data cleaning tasks?
**David Mertz**
I very deliberately eschew “recipes.” Understanding the specific problem and specific dataset is crucial, and reducing a solution to a cargo-culted recipe is almost always going to do the wrong thing for your specific purpose.
What I do discuss in considerable depth is how to _think about_ the problem in hand, and likely techniques with which to approach it.
**Tim Becker**
What are in your opinion the most important things to improve in order to become more efficient at data cleaning?
**David Mertz**
Practice and an understanding of your domain. I mean, of course understanding particular tools like Pandas or R tidyverse are of huge value… but knowing what you _want_ to do must come before simply learning APIs.
**Ken Lee**
Been seeing APIs appearing continuously.. just wondering what exactly is an API? David Mertz
**David Mertz**
Oh… Application Programming Interface. It just means the names and arguments of the functions and methods that some particular software library or tool provides.
**Tim Becker**
thank you 🙂 David Mertz I guess, there is no easy way around data cleaning and having domain knowledge.
**A McCauley**
Hi David Mertz data cleaning is quite a repetitive task, are you a fan of trying to automate as much of the process of data cleaning, such as stages of detecting areas that need cleaned etc?
**David Mertz**
Obviously automation save a whole lot of time. I’d have a certain caution about some of it though. If you really are automating cleanup/transformation of the SAME data source that has characteristic problems, of course it’s good to reuse the scripts you wrote in your first analysis.
But applying those to very different data is likely to miss crucial issues. Moreover, sometimes data coming from the same source can change character in certain ways, which may mean that you need to be sensitive to new kinds of cleanup issues that arise.
**WingCode**
Hi David Mertz,
Thou book shall be hailed amongst thy mortal Data Engineers! Humor coupled with anything, I find is a fun way to learn about anything. I would love to read your book! I had a few questions:
1. Does one of the reasons. that data cleaning issue stems from the fact of not having proper schema registry or API format (OpenAPI) ? If we have proper schema registry in place everywhere, we can track the evolution of schema and schema registry itself ensure that missing fields, new fields, changes to schemas are handled. Using codegen coupled with API format, we can ensure that clients don’t break with data changes.
2. One of the biggest problems I have faced with respect to data cleaning is deduplication. The same information can be represented in magnitudes of different ways. For example: 2 Product description of the latest & greatest phone. “This is the fastest phone from Pineapple with 256GB of RAM, 1GHz processor and available in all color spectrum under the RGB space” Vs “Coupled with 256GB of RAM, 1GHz processor and available in all colors you can dream of, this is the Pineapple phone”. Is there any good library, frameworks which you found useful in deduplication of data?
3. Assume a Utopia ( that one I am dearly waiting for) every single person & organisation follows the same proper data standards (naming conventions, schema standards, JSON response standards). Do you think we will have data cleaning issues?
**David Mertz**
I’ll go by parts.
1. Having (versioned) schema registries might help a certain proportion of data integrity issues. My feeling is that it’s a fairly small percentage though.
I find that errors because of instrumentation faults, human transcription errors, bias in selection, recording failures, etc. are much more prevalent. All of those apply just as much even when the schema is clearly understood and followed by all ends of the data processing.
**David Mertz**
1. I think the question is about plain text descriptions that may describe the same thing but be worded differently. I’m not sure. I discuss in some moderate detail analyzing string similarity in the book.
That said, for something like a product description, anything mechanical will probably fail. Version 16 and Version 17 of the same line are likely to use largely the same marketing language (and hence have similar strings). More sophisticated semantic concept clustering doesn’t improve that problem. In fact, even a different product, in whatever version, is likely to use similar marketing language, especially from the same maker.
Anything you do that says “equivalent descriptions name the same product” will get as many false positives as true positives.
**David Mertz**
1. In your utopia, do all instruments record measurements correctly? I guess some religious doctrines have a notion of an afterlife of such perfection, so I suppose if you follow one of those, it’s something to hope for.
**WingCode**
Thank you David for your time and replies. I wish you all the best for the book!
**David Mertz**
It’s been nice having folks comment this week. I’m happy to reply for however many more hours the timezone indicates, or indeed in general personally, to any questions.
Can I ask participants a favor? Apparently more Amazon reviews helps convince them the promote my book, and Packt essentially makes itself a subsidiary of Amazon (not happy about that; but such is life). So if contributors to this thread write reviews (and buy the book, of course), it helps me.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Science on AWS – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Science on AWS
-------------------
#### by [Chris Fregly](https://datatalks.club/people/chrisfregly.html)
, [Antje Barth](https://datatalks.club/people/antjebarth.html)
##### The book of the week from 28 Jun 2021 to 02 Jul 2021
With this practical book, AI and machine learning practitioners will learn how to successfully build and deploy data science projects on Amazon Web Services. The Amazon AI and machine learning stack unifies data science, data engineering, and application development to help level upyour skills. This guide shows you how to build and run pipelines in the cloud, then integrate the results into applications in minutes instead of days. Throughout the book, authors Chris Fregly and Antje Barth demonstrate how to reduce cost and improve performance.
* [Book's page](https://www.oreilly.com/library/view/data-science-on/9781492079385/)
* [Book's page on Amazon](https://www.amazon.com/dp/1492079391/)
* [Book's website](https://www.datascienceonaws.com/)
* [Book's GitHub repository](https://github.com/data-science-on-aws/workshop)
Questions and Answers
---------------------
**Antje Barth**
Hi everyone! Chris Fregly and myself are here to answer your questions!
**Antje Barth**
We’re also running a live, 4hrs hands-on workshop tonight, starting at 6pm CET. The workshop is based on our book and code repo: Build an End-To-End Pipeline With BERT, TensorFlow, and Amazon SageMaker. You can RSVP [here](https://pages.awscloud.com/NAMER-field-OE-Machine-Learning-Dev-Day-2021-reg-event.html)
.
**Lalit Pagaria**
Thanks for sharing link. Is it possible to share link with other community?
**Antje Barth**
Sure, please do!
**Kshitiz**
Hi Antje Barth , Would this session be recorded? It will get pretty late in IST.
**Ken Lee**
Hi same question here Antje Barth, it would be pretty late in SGT too
**Antje Barth**
We do have previous workshop recordings shared here:
[https://youtube.datascienceonaws.com/](https://youtube.datascienceonaws.com/)
For example, [the May workshop](https://www.youtube.com/watch?v=sL8wpFo7LaQ)
**Chris Fregly**
yesterday’s workshop here: [https://youtu.be/R0dyrBQMAnQ](https://youtu.be/R0dyrBQMAnQ)
**Ken Lee**
That’s great stuff! Thanks
**Agrita Ga**
Hi Antje Barth and Chris Fregly. 👏
Is there currently any standard (or trend) for MLOps _best practices_ in context of AWS tech stack?
**Antje Barth**
I think it’s going to be a phased approach
1/ Moving away from manually building models to
2/ Orchestrating the individual model building workflow steps (ie. data preprocessing, training, evaluating) in a pipeline, and automating tasks within each step
3/ Automatically re-running training and/or deployment pipelines on triggers such as model decay, or code changes
The tools and technologies will likely vary based on each use case and team experience. AWS recently launched SageMaker Projects, which automatically sets up a CI/CD automation for your ML pipelines. So whenever you commit new code into a code repo, it would re-run the model building and deployment pipeline.
On manual approval, you could deploy into a staging environment, on a second approval, after running integration tests, you could deploy into a production environment. This is just a sample setup.
We also describe this approach in our book!
**Bayram Kapti**
Hi Antje Barth & Chris Fregly! Thanks for sharing your thoughts here with us!
While every problem requires different approach and solution, it’s said that most of the times, fundamental algorithms in DS are the most effective solutions to actually getting results.
1-) Do you agree? How has it been working for you so far?
2-) How does it work at #aws? Are there systems, procedures built at Amazon to make sure best solution is used when approaching a problem?
**Antje Barth**
You should always go for the solution that solves your problem the fastest and in the most efficient way. For example, if you need a simple image recognition model, why build the model from scratch yourself, if there’s plenty of algorithms and models in the industry that have already solved this problem? Re-use and customize it to your needs. You should focus your time on solving the actual business problem, not figuring out how to train a model or managing infrastructure.
Coming back to AWS, I would always recommend looking at the pre-built AI services first, ie. Amazon Rekognition for object/video detection, Amazon Comprehend for NLP, Amazon Textract for document text extraction, Amazon Personalize for personalized recommendations etc. If they already solve your problem, great.
If you need to build and train your own model, go to the next level, Amazon SageMaker. This service gives you the freedom to either use built-in algorithms, write your own code, or even bring your own Docker images with your custom libraries. This layer still takes care of all of the infrastructure management for you.
AWS always works back from the customer in solving customers’ challenges.
**Bayram Kapti**
3-) Also, how does your day look like working at aws?
**Antje Barth**
Getting up, a lot of coffee ☕😅, checking emails and Slack, a few calls with team members, working on demo code, presenting at a conference, running a workshop, dropping in here and answer #book-of-the-week questions! 🙂
**Alex**
Hi there Antje Barth & Chris Fregly! Pleasure talking to you 🙂
What are the three most important things that someone using AWS (instead of any of the other main cloud providers) for ML/AI purposes can brag about?
It’s known that top-tier cloud providers offer very similar services, so how would you convince someone who doesn’t know which one to start using to get into AWS? And more specifically, to AWS _for data science_?
**Antje Barth**
AWS always approaches everything they do by focusing on their customers. This customer obsession drives 90 percent of the product roadmap, leading to the invention of a bevy of machine learning services at all three layers of the stack that I mentioned earlier:
For example, at the bottom layer of the stack AWS builds their own custom silicon designed to accelerate deep learning workloads with AWS Inferentia.
In 2017, AWS launched Amazon SageMaker – a fully managed service that enables developers to quickly build, train, and deploy machine learning models at scale.
And at the top layer of the stack, AWS offers the broadest set of artificial intelligence services – with little to no ML experience required.
This includes entirely new categories of services enabled through machine learning including Amazon Kendra to reinvent enterprise search, Contact Lens for Amazon Connect to embed intelligence into contact center operations, Amazon HealthLake to help healthcare organizations make sense of their data, and machine learning services purpose built for industrial and manufacturing companies just to name a few.
Also, AWS is the only cloud provider to offer a choice of Intel, AMD, and Arm processors. And for running machine learning at scale and in production, AWS introduced the Inf1 instances for EC2, which are powered by AWS Inferentia chips.
So I’d say it’s the combination of the depth and breadth of services, together with the fast pace in innovation, always working backwards from customer challenges, and the experience and maturity after 15 years in business paired with millions of active customers and tens of thousands of partners globally.
**David Cox**
Very excited to see this book here! Riffing off Alex’s post, my question is why AWS? Why contain everything within the AWS ecosystem?
**Chris Fregly**
I use the AWS ecosystem everyday - works very well for me! 🙂
**Chris Fregly**
We work with a lot of customers that use open source container (ie. Kubernetes), analytics (ie. Apache Spark), and machine learning (ie. TensorFlow/PyTorch/Scikit-Learn) technologies with AWS. AWS handles the “undifferentiated heavy lifting” of managing your own infrastructure. Would you like to spend your time debugging lower-level infrastructure issues? Or would you like to spend your time solving higher-level business problems? I personally like to solve business problems, so I use AWS for my analytics and ML infrastructure needs.
**Heeren Sharma**
Hi Antje Barth Thanks for the heads up regarding workshop. I have not played around with Sagemaker but have experience with Azure ML. One challenge that I see time and again is integration among services e.g. for a full blown data pipelines which leverages ML realtime scoring ( and maybe training), something like a feedback loop related, how well AWS Sagemaker integrate well with overall AWS services (e.g. with a Glue job, etc.)? Another quick follow up question, how well code integration work e.g. can one connect with jupyter servers with their local IDE, are there extension? Many thanks.
_Disclaimer: I am primarily coming from AWS world, but since past 1 year working over Azure ML for a client project. So I am curious how AWS solves these problems_ 🙂
**Chris Fregly**
Amazon SageMaker integrates with many of the AWS services - and we are always adding new integrations based on customer feedback. More specifically, SageMaker Pipelines integrate with [EventBridge](https://docs.aws.amazon.com/sagemaker/latest/dg/automating-sagemaker-with-eventbridge.html)
to trigger a Start Pipeline event or respond to state changes within your pipeline execution. Also, SM Pipelines supports a flexible [CallbackStep](https://docs.aws.amazon.com/sagemaker/latest/dg/build-and-manage-steps.html#step-type-callback)
that lets you call any service directly using Python, Java, etc.
**Heeren Sharma**
Many thanks! Sounds awesome.
**Fernando Lichtschein**
Hello! Is there a workshop that can be done by an IT student using an AWS Educate classroom account that can demonstrate a real pipeline? Ideally it should not involve too much knowledge of AWS services,
**Antje Barth**
I’m not super familiar with the Educate account, but we recently launched a Coursera specialization in collaboration with [DeepLearning.AI](http://deeplearning.ai/)
, called Practical Data Science.
It comes with 3 courses 10 weeks of lecture videos, quizzes and on-demand, self-paced hands-on labs! (no personal AWS account required, you get access to a demo account): [practical data science](https://www.coursera.org/specializations/practical-data-science)
This might be a good path to start!
**Ken Lee**
Looks great! Thanks Antje Barth will start on this
**Fernando Lichtschein**
Great, thank you!
**Asmita**
Hi Antje Barth and Chris Fregly, will the book be helpful for beginners who are currently learning AWS and do not have much hands on experience? What advise would you give on how to start working on AWS?
**Antje Barth**
If you are completely new to AWS, you might want to start with a general, intro-level AWS course such as the [AWS Cloud Practitioner](https://www.coursera.org/learn/aws-cloud-practitioner-essentials)
, or [AWS Fundamentals](https://www.coursera.org/specializations/aws-fundamentals)
course.
As for our book, we always start with an introduction and discussion of the relevant concepts first before we go into the AWS specifics to help readers follow along. So as long as you bring some basic cloud and ML knowledge, readers should be able to follow along easily.
**Philip Dießner**
Hello Antje Barth and Chris Fregly.
Thanks for being here! Where do you see your book on the way to becoming AWS ML Certified? Is it + some more practice and experience a good basis? Or are there areas where other learning resources should be used?
**Antje Barth**
The book definitely covers a good amount of ground towards the ML certification. In addition, I’d spend some time hands-on, familiarizing yourself with the services and concepts. And as a final check, scanning through the specific ML cert preparation materials!
**Livsha Klingman**
Hi Antje Barth & Chris Fregly! Thank you for being open for Q&As!
Being a trained data scientist but my employment pushing me more into the data engineering spectrum, due to the complexity & velocity that data is now flooding the ‘market’ with, and the greater need for pipelining data to apply ML/AI applications. After successfully building a few pipelines, both in Azure and AWS, I find myself quite bewildered by the array of AWS platform tools and almost unsurmountable amount of syntax surrounding each. Does your book provide a method of selecting the correct tools or combinations of for each given data scenario, to optimize the pipeline, as well as maximize the data prep needed for ML in a way that speaks not just to professionally trained data engineers? The documentation ‘out in the market’ I find quite overwhelming coming from a data science stance and designed for professional data engineers at large. My role is purely ML/AI goal oriented data pipelining.
**Chris Fregly**
the book is focused on ML/AI-oriented data pipelines, yes. the first couple chapters show the breadth of services available for different use cases. the remaining chapters dive deep into building an end-to-end pipeline for a single ML/AI use case. we preferred this approach because it provides a specific slice through the broad set of AWS options across the analytics, machine learning, streaming, and security services on AWS.
**Livsha Klingman**
Thank you!
**Ganeshkumar**
Hi Antje Barth and Chris Fregly , thank you so much for Q&As.
1. Currently AWS does not offer drag and drop feature in UI, which may simplify things while model creation. When can we expect such feature in Sagemaker?
2. In your view how much of data architecture skill is required for data scientist to excel in building an effective model for consumption
?
**Chris Fregly**
AWS offers drag-and-drop for the data/feature engineering step of the pipeline. This service is called SageMaker Data Wrangler and it’s integrated with SageMaker Studio. we cover this in the book. I can’t comment on the roadmap for SageMaker, but we prioritize features based on customer feedback - and i will pass this request on to the SageMaker team!
**Chris Fregly**
having a good data foundation is very important to building effective models. everything starts with your data.
**Ganeshkumar**
Thanks very much !
**Livsha Klingman**
Hi againAntje Barth & Chris Fregly!
Taking full advantage of this opportunity… Due to my novice data engineering stance and my ‘on-the-job’ training. I basically devised a ‘split personality’ method of focusing on the engineering -Part 1, the most optimal ETL/ELT pipeline to a final data structure, and then and only then focusing on the specific data requirements to implement the Part 2- Data scientific or analytical ML algorithms or visualizations applications.
My question is…Is this really in the long run (the final goal in mind) inhibiting my optimized process. I see the benefits of a final comprehensive data structure at the end of the ETL/ELT process, but maybe the final data manipulation output (ML or AI) specifications should really be incorporated into the initial pipeline? Does it make a difference whether processing BIG data or not? Within the AWS platform or not?
**Chris Fregly**
definitely depends on the dataset (where is the data located?), model type (how difficult/time-consuming to re-generate the features for this model type?), organization structure (are you sharing this data with other teams outside of yours?), security needs (do you need to lock down certain features to certain teams?), etc. data pipelines can get complex, for sure. if you the data transformations are being re-used by many teams, you might want to use SageMaker Feature Store to store and manage your transformed features - [feature-store](https://aws.amazon.com/sagemaker/feature-store/)
**Chris Fregly**
SageMaker Feature Store is supported by SageMaker Pipelines, btw.
**Livsha Klingman**
I am going to look into this - thank you for the direction and your time!
**Livsha Klingman**
Another… Antje Barth & Chris Fregly!
Do you have any ideas how I can boost my knowledge and professionalism within my constraints of my full time job?
When considering the whole picture, the pipeline AND the final output. What should the list of focus points and considerations in order of importance? What are the typical pitfalls to be aware of when integrating into the AWS platform?
**Chris Fregly**
data is the most important. useful data transformations that produce useful features.
**Livsha Klingman**
Does this mean that the pipeline should be adapted for the features involved and not adapt (slice) the data after the data loading?
The ideal, for producing a data science goal, would be to pipe the necessary features and not the whole data structure, like I am currently doing, only filtering out data (features) after all of the important data has been cleaned and stored in the correct structure?
**Livsha Klingman**
Thanks so much your time and availability…Antje Barth & Chris Fregly
How do you see the tools in AWS providing an optimal medium for building & training a model? Is using the AWS infrastructure and their given tools really more effective for modeling, than experimenting with the vast assortment of other algorithms, NLP, CNNs and general deep and unsupervised learning tools?
**Chris Fregly**
Amazon SageMaker - and the broad AWS set of analytics tools such as Amazon Athena and Redshift - support all available algorithms for NLP, CNN’s, deep learning, supervised learning, unsupervised learning, reinforcement learning - everything! AWS provides the managed infrastructure that allows you focus more on the business problem and less on the infrastructure. You can also scale out your data processing and model training/tuning/deploying with a simple API call or click of a button. In other words, AWS does not limit your options in any way. You can easily convert the Python code running on your laptop to a scalable SageMaker Job. If it runs locally, it can run on the SageMaker infrastructure.
**Livsha Klingman**
Amazing - almost too good to be true… so what’s the snag? Why am I not crossing paths with others using AWS in this way?
**Sara Lane**
The AI field is constantly evolving, and Amazon keeps adding new features to its AI suite. I think that many people are actually taking advantage of these features, and I think that usage will grow as time goes on.
**Tim Becker**
Hello Antje Barth and Chris Fregly, I would like to ask you a few questions concerning the book. Of course, the book focuses on AWS, but is the covered knowledge transferable to other platforms? For example, if my company is using google.
**Chris Fregly**
Sure, the book discusses the concepts/problems at a high level, then dives deep into how to implement the solution on AWS. throughout the book, we provide best practices. Some of the cost-savings and performance-related tips are AWS specific, but the concepts are transferrable across all environments.
**Tim Becker**
Do you have some recommendations on how to practice cloud services for free?
**Chris Fregly**
SageMaker supports free tier. Just make sure to clean up the resources when you’re done to avoid extra cost beyond the free tier. ie. shutdown the notebooks, etc.
**Tim Becker**
What kind of performance can we expect from models developed with SageMaker Autopilot?
**Chris Fregly**
SageMaker Autopilot builds a set of model pipeline candidates using various algorithms and feature-engineering steps using Automated ML (AutoML). Performance varies depending on your dataset and type of problem (classification, regression, NLP, multi-layer perceptron, etc). The cool part is you can just point Autopilot to your dataset and try it out. Your mileage may vary, etc.
**Tim Becker**
thank you for your answers, is testing of autopilot included in the free trial?
**Ken Lee**
Hello Antje Barth and Chris Fregly. What are the advice on a proper machine learning project that accumulates as little of technical debt as possible? A lot of the tutorials found online taught us on the HOWS to build a model, but did not cover most of the stuffs other than that. Or can we find answer in the book?
**Chris Fregly**
In an ideal world, you would have clean data, perfect feature transformations, the best algorithm, and the optimal set of hyper-parameters. This way, you would just train your models and push them to production. This is rarely the case. Anything beyond the ideal scenario will incur some technical debt. The ideal state would be a perfectly-tuned experimentation pipeline that lets you easily try, track, and compare different combinations of data, feature transformations, algorithms, and hyper-parameters. This is what SageMaker Pipelines (integrated with SageMaker Experiments) offers as a managed service to let you focus on the business problem.
**Chris Fregly**
short answer: the less infrastructure code you need to write, the less technical debt you will incur as you are focusing only on the business problem.
**Chris Fregly**
and yes, the book covers a clean, end-to-end implementation of the common machine learning task of text classification using natural language processing (NLP) and BERT.
**Ken Lee**
Great! Thanks for the tips! Appreciate jt
**Ken Lee**
also particularly curious on the Model Monitoring aspects of the deployed model, aware that ML models invariably suffer from performance degradation. Which AWS services cover the monitoring part? Also regarding the containerization of application on AWS, do you have a good resource to point us to or it is already covered in the book?
**Chris Fregly**
SageMaker Model Monitor is the AWS service that monitors models for drift and degradation in production. Model Monitor uses an open source library called `deequ` (a library on top of Apache Spark) to continuously monitor the live model-prediction inputs for statistical drift against the baseline training dataset - as well as model prediction drift. For other types of statistical analysis, check out SageMaker Clarify and the accompanying open source library `smclarify` .
**Chris Fregly**
Containerization of applications should be very well-documented as containerization is so common these days. If you hae trouble finding examples, ping me and i can point you to some references. If you mean containerization of the trained models, we cover this in Chapter 10 when we talk about deploying models.
**Chris Fregly**
Oh, and it’s worth highlighting that SageMaker Model Monitor is integrated with Amazon CloudWatch to automatically notify you if your model starts to drift or degrade outside of a given threshold. This is a very powerful mechanism to help wake up the data scientist at 2am! 🙂
**Ken Lee**
Great stuff! Thanks Chris Fregly 👍👍👍
**Ricky McMaster**
Hi Antje Barth & Chris Fregly, many thanks for doing this!
My first question is on tool choices - I see from the preface that you cover both Kinesis and Kafka, but generally speaking you understandably stick to AWS products. Do you discuss (or can you imagine) use cases where it’s best to opt for a more customised solution? For example, more than one company has had scaling issues with Athena - how do you address this topic?
**Chris Fregly**
AWS offers Managed Streaming for Apache Kafka (MSK) which uses open source Apache Kafka as an alternative to Amazon Kinesis. However, I also see customers that prefer to manage their own Apache Kafka clusters on EC2 directly. These are all valid options - and the choice depends on many factors including the skillset of the team, etc.
**Chris Fregly**
I’m not familiar with the specific scaling issues that you mention. In those cases, the customer should work with AWS Support to diagnose the issue and find the right solution forward. Amazon Athena is based on - and compatible with - Apache Presto, so the customer could choose to manage their own Presto cluster on EC2 with minimal changes to their data pipelines.
**Ricky McMaster**
Ok thanks for your take on this Chris Fregly, appreciated. The Presto approach you cite is something I’m a little familiar with.
**Ricky McMaster**
Secondly, do you see any scope at AWS for developing its own framework(s) to rival either TensorFlow or PyTorch?
**Chris Fregly**
Amazon has contributed Apache MXNet to the Apache Foundation which is similar to TensorFlow and PyTorch. Just like any large enterprise, we use many different machine learning, deep learning, and reinforcement learning frameworks for our business needs.
**Ken Lee**
Correct me if I’m wrong but I recall that Amazon is a major contributor of Pytorch, alongside with Facebook?
**Alper Demirel**
Hi, first of all thanks for being here. What should be considered when migrating an ML project from local or another cloud platform to AWS? What are your suggestions about this?
**Antje Barth**
Depends on the complexity of the workloads you want to migrate.
It could be as simple as using your local IDE and configuring a connection to AWS, importing Python SDKs and start launching jobs with Amazon SageMaker.
For fully operational ML workload migrations, the plan would be more like:
1/ Plan the migration
* Validate source code and datasets
* Identify target build, train, and deployment instance types and sizes
* Create capability list and capacity requirements
* Identify network requirements
* Identify the network or host security requirements for the source and target applications
* Determine a backup strategy
* Determine availability requirements
* Identify the application migration or switchover strategy
2/ Configure the infrastructure
* Network, security, storage
3/ Upload the data and code
* Migrate datasets to provisioned S3 buckets
* Package ML training/hosting code as Python packages and push to provisioned code repos
4/ Migrate the application, and cut over
**Alper Demirel**
Thank you very much, what you said means a lot to me. It will work great for me 🙏
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Engineering MLOps – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Engineering MLOps
-----------------
#### by [Emmanuel Raj](https://datatalks.club/people/emmanuelraj.html)
##### The book of the week from 05 Jul 2021 to 09 Jul 2021
MLOps is a systematic approach to building, deploying, and monitoring machine learning (ML) solutions. It is an engineering discipline that can be applied to various industries and use cases. This book presents comprehensive insights into MLOps coupled with real-world examples to help you to write programs, train robust and scalable ML models, and build ML pipelines to train and deploy models securely in production.
The book begins by familiarizing you with the MLOps workflow so you can start writing programs to train ML models. Then you’ll then move on to explore options for serializing and packaging ML models post-training to deploy them to facilitate machine learning inference, model interoperability, and end-to-end model traceability. You’ll understand how to build ML pipelines, continuous integration and continuous delivery (CI/CD) pipelines, and monitoring pipelines to systematically build, deploy, monitor, and govern ML solutions for businesses and industries. Finally, you’ll apply the knowledge you’ve gained to build real-world projects.
By the end of this ML book, you’ll have a 360-degree view of MLOps and be ready to implement MLOps in your organization.
* [Book's page](https://www.packtpub.com/product/engineering-mlops/9781800562882)
* [Amazon](https://www.amazon.com/dp/1800562888)
* [Book's GitHub repository](https://github.com/PacktPublishing/EngineeringMLOps)
Questions and Answers
---------------------
**Tino**
Hello Emmanuel Raj 🙂 Thanks for taking the time! As it feels like MLOps is currently on the rise when do you think it is really needed for a company to focus on MLOps? I often feel it is important to get something out there to see the impact but the operational part is often only the 3. or 4. steps whereas model drift, ect. can cause a negative business impact right away. Would you recommend to set up a good MLOpy framework before going live?
**Emmanuel Raj**
Hello Tino,good question! For any tech company focusing on making intelligent products (or data powered) setting up MLOps is recommended (to save time, money and energy). It saves a lot of time for data scientists and SE team from mundane tasks (deployments, manual tests, repetitive data engineering tasks, manual debugging etc) and enables them to focus on what truly matters (making best models and learning in realtime from data) by taking benefits of automation via CI-CD pipeline to avoid mundane jobs. Yes, before implementing in live/production it is recommended to test the pipeline and monitoring features (model, data, feature drifts etc) in DEV and QA environments to validate if the MLOps pipeline provides business value or not (measure with business KPI’s), only then go live/to production.
**Tino**
Okay cool 🙂 Got it 🙂 Thanks!!
**Emmanuel Raj**
Hello everyone! I am glad to answer you questions 🙂 looking forward to hearing your thoughts!
**Agrita Ga**
Do you believe there should be a difference between implementing MLOps in a smaller organization (let’s say startup focusing on ML solutions) and bigger organization?
**Emmanuel Raj**
Yes, there will be some differences b/w MLOps pipelines for smaller vs big organisations based on their business needs, teams and data processing abilities. Check out chapter 2: _Characterizing Your Machine Learning Problem_ in the book. It explains on this on detail 🙂 (screenshot from chapter 2, figure 2.9)
**Agrita Ga**
I’ll expand this question a bit, - does best practices (or some tips&tricks, or even tech stack?) differ for small vs big teams?
**Emmanuel Raj**
Yes, they differ case by case (company by company) especially the tools. But some of the the principles remain same on high level (MLOps pipeline to build, deploy and monitor ML models). For small companies, most likely they need to build the MLOps platform with limited, budget both in money and time and their operations might be small on scale. Big companies have high volume of data, operations and teams so the setup of MLOps pipelines/platform will differ in most cases 🙂
**Agrita Ga**
Thanks for your input! Appreciate a lot! 🙌
**Diego**
Hi Emmanuel Raj, thanks for this opportunity of q&a. to Do you think that MLOps is definitely a skill that every data scientist needs to have if he/she wants to keep relevant in the job market or, on the other hand, data scientists should just focus on data/statistics/algorithms because otherwise they are ‘biting off more than they can chew’?
**Emmanuel Raj**
Good question Diego! Knowing how to setup MLOps pipeline/platform (infra and architecture) is a bit too much for data scientists. However it is recommended for data scientist to know how to work with features of MLOPs pipelines/platform (once they are setup) such as registering datasets, models and packaging them on the MLOps platform. This way DS’s can take the benefits of MLOps and focus on what they are good at ‘data/Stats/algorithms’. I hope this answers your question 🙂
**Chetna**
Hi Emmanuel Raj, what’s your take on the importance of cloud technologies certifications? do they make a resume more relevant for MLOps role?
**Emmanuel Raj**
Hello Chetna Cloud certifications are worth it, they give a 360 degree view on what cloud has to offer so you may pick and choose best services to solve your business problems (optimisation). They surely make the resume standout for MLOps role, companies are looking for ML engineers who know data engineering and infrastructure setup well (certified is better) 🙂
**Chetna**
thanks 🙂
**Lalit Pagaria**
What is importance of choosing right Cloud Provider in implementation of MLOps?
What things to take care of while implementing MLOps?
In your experience, which providers do you suggest for small and medium startups?
**Emmanuel Raj**
Lalit Pagaria Choosing right tools for the business problem is most important (not the other way around). Any cloud which has capabilites to serve your needs will do (these days most of them are good enough). Down side of cloud though is vendor lock, to avoid that we can use cloud agnostic/open source MLOps tools e.g. MLFlow and Valohai which can work with most of the clouds. So choosing right cloud/tools depends on the business problem at hand 🙂
**Matthew Emerick**
Hey, Emmanuel Raj! Thanks for doing this!
In an open world where both the data and the environment itself are constantly changing, how does MLOps keep up?
**Emmanuel Raj**
Good question Matthew Emerick! MLOps addresses that constantly changing environment by adapting to the changing data/environment, optimising performance for changes, auto scaling and being relevant for the changing environment 🙂
**Neal Lathia**
❔ How uniform do you think MLOps workflows is across companies?
**Emmanuel Raj**
Hard to generalize at this point as different companies are at different stages in their ML adoption 🙂
**Mansi Parikh**
Thank you, Emmanuel, for sharing your thoughts!
Should MLOps be a concern during early stages of an organization or only when it becomes necessary? (More specifically, at what stage of growth of a data department does this become top of mind?)
**Emmanuel Raj**
Nice question Mansi Parikh! If the company/organisation is sure of having ML models in their workflow then the sooner the better it is to think of implementing MLOps. Otherwise, When data pipelines are set up and the organisation has the needed data setup in place. The sooner the better it is 🙂
**Mansi Parikh**
thank you so much, Emmanuel! this is great. I appreciate the thoughtful response. 🙂
**Rushanthi**
Hi Emmanuel Raj thanks a lot for the golden opportunity on QnA.
What’s your point of view on MLOps when it comes to job market? To be more elaborative, there are number of roles when it comes to data science which involves a data Analyst, data scientist and as well as a machine learning engineer, when taking these roles into consideration which job role requires experience on MLOps?
But then again there arises another question where we are headed towards an automated ML what would be the outcome of MLOps with relevance to the job roles in the market?
**Emmanuel Raj**
Rushanthi It’s good for an ML Engineer to have experience in MLOPs (especially data engineering and platform setup). Data Scientist is the user of MLOPs platform, so it helps if they have some exp using MLOPs platforms to build and deploy models. Data analysts can do without it. Good question on where are we headed with automation - Time will say but probably MLOPs will impact every Data science/Engineering job roles (let’s hope positively) e.g. more efficiency, less time and resources.
**Rushanthi**
Thanks a lot for the enlightenment Emmanuel. Appreciate it a lot for clearing up my puzzle 🙌
**Oleg Polivin**
Hi Emmanuel Raj, thanks a lot for this opportunity! I would like to ask you two questions that are a bit related.
1. In your opinion, what is the main added value that an MLOps person brings to a company?
2. Is it easy to replace an MLOps engineer?
A brief thought that was the reason to ask the questions is in the thread.
**Oleg Polivin**
When I was working on projects that needed something I would call MLOps: creating a docker application, deploying on a google k8s cluster, making a pipeline using gitlab ci, I realized that I do not understand how it is working under the hood, but just looking for “recipes”, keywords and using tutorials or to a lesser extent documentation (written in a form of a “recipe” as well). Like: put this into `gitlab-ci.yaml` file, click on this, that and that in google cloud. Sure, it took a long time to make all the parts work together.
However, it makes me think that:
* there is no special knowledge involved into MLOps vs. ,say, data science where one is expected to know math or statistics.
* Therefore, it makes be a bit “afraid” that MLOps engineer will be either replaced by some automatic deployment solutions or
* simpler, young people who tend to grasp many new tools that appear.
Thank you!
**Doink**
+1
**Emmanuel Raj**
Good question Oleg Polivin! MLOps person brings added value to a Data science team mainly in terms of infra setup/maintenance, monitoring ML Models/systems and maintaining pipelines. Sure it can be done/learned by others and MLOps engineers can may as well be replace (e.g. with SRE engineers, DEVOps engineers etc). For now it looks like data scientists are more on the verge of replacement (with AutoML) 😃 it’s not as easy to replace MLOps engineer though but maybe with time and more automation tools we might get there where MLOps engineers can be easily replaced.
**Lamjed Debbich**
Hi Emmanuel Raj, thank you for this nice book, it covers one of the subjects that interests me a lot. As you know, there are many methods of MLOPS on the market, the new user can get confused for which method should we use? Do you have any tips for getting started?
**Doink**
+1
**Emmanuel Raj**
Hi Lamjed Debbich Thank you! To begin with I suggest to get a good theoretical understanding of MLOps workflow, learn how to build ML microservices using docker and deploy them on various deployment targets (Engineering MLOps book will give a great headstart on this). After that decide on MLOPs tools (cloud or opensource, e.g. Azure, MLFLow etc) you would like implement and then find resources that teach you implementation. Learning by doing is the best way to learn MLOps 🙂
**Alexey Grigorev**
What do you think about the role “MLOps engineer”? Does it make sense? Should it exist?
**Alexey Grigorev**
I see that it’s often synonymous to ML engineer, which I don’t agree with. What’s your opinion about it?
**Chi**
From my understanding, there are (at least) 2 types of MLE defined by most of the companies. Some MLEs focus on the ML models algorithms, other MLEs focus on designing “data intensive application” or wee can call it MLOps? Again, ML in production is not just the ML algorithms — Andrew Ng.
**Emmanuel Raj**
Depends. If a person is needed for team to setup and monitor infra and operations it’s a good idea, otherwise Devops (or SRE) engineers with some knowledge of ML can enable MLOps 🙂
**Alexey Grigorev**
Also, what’s your opinion about the new course from Andrew Ng? I’m taking about MLEPs
**Emmanuel Raj**
Haven’t looked at the content, can’t say much but for the look of it, looks like it’s more focused on ML Engineering/DS problems (e.g optimisation, robustness, efficient modelling etc) and not much on MLOps 🙂
**Doink**
I see different MLOps course one from Made with ML, another on Udemy which is quite popular then we have Andrew Ng and then there is full stack deep learning course covering stuff. Which course or path do you recommend for a noob?
**Emmanuel Raj**
Made with ML looks good (recommended), not sure of Udemy course or Andrew NG’s courses (haven’t looked at the content) 🙂
**Tino**
Hey Emmanuel Raj Would you rather suggest to build an MLOps system on your own or buy it from an external provider? I saw that Fiddler has an amazing solution
**Emmanuel Raj**
Hey Tino, it’s hard to generalize for all cases but depends on the use case we work on. For some cases data is too sensitive and can’t be worked using external tools. For those cases it is better to build on your own but otherwise plug and play solutions like fiddler, mlflow, valohai etc are awesome, using them will save a lot of time and energy 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Relevant Search – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Relevant Search
---------------
#### by [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
, John Berryman
##### The book of the week from 12 Jul 2021 to 16 Jul 2021
Relevant Search demystifies relevance work. Using Elasticsearch, it teaches you how to return engaging search results to your users, helping you understand and leverage the internals of Lucene-based search engines.
* [Book's page](https://www.manning.com/books/relevant-search)
* [Book's GitHub repository](https://github.com/o19s/relevant-search-book)
Questions and Answers
---------------------
**xnot**
How do you see the likes of Vespa/Jina in this space? Are they viable candidates to base your first search system off of vs ES/Solr ?
**Doug Turnbull**
Sure, I think they are great tech. And the Lucene world is falling behind in a lot of information retrieval. Such as doing good dense vector retrieval and thinking in terms of multiple reranking phases. These technologies are good rethinkings of the space (hopefully push ES/Solr to be better too)
**xnot**
Which piece of tech are you most excited about in this space?
**Doug Turnbull**
🤔 I still like Elasticsearch a lot, Elastic has invested a ton in scaling the product, and it also has forks. IE if something happened to Elastic, I feel comfortable that some other version of Elasticsearch (ie open distro) would take over. I also really like Elastic the company, the people there, the culture. Maybe cause I know them well as people 😂 . So it would take a lot for me to switch. I also am comfortable extending the engine in great depth to do what I need (such as the Elasticsearch Learning to Rank plugin). Some of the features of newer tech do find their way into community plugins, etc for ES.
That said, it does take longer for cutting edge features to make it into the more mature product. And the JSON syntax can be very verbose if things reach a certain level of complexity. They are also obviously biased more towards analytics than search. I give them lots of feedback, and maybe sometimes they listen, lol.
**Doug Turnbull**
Lately I have pondered how well a completely (unsharded, but replicated) [pylucene](https://lucene.apache.org/pylucene/)
cluster would work, as then I could manage other data structures on the side for different needs, and most of my colleagues are Pythonistas
**xnot**
I’ve always felt that companies with huge amount of traffic can have a big advantage when it comes to being able to provide a superior search experience. Do you agree?
**xnot**
As a follow up, how can we close that gap?
**Doug Turnbull**
Certainly agree. It’s the secret sauce. I think there’s a famous phrase that with enough data you don’t need fancy models, you can do much more with simpler methods
**Doug Turnbull**
Closing the gap? Probably the lowest hanging fruit is investing in crowdsourcing like from a firm like Supahands ([http://supahands.com](http://supahands.com/)
)
**Andrew Kornilov**
Q: Any plans for the second edition? Some elascticsearch examples are outdated a bit already
**Doug Turnbull**
They finally have a new Space Jam movie… hehe I won’t rule it out!
A second edition would be quite an undertaking, as I’ve learned so much since then. I’d want to incorporate so much more about relevance measurement, about dense vector retrieval, learning to rank, more machine learning… probably more query understanding types of things
**David Cox**
I’m curious the contexts / types of organizations for which this book is relevant?
**Doug Turnbull**
hmm I can’t think of any that aren’t? Maybe risk averse orgs where ‘experimentation’ is not advisable. Like regulatory data that just has to meet certain laws. Also when you’re using search for just log analytics
**Kshitiz**
Hi Doug Turnbull and John,
Q) I know that search is like one of the fundamental component of web. Apart from the common use case which is searching for content on the web, are there any other use cases of search algorithms?
**Doug Turnbull**
there are a ton, because (a) every field is an index which is not common in other databases and (b) the rich package of text analytics.
For example I’ve used it to automate drug recovery, or to do “fuzzy joins” on different databases that have different variants of names or identifiers that can be normalized with text matching.
Also when you write flat data, but you have no idea how you’ll look it up later, and could want to look up on any attribute
**Kshitiz**
I have another question -
Q) What are the bottlenecks/improvement areas in the field of search?
**Doug Turnbull**
bottlenecks: creating the index is time consuming. Generally a search engine is much more read scalable than write scalable
**Doug Turnbull**
improvement areas: dense vector retrieval (like embeddings) is an active area of data structures research. Like the “Approximate nearest neighbors” topic
**Doug Turnbull**
Shameless plug, if you like my book, be sure to also check out [AI Powered Search](http://aipoweredsearch.com/)
which I’ve contributed 2 chapters to so far (on Learning to Rank and gathering search training data from clicks)
**Doug Turnbull**
and ping me if you’re curious about working with me at Shopify!
**Wendy Mak**
at what point would you decide it’s worthwhile to add your custom ML on top of elasticsearch?
**Doug Turnbull**
Usually, when I can trust the training data. But there’s lots of forms of “ML”. There’s augmenting search with ML generated assets. And there’s Learning to Rank, ML based ranking. There’s also tools like dense vector indices that help you do embedding based retrieval. So you might have some enrichment activity that is ML driven making the docs more findable, but the ranking itself might be manual, for example
**Wendy Mak**
also, maintaining a large ES index can be a real pain– are there optimizations you can do around this? (and what sort of metrics would you use to decide a certain piece of information is too ‘old’ and can be moved off to a different storage?)
**Doug Turnbull**
Ah interesting, hmm… I’ve seen some patterns of [hot / cold indices](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management)
. Often this happens in log analytics. In product search, usually merchants explicitly delete their old products.
**Doug Turnbull**
maybe you are thinking like document search, something is old and stale? I am a big fan of trusting user’s behavior. Is there a way to know whether users find the document useful? Do they share it? Copy the link? Does the link show up on slack? Can you see if it’s still used?
**Wendy Mak**
yeah, I am sort of thinking of document search– I used to work for a newspaper publisher and we use ES for recommendation + various other stuff (and the dev in charge of it had a lot of headaches trying to find the correct balance of what to keep or not in the index)
**Alexey Grigorev**
What do you think about Solr vs elasticsearch? Are they almost the same or elasticsearch is a bit ahead?
**Alexey Grigorev**
Which one would you recommend for a new project and why?
**Andrew Kornilov**
or solr is a bit ahead 🙂
**Alexey Grigorev**
Indeed!
**Doug Turnbull**
I usually think people worry too hard about what their current search engine is, with some exceptions - [stop worrying about Solr vs Elasticsearch](https://opensourceconnections.com/blog/2019/02/28/stop-worrying-solr-elasticsearch/)
**Doug Turnbull**
Solr == a bit buggier, weirder, rarely had a Solr project where I haven’t looked at Solr source code
ES == more modern, more opinionated, not going to bend the project to your will as easily, but lets you do almost anything you need in a sane way
**Andrew Kornilov**
and they (elastic) also break things quite often
**Doug Turnbull**
true, though they say “we are now breaking X”. And then I’m like “stop taking my toys away” 😂
**Andrew Kornilov**
my upgrate of elastic (between minor versions!) - aggregations latency
**Andrew Kornilov**
but they managed to fix :-)
**Andrew Kornilov**
well, “fix” == find a workaround
**Doug Turnbull**
ouch
**Alexey Grigorev**
Okay, thanks! And what about managed elasticsearch from AWS? It seems to be the easiest option to get started
**Andrew Kornilov**
if you are not aws certified guy, [this one](https://www.elastic.co/cloud/elasticsearch-service/signup)
is way easier to start
**Alexey Grigorev**
That’s nice, thank you!
**Dmitriy Shvadskiy**
With tons of work/research going into vector search what do you think of Learning to rank. Is it still relevant? What are good usecases?
**Doug Turnbull**
Depends if by Learning to Rank you mean “the problem of ranking solved by ML” -> then yes. In the end, we’re just training neural nets against the same loss functions.
If you mean “the traditional family of models” (LambdaMART, RankSVM, etc)… I think both are pretty well understood. In particular RankSVM is nice because its a set of linear weights, so easy to interpret/understand.
Plus vector search is only one part of a search solution, the inverted index is still valuable for many use cases. So all those features combined into a final ranking function is super valuable.
**Dmitriy Shvadskiy**
Thanks. I meant “the problem of ranking solved by ML”. But second part makes sense. Don’t change it if it works 😀
**Bayram Kapti**
Apologies if this is too rookie, but I’m braNd new for search experiences.
Where does someone start for improving search experience on a given platform?
What’s the place of AI and ML for improving search experience?
Are categorization & labeling considered part of improving search experience or are they completely different areas of expertise?
**Doug Turnbull**
No worries!
Best place to start:
* How will you define “good” and can you get data that defines what ‘good’ is for a query. There’s different strategies. One is to use expert, human raters if, for example, your app is very domain specific. Another is to use clicks, etc, which involves its own complexities (see Ch 11 AI Powered Search). Another is crowdsourcing… That strategy is very specific to the problem
* Then try to iterate and get better!
Like all kinds of ML, it comes down to the underlying data. How good / clean is data for an ML problem. At Shopify, all our merchants have vastly different ways they use our underlying data. So that’s a challenge for us that would lend us to lean to hard on some parts of content to really “learn” anything to help merchants.
Absolutely categorization / labeling are HUGE. We have a team member that just manages a team that does this.
**Bayram Kapti**
Thanks appreciate thw answers
**Dmitriy Shvadskiy**
What is a good way to scale judgements collection or ranking feedback for Enterprise search. Clicks do not mean the same things as in online shopping scenarios. Is thumb up/thumb down on the search result a good way to collect feedback? I had to build a judgement list for 1 experiment and it is a slow and painful process
**Doug Turnbull**
Enterprise search is _tough_ in part because nobody on your company in really that incentivized directly to make themselves/their content findable. If anything, they have the opposite incentive. They want to not be bothered, lol. Contrast this with Web search, where everyone is trying to SEO their content to get in front of as many eyeballs as possible
**Doug Turnbull**
It’s _also_ domain specific, making things tougher 😬
**Doug Turnbull**
For a large org, with a heavily used app, you could use clicks for your head queries. BUT of course, as you get down the long tail, that gets tougher
**Dmitriy Shvadskiy**
Definitely agree about incentives. It is a pain
**Doug Turnbull**
yeah not my favorite domain…
I would _probably_ gather feedback informally myself from qualitative feedback and turn it into judgments, rather than make users rate items
**Doug Turnbull**
And likely give people a very simple button to complain about a whole page of results (with very few fields to fill out, if any). Then when they click, capture the query + results. And if you can, you can find that person, and get more info, and turn that into relevance judgments
**Dmitriy Shvadskiy**
Great. Thank you for the idea
**Doink**
Transformer based Search engines vs ES based search engines which to choose when, what types of challenges each have? Cause I think there are lots of BERT models used for search
**Doug Turnbull**
I think in a way BERT-like search will take over for first pass search, that’s text heavy. However, in my experience, no technique ever really “dies” in search. So lots of classic relevance approaches, that people use to explicitly manage and understand queries - like simple dictionaries/taxonomies managed by specialists - I think will continue. In part this is because of the often specialized nature of the search app. Another reason is the focus on precision, and in many cases the kind of explicit control a traditional taxonomy based approach built on an index like ES is preferred to have really fine-grain precision
**Doug Turnbull**
Also not all search is very text-centric, the text might be unreliable/noisy, and BERT seems ideal for text passages
**Kim Falk**
Hey Doug Turnbull, What is the difference between a search engine and a recommender system?
**Doug Turnbull**
I have _no idea_ 😆
**Doug Turnbull**
Probably the difference is really how explicit and active the users intent is from the user’s PoV, but implementation wise they are just on one spectrum
**Kim Falk**
Good answer 🙂
**Alexey Grigorev**
Doug Turnbull maybe Kim Falk knows? I’ve heard he’s answering questions in August [here](https://datatalks.club/books/20210802-practical-recommender-systems.html)
.
**Alexey Grigorev**
Do you know how it works? The query is “movie where a man wakes up on the same day” and it correctly identifies it as “Groundhog Day”
**Doug Turnbull**
google just amazes sometimes 🙂. But as it’s a long form query, and likely in the realm of BERT/transformers/question answering kinds of solutions.
**Doug Turnbull**
To me, I’m always first interested in what the loss function was to train such a thing, and the specific training task to predict missing terms in a piece of text by randomly masking them is quite interesting. [Link](https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c?gi=8f06ba1d9231)
.
**Alexey Grigorev**
Even before defining the loss function, I imagined they needed a lot of training data for that. Do you think just relying on clicks would be sufficient in this case?
**Alexey Grigorev**
Perhaps there’s a separate flow for movies and books
**Doug Turnbull**
yes google has looooooots of training data (and lots of smart people working on the prob). At high scale, like web search, they probably can rely on clicks.
I doubt there’s a separate flow, I bet you can generalize these patterns regardless of topic…
**Lalit Pagaria**
How to evaluate search system?
How to achieve micro second search result on BERT based system with TBs of data?
**Doug Turnbull**
I’m not sure you can achieve micro second search results using BERT / TBs of data 🙂
**Doug Turnbull**
[Link](https://softwaredoug.com/blog/2021/02/21/what-is-a-judgment-list.html)
**Doug Turnbull**
On evaluation
**Lalit Pagaria**
I thought there might be magic way to achieve it 😄
**Shankar Somayajula**
Doug Turnbull
How do you incorporate fuzzy matches within the search function? Search for relevance using entire phrase and overlay with “# of Partial hits based on keywords” within phrase? Do you need to maintain a set of tags or keywords (based on topics, say) for each inventory item to match against/with the user ask? Is there a hierarchy of sorts within the various independent search tasks while combining to get the final return resultset?
**Doug Turnbull**
I prefer to learn common misspellings by looking at query -> query refinements in log. Or by suggesting spellchecking and seeing what they click thru/what they don’t. Then you can build a dictionary of these common corrections
**Shankar Somayajula**
Doug Turnbull I dont mean mis-spellings but regular words but the match being partial, not a full or exact match. So if user gives “auburn hair color with lavender perfume” … match to “auburn hair color” with any scent/perfume, “red hair color with fast drying feature” etc.
**WingCode**
Hi Doug Turnbull ,
1. How do you handle:
a. Different languages? Example: Input text is in English but documents to be queried over is in Spanish.
b. Phonetic variations?
c. Transliterations in search?
2. How do you search:
d. Text input over media content (image, video, audio). Example: Input -> “Lots of explosion”; Results , , ;
e. Media versus media (Search for similar images given an image, video given an video, audio given an audio).
**Doug Turnbull**
Ah these are all great and each very complicated questions. For 1b & 1c, assuming there’s no homonyms, a good starting point is a simple dictionary approach. For example, I’ve done that for British <–> US English variants
Everything else feels very much like a deep learning problem… You build a model transforming each to a vector space, and try to build an embedding that moves more similar content together, and push dissimilar content apart using whatever training data you have.
**Doug Turnbull**
then with that embedding space, I’d use an ANN index (or hack one out of an inverted index if possible)
**WingCode**
Thank you for the answers Doug 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Interpretable Machine Learning with Python – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Interpretable Machine Learning with Python
------------------------------------------
#### by [Serg Masis](https://datatalks.club/people/sergmasis.html)
##### The book of the week from 19 Jul 2021 to 23 Jul 2021
Do you want to understand your models and mitigate risks associated with poor predictions using machine learning interpretation? Interpretable Machine Learning with Python can help you work effectively with ML models.
* [Book's page](https://www.packtpub.com/product/interpretable-machine-learning-with-python/9781800203907)
* [Page on Amazon](https://www.amazon.com/Interpretable-Machine-Learning-Python-hands/dp/180020390X)
* [Book's GitHub repository](https://github.com/PacktPublishing/Interpretable-Machine-Learning-with-Python)
Questions and Answers
---------------------
**Carsten Schnober**
For a next generation of state-of-the-art ML models, do you think explainability will be somehow “embedded” into the models (as opposed to ANNs)?
**Serg Masís**
Hi Carsten. Don’t know what exactly you mean by “embedded”. There’s an issue of transparency through the property of model explainability which means so called “black-box” models are inherently disadvantaged in this regard. However, it’s not the only important transparency property available nor a reason to disqualify ANNs because they can be queried in other meaningful ways. That being said, if you want state-of-the-art models that retain the same explainability properties of white box models, there are glass box models like this one: [https://interpret.ml/docs/ebm.html](https://interpret.ml/docs/ebm.html)
and many more derived from Bayesian Rule Lists or Trees, not to mention new causal modeling methods.
**Shankar Somayajula**
Hi Serg Masís How is Interpretable ML different from BI with ML specific visualizations? Isn’t this EDA on model outputs with KPIs relating to model performance – ML model scoring generates predictions/outcomes which one slices and dices as per interest/motivation.
**Serg Masís**
Hi Shankar. You pose a very interesting question. Indeed I see Interpretable ML as a very similar exercise as BI. It can slice and dice model performance (often called error analysis) but should ideally delve into issues of fairness, uncertainty, robustness, consistency, etc. Of course, many of these look at the distribution of model performance from other angles but you can also learn from feature importance, feature interactions and partial dependence. These help inform model improvements and even business decisions (much like BI can)
**Shankar Somayajula**
Serg Masís Thanks a lot for your response. Yes, i agree that there are many other areas of focus in addition to performance. Also the feature related functionality isn’t as straight forward as typical BI use cases and falls more into the realm of Data Science/ML.
I come from a BI background and found the ML Classification based Confusion Matrix Metrics good candidates for leveraging BI self service practices … this exploration led to Interpretable AI which is indeed a vast field.
One of the ways that BI can add value is by way of data modeling (not ML modeling) … by leveraging a data model (sql schema) to store the ML outputs with various dimensions/attributes, we should be able to do things which are now done via UI/Self Service vizualizations via a backend/api call and automate many manual operations.
Say, we have an ML model and its predictions, given a new Dataset for scoring, and a candidate list of relevant attributes (col1, col2…col8) for the exercise in question, find the subset/sub-population within these attribute slices where falls below 10% of the value evaluated over … This can throw up 3-4 slices like col4=, col6=, col6= where this model degradation does indeed happen and then the user/expert goes on to visualize/explore those sections of the data using the UI. Otherwise discovery depends on analysts stumbling upon the right subset of interest visually in a self-service but manual mode of operation.
**Serg Masís**
Yes! We definitely need to store model meta-data much more than is practiced. This would include things like performance metrics, fairness metrics, variance of performance against hold-out datasets to show reliability, sensitivity analysis for uncertain inputs hyper-parameters used in training, data provenance, and much more. Ideally we could use data like this to certify models in all sorts of dimensions.
**Akshaya Natarajan**
Hi Serg! The book looks really interesting. I have a couple of questions:
1) I’m currently working with ResNets and most of the times don’t get how to interpret certain results. What’s the best way to go about interpreting DL models?
2) How can we use Interpretability to detect bias in the ML model?
**Serg Masís**
Hi Akshaya! Thank you for your questions 1) There are many ways of interpreting DL models depending on the data type (image, text, tabular, time series,…) and/or model architecture (CNN, RNN,…). You mention Resnets so I take it you are interested in Convolutional Neural Networks (images). I would start by understanding the layers with intermediate activations. Then chose examples to study and apply gradient based methods (GradCam, etc) and Permutation based methods (SHAP,…) to interpret the outcomes of the model. 2) It is tough to detect bias in any model but images is particularly challenging because so far the only methods available focus on disparities of representation and outcomes, and don’t go deeper. If you are interested I explain fairness in more detail in [the talk I did for deeplearning.ai](https://www.youtube.com/watch?v=A0ADFiiZU0k)
.
**Akshaya Natarajan**
Thanks a lot for answering Serg!! Will definitely checkout the video.
**Serg Masís**
Your welcome!
**Dr Abdulrahman Baqais**
Hi Serg Masís. Thanks for the book and for your time to answer our questions.
1) Interpretable models is a general term that different stackholders define it differently. It is different for a technical team, than a business users, than a client than an auditing entity like a government or ethical board.
The question is : to whom we should provide Interpretable models? Should we have different Interpretablity to different types of the above stakeholders?
2) In today enterprise ML pipeline, shall we include Interpretablity in the ML pipeline or operatingodel. Probably after the deployment or maybe as part of ethical auditing or should we do it only when it is required.
3) which roles of ML team take care of Interpretablity: DS, ML engineer , a combination of both or we need a separate role with advanced math skills probably to do the task?
4) Can Interpretablity be evolved into a separate business model by itself: something like Interpretablity as a service.
5) Shall we consider Interpretablity as a default embedded part of the product shipped to the customer? Or is it a separate entity in which we can charge the customers differently?
6) Shall we sacrifice little of accuracy to obtain higher Interpretablity even though can sometimes result in profit reduction?
Thank you so much.👍 💯 🙏
**Serg Masís**
Hi Abdul. I much appreciate your questions. I saved you questions for last. They are very good. It’s been a long day for me so I hope you don’t mind if I answer tomorrow.
**Dr Abdulrahman Baqais**
Thank you Serg Masís. Sure. Whenever you have time.
**Serg Masís**
Hi Abdulrahman. 1) Indeed. You would use different interpretation methods for different stakeholders. The technical team tasked with training and deployment should implement and test all pertinent methods to their models. Business stakeholders might be interested only in predictive performance and feature importance. However, ethical board and auditors might be only interested in results from fairness and robustness tests. Government might be interested in sensitivity analysis or whatever compliance tests they have coded in their regulation. As for the end-user, they might want an explanation attached to their predictions. 2) I hope that it will become standard practice to use ML interpretability methods in each step of the pipeline. I think in the future modeling will be done primarily through drag and drop interfaces which will have more a cockpit feel to it than it does right now alerting practitioners of all sorts of issues with data, models and outputs and allowing them to correct these as they go along. This embedding-interpretable-ML-in-the-pipeline approach will make ML more responsible - although I believe making it ethical will require often a better understanding about the data collection process which is often outside of the purview of the ML pipeline, so it won’t cover all bases but I’m hoping that by freeing up some time from ML practitioners from the nuts and bolts of programming modeling they can focus on other issues.
**Serg Masís**
3) Right now engineering is an important skill to have in ML modeling. However, in a world where most data engineering and modeling doesn’t require programming but more interpretation of statistics, engineering belongs more in MLOps and ML research roles. However, it would open up the floodgates for folks that are data-savvy non-programmers but domain experts to participate in modeling. I think it still makes sense for these folks to be knowledgeable in statistics (ML engineers aren’t necessarily). The focus on interpretation, will heighten the importance of statistics, and it is the more the domain of data scientists than ML engineers. 4) It already has. I’ve seen a few startups offer this service. 5) It should eventually in a model that comes with something like a manifest with everything from data provenance to feature importance pre-defined and an explanation given automatically with every prediction 6) Definitely. I see the value of models that can perform more transparently, fairly and reliably outweigh the risks of just having a higher predictive performance. Higher profits may come with many risks. A model that is right 99.6% of the time might be right for the wrong reasons 0.3% of the time and that can eventually backfire. I rather take a model that is right 99.4% of the time but only right for the wrong reasons 0.01% of the time.
**Dr Abdulrahman Baqais**
Thank you Serg Masís for your time and detailed answers. Very insightful.
**Jeff Herman**
Hi Serg. There are a lot of different machine learning interpretability techniques (LIME, SHAP, Anchors, Permutation Importance, etc), is there one such technique that you find yourself using more? Follow up question, is there one such plot or visualization that you use a lot? For example - I see feature importance with tree based models in sklearn used a lot, do you find yourself using one technique a lot like SHAP summary plot?
**Serg Masís**
Hi Jeff. I personally use SHAP the most during my modeling pipeline but I’ve implemented Anchors into the inference engine of my projects so it has probably been used more times by end-users than I have used SHAP making the models. SHAP’s summary plot I use a lot but I use interaction plots probably slightly more because in my line of work understanding feature interactions is critical.
**Alper Demirel**
Hi Serg Masís, thanks for being with us.
What was your goal in writing this book? Do you believe you have achieved your goal?
**Serg Masís**
Hi Alper. Thank you for your question. My goal was to 1) create a comprehensive book about interpretability methods 2) while doing it convince ML practitioners of its importance. The first goal was acheived I think although I had to leave some important topics out (namely, privacy). As for the second goal, I think for years now Ive seen it make appearances in conferences but it still was a bit under the radar for industry. You had AI Ethicists and Academic Experts and some business folks champion XAI / IML but all the people that actually work with data in industry weren’t getting that involved. Its hard to believe because my book was only published this year but it was the third book ever on IML / XAI for practitioners but I think the topic is becoming more top of mind to the audience the book was written for. So its a work in progress but I believe its making in roads.
**Alexey Grigorev**
Is there any difference between explanability and interpretability? (I.e. between explainable AI and interpretable AI)
**Serg Masís**
In theory (i.e. academic literature) it does because explainability and interpretability are usually not used interchangeably but, strangely, Explainable AI and Interpretable ML are. Not sure about variations like Interpretable AI and Explainable ML but they likely, in practice, synonyms. It gets confusing because interpretation methods output explanations which we interpret. As for definitions of all the terms you mentioned. There are many definitions of _explainability_ but it’s more focussed on the inner workings of the model (the how the prediction is made), whereas definitions of interpretability specially when referred to as _post-hoc interpretability_ is more about why is a prediction is made. For that reason it is perfectly content with black box models. As for _Interpretable ML_ (aka _Explainable AI_) its the collection of methods used to understand/debug models on three levels (Fairness, Accountability, Transparency) and even make improvements in those aspects.
**Alexey Grigorev**
Thanks!
so explainability = I can explain how it works internally
while interpretability = I can interpret the output and understand why model made this particular prediction
correct? Or not?
I think I’m still a bit confused…
Maybe we can try an example? Let’s say we have a resnet model which predicts cats and dogs. What would be explainability and interpretability for this case?
**Serg Masís**
Yes correct, Alexey, although in a Venn diagram Intepretability includes explainability because transparency is a property of interpretability. In other words, it helps to understand how the model works to explain why it made predictions. As for your example regarding the ResNet, methods like intermediate activations help you visualize each layer in action (and activation maximization per filter) . These help you with the explainability because you can understand how they work. On the other hand, methods that just show you a map of what parts of the image, according to the model, tell it is a cat or a dog (such as Integrated Gradients, GradCam or SHAP) help with interpretability because they don’t tell you how the model works but why it is working (or not!).
**Alexey Grigorev**
Clear, thank you!
**Leonoor Tideman**
I love the question Alexey Grigorev because I used to find this genuinely confusing 🤣
Based on what I have seen in XAI & IML, most researchers and practitioners indeed use “explainable AI” and “interpretable ML” interchangeably. I guess it is partly because the field of XAI & IML is still relatively new and pp have not get agreed on one technical definition. When I entered the field (2019), one of the few general references was the _Interpretable Machine Learning Book_ by Molnar (PhD student in Germany). Molnar was one of the first pp to organize all sorts of (sometimes very different) methods under the name of “interpretable ML”.
The main proponent of differentiating between “interpretable ML” and “explainable AI” is Cythia Rudin (professor at Duke University in the USA). She defines IML as being about models that are inherently interpretable (e.g. linear/logistic regression, decision trees), whereas XAI is about generating post-hoc explanations (generally in the form of feature attributions) for arbitrarily complex black-box models. So, according to Rudin, XAI is about using a interpretable model to explain the original black-box predictive model. I highly recommend [Rudin’s paper](https://www.nature.com/articles/s42256-019-0048-x#Sec2)
. I like her approach, but as mentioned by Serg Masís her perspective is not widely shared within the community.
**Serg Masís**
Yes call me one of those that don’t agree with her view. If anything her definitions for XAI and IML should be flipped since the term explainable is more optimistic, for lack of a better world, than interpretable. I honestly don’t think the term explainable belongs anywhere near statistics (and much less neural networks) since even the most basic methods from hypothesis testing to linear regression coefficients are not infallible and self-explanatory and thus require an interpretation within a margin of error.
**Emily Tran**
Do you need to understand the business 100% to be able to build interpretable ML model/neuro network?
**Serg Masís**
Maybe not all the business, but the better you understand the business related to the data, the better you can interpret the models trained with that data. However, in the absence of domain knowledge you can learn so much about the data through the model. It can help with EDA.
**Marcello La Rocca**
Hi Serg Masís
Thanks a lot both for Q&A!
Your book looks great, congrats!!!
Question: as of today, how far would you say we are with the adoption of interpretable machine learning? E.g. what percentage of models used in production would you estimate are interpretable?
**Serg Masís**
Hi Marcello. Interpretable and explainable are loaded terms. In the context of my book, all models are “interpretable” in the sense that they can be interpreted through model-agnostic or deep learning specific methods. However if by “interpretable” you mean that transparent, I would estimate that 90%+ of production models are “black box models” so they are more opaque. My book doesn’t discriminate against this kind of model since post-hoc interpretability is possible in a model or system with known inputs and outputs, although I wont lie that transparency helps with the reliability of the methods, and ease of interpretation. Therefore, what makes a model interpretable is not exclusively its nature (class, architecture, etc) but that ML practitioners know how to and are actively interpreting models.
**Marcello La Rocca**
Thanks!
**Marcello La Rocca**
Question 2: What’s your favourite interpretable alternative to NNs? What about GAMs (generalized additive models, with or without pairwise interaction), do you think they can be as effective as claimed, in comparison to NNs?
**Serg Masís**
Neural networks come in many flavors so it depends. I wouldn’t use anything other than CNNs for images - it actually surprisingly interpretable. With graph data you have more options like SVMs but NNs are my default. For univariate time series, statistical methods (Garch, ARIMA, etc) are much better than RNNs but not necessarily for multivariate. As for text, transformers rule in accuracy and even interpretability (an improvement from RNNs - even bidirection ones). Last but not least, for tabular data I do so as an example in the book because I know lots of people use NNs with tabular but honestly I can’t think of any good reason to use NNs on tabular data considering better alternatives. My preferred model classes for tabular are decision trees and ensembled decision trees (random forest, XGboost, etc). They tend to perform equal or better than NNs, overfit much less and are easier to tune - you can use monotone and interaction constraints also to place guardrails. As for GAMs, they are good alternatives if you require more transparency however there’s a trade-off with predictive performance. That being said, there’s a gradient boosted GAM that attempts to not compromise on performance which I cover in my book: [https://interpret.ml/docs/ebm.html](https://interpret.ml/docs/ebm.html)
**Marcello La Rocca**
Thanks a lot Serg, that’s super thorough!
I’ll take a look at EBMs, thanks!
You mentioned transformers: I haven’t had a chance to dig into the topic, but I am extremely curious to, as I have read (high level) how they improve over RNNs. And I didn’t know they are even more interpretable! I’ll be eager to read about this aspect too.
**Serg Masís**
Yes, the problem with transformers is since they scale much better, they are pretty much fed the entire Internet 😂 and that has inevitably led to many questions about bias (see Timrit Gebru et al’s paper [https://dl.acm.org/doi/10.1145/3442188.3445922](https://dl.acm.org/doi/10.1145/3442188.3445922)
)
**Lalit Pagaria**
Is there any legal framework/requirements for interpretable AI mandate by governments?
**Serg Masís**
Good question Lalit. Many governments throughout the world have “frameworks” of some kind but in most cases they aren’t requirement and when they are not (namely, European Union) wording is very ambiguous. For instance, GDPR for the EU has enacted a “right to meaningful information about the logic involved” for algorithmic decision-making but an explanation can come in many forms and levels of detail. What precise methods to use and how to deliver this explanation is not defined.
**Doink**
Serg Masís is GDPR written by technical folks or bureaucrats?
**Serg Masís**
I think more of the latter but even technical folks don’t have all the answers yet. We still haven’t figured out what constitutes the best framework for providing explanations on all levels (fairness, accountability and transparency). This might also vary according to the use case since some need a bigger focus on fairness and others on accountability, for instance
**Krzysztof Ograbek**
Hi Serg Masís, thank you for doing this! 2 Questions:
1. Do you think that ML world would benefit from having more people with a non-tech background. I mean like artists, philosophers, etc. How could we attract such people into the field?
**Serg Masís**
Hi Krzysztof. No problem. 1) definitely! I think we can open up the floodgates once we remove the programming requirement. I envision future ML systems would be drag and drop so that anybody that has an interest (hopefully more domain knowledge) can partake in the process.
**Krzysztof Ograbek**
Just a follow quick follow up question. Don’t you think that math is the main reason why AI is so scary to non-tech?
**Serg Masís**
Math is daunting yes to many but don’t think its the main deterrent. I think there are more scientifically-minded data-savvy people without the programming skills than those without the math skills. Mind you by math skills I’m not talking about graduate physics level math skills but college statistics + linear algebra + calculus. Of those three, statistics is the most important for data science. I think it’s better to understand linear algebra and calculus intuitively for practical use cases than know how to prove a theorem.
**Krzysztof Ograbek**
1. Is your book beginner-friendly?
**Serg Masís**
2) yes but It’s not like a zero to hero book. It’s starts at around 0.01 😂 because I don’t have a glossary for basic concepts like what is machine learning, supervised learning or Python. The expectation is that the reader has some ML 101 knowledge. Besides that the first three chapters are introductory and go over the most basic models and their properties
**Doink**
Serg Masís 2 Questions
How to deal with the tradeoff between fairness and explainability
If I apply differential privacy then how to maintain fairness, explainability and make sure that there isn’t a high privacy loss.
**Serg Masís**
Hi Doink! 1) it depends on the type of fairness. One could argue that procedural fairness is upholded better the simpler the model. For instance it two Branch decision tree for credit scoring based on income is very straightforward, and it’s fair because it’s the same rule for everyone. However it’s outcome fairness could be dismal because people that can pay back their loan will get rejected. Personally the level of Procedural fairness I look for is no double standards among similar people but otherwise outcome fairness is what we should shoot for which is statistical parity among demographic groups. The problem is Explainability is at odds with outcome fairness but not procedural fairness. That being said you have to reduce model complexity as much as possible by feature section/engineering, regularization, etc to help with explainability and generalization
**Serg Masís**
2) for fairness there are many bias mitigation methods you can try and assess using fairness metrics. For differential privacy you can tweak the strength (eps) and then assess its effectiveness. There are many ways to assess explainability. There aren’t great metrics for it but if the model overfits minimally and also is outcome fair it is likely very explainable. How to get all three? Combine fairness methods + assessments, differential privacy (inc assessment) and standard performance metrics (compare with hold out) into your pipeline and optimize with all metrics in mind. You can make your own weighted metric that combines several metrics in one and hyper parameter tune for that. There’s definitely a trade-off so bear that in mind with your weighting.
**Alexey Grigorev**
Was there something you wanted to include in the book but eventually decided not to?
**Alexey Grigorev**
If yes, what was it and why you decided not to cover it?
**Serg Masís**
So much! First there was two chapters originally planned on Interpretation methods for NLP but two months after I started writing the book I found out that Denis Rothman was about to release his XAI book (and it included some NLP because that’s his area of expertise) so I decide not to. I regret this decision and I’m considering adding one NLP chapter in the 2nd edition. The other mayor decision was not to include a chapter on privacy preserving ML because the book was getting too long and this topic is so big. I might just carve some on space for an introduction to this topic in the second edition but would have to remove a chapter. Lastly, there’s a bunch of other topics like error analysis, data augmentation techniques, causal explanations through advanced counterfactual methods and semantic segmentation that I considered covering, but there’s only so much I can cover in a 700 page book! There’s so much more that meets the eye with IML. When I told some of my colleagues that I was writing a book on IML they said sure “I know SHAP and LIME” but it’s much more than that (mind you SHAP and LIME is only 2 out of the 14 chapters, and if I truly wrote about every method available it would had been easily at least 40 chapters).
**Alexey Grigorev**
Maybe instead of second edition, you could have another book “advanced IML” =)
**Serg Masís**
Good idea, Alexey!
**WingCode**
Hi Serg Masís. Thank you for the Q&A!
How do you determine bias when you don’t have access to the model but have access to the input (training data) & output of the model (model predictions), in case of machine learning as a service platform (example: [AWS Personalize](https://aws.amazon.com/personalize/)
) ?
**Serg Masís**
Hi WingCode. Great question! There are two kinds of fairness that pertain ML: outcome and procedural. Although for procedural you would at least have to train a proxy model (with some disclaimers), the model is really not necessary to ascertain bias when measured on the outcomes alone because you can use the labels to determine bias on the data level and the predictions to do so on the model level. What the metrics do is statistically measure disparities between groups. What can make this challenging is when you don’t know what the groups are or don’t even have the data pertaining the groups. Say, gender feature was removed or differential privacy was applied to the training data.
**WingCode**
Thank you for the answer Serg! 🙂
**Agi Kajanaku**
Hi Serg Masís, can you elaborate more on how you would approach connecting ml model usage to peripheral hardware?
Thank you!
**Serg Masís**
Hi Agi Kajanaku! Sorry I’m not sure what you mean by peripheral hardware? Mouses and keyboards? Or more like IoT devices like cameras, sensors, lights, locks and smart speakers? And what do mean about ml model usage? like reporting usage back to the cloud? Or running models in the cloud?
**Agi Kajanaku**
Hi Serg Masís, apologies for the lack of clarity 😅 I was thinking more of devices like cameras and sensors and mostly running the model but open to reporting usage too. I understand I might have missed the week mark but if you do happen to have time, would love to learn more. Thanks!
**Serg Masís**
Hi Agi Kajanaku I started writing a response and then forgot to press the send button. Now it’s gone 😅 Anyway, the best approach to model training while preserving privacy in IoT devices is federated learning which is a distributed approach. Because of this private data is never sent to the cloud. That being said, this approach has its limitations because IoT devices aren’t known to have a lot of compute. In other words, sensors definitely are feasible - and cameras less so. What can happen with cameras is you pretrain models with generic data and then the IP camera doesn’t send streaming video to the cloud but run inference on the pretrained models. Occasionally it can connect to the cloud to download model updates. Using generic computer vision models work with things like detecting dogs and people, but not specific people or objects like for instance, if you wanted the camera to detect one of the people that lived in the house at the entrance and then open the door. Facial recognition would require training a model with personal data rather than generic and that would mean cloud involvement for now. In my apartment I’m using a Raspberry Pi, which has a more powerful processor, precisely for that but that is of course one more device besides the IP camera.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Applied Natural Language Processing in the Enterprise – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Applied Natural Language Processing in the Enterprise
-----------------------------------------------------
#### by [Ankur A. Patel](https://datatalks.club/people/ankurapatel.html)
, Ajay Uppili Arasanipalai
##### The book of the week from 26 Jul 2021 to 30 Jul 2021
NLP has exploded in popularity over the last few years. But while Google, Facebook, OpenAI, and others continue to release larger language models, many teams still struggle with building NLP applications that live up to the hype. This hands-on guide helps you get up to speed on the latest and most promising trends in NLP.
With a basic understanding of machine learning and some Python experience, you’ll learn how to build, train, and deploy models for real-world applications in your organization. Authors Ankur Patel and Ajay Uppili Arasanipalai guide you through the process using code and examples that highlight the best practices in modern NLP.
* [Book's page](https://www.oreilly.com/library/view/applied-natural-language/9781492062561/)
* [Book on Amazon](https://www.amazon.com/dp/149206257X)
Questions and Answers
---------------------
**Rimma Shafikova**
👋 NLP beginner here. When we speak about progress in NLP, is it always limited to English? Are state-of-the-art approaches transferable to other languages? (of interest: Russian, Mandarin)
**Ankur Patel**
Yes, the beauty of the state-of-the-art approaches today is that they are very transferable to other languages, especially languages where large corpuses of text is readily available (e.g., Russian, Mandarin, etc.). Progress is more limited in languages where large corpuses of text are harder to come by, though
**Ankur Patel**
Basically, to develop very good NLP models today, you need access to massive volumes of text (in any language of your choice), which is easy to come by
**Rimma Shafikova**
thanks Ankur!
**Alex**
Hi there Ankur Patel! First of all, thanks a lot for taking the time to reply our questions 😄
Since this book is aiming to show applied NLP in companies/orgs: Do you consider that getting the executive buy-in is difficult, given the conceived complexity of getting effective NLP models to work in real-world/business problems?
**Ankur Patel**
In general, yes getting executive buy-in is difficult, mostly because this is a new space, and executives may prefer the tried and tested over something new that they do not fully understand yet
**Ankur Patel**
That being said, it does depend on the executive. In my experience, defining the deliverable narrowly and showing value in a modest way fast is a good way to garner the attention and interest of executives fast
**Ankur Patel**
Once they see the return on investment, you can pitch more ambitious projects
**Ankur Patel**
Too often, I see people promising too much, and then things take longer and cost more and executives become disenchanted by the space because of it
**Dr Abdulrahman Baqais**
Thank you Ankur Patel for the book.
Few questions:
1) Most NLP advances now is based on huge preteained model, what about the classical ML models (Logistic Regression, Bayes naive, LSTM..etc).
Do you feel that Today’s NLP practitioners should jump directly to transformers and preteained models.
**Ankur Patel**
I think the older approaches (regex and classical ML models) still have a place even with transformers
**Ankur Patel**
It really depends on the task. For example, if you are trying to process invoices (OCR + text classification), using transformers is the way to go
**Ankur Patel**
If you are performing entity resolution or linking, then a simpler approach based on cosine similarity or regex or Elasticsearch may be better
**Ankur Patel**
I think having awareness of all the different potential approaches to the problem will help you pick the right tool for the given job at hand
**Doink**
how to know which to use when? Is there a mindmap especially for those who don’t have much NLP domain
**Ankur Patel**
Here is a good approach.
1. If your task is the same or similar to one of the core NLP tasks here, use a Transformer-based model. [https://huggingface.co/transformers/task\_summary.html](https://huggingface.co/transformers/task_summary.html)
2. If your task is for a small dataset or relatively simple or requires interpretability, start with the older yet simpler approaches such as rules-based NLP (e.g., regex) or classical ML.
3. If your task is for a large dataset and is more complex in nature, skip the rules-based NLP or classical ML and research some of the state of the art NLP approaches to solving the problem. Implement one of these approaches.
**Dr Abdulrahman Baqais**
2) NLP domain seems daunting with many subdomains: NER, summarization, translation..etc. many tools, many libraries and packages. Yet running a blacbox preteained model can get very high accuracy if it was run by a junior DS who has no clue what is going on.
What kind of skills NLP practitioners should equip themselves in order to be able to digest all these information and still be in demand in industrial market.
**Ankur Patel**
I think spending 20% of your time learning what is new and effective (in an applied setting) is a must. It is true that some of the latest methods are so good that they may leap frog existing approaches even without too much tuning
**Ankur Patel**
For example, we write about spaCy, Hugging Face, and fastai in our book. These are must haves today if you are doing NLP in enterprise
**Ankur Patel**
Older libraries such as StanfordNLP and NLTK are much more dated and less effective
**Dr Abdulrahman Baqais**
3) NLP is taught at an advanced level at separate track in many DS bootcamps assuming a solid DS knowledge.
Can we teach NLP to non-DS to create a citizen NLP practitioners?
**Ankur Patel**
It’s possible now, but it wasn’t really possible a few years ago. Now there are more easy to use libraries available, but we still have a long way to go
**Ankur Patel**
I would say that you need some Python and DS knowledge to begin the NLP journey today, but NLP is not necessarily a crazy advanced field
**Ankur Patel**
Our book assumes that the reader has some basic Python experience but not much more than that
**Matthew Emerick**
Hey, Ankur Patel! Thanks for doing this!
What is the singular biggest challenge for putting NLP into production?
**Ankur Patel**
I find that most NLP specialists struggle with the engineering aspects such as refactoring code, developing Docker containers, writing unit and integration tests, deploying the model as a service on AWS / Azure / GCP
**Matthew Emerick**
How important is the study of linguistics to NLP developers?
**Ankur Patel**
Nope, not necessary today. It helps but it is not required by any means
**Matthew Emerick**
What do you see as the next big step in NLU?
**Ankur Patel**
We need better representations to achieve “longer-term memory.” Without this, NLU is very hard. Attention mechanisms and transformers have helped on this front, but we need machines to hold memory over very long spans (chapters and chapters of a book), and this is really hard today
**Matthew Emerick**
Would you think symbolic AI mixed with today’s statistical methods might help?
**Ankur Patel**
I think so. I haven’t studied symbolic AI enough and it hasn’t been in vogue recently, but I think using just statistical based methods may be limiting. I’m hoping more research is done in this and other areas soon (to complement the statistical approaches)
**David Cox**
I appreciate you taking the time to respond to questions, Ankur Patel! I second Matthew’s questions about with respect to getting NLP products into production. Also, many database systems (e.g., AWS) are offering out-of-the-box NLP solutions. I’m wondering if you have any thoughts or recommendations on cloud computing systems that do simple NLP solutions well and also allow individuals with training in NLP to engage in more advanced analytics?
**Ankur Patel**
I think the major cloud providers have done a great job lowering the barrier of entry to perform NLP
**Ankur Patel**
I find Google best in this regard today, followed closely by Amazon. Today, you can perform some of the core NLP tasks with just basic Python knowledge without knowing ML or DL
**Ankur Patel**
I think we will see this trend continue quite a bit. Easy to use NLP APIs tailored to the developer community at large instead of to just data scientists
**ASHISH SONI**
Hey Ankur Patel! Really curious about the content your book!
What kind of business problems/examples have you covered in the book?
**Ankur Patel**
We cover a lot of the more popular NLP tasks in enterprise today such as named entity recognition, text classification, sentiment analysis, and summarization
**Ankur Patel**
We don’t focus on any one vertical because a lot of these tasks are appropriate across verticals
**ASHISH SONI**
Thank you Ankur! 🙏
**Ajay Arasanipalai**
To add on: there’s been lots of evidence recently that backs up what Ankur Patel says here about the same techniques working across verticals.
**Ajay Arasanipalai**
The real beauty of neural nets and deep learning is that they actually do live up to the promise of generalize across many different datasets. For example - GitHub copilot and AI dungeon, two very successful and popular real world, actually deployed, commercial products who’s domains are quite different (software engineering vs. entertainment), use nearly identical models.
**David Cox**
Ankur Patel Based on the trends you’ve watched over the past decade and that led into your book, what do you think are the next areas for major advancement in NLP? And, what are some lingering challenges that we don’t seem to be close to?
**Ankur Patel**
Let’s start with the next major (applied) advancement in NLP. Combining computer vision with NLP to process documents is and will become a very large area of investment (from the applied / enterprise community).
**Ankur Patel**
For example, think of all the paper heavy industries today that require visual and textual cues to interpret properly (e.g., invoices, health statements, receipts, legal documents, financial documents, etc.). These areas are ripe for automation using not just NLP but also computer vision (jointly).
**Ankur Patel**
The most hyped area is NLG (natural language generation), largely off of GPT-2 and GPT-3 from OpenAI. But, we are still quite a bit away from having human-like generation
**Ankur Patel**
That’s one of the biggest challenges, but GPT-3 has caught the attention of many people, both in academia and in industry, so I expect a lot more innovation here in the coming decade
**David Cox**
Very interesting! Especially tin thinking about businesses that are paper heavy. Do you know, offhand, if anyone has looked at the carbon offsets between creating and using paper as compared to the resources needed to build the models specific to the different business cases and the resulting data storage/use?
**David Cox**
I wonder a lot about the getting to language that is similar to human. In particular because of the known contextual factors that go into human language outside of simply looking at the structure of what was said prior to the generated statement. It seems this will require combining other datasets to get to that structural piece as well as advances in NLP generally.
**Wendy Mak**
Hi Ankur, what business problems do you think would benefit a lot from modern NLP methods but is commonly overlooked in research as too boring or from the businesses not being aware it is possible?
**Ankur Patel**
A few problems come to mind, but one of the biggest areas is information retrieval. Researchers find it “boring” at least compared to NLU and NLG tasks, but informational retrieval (think Google on domain-specific private documents such as legal and finance and healthcare) is incredibly valuable to businesses.
**Ankur Patel**
Businesses are using CTRL-F and manual organization and tagging of data to find what they need but NLP could really unlock a lot of value here. It’s not a Google scale problem so it often gets overlooked by the tech giants. Curious what you think about this (and others).
**Wendy Mak**
yeah, I agree– I think it’s really relevant but you don’t see a lot of papers about document tagging etc in e.g. Neurips… In the last company I worked for there’s a lot of legal documentation that could really benefit from automatic tagging (unfortunately that project got parked when the biz people decided it was low priority…)
**Ankur Patel**
Part of the challenge here is that the effort to build a model to do this well isn’t worth it for any given organization, so software companies will need to provide this service to many clients to make the investment worthwhile. I think we are starting to see more of this now
**Jeff Herman**
Hi Ankur. Thanks again for taking some time to give your great insights on our questions! With transformers how easy is it to really understand how the model is making the predictions that it is? For example if we have a text classification using a transformer, can we see which words were of most importance for a prediction?
**Ankur Patel**
Initially, this was hard to do, but since 2018 there has been some good progress on introducing interpretability to the Transformer models
**Ankur Patel**
You can now see which word(s) the model paid most “attention” to as it was making a prediction
**Ankur Patel**
It’s still difficult to truly understand the black box at scale though (across many predictions) but for one off predictions, this is possible today
**Ajay Arasanipalai**
To add on: I know it wasn’t originally your question, but let me also mention that I don’t think trying to kind “keywords” is necessarily a great idea for interpretability. As Ankur Patel mentioned, I don’t think this approach scales well in practice - when you have thousands of users querying your model with many gigabytes of text, what insight do you hope to get by finding the most “attentive” words?
**Ajay Arasanipalai**
In the case of text classification, it might be smarter to just use basic word/subword counting post-classification (i.e. what are the most common words among novels that have been classified as horror).
**Jeff Herman**
Hi Ajay Arasanipalai. Appreciate your insight! Originally, I was thinking of how to audit the model. For example, if we are looking at novels and we predict a novel as horror when it is actually historical I wanted to know the most important words for why the model predicted it to be horror. I like your approach, I could compare most common words in that novel vs the most common words in the different novel genres
**Krzysztof Ograbek**
Hi Ankur Patel, thank you for doing this.
How will NLP be different in 3 years from how it is today? Are there any tasks that will explode on popularity?
**Ankur Patel**
I think we have seen NLP in good use in consumer applications today (for example, with social apps), but the next 3 years will focus on getting NLP into the workplace, automating tasks that white collar workers are doing today
**Ankur Patel**
For example, analyzing and processing invoices, bank statements, legal memos, health care statements, financial documents, etc.
**Ankur Patel**
The barrier to entry to use NLP will also come down, just like it did for computer vision since 2012
**Ankur Patel**
Another trend that intersects well with NLP is the no-code movement in software development and machine learning. We should be able to load documents, highlight text, and have NLP models perform an array of valuable tasks such as document classification, sentiment analysis, summarization, etc. We are not quite there yet
**Krzysztof Ograbek**
I love the answers. Thank you!
**Krzysztof Ograbek**
Is your book for everyone, regardless the level of NLP experience?
**Ankur Patel**
Yes, it is. In fact, it is positioned best for newcomers and intermediate users
**Ankur Patel**
You will need to know Python and have some awareness of data science and ML though
**Ajay Arasanipalai**
To clarify: we don’t really require any NLP experience, but it’s definitely. If you’re a complete beginner, you can definitely pick up the book and get started - we have a bunch of external resources (which I think is actually one of the more valuable parts) to help. But it definitely won’t be enough on it’s own if you don’t have any experience with Python, PyTorch, or basic deep learning.
**Ajay Arasanipalai**
One thing we realized early on is that there are a lot of resources that help you get started, but less so that dive into the details of how to go from copy-pasting SciKit Learn snippets to implementing and deploying state of the models in production.
**Giuditta Parolini**
Why are chatbots so frustrating for the user, when NLP-based translation tools (I am thinking about DeepL for instance) can do a very good job?
**Ankur Patel**
Part of this has to do with NLU. Chatbots today don’t do a great job of holding relevant context across questions. You can see this firsthand if you try to ask Google Assistant or Siri or Alexa a series of related questions
**Ankur Patel**
Another part has to do with conversational language that many of us use in chatbots (for example, the use of casual language or idioms, both of which are very hard for NLP models today)
**Ankur Patel**
I think both of these items will have to get solved before chatbots appear “intelligent”
**Ajay Arasanipalai**
The other thing to consider that most chatbots deployed today probably aren’t using the very best of modern deep learning techniques. While the GPT-3 demos certainly look convincing, keep in mind that they require 22 GPUs for inference…
**Ajay Arasanipalai**
Especially for those not utilizing the latest tools (like Hugging face’s transformers library), the engineering effort and compute resources required to run a medium-sized model like BERT for a simple chatbot that really only needs to cancel orders every once in a while may be prohibitively expensive.
**Giuditta Parolini**
Thanks for your answers. At this point one can only say that, most of the time, chatbots are the fancy equivalent of switchboards with intolerably long recorded instructions. Both chatbots and switchboards do not give the user what (s)he needs, but they gain time when companies do not have enough customer service staff. I will try not to be too upset next time a chatbot wastes my time.
**Ajay Arasanipalai**
Hi everyone, I’m Ajay, Ankur Patel’s coauthor for “Applied Natural Language Processing in the Enterprise.” Thanks for setting this up, and I’m happy to stick around and answer any questions you all may have.
**Ajay Arasanipalai**
Quick comment: while no-one specifically asked this question, it seems like many of you here are asking about something along on the lines of what NLP applications are going to be hot in the next few years. Note that this isn’t _just_ about completely new products and services, but things that we’ve all been using for a while from established companies may also start getting better as they incorporate transformer models.
**Ajay Arasanipalai**
You might have noticed that Gmail’s autocomplete and predictive keyboards have been getting better over the years. I think this trend will continue, and we’ll start to see even more autosuggestions and prompts popping up across different applications/domains. A great example of this is GitHub Copilot - [https://copilot.github.com/](https://copilot.github.com/)
**Alexey Grigorev**
Which NLP papers are your favourite? Why?
**Ankur Patel**
I personally love this paper: [https://arxiv.org/abs/2012.14740v1](https://arxiv.org/abs/2012.14740v1)
**Ankur Patel**
I’m working on document understanding problems, and this is an excellent paper on the topic
**Krzysztof Ograbek**
What are the characteristics of a great NLP Engineer? Can you tell someone has a potential despite lack of experience?
**Ankur Patel**
Most of the time it comes down to practical, hands-on experience for me
**Ankur Patel**
I love to see Github repos and projects, even if the work experience isn’t quite there
**Ankur Patel**
Any indication that the individual is learning new materials and experimenting with them is a huge positive
**Ankur Patel**
Thirst for the space is big
**Mansi Parikh**
Hi, Ankur and Ajay. Thanks for interacting with the community! We appreciate your time and enthusiasm.
At what point does it become crucial for an organization to adopt an NLP-focused analytics strategy? Is it only when you’ve exhausted all other analytical opportunities for non-text data that you need to dive into this and developing new capabilities to continue to add incremental value to your business? Is it based on the actions of competitors in the market? Basically, how do you know when to seriously introduce this to an organization?
**Ankur Patel**
I would frame it a bit differently. If your organization uses text or documents today at reasonably high volume, then it is time to invest in NLP.
**Ankur Patel**
If your organization does not have high text or audio needs, there is no need to dabble in NLP
**Mansi Parikh**
Thank you!! I just wasn’t sure if NLP was essentially a right step forward for even organizations that may not have this type of data yet but now are realizing that they have to collect it and if NLP techniques could eventually be valuable to them. I was considering it like that, but maybe the business should just focus where it’s meant to focus.
**Ankur Patel**
Yes exactly. Let the business need drive the technology instead of the other way around
**Mansi Parikh**
One more, please, for both of you.
How can you estimate the value of an NLP undertaking to a business? Given its popularity, it might not take much to convince leadership that this is a promising route forward, but as there are usually many options for ways an organization can proceed in the future, you may still need to justify building expertise around this subject compared to alternatives and that can be done by estimating potential (or expected) business value, I suppose.
**Ankur Patel**
I would start by framing the problem in terms of savings to the organization if you could introduce x% automation
**Ankur Patel**
By itself, applying new tech for the sake of using new tech isn’t going to be too convincing, no matter how hot the new tech is
**Ankur Patel**
If you could deliver some value fast, that will also help with buy in
**Mansi Parikh**
Great! That sounds reasonable to do and provide. Sometimees fast POCs are difficult, but necessary to win over the decision-makers. Thanks again, Ankur.
**Ricky McMaster**
Hi Ajay Arasanipalai/Ankur Patel, thanks for doing this! Just following from the point Ajay makes above about improvements in autocomplete/predictive text, which is something I’ve been thinking about anyway recently.
As we as users become more reliant on such features, how do NLP models account for their training data potentially becoming more and more machine-generated (or at least machine-influenced), whilst humans might lose more of their standards in grammar or general literacy?
Is there a risk that it makes it more difficult for models to keep up with developing linguistic trends whilst remaining grammatically ‘correct’?
**Ajay Arasanipalai**
Good question. I think this is still an unsolved problem, but there are a few things that we might be able to do short-term as a quick fix. The most obvious solution is flagging the data your model generates, and avoid training on that. High quality testing and validation sets also help a lot here. You could also measure user satisfaction with the prompts and use that as a metric.
**Ajay Arasanipalai**
As for humans loosing grammar standards, I suppose that’s more of a social problem. The same could be said of self-driving cars, but we work on that anyway. I think worrying about polluting the training set with bad grammar due to mass normalization of autocorrect is a fairly out-there long term concern.
**Ricky McMaster**
Thanks, appreciate the response.
**Laia**
Hi Ajay Arasanipalai and Ankur Patel very interesting topic!
NLP products have become better and better, but what are the current NLP frontiers?
**Ankur Patel**
The two frontiers that I see are NLU and NLG, both of which are sub-areas in NLP more broadly. Here is [a good post on it](https://www.ibm.com/blogs/watson/2020/11/nlp-vs-nlu-vs-nlg-the-differences-between-three-natural-language-processing-concepts/)
**Ankur Patel**
NLP is much more mature today, and it is being used to process and structure all sorts of text and audio data (e.g., invoice processing).
**Ankur Patel**
NLU and NLG are more open areas of research that aren’t ripe enough for broad industry application just yet, partly because of the inference costs as Ajay Arasanipalai mentioned yesterday and partly because the tech isn’t quite good enough yet to be applied to broad settings (above and beyond autocomplete, for example).
**Ankur Patel**
NLU can unlock a lot of value because, once it has matured, we will see more context-aware conversational bots that handle queries much more like humans would, instead of the fairly “dumb” question and answer bots today
**Ankur Patel**
NLG will help us generate more open-ended text and audio, perhaps even assist in the creative fields such as novel writing
**Ankur Patel**
We have a ways to go before we get to that point though
**Laia**
Thanks for your answer!
**WingCode**
Thank you Ankur Patel & Ajay Arasanipalai for the great Q&A.
Do you think the future will be ruled by compute intensive models (example: Transformers, GPT-3) ?
Will there be more efforts put into less compute intensive techniques (example: distillation) thereby making the state of the art accessible to all?
**Ajay Arasanipalai**
I’m conflicted on this. Note that the other way you can get accessibility is by making compute cheaper. We’re started to see many accelerator and deep learning hardware startups that promise huge gains in efficiency and affordability. Even Nvdia, who has an undisputed monopoly on this market, continues to make better GPUs year over year. It’s entirely possible that what we consider “compute intensive” today will be very easy for the average practitioner to run 5-10 years from now.
**Ajay Arasanipalai**
The reason we’re seeing an interest in larger language models is that at the moment is because they are continuing to scale nicely and it’s a relatively “safe” bet to improve your model’s performance. Would you rather hire a team researchers to work on distillation for a year who may or may not be able to produce a ~20% improvement, or just change a parameter in your initializer that doubles the number of layers?
**Ajay Arasanipalai**
But even then, the only company I know of that regularly scales language models is OpenAI. I think most others have settled on BERT and it’s variants for practical use. I think it will be similar to the story in vision today - ResNet50 and YOLO are the standard, but there are better alternatives if you’re willing to invest the time, energy, and compute.
**WingCode**
Thank you for the answer Ajay 🙂 Interesting take.
Followup question. Do you think the future of GPU hardware for DL will be dominated by NVIDIA GPUs because of the wide adoption of their CUDA language? Do you think any other player can pull an “Apple” (with their ARM transition) and we get a new open source API interface?
**Ajay Arasanipalai**
I think the lesson we’ve learned from Nvidia is that programmability > raw performance. If developers don’t like using your platform, it won’t work.
**Ajay Arasanipalai**
Whether or not these hardware startups end up providing a competitive products remains to be seen (and is probably 10+ years away, so is hard to predict). But I think the most promising thing to look for short-term is software/compiler improvements to Google’s TPUs. The really big issue there is that Google doesn’t want to sell their TPUs though.
**Ajay Arasanipalai**
As for an open source CUDA alternative, I don’t think we’ll see anything promising within the next 1/2 years unfortunately. But that’s just my guess.
**WingCode**
Thank you again for the answers Ajay
**Ankur Patel**
Thanks everyone! And big shoutout to Alexey Grigorev for inviting us and organizing this. If you want to follow me personally on trends in the AI/ML and NLP space, please feel free to [subscribe here](https://www.ankursnewsletter.com/)
. Hope to see you all again soon
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Practical Recommender Systems – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Practical Recommender Systems
-----------------------------
#### by [Kim Falk](https://datatalks.club/people/kimfalk.html)
##### The book of the week from 02 Aug 2021 to 06 Aug 2021
Online recommender systems help users find movies, jobs, restaurants – even romance! There’s an art in combining statistics, demographics, and query terms to achieve results that will delight them. Learn to build a recommender system the right way: it can make or break your application!
* [Book's page](https://www.manning.com/books/practical-recommender-systems)
* [Book's GitHub repository](https://github.com/practical-recommender-systems)
Questions and Answers
---------------------
**Emily Tran**
I have never built a recommendation model but I’m curious 🤩
1, how can a recommender destroy an application? In how many ways is it possible?
2, did you experience some destructive recommenders before? What can you do to save the application?
3, how can you detect such recommenders?
4, which mistakes can I as a beginner do to lead to those destructive models?
You see, I’m a Master student and might or might not build something like that in the future 🙊 thanks in advance 😄
**Kim Falk**
Hi Emily Tran,
Thank you for your questions 🙂
1) The ways a recommender system can destroy an application are endless. From a pure engineering point of view, a RecSys can be very resource demanding and slow down everything, which will hurt the application. From a recommender point of view, there is nothing that scares people away faster than bad recommendations. The contrary can also be the case - if a recommender knows the user too well, the user might find it scary. Just to mention a few.
**Kim Falk**
2) Again, it’s hard to say what the solution is if you don’t know the exact problem. In the case where it’s low-quality recommendations, you should have a look at getting better data and tune your recommender. If it is an engineering problem, it is worth considering adding more resources or look to see if you can do more things offline such that it won’t hurt the performance of the system
**Kim Falk**
3) again depends on the problem. But rule of thumb look for whether you have unhappy users
**Kim Falk**
4) consider what metrics you want to optimise for and be sure you test your system well. Not just offline, but more importantly, online in production. A lot of research indicates that good offline test results do not ensure good online results. Be sure to have monitoring up and running before adding or updating a recommender such that you can measure any changes in performance.
**Emily Tran**
Thank you very much for your answers. Nr 4 is absolutely helpful for every models! I will apply it from now on😤
**Bayram Kapti**
Thank you Kim Falk !
Is it possible to do client side recommendations instead of server side? If so, how does client side recommendation compare to server side ?
**Kim Falk**
Anything is possible 🙂 As with any data applications it depends on what data you have.
There are different ideas about how to do it. One way that seems to be discussed a lot recently is to do federated learning. Where basically you start out with a model trained on the server, this will be sent to each client, where the model will be updated according to the clients actions. This model will then be sent back to the server where the client models will be assimilated, and then it starts over.
Have a look at [this article](https://arxiv.org/abs/1901.09888)
.
**Ana Data Science for Social Good Portugal**
What is the product lifecycle of a recommendation system (how does it evolve in time)? And recommendation engines trends/ usecases that you have seen across industries?
**Kim Falk**
As with every data driven product, the life cycle is something the lines of:
1. Understand data/business
2. Collect data
3. Prepare data
4. RecSys Modelling
5. Offline evaluation
6. Recsys deployment
7. A/B testing
8. Repeat
**Ana Data Science for Social Good Portugal**
What specificities exist in content and session based recommendation engines?
**Kim Falk**
I would like to answer this question but it would be many hundreds of pages. In short Content-based recommenders mostly use NLP to create embeddings of descriptions of the content, and then use those embeddings either to train recommenders or simply recommend by finding nearest neighbours to items consumed by the user.
**Kim Falk**
Session-based recommendations on the other hand are done a bit like the language models that predicts next words in something you are writing. If you watched movie A, B, C then the next best movie to suggest would be D.
Have a look at [this survey](https://arxiv.org/pdf/1902.04864.pdf)
for more information.
**Bayram Kapti**
What are some indications for an organization to start serving recommendations to their users?
Similar question, when do you think an organization should stay away from serving a recommender system to their users?
I’m looking at this question from two concepts: 1-) organization maturity
2-) product type (user behaviour on the product etc)
**Kim Falk**
I am not sure what to answer here. It depends on the product/content organisation wants the user to consume.
If there are too many products for users to overview themselves, then a recommender system is recommended 😉
**Kim Falk**
It depends on whether the organisation has data enough to implement a recommender system, either usage or content data. And if they monitor their system. If they don’t monitor the system, then they won’t be able to measure if the recommender improves anything, then they might as well not do it.
**Bayram Kapti**
Got it! Sorry for the vague question. But I got a good answer that I was looking for.
**Ana Data Science for Social Good Portugal**
What are the challenges of creating and maintaining a recommendation engine from 2 perspectives:
A) data engineering
B) data science
**Kim Falk**
A) There are two aspects for a data engineer, the _offline_ part one where you should have a system which collects the data needed to train the model, and then actually have the model trained, ensuring that it doesn’t blow up or worse create bad recs because of some new trends in user behavior. Secondly is the online part where you have to create a pipeline which provide personalised recs quick enough for the user not to find another page that loads quicker.
**Kim Falk**
B) The data scientist needs to understand the data, the business domain and come up with good models (and tricks) to make the recommendations good.
**xnot**
Does the book cover graph based algorithms ?
**Kim Falk**
In a way:) graph-based recommenders are based on the same concepts described in my book. But actual graph-based algorithms are not described in detail. For that, I would refer to Graph-Powered Machine Learning by Alessandro Negro.
**xnot**
What’re the most common mistakes you’ve seen people make ?
**xnot**
Do you also go into handling diff types of biased datasets / detecting such bias ?
**Kim Falk**
Popularity bias is discussed, as well as item and user bias in collaborative filtering.
**xnot**
How do you tackle long tail / niche interests with little data ?
**Kim Falk**
Usually, you would use a content-based recommender algorithm if you have little data. But since you say long tail, it has to do with something that also has popularity (because if there were no popular items, there would be no long tail either). It will always depend on which algorithm you use, but a way to do it is to penalise recommendations by popularity. Be careful because the idea is to recommend stuff that is not too popular but still something the user would like to watch.
**Doink**
how to do recommendation engine on edge devices using federated learning?
**Kim Falk**
Have a look at [this article](https://arxiv.org/abs/1901.09888)
**Doink**
yes I have seen some papers on this but what is your take on actually getting this stuff working in production?
**Kim Falk**
Doink Given the chance, I would like to try it out because it sounds very interesting. The thing is that most apps/user only interacts with few items in a catalogue, so I am not sure how applicable it will be in a production system. Because of the amount of data, but also the resources used to train the model. Again I’d rather not say it wouldn’t work, I just think that many other things also need to be working to make it work.
**Neal Lathia**
❔ What are the most common mistakes people make when building a recommender system?
**Kim Falk**
The list is long. The biggest mistake is probably to think it is easy 🙂. As with any data-driven product, it is very hard to evaluate the quality of what you are implementing. It’s only ever in production your recommender gets an accurate estimate of performance.
**WingCode**
Hi Kim. Nice to meet you again here.
1. Are you planning for a 2nd edition of your book?
2. If yes, what are the additional topics you are planning for your second edition of your book?
3. If all above yes, when can we expect the 2nd edition? 😛
**Kim Falk**
Hi WingCode,
Yes, I am planning a 2end edition. What exactly will be the content is still up for debate as I’m no further than working on the table of content. But on my wish list are chapters on deep learning algorithms, multi-armed bandits and reinforcement learning, and Sequence-based. Besides generally updating the rest of the material. Do you have any other good suggestions?
It took my four years to write the first edition, I will be faster the second time around, but I wouldn’t expect it before a couple of years.
**Kim Falk**
I better note that I don’t think my book is outdated, I think the content is still very relevant for somebody who wants to build recommender systems.
**WingCode**
Kim Falk Probably more methods and techniques in extracting features from non-text media ( video, audio, picture) which can be used for recommendation systems?
**Alexey Grigorev**
Doug Turnbull and I were wondering if you know the answer to this question
\> What is the difference between a search engine and a recommender system?
**Kim Falk**
This will be a longer answer 🙂
**Kim Falk**
A search engine takes a seed, and the search engine returns a list of content using that seed. In most cases, the seed is a sequence of words. Nowadays, Google also allows for searching for images and returns images. A recommender system takes a user profile or an item as a seed and returns a list of content items.
Both search engines and recommender systems can personalise the result. So I would say a search engine and a recommender system differs only in what the seed is.
A recommender system can be viewed as a pipeline that does the following three steps:
* candidate selection
* apply filtering rules
* rerank result.
Where candidate selection is a rough cut of items that might be interesting, it filters them based on some internal set of business rules and then reranks the remaining items. If you view it this way, then the candidate selection could be made by a search engine. In this view, a search engine could be a component of a recommender system. I’m sure others would look at it as the other way around 🙂.
**Alexey Grigorev**
Great, thank you!
**Doug Turnbull**
Oh I am glad Alexey Grigorev thought to do this as I was on vacation last week 😉
**David Cox**
Kim Falk I work primarily in the healthcare domain. I’m wondering if you know of anyone doing good work in this realm related to recommendations on patient-doctor matching or patient-intervention matching?
**Kim Falk**
I am not familiar with any work applying recommender systems on that problem. I heard about patient-medicin matching, but I would have to go and search for it again.
**David Cox**
Oh, interesting. Thanks!
**Mansi Parikh**
Hi, Kim! Thanks for being available to interact with the community.
Please excuse my question if it sounds amateurish, but what are the easiest recommendation algorithms to implement in a business vs the hardest? And, if you could please provide a few options of intermediate difficulty as well, that’d be great. I imagine that algorithm selection depends on the quality, structure, and volume of data among other factors which you would detail in your book, but if you were to just develop a quick proof of concept in a business to demonstrate the power of these sophisticated analytics initiatives, what would you choose?
**Kim Falk**
Hi Mansi Parikh
Thank you for the question. It depends on what type of data you have. Here are two quick ideas on how to start.
If you have usage data, then have a look at the framework called Implicit or LightFM. Both are pretty straightforward to use.
On the other hand, if you have content descriptions. Then you can use Gensim package to create word2vec embeddings and then find similar items based on cosine similarity.
I think the more advanced methods depends on the problem domain.
Hope that helps.
**Mansi Parikh**
I love this answer. All of this is fairly new to me, and now I have a real plan to learn it or at least get some exposure. Thank you, Kim!!! Much appreciated.
**TCG**
Hi Kim Falk, I couldn’t understand the difference between `usage data` and `content description` with respect to recommendation.
Like I’m chalking a plan for building a Hotel Recommendation System for which I have data around `1gb` with `0.9 million reviews` of `4000 hotels`. Im still a beginner and I cannot decide if I should use `LightFM` or `TensorFlow Recommender` or other ML algo.
**TCG**
Now that you have given 2 approaches to build on, i.e. `usage data` & `content description`. In which domain the Hotel Recommendation falls in?
**WingCode**
Kim Falk What are features which can be extracted from the content of:
1. Audio
2. Video
3. Image
Other than metadata of the content (aggregated features like genres, duration, list of cast etc)?
**Kim Falk**
The sky is the limit. Audio could be something like is it music or talk, is it one person or many, are they angry or happy. The video could be cut into scenes, and count number of explosions, kisses, cars, houses ect. generally do object detection, if there are humans what are they doing ect. It is also relevant how much the camera moves, how dark the images are.
**Kim Falk**
With the images you should do object detection, atmosphere, style.
**Kim Falk**
This is just a few ideas.
**WingCode**
What libraries do you recommended for the same?
**Kim Falk**
Extracting features from Audio, Video and Images are not something I have tried, I have used data which was extracted as embeddings, but never done it myself. Im not sure which framework to recommend you.
**WingCode**
Thank you Kim for all the answers
**Shankar Somayajula**
Kim Falk Hi, What are the ways by which recommender systems can react in real time to topical/dynamic trends? How does one balance the weightages to recommendations derived from historical data and those pertaining to recent/near term trending data? Basically be data driven (so start off giving mostly historical recommendations to user) but with ability to take feedback and react to user preferences which are recent (today/this week/…) … e.g. most of the historical recommendations are not working/degradation, so surface near term recommendations/trends instead of historical trends/recommendations. Any framework to do this in a data driven way (with or w//o manual tinkering)?
**Kim Falk**
What are the ways by which recommender systems can react in real time to topical/dynamic trends? As with most of the answers I have given, this also depends on what is the specific domain and scenario. A way could be to to rerank your recommendations based on which items have changed most in frequency the last hour compared to the last day (last day compared to the last week/month). I think a better answer would need more context.
**Kim Falk**
How does one balance the weightages to recommendations derived from historical data and those pertaining to recent/near term trending data? Depends on your domain, but you can create a hybrid model as a linear function of the output of the two models and then run tests to see which weights will provide you with the better recommendations.
**Kim Falk**
To be data driven you will have to create a platform where you allow for data to be collected about different settings and then let the data decide what the right values should be. Bayesian Multiarmed bandits would be away to leave the weights completely up to the machine.
**Kim Falk**
Any framework to do this in a data driven way (with or w//o manual tinkering)? Im not sure what this framework should do exactly?
**Shankar Somayajula**
Kim Falk Thanks for the answers/pointers. Domain is shopping/retail primarily. I’ll check your answers to other questions also to pick up some useful info.
**Lavanya M K**
Hi Kim Falk How to select A/B test users for realtime recommendations?
**Kim Falk**
The simple answer is that you just take 10% of the traffic for the test group. The somewhat harder answer is that it needs to be a random sample which is representative of the whole population. If the users has a randomly created identifier you can create a hash function which can split the users into 10 buckets and then simply take all users from that bucket, There are several books written about how to do it best, Fx _Tuning Up_ by David Sweet.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Tuning Up – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Tuning Up
---------
#### by [David Sweet](https://datatalks.club/people/davidsweet.html)
##### The book of the week from 16 Aug 2021 to 20 Aug 2021
Tuning Up: From A/B testing to Bayesian optimization is a toolbox for optimizing machine learning systems, quantitative trading strategies, and more. You’ll start with a deep dive into tests like A/B testing, and then graduate to advanced techniques used to measure performance in highly competitive industries like finance and social media. The tests in this unique, practical guide will quickly reveal which approaches and features deliver real results for your business.
* [Book's page](https://www.manning.com/books/tuning-up-from-a-b-testing-to-bayesian-optimization)
* [Book's GitHub repository](https://github.com/tuningup/tuningup)
Questions and Answers
---------------------
**WingCode**
Hi David Sweet, nice to meet you!
Should A/B test users selected randomly or should we select them based on specific attributes?
**David Sweet**
Short answer: randomly.
Long answer: Selecting based on specific attributes can improve the precision of a measurement. This is a technique called _blocking_.
Let’s say you’re running an A/B test. You have two versions of an application, A and B, that you want to compare. You’ll measure “time spent on the app” for each.
The attribute you know about your users is their age: They’re either “over 25” or “25 or younger”. The “over 25” crowd generally spends less time on the app.
If you assign users completely randomly to version A or B you might, by chance, have more “over 25” users seeing version A. These extra “over 25” users would bias downward the measurement of time spent — but you’d attribute that bias to version A. If you were to rerun the experiment you might, this time, assign more “over 25” users to version B. In that case you’d attribute to the bias to version B.
So sometimes the age attribute makes your measurement biased towards A and sometimes towards B. On average, it’s unbiased, thanks to randomization.
Unfortunately, there’s variability from run to run. You can mitigate that by assigning the same number of over-25 users to A as to B and
the same number of 25-or-younger users to A as to B. Then age-related variability will disappear.
That being said, when you select users from the over-25 group, you should select them randomly, and the same goes for the under-25 group. That way your experiment is still unbiased to all of the other factors that might affect time spent.
**WingCode**
Thank you for the super detailed awesome answer David 🙂
**Denis L.**
Hi David Sweet! Thanks for doing this.
Bayesian techniques require specific likelihood function assumptions. From your experience, how often do you think these assumptions are violated by end-users and how robust are Bayesian methods under misspecification of likelihood (or prior)?
To give a concrete example - if we run Bayesian A/B test analysis on a metric following what we think Poisson distribution, however upon closer inspection it violates some assumptions of the Poisson (but we model it as Poisson regardless).
**David Sweet**
The Bayesian method I discuss in the book uses a non-parametric method call Gaussian process regression (GPR) to model what’s called the _surrogate function_, the function that maps system parameters to a business objective. For example, you might serve ads based on a prediction of click-through-rate (CTR) with a rule that says, “If the predicted CTR < _threshold_ don’t show the ad.” The parameter _threshold_ affected how much ad revenue (the business metric) you earn per day.
You could use GPR to model the function mapping _threshold_ to daily ad revenue making (essentially) no assumptions about the shape of the function.
Once you have that surrogate model, you can ask it, “Which value of _threshold_ would give me the largest daily ad revenue?”
**Denis L.**
Thanks for the answer!
**Bayram Kapti**
Thank you David Sweet!
It’s easier to calculate the statistical significance of an uplift in the test group’s performance against the success metric(s) during an AB test when the success metric is a rate -> such as CTR.
However, I find it challenging to measure increased performance after a change in the product. Example Hypothesis: If I add this new feature to my app, It will increase the duration on the app.
**Bayram Kapti**
A follow up question to above, what are the best practices to come up with “X % increase” in the hypothesis statement. This varies for industry or the metric itself, but is there a methodology to come up with The X percent in the hypothesis statements?
**WingCode**
I would probably like to add onto the question 🙂
Is there some technique to predict the “X % increase” even before running the experiment?
**David Sweet**
Generally speaking, we’ll hypothesize “no change in the metric”. This is called the null hypothesis. The A/B test will measure the change in the metric and, potentially, reject the null hypothesis.
The good news is that you don’t have to guess, beforehand, by how much the metric will change to run an A/B test.
You do need to know the level of variability (the standard deviation) of the metric, however. Usually you can estimate that number from existing data. For example, if your metric were time spent by a user during a session, you could look at existing logs of users sessions and compute the standard deviation of the time spent.
That being said, it would be nice to have a prediction of the A/B test result beforehand. If the prediction said, “This new feature probably won’t change time spent by enough for anyone to care”, then you could avoid running the A/B test altogether.
One can sometimes use a domain-specific simulation of the system to make such a prediction. In quantitative trading, for example, it is common to run a trading simulation (aka, a backtest) to estimate the profitability of a trading strategy before deploying it. Simulations are sometimes used by ad-serving systems, recommender systems, and others, too.
**Bayram Kapti**
Appreciate the answer David Sweet!
When you have seasonality in your users behavior, I guess than you could intentionally only look at the variance for relavant season.
Imagine an e-commerce store having higher durations during Christmas & Holidays season when compared to Summer. In this case, would it makes sense to include Summer data to calculate variance for an AB test during Christmas time?
**David Sweet**
Yes.
The goal is really to predict the variance that will be realized during the experiment. Using (in your variance calculation) only data expected to be similar to the data you will collect during the experiment would achieve that goal.
**Wendy Mak**
hi David, would you use an ‘off the shelf’ system to run your A/B tests, and at what point would you decide that it’s better to build your own?
**David Sweet**
Sure. I would look at it like any other build-vs-buy decision. Does the product do what you need? Is the price right? (Does the do _more_ than you need? If so maybe you’re paying for features you won’t use.)
I think it would be useful to initially run A/B tests in the lowest-effort way possible — whether that’s with a web-based tool or some simple manual calculations in Jupyter. You’ll learn about the details that are specific to your system (ex., deploying “B” versions, logging measurements) and get a sense for the value a piece of commercial software could provide.
**Wendy Mak**
also, how do you decide whether something is worth running A/B tests on?
**David Sweet**
One way to do this is to run a simulation before hand.
**Alexey Grigorev**
Hi David Sweet!
What are your favorite blog posts and videos from companies that talk about their experimentation platforms and the way they run A/B tests?
**David Sweet**
This is a good one: [https://research.fb.com/videos/the-facebook-field-guide-to-machine-learning-episode-6-experimentation/](https://research.fb.com/videos/the-facebook-field-guide-to-machine-learning-episode-6-experimentation/)
I especially like the comments about how improvements in offline/modeling metrics (ex., MSE of a regression, or cross-entropy of a classifier) don’t exactly translate to improvements in online/business metrics (like revenue, clicks/day, etc.)
**David Sweet**
For a nice overview of Bayesian optimization in practice: [https://research.fb.com/blog/2018/09/efficient-tuning-of-online-systems-using-bayesian-optimization/](https://research.fb.com/blog/2018/09/efficient-tuning-of-online-systems-using-bayesian-optimization/)
**David Sweet**
Also, most large companies have internal, custom experimentation platforms:
Uber: [https://eng.uber.com/xp](https://eng.uber.com/xp)
/
Netflix: Netflix [https://lnkd.in/dmKdFJ8](https://lnkd.in/dmKdFJ8)
Twitter: [https://lnkd.in/dFHZxSM](https://lnkd.in/dFHZxSM)
Facebook: [https://lnkd.in/dG7\_8GV](https://lnkd.in/dG7_8GV)
LinkedIn: [https://lnkd.in/dCc8aQN](https://lnkd.in/dCc8aQN)
Spotify: [https://lnkd.in/dKcyyuM](https://lnkd.in/dKcyyuM)
Spotify: [https://lnkd.in/d9E-5BC](https://lnkd.in/d9E-5BC)
**Alexey Grigorev**
Oh thank you!
**Oleg Polivin**
Hi David Sweet
Thanks a lot for the opportunity to ask you a question!
Shall a data scientist know A/B testing well?
Here is my reflection/speculation:
I’m working as a data scientist for some time now, and I didn’t have to use A/B testing so far. I know some basics, but surely there is much more to applying A/B tests in production. It looks like it is a must for “data analyst” positions. And I feel uncomfortable about this because:
* I would speculate that A/B testing itself brings more value to a company than a bunch of data scientists.
* I do not have that knowledge => am I/will be still relevant to the industry?
**Eimhear Rainey**
Great question Oleg. I’m a data analyst venturing out into the word of unit testing and wondering if A/B testing is something I should be learning.
**David Sweet**
A/B testing — and related methods — help you translate your data science work into concrete, business terms.
For example: You might design a new feature and find that it reduces a model’s RMSE by .1%. Is that good? How good? Would your boss care if s/he’s not a data scientist? Would a shareholder care?
The question you need to answer is: How much impact does your new feature have on the business? How much extra revenue can the business generate by using your new feature? How much more do users enjoy the product with your new feature?
You answer questions like that by running an A/B test comparing the original model (without your feature, version A) to the new model (with your feature, version B). You run that test on the production system — the web site or mobile app or whatever — and measure the business impact directly.
You could think of it this way: When you write a self-assessment at the end of a quarter or year, would you rather write, “I improved RMSE by .1%” or “I added $XX million/year to the bottom line”?
**Oleg Polivin**
David thank you for this detailed answer, it is great!
**Oleg Polivin**
And a somewhat related question.
Would it be correct to say that A/B testing is mostly used in e-commerce, advertising industries, and, more generally, where some kind of recommendations are involved?
That would explain my lack of knowledge of A/B tests since I have not worked in those industries.
**David Sweet**
Yes, A/B testing and related experimental methods are used in advertising and on recommender systems. They are used to improve web sites, web and mobile applications (think Google, Facebook, Instagram, Twitter, Spotify, Amazon, Uber, Apple products, and so on) and on trading systems.
In medicine, A/B tests are called random controlled trials (RCT), and are used to test the efficacy of new medications and other types of treatments. Anywhere you see “Six Sigma”, “process improvement”, etc. you’ll find A/B tests and related experimental methods. And, of course, you’ll find experiments in the sciences.
A/B tests may be applied anywhere you need to make a comparison in the face of complexity and uncertainty.
**Oleg Polivin**
I have some experience in running experiments in Economics, but we never had to use multi-armed bandits and we didn’t call it A/B testing, rather RCTs as well.
It also seems to me that organizing A/B tests in industrial setting is more complicated.
Thanks a lot!
**Alexey Grigorev**
How much statistics do you think data scientists and analysts need for running A/B tests?
It feels that data scientists are not always good with stats and focus more on ML.
**Alexey Grigorev**
And often experimentation platforms take care of things like calculating the sample size. But do we need to understand how these things work to be able to use them properly?
**Tim Becker**
Hi David Sweet, really interesting book! When I went through the introduction, I was wondering what the biggest pitfalls are when evaluating models? What kind of mistakes are frequently made when doing the final test in production. Also, when evaluating financial models, e.g. for stock trading, isn’t there a lot of randomness involved? How can we be sure about selecting the better model for the future? Extending the period over which we compare both models?
**David Sweet**
A mistake that is common, has a big impact, and is easy to avoid is early stopping. If your A/B test design says “run for 10 days”, and you stop before 10 days because the t statistic looks good, you’ve made the mistake of early stopping. The t statistic itself is noisy and takes time to settle down, so you really need to wait it out.
When you design an A/B test you’ll, in part, try to limit your false positive\* rate to 5%. Early stopping can easily make that rate much higher (like 50% or 75% or more).
\*A “false positive” is when you think B looks better than A, but, in fact, it’s not.
—
There is, indeed, a lot of randomness involve in financial models. It’s common to run an experiment on high-frequency strategies or execution strategies (which also run on high-frequency data) in 1 week - 1 month, depending on the specifics of the system.
**Tim Becker**
David Sweet thank you for answering my questions. It is a quite challenging and interesting topic.
**Mansi Parikh**
Hi, David! It’s nice of you to do this - thank you!
When easy-to-get large sample sizes generally cause most tests to be significant when means or proportions are actually only slightly different, what do you do? Do you have to look into other parts of the output, such Cohen’s D, to put the test in perspective?
**David Sweet**
If you have very large sample sizes, then you are fortunate. You’ll have very small standard errors of your measurements of your business metric and can, thus, precisely measure the difference in business metric between versions A and B of your system.
The question that remains is: How big of a difference you care about? The answer to this question is specific to your system. For example, if “version B” of an ad serving system produced $1,000/day more than version A, would you care? It depends. If your company is a small startup that just started serving ads and has little revenue then, yes! You need that $1k/day.
But if you work at Google, where ads produce $150B/year (I think), maybe you’d have to consider whether $1k/day extra is worth the effort it takes to modify the code and the risk (however small) of making a change to the system.
I like to think of this as the question of “practical significance” to differentiate it from statistical significance. Statistical significance tells you how much to believe the measurement. Practical significance tells you how much to care (from a business perspective) about the the value you measured.
**Mansi Parikh**
Also, whenever you are cautious about your new variant possibly performing poorly compared to your control, wouldn’t you always go with a split that allocates less traffic/money/etc towards the variant being tested? [https://geoffruddock.com/run-ab-test-with-unequal-sample-size/](https://geoffruddock.com/run-ab-test-with-unequal-sample-size/)
**David Sweet**
Yes.
Generally, it’s a good idea to start very small. With a small size running you can detect bugs in the new code, bugs in the measurement tooling, and very large, adverse changes in your metrics. Then you can scale up to the full testing size. Even that doesn’t necessarily need to be very large. Like you said, you might want to keep it small for safety’s sake.
The tradeoff to keep in mind might be that small sizes will take more hours (or days) to run to completion.
**Mansi Parikh**
In most cases, we are running experiments with multifactorial designs. Is it appropriate to still compare all treatments to a common single control (T1 vs C, T2 vs C, T3 vs C…) or perhaps create many different single treatment + control cuts (T1 vs not T1, T2 vs not T2, T3 vs not T3…) and use basic statistical hypothesis testing or should we graduate to something more sophisticated like an ANOVA (though to derive meaning out of those, we’ll typically run Tukey paired tests anyway)?
**David Sweet**
You can compare all of the treatments to a common control, but you’ll need to use a Bonferroni correction (or some other family-wise approach, if I understand correctly, is what you’re accounting for with the Tukey paired tests) to get the right p values.
**Kyle Shannon**
❓ Hey David Sweet, how do you handle evaluating multiple tests at once that could have interference with each other? Perfect world is to isolate, but that rarely happens. I was wondering if you have learned any tips or used any frameworks you liked
**David Sweet**
This is a tough one. My knee-jerk reaction is to suggest finding a way to decouple them. 🙂 For example, if you have a version A and version B of a web app, each day (or each hour, or whatever) you could flip coin and say, “heads we run A, tails we run B”. It’ll be lower precision that running simultaneously, but at least they won’t interfere.
But if interference is unavoidable, I don’t have a good answer. Perhaps you could find a way to model the interference. I’m picturing something akin to “y ~ chi\_A + chi\_B + chi\_AB” where y is your business metric and the chi’s are indicator variables. If you could fit a model of (roughly) that sort to your measurements, then maybe you could separate the effects of A and B from the effect of the interaction.
**Kyle Shannon**
Thank you for the response, haven’t done much with modeling for interference that’s interesting. Gotta love the ol coin flip 🙂
**Hironori Sakai**
Hello David Sweet, I have a question about the early stopping. In an ordinary (frequentist) A/B test, we compute the (minimal) number of observations of the groups for the experiment for the stopping condition in advance. How can we determine such a number in a Bayesian A/B test?
**David Sweet**
In the Bayesian approach — aka., multi-armed bandits — you don’t.
When you design an A/B test you place two constraints on your measurement: (i) the false positive rate is limited (usually to 5%), and (ii) the false negative rate is limited (usually to 20%). You calculate the minimum number of observations needed to satisfy those constraints, given that there is variation (error) in your measurement.
Bandit methods optimize for business-metric impact. Bandit methods will monitor the business metrics and their standard errors for A and B and allocate more observations to A or B as needed to capitalize on business metric and/or decrease the standard error.
Bandit methods will likely lead to higher false positive & false negative rates — especially when the business metric performance of A & B are similar. But the more similar they are, the less you care about telling them apart (if business metric maximization is your goal).
**Kyle Shannon**
❓ Do you have any suggestions of frameworks, tools or resources for working with stakeholders to help them better plan out their A/B tests?
**David Sweet**
Names that come to mind are Optimizely and VWO, but I have not used them personally. A/B testing is a big space, so you’ll find many other commercial and open source tools.
**Kyle Shannon**
Cheers, thanks!
**Kyle Shannon**
We’re using a homegrown and looking to get something more managed currently checking out optimizely heard some good things
**Alexey Grigorev**
I’ve just come across this:
[https://docs.google.com/document/d/1uyQwvvFAa6I5aexiEmKWdHLoBzNPPskDKwEbAaOKe6M/edit?usp=sharing](https://docs.google.com/document/d/1uyQwvvFAa6I5aexiEmKWdHLoBzNPPskDKwEbAaOKe6M/edit?usp=sharing)
**Eric Sims**
Ahhh! I need this book in my life! 😅 My co-worker and I have been breaking our brains over A/B testing this week.
**David Sweet**
Re: peeking
It’s terrible! Resist it! It can send your false positive rate through the roof.
Now, simply _looking_ at the t stats and measurements, of course, won’t cause any problems. In fact, you should watch to make sure that something isn’t going very wrong. After all, you’re testing something new. It could have an adverse effect on your metrics.
The problem is really if you say: “The t statistic is high, and the B version looks better, so I’ll stop now and switch over to B.” Had you waited, the t stat might have come back down, and you might not have been so excited about the B version and just stuck with A. That’s how you drive your false positive rate up. That’s why you need to wait.
**Eric Sims**
Thank you!
**Eric Sims**
What is your opinion on ‘peeking’? Is it terrible? Or is it something that people are going to do no matter what so you try not to lose sleep over it?
**Eric Sims**
Do you have any good analogies/examples to help people understand why ending a test when they see what they want is not a good idea? …asking for a friend 🙂
**Lavanya M K**
David Sweet what does “tuning up” in the book title mean🙂? Is it tuning abtest system or optimising the business metrics?
**David Sweet**
I’m using “tuning” to refer to the optimizing the business metrics. (Like setting the tuning knob on a radio.)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Grokking Machine Learning – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Grokking Machine Learning
-------------------------
#### by [Luis Serrano](https://datatalks.club/people/luisserrano.html)
##### The book of the week from 09 Aug 2021 to 13 Aug 2021
It’s time to dispel the myth that machine learning is difficult. Grokking Machine Learning teaches you how to apply ML to your projects using only standard Python code and high school-level math. No specialist knowledge is required to tackle the hands-on exercises using readily-available machine learning tools!
* [Book's page](https://www.manning.com/books/grokking-machine-learning)
* [Book's GitHub repository](https://github.com/luisguiserrano/manning/)
Questions and Answers
---------------------
**Kashan Ahmed**
Will this book cover unsupervised learning as well?
**Luis Serrano**
Hi Kashan Ahmed !
No, the book only covers supervised learning. Perhaps a future book will cover unsipervised
**Luis Serrano**
I mention some things about unsupervised learning at the beginning, but never cover the algorithms in detail
**Kashan Ahmed**
Thank you for answering. 🙂
**Kashan Ahmed**
will it cover scikit learn?
**Alper Demirel**
Hi Luis Serrano, Firstly thank you for your time.
What makes this book special from other related books on the market?
**Luis Serrano**
Hi Alper Demirel , thank you for your interest!
I wrote this book to be understandable by readers without an extensive background in math and programming, so the algorithms are explained in a more conceptual way with examples, figures, and stories, as opposed to formulas. (The formulas are there too, but they appear after the conceptual explanations)
**Alper Demirel**
Thanks a lot for the answer, I’m so glad it’s not a book full of formulas. I hope I get the chance to read it!
**Wendy Mak**
hi Luis, how is a ‘grokking xxx’ book different to the other manning books? and what made you write a book in this style?
**Luis Serrano**
Hi Wendy Mak !
The grokking books tend to make a big effort in making the topic understandable for most people, not just those with a super technical background. I always like to explain things in that manner, as that is the way I like to understand things. When I started writing this book it wasn’t meant to be a grokking book, but when the editors saw it had that style, they asked me if I wanted to make it part of the series and I thought it was a good idea.
**David Cox**
Riffing off this thread, how did you go about identifying (a) that you wanted to write this book and (b) connect with Manning for publishing?
**Wendy Mak**
also, how did you get into teaching ML, and what do you enjoy about teaching?
**Luis Serrano**
Wendy Mak I have always loved teaching. One reason is that I am a slow learner, so I always have to digest everything I learn until I make it very understandable for myself (with figures, examples, etc), and so when I explain it to others it has already been digested in my mind, so teaching comes out easier.
I started as a mathematician, my PhD was in math and I was teaching and working on research. The ML bug bit me when I started hearing about it and I decided to switch careers and started working at google, that’s when I started learning ML more seriously.
**Neal Lathia**
❔ What does _grokking_ actually mean? 😂
**Alexander Seifert**
[http://www.catb.org/jargon/html/G/grok.html](http://www.catb.org/jargon/html/G/grok.html)
🙂
**Luis Serrano**
Hi Neal Lathia ! To grok is to understand something very well in a conceptual and intuitive manner. This is why the grokking books try to explain things in such a way for most people to understand even if they don’t have a technical background.
**Luis Serrano**
Thanks Alexander Seifert !
**Eric Sims**
Do you have a favorite ML algorithm at the moment? 🙂
**Luis Serrano**
Hi Eric Sims !
Great question! I like many algorithms, I gotta say I really enjoy the kernel method (SVMs) because it’s so clever. Also naive Bayes, and anything that is purely probabilistic, since all they do is playing with conditional probabilities and they can do wonders.
**Eric Sims**
How did you determine where to draw the line between “essential” math for understanding ML and “extra stuff” that can be learned later if needed? I often worry that I will miss something important if I don’t have a decent grasp on what’s actually happening. Of course, my decent grasp is not necessarily super strong - I don’t really know any calculus.
**Luis Serrano**
Eric Sims , yes, that’s a huge part of the book, and the reason is that I believe that to start in ML, one doesn’t need that much understanding of math. The math can be learned as you learn ML, hand in hand. Most experts recommend learning a lot of math and then starting ML, which I don’t fully agree with, which is why I wrote this book.
As for essential math to read this book, high school math is enough. Having an intuition of what a formula is, basic equations like the equation of a line, and basic probability (e.g., being able to calculate a probability as a ratio of numbers) is enough, as everything else can be developed as we go.
**Chetna**
Hi Luis Serrano, thanks for this QnA.
Is this book about ML system design?
**Luis Serrano**
Hi Chetna ! Great question, no, the book doesn’t talk about system design. It talks about the algorithms, how they work, and how to apply them. There is code, but mostly to study datasets, not to put in production.
**Chetna**
got it, thanks 🙂
**Luis Serrano**
Hello everyone! It’s such a pleasure being here, and thank you for your great questions! Answering them now in the threads.
**Kashan Ahmed**
Will the person who completely read and implement all the example from the book be able to consider themselves intermediate level or will need more practice and familiarity with high level libraries?
**Luis Serrano**
Hi Kashan Ahmed! Great question, yes, the person who finishes the book will have a working knowledge of most supervised learning algorithms, and the packages to use them (including a deep learning package), so they’re definitely an intermediate level ML practitioner.
**Luis Serrano**
More can always be learned after the book, including other similar libraries, other fields such as unsupervised learning, generative learning, and reinforcement learning. Also, ML production and system design is a field people can learn more about outside of the book, in case they want to implement models in production.
**Kashan Ahmed**
Thank you for detailed answer.
**Krzysztof Ograbek**
Hi Luis Serrano. I have to admit I never heard of your book. Thanks for doing this!! Which libraries are you using for projects in your book?
**Luis Serrano**
Thanks Krzysztof Ograbek!
The ML libraries are scikit learn, Turi create, Keras (Tensorflow), and xgboost. For other stuff, matplotlib, pandas, numpy
**xnot**
How much important would you place on developing geometric intuition behind linear algebraic concepts? Do you think you can do without it when developing / maintaining ML projects in production ?
**Luis Serrano**
Hi xnot!
I think geometric intuition is more important than the formulas and the math. And yes, I think this can be done while learning and developing ML.
When I talk about geometric intuition, I mean things like drawing lines that pass close to a group of points, lines and planes that separate points of different colors, rotations, transformations, etc. Most people have this type of intuition, it’s only a matter of tying it to the algorithms and the applications.
**xnot**
Thanks Luis Serrano. Does the book cover ways of approaching this?
**Quynh Le**
Hi Luis Serrano, I am glad to know about the book. Thanks for writing it! I learned about regressions in school but have never done machine learning yet. How would you suggest me approach machine learning in general? Can I implement machine learning with only Python?
**Luis Serrano**
Thanks Quynh Le!
If you learned regression, then you know machine learning. Most of the main algorithms are similar to that one, only with some small tweaks. I think a book or introductory course in ML can get you up to speed. And yes, one can implement everything only in Python, as a matter of fact, it’s the one I recommend the most, since most of the important packages are written there.
**Quynh Le**
Do you suggest any intro ML course or book (other than Andrew Ng course)? Can I read and implement projects from your book using Python?
**Luis Serrano**
I have a bunch of videos here that you may enjoy: [https://serrano.academy](https://serrano.academy/)
**Luis Serrano**
Also, there are some interesting courses at Udacity. Here is one on deep learning that I teach with a few other people:
[Intro to Deep Learning with PyTorch](https://www.udacity.com/course/deep-learning-pytorch--ud188)
**Luis Serrano**
Quynh Le ^^
**Quynh Le**
Luis Serrano Thanks for the suggestions, I am a big fan of Udacity courses! I’ll check your website as well!
**WingCode**
Hi Luis Serrano, Your book looks like a fun read!
Do you find it difficult to strike the right balance between complicating vs oversimplifying while explaining a data science concept?
**WingCode**
Can your book used for communicating data science concepts to higher stakeholders who necessarily don’t understand the nitty gritty of data science?
**WingCode**
What is the topic you found the most difficult to grok? Have you ever felt that any data science topic cannot be just “grokked” ? 😅
**Luis Serrano**
Hi WingCode, great questions!
1. It is challenging to strike the right balance between oversimplifying while explaining, this is why I try my explanations on people, including experts and non-experts. If I can get a non-expert to understand it while not boring an expert, I feel that I’ve hit the right spot.
2. Yes absolutely, the book can be used to communicate data science to stakeholders, since it has lots of real-life applications, which they can use to see the value of these algorithms. It also explains the details, which can show them they’re not that complicated or hard to implement, and that they’re not voodoo, just simple math used with the purpose of solving the problem.
3. Ah interesting. There are many topics I’ve found hard to ‘grok’. In the book, some of the ensemble methods such as xgboost or gradient boosted trees took me a long time to understand and to simplify, since they have lots of ins and outs, and some of them sound a bit arbitrary. Once you really get into the details, they are not arbitrary, but it takes a while to realize. Other topics outside of the book scope, such as reinforcement learning, take me quite a long time, for the same reason. But I have a believe that everything can be ‘grokked’, and the ones that haven’t yet, is because we don’t fully understand them. 🙂
**WingCode**
Thank you Luis for the great answers! :)
**WingCode**
Luis Serrano I have few more questions 🙂
1. What are the next upcoming topics under your “grokking” radar (excluding the ones in MEAP) ?
2. How do you perform candidate selection of a topic for “grokking”? Is it the hype or general consensus that something is inherently difficult?
3. Any plans for quantum mechanics or quantum computation grok? I asked because in pop culture adding “quantum” before any word is generally a good way to mystify a topic.
**Luis Serrano**
WingCode great questions, keep them coming! 🙂
1. Unsupervised learning is a big one, since it covers generative learning. Aside from that, reinforcement learning is one.
2. For this book, I picked the most popular algorithms of supervised learning, so it was pretty straightforward. As for other type of content creation (videos, etc), I normally have a list of things that I’m interested on because I’m trying to understand them, in topics like probability, ML, statistics, etc., so as I understand them, I create content about them. Some of them come out of projects at work, and others from things I watch or read, etc. It’s pretty random. 🙂
3. YES!!! Quantum is definitely next on the list. Right now I’m working on quantum ML, and learning a lot of stuff. My goal is to understand it in a simple way, just like it is with ML. Definitely keep an eye for that material, because I’ll be grokking quantum a lot in the near future.
**WingCode**
Haha, thanks Luis. It is a pleasure to ask you questions & also to get your answers :)
**Alex**
Hola Luis Serrano, super excited to have you in here!
What is actually the target reader of your book? Who is it aimed for?
Muchas gracias!
**Luis Serrano**
Gracias Alex !
The book aims to be a one-size-fits-all, as it offers beginners a chance to get into ML, and the more experts a different and simpler view of the algorithms that perhaps they haven’t seen before.
But in a nutshell, the reader who’ll enjoy this book the most is the beginner in ML who comes without a very heavy knowledge of mathematics and programming. The reason is because in the book we explain the methods and algorithms from a more intuitive perspective, where the math is there, but more in drawings and examples than in formulas.
**Lavanya M K**
Hi Luis Serrano What are the tools you use to create illustrations in the book?
**Luis Serrano**
Hi Lavanya M K!
For the illustrations I used mostly keynote. For some of the plots and graphs I used matplotlib.
**Utkarsh Agrawal**
I recently read Machine Learning Yearning by Andrew NG and I particularly liked how he describes multiple scenarios and then suggest strategies for tackling them.
Would you guys recommend similar books?
**Alexey Grigorev**
I like “Rules of machine learning” from folks at Google. I find it somewhat similar to ML Yearning
**Andrea Mordenti**
Hi Luis Serrano
it is really exciting to have the possibility to chat with you. I am an adjunct professor both at university in Italy and high school and I believe the way you explain everything is straightforward for an audience that can span from very entry levels to people that are already approaching the world of AI. I’d like to suggest you book to my classroom and use it as a reference for the course 🙂 where will be available the printed version?
And, are you thinking about also a book for advanced users? If so, what would you like to focus on? Thanks!
**Luis Serrano**
Thank you Andrea Mordenti, glad you enjoy the explanations, and thank you for considering the book for the course!
The printed version will be available soon, hopefully in a month or two, I’ll keep you posted.
**luckylittle**
Welcome Luis Serrano - I like the _grokking_ series of Manning and your book definitely caught my attention. I would be very interested in reading it. I checked your website [https://serrano.academy/](https://serrano.academy/)
to learn more about you. It is very impressive that you were part of the Google video recommendations team at YouTube, where you trained machine learning algorithms to recommend videos. My questions are:
1. Quickly looking inside the book live preview, I see the chapter about overfitting/underfitting. The _Fukushima power plant disaster_ when predicting the probability of a very strong earthquake is a devastating example of overfitting. What is the best way to train a predictive model that does not “follow” the training data too closely (and thus prevent potential disaster)?
2. Has any of your experience from the _YouTube_ recommendation team reflected chapters in the book?
3. You have been writing it since ~2019, do you enjoy writing the book and is it going to be released soon?
**Luis Serrano**
Hi luckylittle, thank you for your interest in the book and the page! I hope you had a chance to check out [the youtube videos](http://www.youtube.com/c/LuisSerrano)
.
1. For anomaly detection, such as Fukushima, I would use unsupervised learning techniques, such as clustering, etc. Some of them, like DBSCAN, have a way to detect outliers.
2. I add examples of recommendation engines such as youtube in several places in the book. For example, you can use linear regression to predict how long a video will be watched by a user based on how long that user has watched other videos. Also, classification methods can be used similarly to predict if the video will get watched or not. Also, there is an example of analyzing the text in Netflix reviews of movies, using python.
3. The book will be released very soon, it’s in the last stage of production. I think there’ll be a physical version in about 1-2 months. I have enjoyed the writing (and re-writing) process, as it has taught me to understand things in a different way.
**luckylittle**
Luis Serrano Yes, I have checked out your YouTube videos - very impressive!
**Luis Serrano**
Hello everyone here! As I’ve been speaking to people about other types of content, I want to invite you to check out [my youtube channel](http://www.youtube.com/c/LuisSerrano)
, where I have videos about machine learning explained in friendly conceptual ways (not with crazy formulas).
**Krzysztof Ograbek**
Luis Serrano What are other grokking books you would recommend to aspiring Data Scientists?
**Luis Serrano**
Krzysztof Ograbek I recommend grokking deep learning, grokking deep reinforcement learning and grokking algorithms. All very good. DL and DRL are written by two friends and former coworkers of mine (Andrew trash and Miguel morales), who are great at explaining
I also recommend advanced algorithms and data structures by Marcella la Rocca, also from manning, great for the fundamentals
**Matthew Emerick**
Hey, Luis Serrano! Thanks for doing this.
Do you talk about graph machine learning in your book?
**Luis Serrano**
Thanks Matthew Emerick !
No, the book focuses on supervised learning, and there is no graph machine learning
The books I recommend after this would be grokking deepnleanring and grokking deep reinforcement learning. They are a good continuation of the topic, and written in a similar style
**Matthew Emerick**
What do you consider the best follow up book after reading yours?
**Alexey Grigorev**
Hi Luis, you’ve had quite an interesting career so far. I’m curious to know a bit more about your experience as the content lead at Udacity and lead educator at Apple. How did you transition to this kind of role from being a ML engineer?
**Alexey Grigorev**
And as a follow-up, how did your day back then look like?
**Luis Serrano**
Thanks Alexey Grigorev!
Being at udacity was great because I really enjoy teaching and that’s what I was doing. Being able to lead a team that produces educational content was great because I would design the courses and was able to propagate my style to others, who would also inject their style into the courses, getting the best of both worlds. The transition to Apple was smooth since at Apple I was teaching workshops, it was actually fun because they were in person, and I enjoy that.
Coming from ML engineer to educator was very rewarding, because I think as a teacher more than as an engineer. The engineering mindset is more of a “get things done” mindset, while I like to understand things slowly and thoroughly, which is more suitable for teaching.
**Luis Serrano**
My day back then had a lot of meetings with instructors in my team, I would check their material and give them feedback. There were team meetings to decide on curriculum etc. I would try to crunch the meetings in some days, and leave some days empty so that i could sit down and create educational content for hours, as this can be a lengthy process.
**Alexey Grigorev**
Thanks for sharing it! Sounds quite fun
**Luis Serrano**
Thank you Alexey Grigorev for the invitation, and thank you all so much for your great questions! It’s an honor to be a part of this group!
Here is my contact for anyone who’d like to stay in touch, or has any further questions/comments!
email: luisgui.serrano@gmail.com
[Webpage](http://serrano.academy/)
[LinkedIn](https://www.linkedin.com/in/luisgserrano/)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Practical MLOps – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Practical MLOps
---------------
#### by [Noah Gift](https://datatalks.club/people/noahgift.html)
, Alfredo Deza
##### The book of the week from 30 Aug 2021 to 03 Sep 2021
Getting your models into production is the fundamental challenge of machine learning. MLOps offers a set of proven principles aimed at solving this problem in a reliable and automated way. This insightful guide takes you through what MLOps is (and how it differs from DevOps) and shows you how to put it into practice to operationalize your machine learning models.
Current and aspiring machine learning engineers–or anyone familiar with data science and Python – will build a foundation in MLOps tools and methods (along with AutoML and monitoring and logging), then learn how to implement them in AWS, Microsoft Azure, and Google Cloud. The faster you deliver a machine learning system that works, the faster you can focus on the business problems you’re trying to crack. This book gives you a head start.
* [Book's page](https://www.oreilly.com/library/view/practical-mlops/9781098103002/)
* [Amazon page](https://www.amazon.com/Practical-MLOps-Operationalizing-Machine-Learning/dp/1098103017)
* [Book's GitHub repository](https://github.com/paiml/practical-mlops-book)
Questions and Answers
---------------------
**Alex S**
I wasn’t sure how it’s possible to read this book as it isn’t published until October this year. Could you let us know, Alexey Grigorev?
**Alexey Grigorev**
Probably you should ask Noah Gift about it 😃 But you can read it through OReilly Learning, the early release version is already available there
**Alex S**
Ah ok I didn’t realise that you could read the book before it was published!
**Mahmoud Jalajel**
Same in Germany.
And I _think_ my trial with O’Reilly learning expired. So I’ll have to wait for the book release!
**Maja**
Me too. I can’t wait till October to by this book fromNoah Gift. His previous two books (Python for DevOps and Pragmatic AI) are great and have been a huge help for me. Also, his co author Alfredo Deza has such an inspiring life story.
**Noah Gift**
Yes, you can read online in rough draft form on the O’Reilly website: [https://learning.oreilly.com/library/view/practical-mlops/9781098103002/](https://learning.oreilly.com/library/view/practical-mlops/9781098103002/)
**Noah Gift**
It also should be in kindle form in around 30 days or so and in print soon after.
**Praveen**
Noah Gift
Are there any generic rules behind selecting MLOps tools for a given ML task ?
**Noah Gift**
A good place to start is by using the tools on the platform you are already on. All major cloud platforms have an MLOps solution and this is a great place to start. AWS Sagemaker, GCP Vertex AI, and Azure ML Studio
**Kshitiz**
Noah Gift and Alfredo Deza - First of all thanks for doing this. I want to discuss couple of things here -
1. Should MLOps be applied to all data science/ML projects or should people be looking at some sort of maturity in the project? To put it simply - Should there be any minimum requirements in terms of size of data, number of users if it’s used in an application, how long in a problem do people have to wait to get the results validated etc. ?
2. In what sort of problems/use cases are feature stores useful? How is feature store different than a database?
**Noah Gift**
1. I do think the process of MLOps should be applied to all projects because it is an extension of DevOps. All software projects should have CI/CD and you can even do this with notebooks: [https://github.com/noahgift/myrepo](https://github.com/noahgift/myrepo)
2. For feature stores they have raw materials in a form easily consumed by a ML pipeline. I.E. Containers package the runtime with the code, Feature stores package the raw ingredients for ML into a metadata system. A database is too low level by itself to be a feature store.
**Eunice**
Noah Gift What are the common skills between an MLOps and a Data Engineer ? And what skills are specific to MLOps ?
**Noah Gift**
There is a strong overlap between Data Engineer and MLOps with perhaps as little as a 5% overlap. The key 5% is that a MLOps practitioner also knows a bit about ML and can train models, diagnose their output and knows about ML Platforms like AWS Sagemaker, MLflow, etc.
**Denis Volk**
Larger companies are using in-house MLOps platforms, while for smaller teams, it is hard to dedicate lots of development time to set up similar machinery. On the other hand, some level of MLOps is just necessary to keep an ML project useful to business users. How to determine the right amount of MLOps for a project?
**Noah Gift**
I would start with whatever platform is available and use their offerings: i.e. Google, AWS, Azure. Let’s take AWS for example, if you have gigantic data and gigantic teams, say over 250 people in your company then a “big” platform like Sagemaker probably makes sense because of how much it offers.
If you use AWS but have a 3 person team, Sagemaker may or may not be the best easy win. Perhaps AWS App Runner with open source MLOps tools might be a better fit.
**Alexey Grigorev**
Which open source tools can you recommend?
**Jon Exume**
Can you talk about the specific careers that MLOps plays a big role in?
**Noah Gift**
Autonomous driving is a good example. I went to Tesla AI Day last week and 90% of the people I spoke with did MLOps, i.e. tools/infra around computer vision.
**Jon Exume**
Thanks
**David Cox**
I appreciate your taking the time to answer questions, Noah Gift! From your experience, what is the background of the primary people you see getting into MLOps?
**Noah Gift**
People with a strong DevOps/Infrastructure skill set can easily make the transition to MLOps. They just need to pick up a bit of ML training. One way to do this is to read the book I wrote and also to get AWS ML Certification certified (or similar). Note, I helped create the AWS ML certificaiton….
**David Cox**
Thanks, Noah!
**David Cox**
A follow-up question to the one above. Sometimes “new” jobs in technology are just the same skills from past positions but combined in a new way or centering around a new tool. What do you think distinguishes MLOps from past, similar areas? And, what similarities does it share with other areas/processes?
**Noah Gift**
I think MLOps is essentially an evolved DevOps but with the addition of ML.
**Duverger PETGA**
Hi Noah Gift I really appreciate your work but I have one question : between “Cloud Computing for Data Analysis” and your actual book “Practical MLOps” or “Python for Devops”, in what order we have to read your books ? For a beginner in MLOps ?
**Noah Gift**
You can read in any order. Since both Python for DevOps and Cloud Computing are start with either then move on to Practical MLOps. They all have a similar theme with more depth on cloud, devops or mlops depending on the book
**Doink**
How to decide which tools to choose? Should one choose for an open source alternative or choose a tool by a cloud service provider?
**Noah Gift**
How to decide which tools to choose?
whatever is simple to get started with an improves automation and quality.
Should one choose for an open source alternative or choose a tool by a cloud service provider?
I personally prefer to pay a vendor, so I would start with a cloud offering.
\[10:03 AM\] There are a plethora of tools coming out, how do you make a framework on choosing which tool to choose and how to choose?
If you are on a cloud platform start with what they offer and go from there.
\[10:04 AM\] How to practically navigate through the MLOps cycle? Some nuggets of wisdom like MLOps isn’t a tech problem but a people problem etc
Make sure you have CI/CD working and iterate from there.
\[10:04 AM\] Do small startups really need MLOps or is it over engineering?
MLOps is a behavior/methodology that focuses on Kaizen (continuous improvement). So it applies to anything small or big.
A. Automate everything
B. Make it better quality daily
**Doink**
There are a plethora of tools coming out, how do you make a framework on choosing which tool to choose and how to choose?
**Doink**
How to practically navigate through the MLOps cycle? Some nuggets of wisdom like MLOps isn’t a tech problem but a people problem etc
**Doink**
Do small startups really need MLOps or is it over engineering?
**WingCode**
Hi Noah Gift,
Why did you choose the cheetah as the book cover? How is it related to MLOps? Does it portray the advantages given by MLOps ? 🙂
**xnot**
Looks like a 🐕, probably dalmation
**Noah Gift**
We don’t have control of the animals.
**Alper Demirel**
Hi Noah Gift, thanks for being with us.
What should be the starting point for our current project for MLOps? And what are the biggest disadvantages that MLOps bring?
**Noah Gift**
To start with I would make sure you have CI/CD, i.e. the foundation of modern software engineering. This is the first step.
I don’t believe there are any disadvantages to MLOps. In a nutshell it just means “Kazien”, i.e. continuous improvement. Make everything better and more automated.
**Lalit Pagaria**
Thanks Noah Gift for this session. I have following queries
What are good observability tools are there in MLOps space? (Specially open source tools)
What is most important MLOps checklist for business critical model serve pipeline?
Do you believe current set of lowcode/nocode MLOps solutions are good enough to be used for mission critical usecase?
**Noah Gift**
I would start with traditional monitoring/instrumentation for you platform using whatever tools are already in place. Then add additional business logic for ML.
Additionally if you use Cloud Platforms they have default monitoring like for example Azure ML Studio which does model versioning and experiment versioning.
**Noah Gift**
“What is most important MLOps checklist for business critical model serve pipeline?”
Start with CI/CD, if you don’t have this you cannot do MLOps
**Noah Gift**
“Do you believe current set of lowcode/nocode MLOps solutions are good enough to be used for mission critical usecase?”
Yes, in many cases you don’t need to write code. A good example is Azure ML Studio AutoML.
**Eunice**
Hi Noah Gift, Alfredo Deza thanks for the quick answers. When a team starts using the Agile framework, they may need a Scrum Master to facilitate and help to implement Agile. Do you think an MLOps specialist may be necessary for big organizations used to other frameworks to start using MLOps? Or hire an ML Engineer and have a Lead Data and Project Manager aware of the subject may be sufficient?
**Noah Gift**
I think it may help to have someone who has some form of MLOps certification. One good example of this is course I just created on Coursera: [https://www.coursera.org/specializations/building-cloud-computing-solutions-at-scale](https://www.coursera.org/specializations/building-cloud-computing-solutions-at-scale)
**Noah Gift**
Btw, you can also help promote a lot of my content and contribute to charity with this humble bundle, including PSF and women who code: [https://www.linkedin.com/posts/noahgift\_humble-software-bundle-python-2021-activity-6838263509390807040-zJ98](https://www.linkedin.com/posts/noahgift_humble-software-bundle-python-2021-activity-6838263509390807040-zJ98)
\>. Help spread the word.
**Kamran Ali**
Noah Gift Is this book covers any specific Cloud Platform (e.g. AWS ) or any specific tool (e.g. MLFlow) etc
**Noah Gift**
We cover AWS/Azure/GCP very heavily
**Kamran Ali**
Thanks for the response ! 🙂
**Alexey Grigorev**
By the way, we have another celebrity appearance - Alfredo Deza himself! Welcome Alfredo!
**Maja**
Hello Alfredo Deza ! Thank you so much for joining us. I am so happy to have this opportunity to e-meet you and to ask questions. From your inspiring life story we can learn that anything is possible and that geat tihngs do happen. You just have to love what you are doing and to do it in the best way you can. From your book “_Python for DevOps_” we have learned how to do DevOps in Python. But, I have to ask you considering that ML pipeline is more complex, what are things we shouldn’t ever do - bad practices that happen due to the lack of knowledge, or experience?
**Alfredo Deza**
Hi Maja! Thanks for the super kind words. This is a great question! I think that there are a few things from seeing the opposites of the core pillars of operations (DevOps/MLOps in general) like automation, monitoring, testing, and CI/CD. For example: no (or little) automation, doing things manually, no pipelines, no monitoring.
Aside from those, you have other red-flags like over-engineering. Fast, iterative processes are far better than waiting 3 months to design the perfect thing
**Alfredo Deza**
There is always room for improvement. I keep hearing people say “what if everything is already automated?” - well… there is always stuff to automate and improve. You are asking a critical question here, and not asking critical questions (see critical thinking section at the beginning of the book) is a tremendous problem.
**Maja**
I will read it as soon as I get the book. Thank you Alfredo Deza so much for your guidance!
**Livsha Klingman**
Alfredo Deza Noah Gift I’m a REAL beginner, but majorly interested and so far got a good repertoire of success in a few beginning projects (maybe beginners’ luck)!
Your books are all touching on my work topics and what I am facing daily and now you have exposed them for me to read up on!
As I develop, slowly, my knowledge and experience, I am discovering how much breaking into the ‘big’ world is an upward struggle between big enterprises and the well-experienced. (As in any professional field!).
What is the correct priority considering the limited manpower for startups and small businesses - veer towards automation or not? Develop pipelines or CI/CD? or using a service tool and focusing on the ML?
Do you have any advice for ‘us’ small businesses to ‘make a dent’ in the big world and gain the skills and experience to be aware of and make the educated decision of tools, methodology and topology, correctly balancing labor, to successfully develop MLOps?
**Alfredo Deza**
Automation is not a one time thing that takes months to achieve and is super expensive. Noah Gift taught me the right path years ago: pick _any one thing_ you do manually and automate it by the end of the week. Rinse and repeat, and suddenly a few months later you have several things automated. It is now CHEAPER to run operations because of it and the team can concentrate in even better automation
**Alfredo Deza**
_Always_ automate
**Alfredo Deza**
Leveraging the cloud for automation (CI/CD or pipelines doesn’t matter) is good. Leveraging anything that is already solved that is not a core competency of your business is crucial
**Livsha Klingman**
Alfredo Deza Thanks for your response! Taking this opportunity further… How do you suggest trying to circumvent issues in MLOps, with compounding model decays through either data discrepancy between CI and CD or training and pipeline data, or models based on a initial wrong hypothesis - collecting biased data, which then exacerbates over time growing in bias?
**Alfredo Deza**
This is a difficult question to get a straight answer. I don’t think there is a one-size-fits-all problem solver here. If you have biased data, but you have automation, tests, pipelines, etc… you still have a biased model in the end. MLOps can’t solve biased data. There is always the human element in all of this, and critical thinking (see critical thinking section at the beginning of the book) is essential
**Livsha Klingman**
Alfredo Deza Thank you for your advice.. Can’t wait to read your book and thanks for all your valuable time!
**Shankar Somayajula**
Alfredo Deza Thanks for taking questions. I like the focus on Automation in your book and answers to questions here.
Can the process of Automation involve an abstraction of the data structures as a data model (schema/objects) so that the artifacts of automation are reusable from one project to another.. facilitating more reuse, making the process of automation more of a Product/Platform service instead of a Project/Task output? How does one facilitate reuse (otherwise) - publishing an API?
**Alfredo Deza**
Reusability is the gold standard. Not entirely sure how to abstract data structures, but sharing/reusing artifacts sounds great to me. As to _how_ to do this, well it depends! Perhaps an S3 bucket would suffice if everything is behind AWS. If you need external access, it sounds like an HTTP API is the way to go
**Tony Gunawan**
Hi, Noah Gift and Alfredo Deza. Thank you for being here to answer the questions. I am newbie in the MLOps field as I am a data engineer right now on financial institutional field with previous experience as ETL developer and hope my questions is not out of context. Is it possible to fully automate all the process of ML end to end, especially in model evaluation? So many data with unpredictable behavior (like in the financial case) that make a model that has been deployed obsolete like during the start of the pandemic, behavior of the people who need to borrow the money from banks or other institutional lenders have gradually changed and need to do some remodeling with new set of data behavior if I would say. In this case, what kind of things that MLOps need to consider when facing this kind of unpredictable phenomena that will happen in the future? Thank you.
**Alfredo Deza**
There is no silver bullet here where everything can be fully automated. You’ve mentioned one of the caveats which is unpredictable behavior. Human interaction+evaluation has to be possible. Pipelines have to be flexible. Any automation/workflow has to easily allow for changes and updates. When automating, you _must_ think about the pitfalls and how to address them. For example, you have a pipeline that normalizes data in small amounts, what can you do _today_ that will allow batching the normalizing if the data is gigantic?
**Alfredo Deza**
alfredinsky
**Tim Becker**
Hi Noah Gift and Alfredo Deza, thank you for answering all our questions! What would you say are the most useful MLOps skills for a data scientist? For example, if I as a data scientist want to increase the collaboration with a MLOps specialist or if I am working for a small company that does not have a dedicated MLOps person and I as a data scientist have to cover the topic as well as possible.
**Alfredo Deza**
if you are starting out then I would pick automation. Anything you can do to start automation is going to be super useful and empowering
**Tim Becker**
Do you have a good idea for a toy project that I could work on to learn more about MLOps? Do you use an example project in your book?
**Alfredo Deza**
The book uses a public Github repository that you can use to see examples [GitHub - paiml/practical-mlops-book: \[Book-2021\] Practical MLOps O’Reilly Book](https://github.com/paiml/practical-mlops-book)
**Noah Gift**
cookbook in particular is a good recipe
**Tim Becker**
thank you guys 🙂
**Luke Garcia**
Hi Noah Gift Alfredo Deza, I’m new to DS and MLOps. Does the book mention Kedro? What role (if any) does Kedro have in MLOps?
**Alfredo Deza**
We don’t have anything related to Kedro (sorry, not sure what that is)
**Luke Garcia**
thank you
**Noah Gift**
If you want a deep dive on the book and how to MLOPs from Zero, watch this 2.5 hour video: [https://www.youtube.com/watch?v=OMv3lkB5W20](https://www.youtube.com/watch?v=OMv3lkB5W20)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Business Skills for Data Scientists – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Business Skills for Data Scientists
-----------------------------------
#### by [David Stephenson](https://datatalks.club/people/davidstephenson.html)
##### The book of the week from 23 Aug 2021 to 27 Aug 2021
It takes more than technical skills to succeed as an analytics professional.
This book explains the foundational business skills you’ll need to deliver business value and grow your career as an analyst or data scientist. Drawing on best practices, published research, case studies and personal anecdotes from two decades of industry experience, David Stephenson lays out a practical and concise overview of foundational skills related to Company, Colleagues, Storytelling, Expectations, Results and Careers – emphasizing how each topic relates to your unique position as an analytics professional within a larger corporation.
* [Book's Amazon page](https://www.amazon.com/gp/product/B08TY93PMC/)
* [Book's page at Bol](https://www.bol.com/nl/p/business-skills-for-data-scientists/9300000024726496/)
Questions and Answers
---------------------
**Alper Demirel**
Hi David Stephenson, thanks for being with us.
How important are business skills in a data scientist’s career? Aren’t these skills considered enough? Do people avoid it or think it’s a waste of time?
**David Stephenson**
If you’re working in a business, they are very important. Have a look at the introduction at the beginning of the book.
**David Stephenson**
Oh, just in case you hadn’t realized this, you can download the free kindle sample of the book and read the preface, introduction and first chapter or two without actually buying the entire book. This will probably give a better answer to your question than I could type in here.
**Alper Demirel**
Thank you very much, I will do as you say 🙏
**Toxicafunk**
Hi David Stephenson, I’ve always considered storytelling to be one of the most underrated skills on the software industry. Beyond the effective use of charts, which is something every data scientist should master, what other storytelling tools do you recommend?
Also, do your think going as far as to read our take courses on writing (i.e. how to write a book or a movie script) may help? Know of any related anecdote?
Thanks for being here!
**David Stephenson**
Hi Toxicafunk, we need to consider the WHO, the WHAT and the HOW of the storytelling process. Making good charts is just part of the HOW. In my experience, many data scientists fail to understand their audience (WHO) and, surprisingly, also don’t have a firm grasp of their own message (WHAT). I talk about these in more detail in chapters 2, 6 and 7 of the book.
**Toxicafunk**
Great insight David Stephenson I think getting the WHAT right comes from experience, but the WHO is always a tricky.
**David Stephenson**
Actually, getting the WHAT right is really difficult, even with experience. Most data science presentations don’t truly have a clear, focused, relevant WHAT.
**Saskia Kutz**
Hi David Stephenson, thank you for being here with us. What do you think are the most common issues for data scientists talking to business people? What business skill do you rank the most important?
**David Stephenson**
I think the #1 skills is ‘empathy’ That is, to understand the background and perspective of the people you are working with. If you get that right, you’ll be able to identify the most important business questions, communicate effectively, and work through difficulties and roadblocks together with those stakeholders and colleagues. It’s the focus of chapter 2 in the book, where I try to illustrate how various horizontal and vertical segments of the company can be completely different. it’s important for us to understand those differences so that we can work effectively.
**Saulius Lukauskas**
Hi David Stephenson , another industry that takes pride in their business knowledge is the consulting industry. The key players in this field have collectively converged to the [“case study” interview format](https://mconsultingprep.com/case-interview-examples/)
where the candidate is given a description of a business situation and the issue at hand (e.g. company needs to understand why it’s losing money) and is asked to apply their knowledge to provide a data-driven advice for the company on how to proceed. I can imagine data scientists facing similar questions in their day-to-day job as well. Do you think practicing such case studies is a worthwhile time investment for data scientists, especially the junior ones, or would you advice against that? Additionally, am I mistaken or such business-problem case studies are not very common in the data scientist interviews as they tend to be more technical? Should companies adopt them?
**David Stephenson**
Hi Saulius Lukauskas, when I was finishing grad school, some classmates looked into consulting and went through that Mckinsey/BCG case study interview process. It’s interesting, and it fits their business model, but I think it’s a bit less relevant to data scientists. It really depends on what skills you expect for the role you’re hiring for, and whether you are expecting the person to identify novel solutions or rather focus on technical progress. This relates to the topic of whether to have a designated ‘Analytics Translator’. More relevant in our case, I think, would be case studies focused on identify data sources, scoping PoCs, and choosing the proper type of model to start with. Also, giving the applicant a scenario where a project was failing and asking them for suggestions as to why they think it failed (I have an exercise like this in one of my training modules).
**Maja**
I can’t miss this opportunity to thank you so much David Stephenson for writing _\*Big Data Demystified\*_. It has been a _\*Huge\*_ help for me. I have read it in a day. It is full with guidelines and advices, practical, clear and precise. 😅 And, of course, thank you for being here with us.
**David Stephenson**
Maja Thanks for the positive feedback on Big Data Demystified 🙂 Glad to hear you found it helpful, and impressed you read it so quickly!
**Maja**
You are welcome! Love the book!
**Pavitra Chakravarty**
Hi David Stephenson - as a data scientist in a startup without a team, I have realized that the step between outlining the business logic and translating to development doesn’t get as much time and effort as it needs. For example the first run of my product I was both getting requirements from the CTO, interpreting things the way I understood it and doing the development work and pushing to production. I did not have a QA team, PM, code review team or even an architect to cross-check me. My documentation left a lot to be desired as I was always putting out fires or getting a product developed to run the company’s business. As a result, even with the best of intentions many mistakes were made. While this is not typical, I often wonder if I could have mitigated this in any manner other than absolutely putting my foot down and refusing to work till I had a team. Does one have that luxury in a startup without a very tech-savvy CTO?
**Rui Ramos**
Hi Pavitra, in my perspective the best thing that could be done is an assessment of Risk and Impact and share that with your C-Levels. What kind of disruptions would it cause if the data doesn’t have enough quality, if that finds to be a big issue that you can present the prof on your assessment that things could go side-ways and don’t simply expect for the best. From your description it seems to me that you required a team in order to provide a production ready product/feature and your C-level should support you on that or grant you the required skillets to accomplish alone what they ask.
**David Stephenson**
Hi Pavitra Chakravarty It’s really difficult to be the only data scientist in a company that isn’t used to working with data scientists. It’s also very common. I’ve spoken with many people in similar situations.
**David Stephenson**
I would say, no, you should not have insisted on first getting a team, especially in a startup, where resources are tight and everyone is expected to be flexible.
In situations like this, you need to be extremely agile, which, at it’s core, means working very closely with your stakeholders and getting input and feedback as often as possible.
They in turn should recognize that mistakes will be made.
It may be that you’d feel more comfortable in an established team at this point in your career. That’s also a legitimate career choice.
**David Stephenson**
It’s also quite possible / very likely, that your CTO was simply very stressed and /or didn’t have reasonable expectations of you.
**Pavitra Chakravarty**
Guys thank you all for the great suggestions David Stephenson - so privileged to have your take on this. I didn’t know if your big data book but am buying it now. Looks really great
**Maja**
David Stephenson What is (are) the most challenging business skills for data scientist to learn and adopt?
**David Stephenson**
I think the hardest is learning to understand and relate well with colleagues, especially non-technical colleagues. I devote four chapters to this subject. We can stay in our technical bubbles all through school, but then suddenly need to relate to completely different types of people. Of course these aren’t typically the top-of-mind skills, such as storytelling, managing expectations, finding the best use cases, etc (ch 6-11), but not relating effectively in certain ways can easily shipwreck a data science effort.
**Lalit Pagaria**
David Stephenson Thank you for doing this. My question is bit reverse. Means with regard to business people, how I make DS team understand ground realities? In my past experiences. mostly DS team want to work on challenging problem, which may not have business sense. Hence how a business team help DS to understand it better? What common language they should use?
**David Stephenson**
Lalit Pagaria Also a great question, and very relevant. I know some data scientists who simply don’t want to work on business problems b/c they aren’t interesting enough!
I really encourage data scientists to work closely with the business stakeholders–to participate in scoping meetings, to present their work in regular demo’s, and to maintain regular contact with these business partners (eg. on Slack or MS Teams). I myself work hard to ensure that the data scientists recognize the importance of the business problem and check that their presentations to the business have relevant content (and are not too technical).
In my experience, not enough effort is put into developing the WHY of a project (I’ve called it the WHAT of storytelling above). When this is done right, it should provide a good basis to motivate the data scientists. Oh, and make sure you don’t hire people who truly want ONLY technically-interesting work and have no interest in delivering business value.
**Lalit Pagaria**
Thank you, great answer.
**Alexander Seifert**
Hi David Stephenson, thanks for being here with us! My question would be the following: With the pandemic accelerating the adoption of remote work, how does this impact some of the skills you identify in your book? I imagine the skills’ relative importance changes a bit with such a grave shift in how teams are organized. Beyond the obvious impact on the importance of clear communication, is there anything else you want to highlight?
**David Stephenson**
The interpersonal skills are a bit more challenging.
I’ve found it more difficult to read people and to forge new relationships when not working together in the same office. It’s also more difficult to follow up on people who don’t respond to emails.
Cultural differences can also be amplified (I think I give the example of cc’ing managers in chapter 3, which is a bigger deal when communication becomes more electronic).
I’m definitely looking forward to face2face work again!
**Doink**
if data scientists develop better business skills do we need a layer of middle management which could be let go for cost-cutting?
How effective are guesstimates in real world settings ?
**David Stephenson**
I wouldn’t say there is an entire layer, or even role, that would be eliminated, but it would make everyone’s lives easier. Consider the product owner role. Can / could data scientists serve as their own product owners? I would say Yes, if they have the appropriate business skills, but that’s perhaps not the best use of their time. Regarding middle management, again No, as its function is broader than simply providing business skills.
**David Stephenson**
Really appreciate all the good questions so far.
If I can make one small suggestion, I’d encourage people to first download the free kindle sample and take 3 minutes to read the book’s introduction. This will give you a better idea of what topics I wrote about, and will help in asking further questions :)
**Alexey Grigorev**
Can you share the link? I will also add it to the book’s page on our website
**David Stephenson**
Here are the links, depending on which kindle store you are using. Note that you can also use the kindle cloud reader on your PC: [US Store](https://www.amazon.com/dp/B08TY93PMC)
, [UK Store](https://www.amazon.co.uk/dp/B08TY93PMC)
, [DE Store](https://www.amazon.de/dp/B08TY93PMC)
**Rui Ramos**
One question David Stephenson is the book only available on the kindle store ? I have a kobo reader trying to understand if i will bump into issues
**David Stephenson**
Ah, that’s indeed a good question. Let me check into that. BTW, you know you can read kindle books on a phone or computer app, as well as online, right?
**Rui Ramos**
Yes, i know. But i’m trying to reduce the amount of hours at the computer and the ebook readers are a good option for me. I’m finishing some other books, and yours on the radar, thanks for sharing with community by the way.
**David Stephenson**
Rui Ramos Can you check the kobo store again now. It should be there either now or within a few hours. Not sure if kobo displays color, but you’ll want color for chapter 7.
**Rui Ramos**
Thanks David Stephenson just bought it, was checking the cap 7, i have a gray scale on the charts so i think i will manage. Many thanks 🙌
**Tim Becker**
Hi David Stephenson, thank you for being available and answering questions. I would like to ask you a few 🙂
**Tim Becker**
* I just changed my employer. Could you give me advice on how to make a good impression during the initial phase and which pitfalls should I avoid at the beginning.
**Tim Becker**
* Recently, I worked on a project that was very important for my employer. There were many stakeholders involved that did not know programming. Everyone expected results yesterday. How would you manage expectations in such a situation? Especially, if you know that building a good product and bringing it into production will still take months.
**Tim Becker**
* Do you have some advice on how to best boost your career in DS?
**Tim Becker**
* Do you have any suggestions on how to best work with stakeholders that are afraid of being replaced by AI solutions?
**David Stephenson**
Hi Tim Becker, congrats on the new job!
*
Here are a few tips
1. Identify what’s currently important to your organization.
2. Actively grow your internal network and identify the colleagues most open to working with you.
3. Work to produce value quickly, even in small ways.
(I talk about this more in chapter 1)
*
Managing expectations is tricky, but really important. It’s really important to have both a stakeholder scoping meeting and a project kickoff at the start, and during those times you need to not only understand requirements but also manage expectations, which includes making clear both the estimated effort and the amount of inherent uncertainly involved. I talk about these topics in the second half of the book.
*
Regarding career, it depends on where you want to go and what’s important to you. I’ve devoted an entire chapter to this one. If you can ask something slightly more specific, maybe I can answer that here 🙂
* .
AI solutions are often not 0-1 replacements, but are tools to make tasks easier. Either way, you’re typically going to be working through a senior manager, and they will manage this tension. From your side, you want to communicate well with the subject matter experts (who may feel threatened) so that you understand their pain points and can work in the initial stages to develop a solution that makes their life better.
**Tim Becker**
David Stephenson thank you for your advice and answers! Yesterday, I read the chapter that is available in the kindle store and in my opinion it is a really fun and interesting.
**Tim Becker**
Concerning the career, at some point I would like to take more responsibility and lead and guide other data scientists. In addition, help the management to decide which project are particularly promising and to develop concept on how to best approach these.
**David Stephenson**
Tim Becker, glad to hear you enjoyed the chapter 🙂
Regarding your career goals, I would say focus on three things:
1. Get to know stakeholders outside of your team and understand what their work looks like. This will help you deliver meaningful projects.
2. Demonstrate a high degree of responsibility, taking initiative and ownership whenever possible. This will help people trust you with additional responsibility.
3. Develop good communication skills and take the opportunity to present outside of your team when possible. Good communication is critical for leadership and influence.
**Tim Becker**
that was fast, thanks again 🙂
**Luis**
Hello David Stephenson. Thank you, I am a project manager in a large aircraft company and one of my main problems is communication between data analytics and business people. What would be your advice on how to generate interest in the business about the possibilities of data?
**David Stephenson**
Hi Luis, this is a very common challenge. Fortunately it’s getting somewhat easier these days, as traditional companies become eager to not miss the boat in this area (and supervisory boards even place pressure on CEOs).
What I sometimes do as an external consultant is to set up data strategy workshops to quick-start projects, but that assumes a certain level of buy-in being there already.
As an internal, I would recommend that you
1. Identify people in the company who are interested in trying out new solutions. Sometimes this is top-down, from executives who latch on to buzz words, and sometimes it’s bottom-up, when practitioners are eager to bring in new approaches.
2. Identify top-of-mind business goals and build a strong case for how data & analytics can meet those goals. This could be in solving a pain point or in making a substantial improvement and might generally fall into the business goals of increased profit, increased market share, decreased cost, or decreased risk. First you’ll need to understand the currents flowing in your company, else you’ll be selling a solution that is swimming upstream.
I might be able to give you more specific advice if you have a few minutes to describe your situation. If you send me a DM, we can continue from there.
**Laia**
Hi David Stephenson!
Thank you for the link to your book preview. I’ve enjoyed what I’ve read! The book is well-written, easy to read, and straight to the point. Congratulations!
No questions per se, just wondering if you have any anecdotes related to how success was measured in projects you have worked on. Dank je!
**David Stephenson**
Thanks, Laia, for the positive feedback 🙂
How success was measured…. that is indeed a challenging one to answer. The ‘obvious’ answer is through primary and secondary KPIs. Depending on the project, these may have been return on advertising spend, forecasting accuracy, increased revenue, higher CTR, lower churn rate, etc.
In reality, I’ve seen many projects where KPIs were not clearly defined (OKRs are even worse!) One thing I really encourage is to push stakeholders to put careful thought into KPIs right at the start, otherwise you find yourself half way through the project but with no clear north star.
But to be blatantly honest, success is typically measured by how happy your stakeholder is at the end of your project, and that doesn’t always depend on how you’ve moved the KPIs.
**Alex**
Hi David Stephenson! First of all, thanks a lot for taking the time to reply to all these questions.
As a business major, I feel like much of the stuff covered in the book might sound very familiar to me. Do you think your book might solely be aimed to Data Analysts/Scientists coming from a STEM background?
Thanks again, and have a wonderful day!
**David Stephenson**
Hi Alex, that’s a fair question. TBH, I expect 80+% of the book is material you didn’t cover in business school, and the other 20% is presented with the additional perspective of an analytic role. The book is very heavily based on experiences of myself and others I’ve known, put in the light of research and industry literature and livened up a bit with personal anecdotes.
That being said, the book is definitely targeted at analytic roles, which typically are filled from STEM backgrounds, but I’ve heard from several non-technical people that the content is very valuable to them regardless. For example, the chapters on scoping projects and managing stakeholders are pretty broadly applicable, as are the chapters on working with colleagues.
**Alex**
Thanks a lot for the reply, David! Will give the book a shot 😄
**Matthew Emerick**
Hey, David Stephenson. Thanks for doing this.
Why do you think that do many analytical types have difficulties with business skills?
**Matthew Emerick**
How would you address the business skills gap from the outset of one’s education (besides reading your book)?
**David Stephenson**
Ok, this is a bit of a dangerous one to answer!
First, analysts tend to process quantitative information faster than their colleagues, which makes it challenging for them to communicate effectively.
Second, the mental precision that enables success in the analytics field can be detractors in the business realm. For example, we find it difficult to cope with ambiguous situations.
Third, analytic work by nature involves less social interaction, which is a big minus when it comes to understanding your colleagues. This hits especially hard in a business environment, where people in marketing, finance and strategy are so very different in their priorities and ways of working (see chapter 2)
What would help in education is to give students more exposure to experienced professionals, ideally including non-analysts but realistically at least analysts with strong business experience.
**Quynh Le**
Hi David Stephenson, I am applying for data analyst position and find your book very informative! As a newbie, I have a basic question about how data analysts could show understanding about the company when working. I think data analysts’ responsibilities mainly involve providing the stakeholders with the data evidence (such as numbers, charts) of what is happening in the company. How could I show more business understanding more than just showing the numbers? What else would business stakeholders like data analysts to discuss more?
**David Stephenson**
Hi Quynh Le, it’s really good you’re asking this question. A common complaint from senior managers is when analysts only provide charts and data but don’t act as ‘thought partners’.
First, it’s important to understand the ‘question behind the question’ When someone asks for data or a chart, try to understand the business motivation.
As you prepare your presentation, try to imagine what follow-up questions your audience might have, and prepare for those questions.
Also, be prepared with recommendations. It’s possible that your audience will not ask your opinion (or even want your opinion!), but it’s good if you have thought through in advance what you think the best course of action would be, based on your analysis.
In addition to doing this for your own projects, take the initiative to chat with colleagues in other teams to hear what their challenges are. This will not only help you identify new applications but will also give you a more wholistic view of the company, which will go a long way in helping you propose solutions that are most relevant for your company,
**Quynh Le**
David Stephenson thank you for sharing your opinions! I find myself often caught up with the analysis and forget about what else next. From your answer, I think that an analyst really needs to think deeply about what data could means to business strategy and to anticipate what stakeholders really ask more from the analysis. I also find your point about chatting with other colleagues really helpful. Thanks for your sharing!
**Michael Kachala**
Hi David Stephenson! Thanks for the opportunity to ask the question. I have rather simple one: how data science can contribute to new business models?
**David Stephenson**
Hi Michael Kachala, that’s an extremely difficult question. DS is often an enabler, rather than a core part of strategy. Not sure if I would succeed in giving a good, concise answer in this thread.
**Michael Kachala**
Thanks for the reply! I meant simple in formulating, but answer indeed can be puzzling. I was thinking on this topic myself and don’t have a definitive answer.
**Tony Gunawan**
Hi David Stephenson. Thanks for giving a chance to ask on this thread. I just checked your book through the links that you have shared before but it seems I can’t see the preview of the book nor I can buy the kindle version (Maybe because I am living in Indonesia right now even though I set my account to US. It says “The Kindle title is not currently available for purchase).
I just want to ask this question, I don’t know whether you covered this topic on this book but sorry if it’s already covered on your book or out of topic: Is it normal for data roles (data scientist, data analyst, data engineer) to get an expectation from other colleagues to be an expertise from end to end process from data analysis, data preprocessing, and deployment the engine to production? I am a data engineer in a startup company and somehow I got some expectations from other colleagues or C levels to know and expert in end to end process and I don’t know if there is some clear enough responsibilities for each data roles and how to deal with them that have such expectations.
**David Stephenson**
Hi Tony Gunawan, I’m afraid I can’t answer your question about kindle availability. I see it available now, but I’m not in Indonesia 🙂 Curious if anyone else on this channel can help?
Indeed, someone asked a similar question yesterday about end-to-end delivery in a startup. It’s generally not an efficient use of resources within a company, but in a startup everyone needs to stretch themselves way past their comfort zones. It’s just the nature of the job choice. If you want to be more focused, it’s probably best to find a company with a developed program and larger teams.
**Tony Gunawan**
So it seems the issue is not in the colleagues per se but on the nature of the startup company itself, isn’t it? It’s encouraging enough (somehow) to know that it is a common problem within the startup culture and not in overall companies. Thank you so much for your reply. Have a great day !
**Víctor Villacorta Elliott**
David Stephenson, hi!, thanks for the opportunity to ask you, I am interested in knowing more about ethics for Data Scientists, I don’t know if you mention it in your book “Business Skills for Data Scientists”, can you comment something about it?. Thanks.
**David Stephenson**
I don’t really write about that topic in this book. My general principle for companies is to align their interests with those of their customers and to not do anything which they would be embarrassed about if it were made public.
**Kyle Shannon**
❓ Hey David Stephenson, have you had feedback from business users reading your book to better understand how to work with data scientists? if so, anything to share?
**David Stephenson**
Hi Kyle Shannon, not specific feedback, but there is a lot of overlap between the target audiences. That is, there are two types of corporate trainings I generally give. The first covers business skills for data scientists. The second covers skills for Analytics Translators, who are, roughly speaking, the non-techies who work closely with data scientists. Some of the chapters in the book are used in both trainings (specifically, chapters 5-11). You might check out the chart at the bottom of [this page](https://dsianalytics.com/data-science-trainings/)
What’s different in the Analytics Translator training is that I also talk about the basics of Statistics / Machine learning, to help business colleagues put everything into perspective and to help them manage some of the pitfalls inherent in DS work.
A third target group would be business executives, for whom I host workshops rather than give trainings. In this case, I focus on the big picture of applied analytics, how to bring value to a company, how to choose technologies, hire teams, form a roadmap, etc. That is really more the subject of my previous book (“Big Data Demystified, How to … “)
**Oleg Polivin**
Hi David Stephenson ! Is communication an objective in itself or is it a means (one of many) of getting a result, getting things done?
**David Stephenson**
yes
**Eimhear Rainey**
Hi David Stephenson I’m a Data Analyst in the public sector (Healthcare) - are the skills and strategies outlined in your book equally as applicable to public sector analysts as those within businesses in the private sector? Can you comment on any specific challenges or barriers public sector analysts face in propagating data driven worflows and how they may overcome these?
**David Stephenson**
Hi Eimhear Rainey, Most of the skills in the book will still be relevant, as they are foundational, but there will also undoubtedly be unique aspects of change management that you’ll have to deal with. It’s hard for me to speak to those nuances, as I have little experience working in the public sector. What the book should still do, however, is to give you the basis for discovering the best way of working in your own surroundings.
If your organization is using a highly structured project management framework, such as Prince2, you may have difficulty using some of the techniques I write about in chapters 8 and 9 (scoping projects and planning kickoff meetings), as well as in ch 11 (agile, kanban, etc,) If that’s the case, the principles in those chapters will still be valuable, but you’d need to adjust how they are used.
**Duverger PETGA**
Hi David Stephenson In a typical data science process, it is business problem that guide the data collection. But many courses and tutorials focus in data collection, data exploration, build model,…etc. I assume that your book want add this important step in the debate. One question now. The title of book relate to “business skill”. It is just problem definition or other skills ? Thank.
**David Stephenson**
Hi Duverger PETGA, the book covers much more than problem definition. I would encourage you to download the free kindle sample and read through the introduction. It will explain the six different topics covered.
**Sepp**
Hi David Stephenson Thanks for providing us with the chance of asking you some questions. I am a spatial data analyst, who recently became the lead of the newly formed data analysis team at my company, consisting 20 colleagues. It is the first time I am in this position and the main reason was that I “have a talend to function as a communitcation bridge between the stakeholders and the team, can prioritize and define tasks efficiently, have the necessary skillset and have a constant open attitude” (my CTO). While I am flattered with these compliments and the opportunity I got I also feel the urge to not leave it like this, but to actually learn more about the communication process between data and business people. This, together with my current private project of creating a small data-driven local firm, is the reason why I would love to know how to approach stakeholders for the first time.
A use-case at my work is that I am often invited to a meeting where some stackholder of any kind (C-level, client, developer, PM etc.) talk to me the first time and expect things that are just totally out of scope for the given time, asking me things that are totally not data-related or think that we are just “some Excel guys”. I am wondering if there are strategies to actively create some awareness in the company / to individuals about what data analysis team actually do. Perhaps you have an example that I could use :)
Regards from rural Germany
**David Stephenson**
Hi Sepp, first, congratulations on the promotion! Sounds like some great feedback.
I’m seeing several questions in this paragraph. For the first, I have a method I typically use in initial scoping meetings. It takes 1-2 hours to run through, but it’s worth it. I describe it in detail in chapter 8, but if you don’t have a copy of the book, here is a link to at least show your the canvas for it: [http://dsianalytics.com/4](http://dsianalytics.com/4)
Your second question is a bit more difficult to answer, as it depends on whether you already have a place at the strategic table or not. If not, see chapter 1 in the book (available in the free kindle sample). If so, it’s more in chapter 10.
It’s difficult to really answer this w/o more context. Send me a DM and maybe we can set up a short call.
**Maja**
Hello again David Stephenson Empathy, understanding, communication are one of the crucial skills for delivering business value in companies. Are there any methods, ways, or practices to improve them quickly? From talking with my colleagues I’m seeing that to be one of the reasons why some companies are failing in delivering good products.
**David Stephenson**
Maja, It’s largely a mindset, from which you develop the skills. In chapter two of the book, I walk through examples of how colleagues differ with respect to Goals, Values and Working Environments, to help get you started with that mindset.
A lot of companies are sending people to trainings in something called Design Thinking (DT), a customer-focused technique for designing relevant products. This might be helpful. I use principles from DT in my strategy workshops, but I’ve never attended one of these specific DT trainings.
Of course I’m biased, but I’d suggest the best way to quick-start this skill for data scientists is by reading my book or attending one of [my trainings](https://dsianalytics.com/data-science-soft-skills-training/)
. I have a few trainings scheduled for this fall. Which reminds me, I should probably mention that in the main channel.
**Maja**
Thank you so much for everything David Stephenson! I’ll tell them about your trainings. Have a great rest of the week!
**David Stephenson**
Message for the channel,
I’d forgotten to mention that I also offer some open enrollment trainings (most of my trainings are in-house). One is a virtual training; others are on location within Europe. If there’s enough interest, I’d consider adding another physical location.
Here is [the link](https://dsianalytics.com/data-science-soft-skills-training/)
**Krzysztof Ograbek**
Hi David Stephenson, I read most of your answers on this channel. Thank you so much for every single one of them. For a Data Scientist who was focused only on the technical part throughout his career: How much more valuable is he becoming once he begins to learn the business part? What may surprise him the most while learning?
**David Stephenson**
Hi Krzysztof Ograbek,
I would say that the business skills make data scientists much more valuable. Without them, you’re dependent on your manager or colleagues for communication, prioritization and stakeholder management, which really puts a roadblock on your career.
A few surprises in store:
* Your non-technical audience probably understands about 5% of your presentations
* Many of your colleagues have completely different definitions of ‘work well done’
* Your network is more important than your knowledge when it comes to advancing your career
In terms of biggest misconception, I would say that it’s very difficult for non-techies to understand the effort required and the risk entailed by DS projects. This is why stakeholder management skills are so critical.
**Toxicafunk**
This is why I said “the who” was the hardest. Many times have I found myself thinking I’ve successfully presented something available for non-techies only to find out I grossly overestimated the minimum tech knowledge required to understand my presentation… 😅
**Krzysztof Ograbek**
Question 2: What is the biggest misconception that non-technical stakeholders have about Data Science job?
**Asmita**
Hello David Stephenson, Can having domain knowledge be one of the business skills? How important should domain knowledge be considered? Should we gain majority of domain knowledge first or start with a satisfactory amount of knowledge and gain it during the execution?
**David Stephenson**
Hi Asmita, yes, DEFINITELY build domain knowledge. Try to use the company’s product, sign up to your own mailing lists, buy your own products online, etc. When you work on a project with stakeholders, ask them questions about the project background, default solutions, how competitors do things differently, etc. You’ll build the domain knowledge while also earning the trust of your stakeholders, as they notice that you are listening. It will also help a lot in designing your solutions.
On that subject, your first ML solutions should be designed in light of current (manual) best practices. They should probably use similar intuition and produce comparable results. I’ve seen analysts deliver complicated models that couldn’t compare with manual methods, and they lose credibility quickly this way.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Software Mistakes and Tradeoffs – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Software Mistakes and Tradeoffs
-------------------------------
#### by [Tomasz Lelek](https://datatalks.club/people/tomaszlelek.html)
, [Jon Skeet](https://datatalks.club/people/jonskeet.html)
##### The book of the week from 06 Sep 2021 to 10 Sep 2021
In Software Mistakes and Tradeoffs you’ll learn from costly mistakes that Tomasz Lelek and Jon Skeet have encountered over their impressive careers. You’ll explore real-world scenarios where poor understanding of tradeoffs lead to major problems down the road, to help you make better design decisions. Plus, with a little practice, you’ll be able to avoid the pitfalls that trip up even the most experienced developers.
* [Book's page](https://www.manning.com/books/software-mistakes-and-tradeoffs)
Questions and Answers
---------------------
**Rui Ramos**
Hi Tomasz Lelek thanks for joining the group 🙂 . Does the book focus on any specific software development language or is it agnostic to it ?
**Rui Ramos**
What is the main audience for the book ?
**Tomasz Lelek**
The book is written in a way that a lot of people involved in the software engineering industry may benefit from:
* Software engineers that have some experience but want to progress to the next level. They may benefit mostly from the trade-offs at the lower level (code)
* Experience software engineers that will be able to relate to mistakes and trade-offs shown in this book.
* System Architects that will benefit from the holistic examples of variety of system, and trade-offs at the architecture level.
**Tomasz Lelek**
The concepts covered in this book are language agnostic. However, to create a book that is not purely theoretical and shows the practical aspects of software development, the code examples are made in a Java language (there is also a couple of C# examples). We wanted to follow the “show Don’t tell” principle. We picked the Java language as it is widely adapted and relatively easy to understand.
**Kshitiz**
Hi Tomasz Lelek and Jon skeet. Thanks for doing this. I was wondering in your experience what are the top mistakes developers/engineers make in this industry? In case if you can elaborate on it from data science/ML perspective, that will be great.
**Tomasz Lelek**
I can point you to a couple of common mistakes in Big Data processing (that you may relate to ML as well):
* Not leveraging data-locality that result in data shuffling (via slow network). It may substantially impact the time your models need to be calculated.
* Data partitioning not optimized for your read pattern. If you are building a data science model, there are big chances that you need to load a lot of data often. If the data you read is not optimized for the read pattern (the way you are analyzing data), it will impact the overall pipeline efficiency (more on that in the chapter 8 of our book)
* Picking a proper format for storing offline data and understanding its trade-offs (Avro, parquet, protobuf)
* Finding the outliers and filtering them out if they may impact the result of your ML processing.
* Dealing with duplicates in your data. Suppose the data pipeline that is producing the data for your ML works in at-least-once semantics (that is very probable). In that case, there is a high possibility of duplicates - you need to handle them (filter out) in your processing.
**Rui Ramos**
Tomasz Lelek can you share the platforms where we can obtain the book as soon as it is available ?
**Tomasz Lelek**
It is already available, please see:
[https://www.manning.com/books/software-mistakes-and-tradeoffs](https://www.manning.com/books/software-mistakes-and-tradeoffs)
It’s in all well-known formats (ePub,mobi,PDF) and on paper.
**Alexey Grigorev**
What are the most common mistakes that software engineers make?
**Tomasz Lelek**
This question deserved a book; that’s why we wrote one 🙂
But to summarize:
* Blindly following some software development principles (e.g., DRY) without realizing it also has drawbacks and costs.
* Handling errors and recovery is a tricky topic; it is easy to make mistakes at this level.
* Overengineering - that is designing components to be super flexible, without considering the problems with this approach: maintenance overhead, guarding against unpredictable usages, and many more
* Doing performance optimizations based on false assumptions. Optimizing code paths that do not impact the overall performance of our applications (optimizing not the hot-path)
* There is a lot of mistakes that are easy to make regarding work with Date and Time; we have a dedicated chapter on that topic.
* In Big Data processing, not leveraging data-locality that result in data shuffling (via slow network).
* Data partitioning not optimized for your write/read patterns.
* Treating 3rd party libraries that we use as something that we don’t take responsibility for. If the 3rd party library has a bug, or we are misusing it, and it will impact our users, it’s our problem.
* Not being aware of the consistency and availability of the systems we use (DBs, queues, etc.)
* Not understanding delivery semantics of the systems that we integrate with; handling duplication of events/messages in a proper way.
* Treating data/APIs compatibility as an afterthought
* Trying to use technology because it is trendy.
**Krzysztof Ograbek**
Tomasz Lelek _Blindly following some software development principles (e.g., DRY) without realizing it also has drawbacks and costs. -_ could you please elaborate on this one? What are the drawbacks? What do you mean by “blindly following”? I mean DRY in particular.
**Tomasz Lelek**
Krzysztof Ograbek
In the era of micro-services, you can sometimes find the duplication of some code between separate codebases. When the code is duplicated, it means that it can evolve independently. The fact that the code is duplicated now does not mean that it represents exactly the same functionality. If we strictly follow DRY, we should create a library or separate services extracting the duplicated code. However, once it is shared between separate codebases, it introduces tight-coupling and may slow down development. It limits the way the code can evolve now. It is no longer independent. This topic deserves more discussion, and in fact, chapter 2 of our book focuses exclusively on this.
**Lalit Pagaria**
How to avoid over-engineering and pre-mature optimization?
I always feel like writing generic code but ends up making it over engineered.
**Tomasz Lelek**
Regarding premature optimization, I am trying to answer this question in chapter 5 of my book.
The critical parts for avoiding it:
* Base your performance data on the SLA and/or empirical data.
* Both data about throughput (req/s) and the expected latency (avg, higher percentiles) are needed.
* Calculate the relevance of the paths in your code using latency and the number of requests
* Having this information, we can calculate the importance of each code path in our code. It may turn out that the small percentage of code is responsible for majority of business value and user traffic (according to Pareto Principle it is often a case)
* We should measure everything after and before changes. Performance tests that validate the system holistically are needed. Also, low-level microbenchmarks will allow us to experiment with different optimizations faster, decreasing the feedback loop.
* Once we detected this hot path in our code, we can focus optimization efforts on the small excerpt of our code and be more efficient with optimizations.
**Tomasz Lelek**
Regarding over-engineering problem, there is no silver bullet. Often, we should not try to come up with a generic solution up-front, quote from chapter 2:
`Sometimes starting from the abstraction and adapting all possible usages to it may not be optimal. Instead, we can implement our system by creating independent components and let them live independently for some time (even if it requires some code duplication). After some time, we may start seeing some common patterns between those components, and abstraction may emerge. It may be a proper time to remove duplication by creating some abstraction instead of starting from it.`
**Krzysztof Ograbek**
Hi Tomasz Lelek, thank you for doing this session 🙂
What are the most common mistakes/misconceptions that software engineers have about Version Control Systems?
**Tomasz Lelek**
I was using only GIT VCS, so I don’t have a good comparison between them; therefore I won’t give the answer here
**Krzysztof Ograbek**
Currently, I am reading Pragmatic Programmer. How is your book different? How can a reader benefit from reading both? If you haven’t read the book, please ignore the question.
**Tomasz Lelek**
Yes, Pragmatic Programmer is a great book - I strongly recommend it. Both books focus on a variety of aspects of how to create better software. My book is more hands-on and practical - it has more examples of recent technologies, and trade-offs are discussed based on those technologies.
**Krzysztof Ograbek**
Glad to see you recommending this book. Thank you, Tomasz Lelek
**Tim Becker**
Hi Tomasz Lelek, thank you for answering our questions. I was really looking forward to ask questions concerning your book. I am a data scientist, however, I spend most of my time writing code and I feel it is quite difficult to anticipate all consequences of my design decisions. I would really like to improve in this area.
**Tomasz Lelek**
I think that the best way to increase awareness is through learning. The first thing that I would read is a book about possible patterns and abstractions in code. I would recommend this one: [https://www.amazon.com/Design-Patterns-Object-Oriented-Addison-Wesley-Professional-ebook/dp/B000SEIBB8](https://www.amazon.com/Design-Patterns-Object-Oriented-Addison-Wesley-Professional-ebook/dp/B000SEIBB8)
. Once your team knows the tools available, it can start applying them in the code.
Knowing the patterns is often not enough because you need the experience when to apply them. This experience can be learned by experimenting in code - to allow safe experiments and refactoring, you should, of course, use a version control system and branches. The second must-have is a “good” test coverage of your code. Without any tests, the fear that change will break something will be too high, and nobody will risk non-trivial refactoring. At this stage, pair programming may be very valuable - cooperation on the same code may result in very interesting ideas and improvements.
You may start refactoring once you have tests, code in version control, and a dedicated branch. The new changes should be submitted as Pull Requests to allow all the team members to review, learn and propose improvements. This will enforce a collaborative effort on the code improvements.
Regarding code duplication, the DRY principle is a good direction for most of the use cases. However, there are some specific scenarios when reducing code duplication and adding abstraction too early may lead to tight-coupling of now necessary related code paths. It will reduce the flexibility of your code and speed of delivery. This is a complex topic, and I encourage you to read the second chapter of our book to learn more.
**Tim Becker**
thank you! I will definitely try this!
**Tim Becker**
* Do you have advice for some general guidelines to get started with that will help to improve the quality of my code?
**Tim Becker**
* I would like to increase the awareness in my data science team that software design is very important. To be flexible, keep the code and models maintainable and many more things. How would you try to convince my team?
**Tim Becker**
* When refactoring a software package that has been writing without former planning and without a specific design in mind, where would you start with the refactoring? How would you approach this task?
**Tim Becker**
* What can be the advantages of code duplication? I recently saw a lot of it and it is creating a mess …
**Krzysztof Ograbek**
Tomasz Lelek, you already recommended Pragmatic Programmer and Design Patterns books. And of course, yours 🙂 Do you have more recommendations? What about Clean Code? Amazon recommend it along with Design patterns 🙂
**Krzysztof Ograbek**
I’ve seen this pro tip for Software Engineers to learn one new programming language each year. What’s your opinion on this? How is it beneficial? Out of curiosity: How many languages do you know? How many do you actually use?
**Krzysztof Ograbek**
One more 🙂 As programmers we get stuck quite often. Do you have any routine, that helps you to move forward when you’re stuck?
**Tomasz Lelek**
Discussion with another person about the problem will allow you to see a different perspective. Also, writing down your ideas will help you organize your thoughts and allow you to come up with a better solution to your problem.
**Tomasz Lelek**
Clean code is also a good book at the beginning of the programming journey.
I think that a programming language is only a tool. It is crucial to learn new things, but we should focus more on understanding our systems than learning a new language every year. I strongly recommend:
[https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321)
**Larisa Biriescu**
Hi Tomasz Lelek. I don’t know if you treat this subject into your book but how do you deal with the more “human” errors that may appear while working on a project? Errors that derive from poor understanding of the customer’s needs, unrealistic requirements (or omission of some of them)? I find it more tricky to deal with people rather than with code 😅.
**Tomasz Lelek**
Of course, that’s a complex topic and deserves a separate discussion (or book 🙂 In my context, I find it useful to have a decision log (document with pros/cons of a decision). In such a document, everyone can contribute and understand the requirements and possible solutions: all its pros and cons. Everyone can take part in the discussion and vote for a specific solution. It may help with those errors, plus everything is explicit, meaning that requirements are discussed in more details.
**Kshitiz**
Tomasz Lelek - I understand that it is not in context of the book, but this maintaining a decision log seems like an interesting approach. Can you explain it a bit how do you do it? Like which tools do you use, what type of decisions are considered for this? It sounds like an exhausting exercise but it can build an excellent mind map for a team if executed well over a long time.
**Tomasz Lelek**
The simpler the tool, the better. The tool used for this needs to allow full collaboration. I found Google Docs useful for it (or wiki page). Each decision can have a dedicated page/document. The person responsible for research can describe possibilities with pros and cons. Next, every team member refers to those propositions and vote (or propose a new one).
**Krzysztof Ograbek**
Thank you Tomasz Lelek for your great answers. Thank you Alexey Grigorev for having this channel 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# DataOps for Dummies – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
DataOps for Dummies
-------------------
#### by Justin Mullen, [Guy Adams](https://datatalks.club/people/guyadams.html)
##### The book of the week from 13 Sep 2021 to 17 Sep 2021
DataOps describes a novel way of development teams working together collaboratively around data to achieve rapid results and improve customer satisfaction. This book is intended for everyone looking to adopt the DataOps philosophy to improve the governance and agility of their data products. The principles in this book should create a shared understanding of the goals and methods of DataOps and `#TrueDataOps` and create a starting point for collaboration.
* [Book's website](https://www.dataops.live/dataops-for-dummies-download)
Questions and Answers
---------------------
**Guy Adams**
Thank you Alexey Grigorev! I’m around all week to answer any questions about DataOps, bringing Agile principles, automation and CI/CD to Data Warehousing - fire away!
**Matthew Emerick**
Hey, Guy Adams! Thanks for doing this.
How important is it to establish DataOps from the beginning of a new product?
**Guy Adams**
Matthew Emerick - great question! Most of the people I work with already have something in place and want to retrofit DataOps to their current ways of working. However, much like DevOps and CI/CD for software, it’s always easier to “start as you mean to go on” and build DataOps in from day 0.
**Matthew Emerick**
What do you think is the greatest obstacle in establishing DataOps in an older, larger company?
**Guy Adams**
If you’d have asked me maybe 2 years ago I would have said the biggest obstacle would be the die hard data veterans - people with 30 years of experience saying “well that’s not how we do things”. The reality has been quite the opposite - those people are saying “thank goodness, I’ve been waiting for this to come along for years”. In a larger company, maybe the biggest obstacle is how the teams are organized and work is done. Inevitably teams are organized around the way that they work (or are forced to work). When a new, better way appears, it can be hard for larger organizations to adopt this quickly. That said, we are working with some of the largest organizations in the world, and they are 100% committed to this sort of transition, because of the benefits they get as a result.
**Toxicafunk**
Hi Guy Adams, thx for being here. I hear SPC as an important component for DataOps, in particular as introduced by Demming. I wonder if you (or other authors) have studied other Quality COntrol Gurus (Juran, Ishikawa, Crosby) and tried to introduce their ideas to DataOps?
**Guy Adams**
Toxicafunk - thanks for the great question. Actual in the world of DataOps, much like Software DevOps, the Statistical Process Control, while very important is actually very simplistic in it’s requirements. In simple terms if very rarely more complex that “only execute D if and when A, B and C have completed successfully”. Therefore the deeper science of Quality Control is rarely needed. What is important and unique in DataOps is how we determine that “A has completed successfully”. In traditional DevOps or CI/CD this is simply based on a return code from a function/module i.e. the piece of code itself tells you whether it was successful or not e.g. the software compiled without errors or it didn’t. In the Data world things can be a little bit different e.g. a software ingestion job MAY return a successful return code that indicates it had no failures, but does this mean that it has done what we EXPECT. If we are ingesting sales data every day and we know even on the worst day we get over 1000 sales records and the sales ingestion job just ran and loaded 2 rows, while the process may call this a success, we have additional information we can apply to post ingestion testing above this to determine that we didn’t get the minimum we expect and therefore to stop the pipeline. I’m interested to hear what you think about the work of such people applies to data as I’m always interest to hear other perspectives!
**Toxicafunk**
I’m a complete noob on dataops which is why your book interests me, but I’ve heard it described as the intersection of DevOps, lean Mgmt, agile methodologies and data analytics. So I thought maybe Ishikawa’s quality circles would play well with the agile component and following your example of a successful job but with less rows than expected… That sounds like maybe a fish diagram could play some role. Hence my question 😸
**Alexey Grigorev**
What’s dataops? 🤔
**Rosona**
Related: how do you distinguish it from data engineering?
**Guy Adams**
I think I can answer both questions at the same time. DataOps is to Data Engineering as DevOps is to Software Engineering. Software Engineering is the process of actually writing software - lines of code. DevOps is the processes and technologies around how the complete piece of software is built, tested and deployed. DevOps supports Software Engineering and makes Software Engineers far more agile and efficient - they can make some changes and have an entire version of the software (including their changes) built and tested, including automatically building the correct environments etc. DataOps is exactly the same for Data. Data Engineering is still required, but DataOps takes all the manual heavy lifting around building environments, getting test data, doing all the automated testing etc and, assuming everything works as expected, deal with the review, promotion and ultimate deployment into production.
**Tony Gunawan**
Hi, Guy Adams. Thanks for being here. Is it an overkill when I want to design the data warehouse to maintain the enormous data that will happen in the future in the company when the current data is not as big as imagined? For example if the company has only 10k-20k of data as of now and use the Hadoop-like design system with big data in mind to handle those current data?
**Guy Adams**
Firstly I wouldn’t be advocating a Hadoop-like design for data volumes of any size - it’s a fairly dated and very expensive and VERY complex approach compared to something like Snowflake. In terms of DataOps overkill - if done right, a DataOps project can be very simple and cost effective/show a good ROI for a project of any scale. In many ways if you are starting small but knowing you are doing to grow - that’s the perfect time to implement DataOps as you have some time to learn and get everything the way you want it before the data volumes explode. Ask a professional Software Engineer the very first thing they do before they write a single line of functional code in a new project - they setup their DevOps and CI/CD system - no matter how bit of small the project is. Starting this way is always a great investment for the future.
**Tony Gunawan**
Great. Yes, I learned a lot about CI/CD implementation from software engineer like using micro services to handle things so it can be scalable, and interesting point when you said that the perfect time to implement DataOps is when you start from small (aka scratch) environment so you can be more “creative” to design and implement it, Thanks, Guy Adams. Cheers!
**WingCode**
Is it easier for a DevOps professional transition to DataOps or a Data Scientist professional to transition into DataOps?
**Guy Adams**
Great question. In the early days of DevOps, DevOps professionals didn’t exist - they ‘converted’ from a variety of disciplines - Software Engineers, Sys Admins even Project Managers. DataOps is in the same place - the pioneers becoming DataOps champions are coming from a variety of places, but of course some will have a slightly easier time in the transition. The most typical path into a DataOps Engineer is from someone historically a Data Engineer, but with some coding/automation background. However a DevOps professional (who typically has some passable database knowledge) is going to have a pretty easy time of it. The Data Scientist is harder to predict since I have worked with a lot of Data Scientists who have very different skill sets - however I would say it’s a less obvious transition that the others.
**WingCode**
Thank you Guy Adams for the great answer!
**Yash Khandelwal**
_Why we need the DataOps ?_
**Guy Adams**
If you are working in the Data/Analytics space and you enjoy building everything manually, having no automated testing, having no integration between systems, being very slow to respond to business requests and having very little governance - then you don’t. However is you want to have the same Agility, Speed, Automation etc as Software Development has enjoyed for 20+ years then you absolutely want DataOps.
**Tim Becker**
Hi Guy Adams, thank you for answering our questions!
**Tim Becker**
* I saw in your book that you mention good collaboration as crucial. Could you provide some advice on how to best collaborate in data centric teams. I mean how to design a framework that helps with better collaboration.
**Guy Adams**
Tim - collaboration requires three main things - overall philosophy, the right technology and the right team structure. One of this biggest thing people seem to struggle with is that no amount of technology can mitigate the need for people to actually talk and work together. The goal of technology in this situation is to support people working collaboratively rather than to magically create collaboration. In my experience actually, if you take the technical barriers away, in general people are pretty good, and work quite naturally in a collaboratively way. There is clearly a lot about how you plan your work, structure your team etc - if you are running your teams in an Agile way - this naturally creates collaboration. Technically, by far the best approach, and the one we follow within the DataOps.live is system, is to follow how the software world solved this - using git. Git is a phenomenal tool to allow potentially massive teams (thousands of people) to all go off, do their own thing, work in little teams, but still be able to bring work together in a very controlled way. With this in place, and good Agile methodologies, it’s actually pretty easy.
**Tim Becker**
* How much test coverage do you recommend?
**Guy Adams**
Enough!!! This is very much “how long is a piece of string question” - there is no simple answer. The software world has been measuring test coverage for 20+ years and still can’t agree on this. The way I encourage customers to think about this as an ongoing process rather than a one off activity. When you start out, spend some time about thinking about the most like, or the most business impactful failure modes and add tests for these. I recommend, as a rule of thumb, if you have 5 tests for a table, you are probably in the right ball park. However, much more important than the number of tests is the usefulness of the tests. It’s trivial to add a set of simple tests to every column in a table unique, not null etc - and doing this you can easily create 10s or 100s of tests - but are these really telling you much. 1 really smart test can be worth 100 tests created “for the sake of it”. However you get there, you will eventually go live with a set of tests and they will be imperfect, and issues can slip through. It’s a fact, just accept it. By catching the majority of issues that would have got into production you are already well ahead of most people. Business users are actually relatively forgiving of things like this. What they are NOT forgiving of is repeition. If they report an issue and you fix it - no problem. If that problem reoccurs and they have to report it again - now you have lost trust and credibility. This brings me to the second part - every time you fix a data issue that you didn’t catch, you MUST add the tests to prevent this issue from getting to users again and you REALLY SHOULD spent 10 minutes thinking a) could this same issue occur again in other places - if so lets catch that now and b) could this issue occur in a slightly different form, if so, lets catch that now. This becomes an ongoing process if improving your tests over time and I believe is the best and most pragmatic approach.
**Tim Becker**
* How to automate documentation in practice?
**Guy Adams**
We have this built into out DataOps.live platform - since we have all of the logic about how we build, transform, test etc in the repository, and we have access to the target database itself, we have all the information needed to build a really good set of automated documentation. I’ve added a few screenshots for this. My team host a weekly office hours session: [https://www.dataops.live/office-hours](https://www.dataops.live/office-hours)
if you’d like to learn more!
**Tim Becker**
Guy Adams Thank you for answering my questions 🙂 automated documentation seems pretty nice
**JGatz**
Hi Guy Adams, thanks for doing this! Regarding overcoming organisational hurdles - have you ever experienced issues between pre-existing DevOps teams and the new DataOps team that an organisation was trying to introduce? For example ownership, best practices etc.
**Guy Adams**
What I have experienced most is that existing DevOps teams have already regarded/been told “sorry the Data team is special - you have no business here”! Once they actually see an organization start to move towards DataOps and embrace DevOps principles, they are usually very happy. One big warning - if you are starting looking at DataOps, ALWAYS involve your existing DevOps teams - if you don’t, they will see this as a Shadow DevOps initiative and the team that should be your biggest supporters may turn into a barrier. They may well have some corporate requirements you need to adhere to, but these are usually straightforward to adopt. Ownership is usually within the DataOps team, because DataOps is a unique thing, but adopting standard company best practices and standards would come from the DevOps team
**JGatz**
Cool, that sounds good advice. As ever, a lot depends on clear and proactive communication.
\> ALWAYS involve your existing DevOps teams - if you don’t, they will see this as a Shadow DevOps initiative and the team that should be your biggest supporters may turn into a barrier
…well said.
**Timur Kamaliev**
Hi Guy Adams, thanks for your detailed answers!
I noticed that for the past 2-3 years, there has been an explosion of different terms related to the world of IT operations (DataOps, MLOps and more). Do you think this trend will stay in the future? And will the business be interested in this and in specialists from such areas?
**Guy Adams**
Great question. I believe when there is new concepts, there becomes an explosion of concepts, terms (and with a lot over overlap, duplication, contradiction) etc and then overtime some of these coalesce and become a better defined and more standard thing. For example I consider MLOps as a subset of functionality within DataOps. As the tools become easier to use and the areas become better defined, most organizations won’t need separate specialists for say DataOps, MLOps etc - one team will be able to handle all of these. I don’t think the wider business will care about any of this - they just want the Data team to be able to deliver quickly, reliably and with good governance!
**Jeanine Harb**
Hi Guy Adams, thanks for taking our questions!
In my few years of experience as a data engineer, I’ve found that the most complicated aspect about testing data pipelines (and overall dataops) is the fact that data models constantly shift (schema changes, different needs…). It is quite time-consuming to adapt automated tests. What would you recommend to simplify this process?
**Guy Adams**
You are very welcome. You are absolutely right - when everything is shifting rapidly, keeping tests lined up can be hard. I have a few recommendations here:
1. Ensure that your tests are stored in the same git repo as your actual configuration and code so that as you make changes and deploy them - the functional changes and tests are deployed together
2. Ensure that your tests are defined in the same place, right along side your functional logic. If you have the data modelling defined in one place and the tests in a different place it’s virtually impossible to keep them in sync. BTW the same here applies to Grants are permissions - if you define them right along side the functional code is MUCH easier to manage and MUCH harder to make mistakes.
3. Deploy your actual functional changes using an automated declarative approach like the Snowflake Object Lifecycle Engine in the DataOps platform - not writing endless ALTER TABLE statements!
**Jeanine Harb**
Thanks a lot, this is great advice!
**Sri**
Thanks Guy Adams for this book and taking our questions!
Site Reliability Engineering (SRE) is being used by many big organizations for operating large software systems. Similar to `class SRE implements DevOps` (as mentioned in [this blog](https://cloud.google.com/blog/products/gcp/sre-vs-devops-competing-standards-or-close-friends)
), what is your thought on `class DRE implements DataOps` DRE-> Data Reliability Engineering with metrics like SLIs/SLOs and SLAs for all data/pipelines related tasks?
**Guy Adams**
Very forward thinking question! I’ve never really come across the term DRE in the real world _yet_ - but I like it a lot. I think some of this is a maturity thing. DevOps took a while to mature before SRE really became a thing - I think full DRE is a little bit down the line for most organizations. However here is one question I always encourage my customers to think about “What is the % availability of your data” - most of them answer very quickly “the uptime of my data platform is 99.99x” - but I challenge them on this. I didn’t ask what the uptime was, I asked what the availability of your data was - by which I mean, what percentage of the time is the the _correct_ data available to the _right_ business users, _up to date_ and _provably_ correct. Virtually no-one is able to measure this today, but I think this sort of metric would be at the core of DRE.
**Guy Adams**
An add on though - if you search for DRE - you get many more matches for “Database Reliability Engineering” than “Data Reliability Engineering” - I think this is totally missing the point - to me Database Reliability Engineering is focusing on that uptime element. Data Reliability Engineering must have a much broader focus - to get good, valid, provable data in the hands of the users that need it requires many things working on concert - the Database is a critical component, but it is only one part of the puzzle so for me “Data Reliability Engineering” >> “Database Reliability Engineering”
**Sri**
Thanks Guy, for your thoughts.. Yes, I too agree that database reliability engineering is completely missing the broader focus on the dataops.. I like the way you put it.. DB is subset of broader “Data”. Thanks again!
**Guy Adams**
Thanks all for having me on book of the week - there have been some great and thoughtful questions and I’ve really enjoyed it. Feel free to contact me here or at guy.adams@dataops.live if you have any further questions about DataOps and I’d be happy to help!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Python Feature Engineering Cookbook – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Python Feature Engineering Cookbook
-----------------------------------
#### by [Soledad Galli](https://datatalks.club/people/soledadgalli.html)
##### The book of the week from 20 Sep 2021 to 24 Sep 2021
Feature engineering is invaluable for developing and enriching your machine learning models. In this cookbook, you will work with the best tools to streamline your feature engineering pipelines and techniques and simplify and improve the quality of your code.
Using Python libraries such as pandas, scikit-learn, Featuretools, and Feature-engine, you’ll learn how to work with both continuous and discrete datasets and be able to transform features from unstructured datasets. You will develop the skills necessary to select the best features as well as the most suitable extraction techniques. This book will cover Python recipes that will help you automate feature engineering to simplify complex processes. You’ll also get to grips with different feature engineering strategies, such as the box-cox transform, power transform, and log transform across machine learning, reinforcement learning, and natural language processing (NLP) domains.
* [Book's page](https://www.packtpub.com/product/python-feature-engineering-cookbook/9781789806311)
* [Amazon](https://www.amazon.com/-/en/Soledad-Galli/dp/1789806313)
* [Book's GitHub repository](https://github.com/PacktPublishing/Python-Feature-Engineering-Cookbook)
Questions and Answers
---------------------
**Guy Adams**
Soledad Galli - thanks for agreeing to answer questions! How automatable is the process of feature engineering against unknown/new data?
**Soledad Galli**
How automatable: that is the million dollar question. I think there is a lot of effort put into trying to automate feature engineering, with different degrees of success.
**Kshitiz**
Soledad Galli - Thank you for doing this. Looking forward to other questions/responses as well.
I want to understand if there is any systematic approach to engineer features for specific problems?
**Soledad Galli**
I think the most advanced in that direction are the libraries featuretools and tsfresh which include transformers that out-of-the-box return a battery of features extracted based on the most used feature calculations. Tsfresh for example returns around 1200 features and then goes ahead and selects the most relevant ones. Featuretools does more or less the same but returns far fewer features.
**Soledad Galli**
Both libraries are geared to time series though.
**Soledad Galli**
For classical so to say data, i.e., tabular data, I have not come accross libraries like this. Instead other libraries like feature-engine, category encoders and sklearn offer a variety of individual transformers from which you can pick and choose according to the needs for your data. And then, you can line up all trasformers within a scikit-learn pipeline, to run them in sequence.
**Soledad Galli**
The thing is, in my opinion, that some datasets may require some special treatment, and you know best your data than any python library, so at the end of the day, domain knolewdge always help
**Soledad Galli**
I think with that I kind of try and answer the 2 questions, no? I don’t know of a systematic approach other than what I mention here.
**Quynh Le**
Hi Soledad Galli, thanks for writing the book! Could you share with us which feature engineering methods are most popular? Also does feature engineering make it more difficult to interpret the model results?
**Soledad Galli**
Quynh Le for imputation, the most popular ones are mean, median imputation for numerical variables, usually accompanied by adding a missing binary indicator. And for categorical variables replacing NA with the string “missing” or “other”. For categorical encoding, the most common is one hot encoding. For variable transformations I would say, log, box-cox and yeo-johnson, alternatively, discretization. And then for scaling: standardization or scaling to the maximum and minimum values, whenever needed. For time series I would say lagging the features, or window features, where you take usually the mean, max or min from a window of a certain number of steps usually behind the point you want to predict. But this is just the tip of the iceberg, there is a lot more that you can do to the data.
**Nikhil Shrestha**
Hi Soledad Galli
Thanks for writing the book.
Feature engineering has always been one of my favorite area in whole machine learning.
I am posting few of many questions here:
1. Depending on data and it’s distribution we decide upon the feature engineering we can perform. As I also learnt that this can be mastered with practice and experience with data. Could you point some resources to understand this better. Lets say for e.g. we have a non linearly separable data in binary classification problem. data looks like a concentric circle where -ve points are in the inner circle and positive are on the outer circle. How can we transform such feature to use Logistic Regression.
2. I did a project on Walmart where I used columns like date, weekly sales(dependent variable), store\_number, to generate two new ordinal columns which actually helped to improve score a lot. I used mean, addition, if else statement to do this. Another example for the same was formula for calculating social media popularity = (number\_of\_people\_reviewed\_for movies / number\_of people\_voted\_for\_movie) \* no\_of\_faceboooks\_likes. It never made sense to me as how this transformation helps in making model better, but it actually did.
Using this technique made me think does FEATURE ENGINEERING limits. Can we use any mathematical function using various columns and come up with a new column and that will improve the model ? or is there a specific pattern in this technique?
3. There are many articles which says you can even use running average of some variables to create a new one out of it. Does this apply to certain specific cases ? or we can do is more often generally?
I hope I didn’t offend or trouble you with these questions, but I couldn’t find answers to these. So thank you again Soledad Galli for the book and your expertise in this area, also thank you Alexey Grigorev for this platform.🙂
**Soledad Galli**
Nikhil Shrestha 1) it is not always possible to transform data to make it linearly separable. Having said this, there are some techniques that help, like some mathematical transformations, or monotonic encodings for categorical variables, or using decision trees to replace the values of the variables by the predictions of the trees. Probably the latter would work best in the scenario you describe. I have a [course on udemy](https://www.udemy.com/course/feature-engineering-for-machine-learning/?couponCode=DATATALKSCLUB)
on feature engineering where I teach monotonic encoding and how that helps with linear models, you might find that interesting. Andrew Ng in his machine learning course in coursera also discusses somewhere throughout the course something about transforming data to make it linearly separable. And for the encoding with decision trees, this article describe its use: [http://proceedings.mlr.press/v7/niculescu09/niculescu09.pdf](http://proceedings.mlr.press/v7/niculescu09/niculescu09.pdf)
**Soledad Galli**
2) you can certainly use any mathematical transformation to create features. The sky is the limit. In fact, the package ts fresh does exactly that, create hundreds of features and then select the most relevant. Some Kaggle competitors (that won several competitions) also create loads of features and then select the most relevant. The thing I would advice here, is to think how are you going to use the features and the model. If it is for an organisation, that is tightly regulated, and needs to be able to explain the models and the variables to the customers and / or auditors, then I would suggest to create features that “make sense”. If it is for a data science competition where the only thing that matters is to produce a high performing model, then you can do whatever you want.
**Soledad Galli**
Also to add to the above, when we build models for organisations, usually someone within the organsation is going to “consume” that model. For example, if we build fraud detection models, the fraud investigators are going to follow up applications or claims that are fraudulent. So they do need interpretable models and thus, interpretable variables. If the variable is some sort of polynomial combination of price, location, and who knows how many other componenents, they can’t really make sense of it
**Soledad Galli**
3) the running (rolling?) average is very frequently used in forecasting.
**Soledad Galli**
I am not offended or troubled by the questions, that is the whole point of the book of the week, right? 😉
**Nikhil Shrestha**
Soledad Galli wow!!
Now that you explained this it all sounds so simple and intuitive.
The problem I was facing was is that I used to hold back when transforming thinking:
* what if this is only computationally heavy, especially when working with kaggle datasets
* my concepts were not that clear before.
Now that concepts are clear I can directly go apply transformation confidently in competition and business case datasets, bcoz I know this will Surely work.
And yes I meant running average (I made a blunder by writing Rolling) 🤭
Hence thank you very much 🙏
I will surely take the course you suggested. I am sure I will learn a lot from there too.🙌
Keep up the good work 🙂👏
**Doink**
Soledad Galli
1. Classical Feature Engineering vs Deep Learning approach, which to choose when?
2. What are the limitations of feature engineering?
**Soledad Galli**
Doink I think the first question here is probably off-the-shelf traditional machine learning models (which need feature engineering) vs deep learning. Deep learning makes sense only when you have loads of data. If your datasets are small, deep learning does not really outperform the more traditional and thus simpler machine learning models. Andrew ng discusses this in one of the first videos in the ai specialization in coursera, in case you want more details.
**Soledad Galli**
Regarding the limitations of feature engineering, I would say the first one is that it is time consuming. The second one is that to make features that make sense, we usually require some domain knowledge of the topic, for which we tend to work together with the experts, in my case, with the fraud investigators. For time series, also scalability is an important issue.
**WingCode**
Hi Soledad Galli,
What are the features usually extracted from:
1. Image
2. Video
3. Audio
**Wendy Mak**
tbh for all 3 above kinds of data you typically run it through a neural net without really doing ‘feature engineering’ in the traditional sense. You might want to do data augmentation for example especially if you don’t have much data, and for video depending on the problem you might or might not want to analyse individual frames, but the old way of e.g. doing CV by extracting edges and whatnot have been more or less replaced by neural nets
**WingCode**
Thank you Wendy!
**WingCode**
Is it common for a feature engineering to break down a complex feature into a naive feature and then apply feature engineering on the naive feature?
example 1: Video (complex type) broken down into a list of images (naive type). Apply feature engineering on each image
example 2: Audio (complex type) broken down into frequency domain(naive type). Apply feature engineering on each frequencies.
example 3: IP address (complex type) broken down into approx geo location coordinates(naive type) . Apply feature engineering on geo-location
**Soledad Galli**
Hello. Sorry WingCode I am not an expert on the field of Video, Audio and Image, so I can’t comment.
**WingCode**
No problems Soledad Galli 🙂
**Tim Becker**
Hi Soledad Galli, thanks for answering our questions 🙂
**Tim Becker**
* How do you recommend to treat outliers? What do you think about clipping outliers and when is this a good idea?
**Tim Becker**
* How do you deal with feature drift? If I re-train a model and e.g., the average of a feature has changed, would you disregard old data?
**Tim Becker**
* How do you investigate seasonality and what features do you use when you find seasonality?
**Tim Becker**
* If you have timeseries data with a high frequency and you guess that consecutive data points are highly correlated, how do you check this and would you disregard some of the highly correlated data points for training?
**Soledad Galli**
Hello Tim Becker, outliers: you can cap the maximum and minimum values, I think this is what I do most often. Or you can remove them from the training data. But you would probably get outliers in the test, live data and then you need to decide what to do with those if you removed them from the train set. If you cap, then that is handled automatically by the feature engineering pipeline. When to cap or to clip depends on the data, and the model that you are training. Tree based models are robust to outliers, so no need to do much about them. Linear models are, so in that case you may chose to remove them. In some cases outliers are important, for example if you are investigating credit risk and a person has been in court already, it is a bad indicator, yet that happens very rarely, so a positive in that variable is indeed an outlier that you don’t want to remove. You can choose to leave in the model and see what happens or perhaps better to create a rule to treat those cases separately (outside the model). Bottom line, I don’t think there is a single way to treat outliers, it depends on the nature of the data and the outcome we are after, as well as the mathematical models we are using.
**Soledad Galli**
Feature drift: again, no single answer. Depends on the data, and what you are trying to model. I think the first thing would be to understand why the drift happened and if we expect this drift to hold or if we expect it to go back to original values. Some examples: a drift can happen because the organisation changed a policy, e.g., before it only served people of a certain age, now they serve all segments. Naturally there will be a drift, but that drift is here to stay. So I would probably try and incorporate more data that resembles the new population. Another example, during the 2009 depression debt indicators signaled risk of default. Several years after the depression, people acquire a lot of debt (not that I agree with that economic model) and they did not default. So debt indicator was no longer a signal for risk of default. What do you expect to happen in the next 5 years? a crisis, say a pandemic? then you could model using the old data. Stability? then I would discard the old data.
**Soledad Galli**
My examples are a bit easy and binary, but I guess you get the yest :)
**Soledad Galli**
Seasonality inspection, I think visual inspection + domain knowledge is what is done more often. If your features have seasonality you can create new features capturing aspects of that seasonality (regression in time, seasonal dummies, seasonal lag). I have not done that myself, so you need to come back with that questions in a few weeks from now if you would like more details, because we are preparing a new course on the topic :)
**Soledad Galli**
You can check autocorrelation with autocorrelation plots. I am not sure I would disregard data. I think you can apply filters to decrease a bit the variation if it is too much so that the autocorrelation is to high at some points, and also create lag features to capture that correlation and use it to your advantage if you want to forecast.
**Sandhya G**
Soledad Galli What are some suggestions on dealing with large number of features which do not have any physical significance. Few things to think of
1. Use lasso to select features
2. Remove features that have low variance (for floating point data)
3. Remove features that are highly cross correlated
4. PCA
Anything else?
Specifically, this is a chemistry problem, predicting properties using chemical structure of the molecule. There are established packages like RDKit which dump a 1000s of descriptors for each molecule.
Thanks!
**Soledad Galli**
You can remove constant or quasi-constant features, for example with this class of Feature-engine: [https://feature-engine.readthedocs.io/en/latest/selection/DropConstantFeatures.html](https://feature-engine.readthedocs.io/en/latest/selection/DropConstantFeatures.html)
**Soledad Galli**
On the same line, sometimes features are duplicated or we introduce duplication after one hot encoding, which can be removed with this other class: [https://feature-engine.readthedocs.io/en/latest/selection/DropDuplicateFeatures.html](https://feature-engine.readthedocs.io/en/latest/selection/DropDuplicateFeatures.html)
**Soledad Galli**
Similarly to Lasso we can use the importance derived from trees to remove less relevant features, one implementation of this is the boruta algorithm, but it can also be done manually with the SelectFromModel from sklearn for example passing RandomForests or other tree based method
**Soledad Galli**
you can perform recursive feature elimination or addition. Shuffle features and determine performance drift. The possibilities are endless… ok maybe not endless but there is quite a bit out there.
**Soledad Galli**
Here are the methods supported by Feature-engine: [https://feature-engine.readthedocs.io/en/latest/selection/index.html](https://feature-engine.readthedocs.io/en/latest/selection/index.html)
**Soledad Galli**
And I discuss all of this in another course that you can find on udemy on this link: [https://www.udemy.com/course/feature-selection-for-machine-learning/?couponCode=DATATALKSCLUB](https://www.udemy.com/course/feature-selection-for-machine-learning/?couponCode=DATATALKSCLUB)
**Soledad Galli**
I have not worked on chemistry problems myself, so I am not sure if people in this field are doing something more specific.
**Sandhya G**
Thank you for the pointers Soledad Galli! I will check out the resources
**Nikhil Shrestha**
Hi Soledad Galli
I am putting my question in 4 steps:
1. Suppose we have some data in CSV and we want to make graph structure or visualization using this data
2. I was thinking of making certain column heading as vertex and the values inside column as edges.
3. For this we can use DictVectorizer to create numerous columns.
4. What can we do for numerical data if want to create a graph structure and end result should be explainable to non technical person
I am presently solving something very similar, so I was thinking of a way out.
So if this approach is completely wrong please suggest me some direction. 🙏🙂
**Soledad Galli**
I am not familiar with graph data (nodes, edges, relationships) so I can’t help with this question.
**Soledad Galli**
Regarding how to present the data to a non-technical audience, I guess it depends on the data, on the project, and what are the key messages to deliver to the audience. Bar plots, line plots, bullets with the key messages. If working with text word clouds.
**Soledad Galli**
I guess many people in this goupr can probably contribute some ideas, so don’t ne shy :)
**Nikhil Shrestha**
Thank you again
I was thinking about presenting the audiance in similar fashion as you mentioned here.
🙂
**Avishek Datta**
_1\. While checking for whether a feature has normal distribution or not, is it necessary to check for the distribution of the Target Variable as well and then transform this Target variable (say, log transform the Target variable)?_
_2\. Again while checking the distribution I observe that one of the features is not normally distributed. So I log Transform it. This is a Predictor variable, now. Now after the entire model has been trained, do I need to use anti-log for that particular feature to take in new feature variables for prediction?_
_3\. Maximum how many transformations can be added in a Pipeline & also a ColumnTransformer?_
_4\. When should one use Winsorization and outlier removal through IQR method? Which is more optimal?_
**Soledad Galli**
For linear regression models, normal distributions in both predictive features and target tend to improve the performance. So I would check the distribution (of both) and if it helps apply a transformation.
**Soledad Galli**
If you log transform a variable, and then train a model, whenever you make a prediction using new / raw data, you need to log transform that same feature before passing it to the model, because the model learned patterns from the log transformed variable.
**Soledad Galli**
you would only apply the inverse of the log transform, if for some reason you need to retrieve the original (raw) values of that variable, say to show some results to stakeholders.
**Soledad Galli**
3) You can add as many transformations as you need. The sky is the limit.
**Soledad Galli**
4) some linear models are sensitive to outliers, in the sense that can be biased by these observations. You would in general cap outliers if a) you are sure they do not add any value, and b) capping improves model performance.
**Soledad Galli**
You could cap finding the limits with the IQR, percentiles or mean and standard deviation. The choice depends mostly on the variable distribution. IQR returns more representative limits when we have skewed variables.
**Avishek Datta**
Thanks for the answers
**Avishek Datta**
_5\. After applying discretization on a feature, is it necessary to check for the skewness of the discretized feature and then transform it, if necessary (say log transform or root transform?_
**Soledad Galli**
There are discretization methods that already, by definition, return intervals that are homogeneously distributed, for example equal frequency discretization. I would apply these methods instead of discretizing into something that is skewed and then transforming the variable further.
**Shankar Somayajula**
Hi Soledad Galli.. Thanks for taking questions.
Using the fraud model as an example, once users of your model see the results of latest fraud model incorporated and see the benefits e.g. successful identification of fraud [instances… How do they respond to gaps in terms of missing fraud? Can they](http://instances.be/)
articulate new features which are variations of existing features and see them incorporated w/o an explicit modeling process (via UI or some other post processing activity)? How does one keep the (domain) features updated? Is Iterative development fast enough to respond to real time fraud behavior/events?
**Soledad Galli**
I think this depends on the organisation. Different organisations may have different procedures. You can have a team of fraud investigators continuously analysing the landscape and creating new features or new rules I would better say. The data scientist could be performing data analysis on recent data and seeing if something new comes up. I think continuously monitoring of the landscape evolution is probably key. And then, you could combine different models to try and detect different aspects of fraud.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Effective Data Science Infrastructure – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Effective Data Science Infrastructure
-------------------------------------
#### by [Ville Tuulos](https://datatalks.club/people/villetuulos.html)
##### The book of the week from 27 Sep 2021 to 01 Oct 2021
Effective Data Science Infrastructure: How to make data scientists more productive is a hands-on guide to assembling infrastructure for data science and machine learning applications. It reveals the processes used at Netflix and other data-driven companies to manage their cutting edge data infrastructure. In it, you’ll master scalable techniques for data storage, computation, experiment tracking, and orchestration. You’ll also learn how to collaborate with data scientists to deliver exactly what they need to succeed.
* [Book's page](https://www.manning.com/books/effective-data-science-infrastructure)
* [Book's GitHub repository](https://github.com/outerbounds/dsbook)
Questions and Answers
---------------------
**Alper Demirel**
Hi Ville Tuulos, thank you for being with us.
In order to be successful in a data science project, the infrastructure we need to set up must have the basic features?
**Ville Tuulos**
great question! Answering the question is exactly the topic of the book 🙂
“Must” is a strong word since every company (or even every project) has different needs. The most common infrastructure layers are:
* Integrations to the surrounding _data infrastructure_ - practically all projects need to ingest data.
* A _compute layer_ that allows you to train models and process data at scale.
* A _workflow orchestrator_ allowing you to run workflows in production reliably.
* A _versioning_ system that allows you to version not only code, but also models, experiments, and maybe keep track of data lineage too.
For prototyping, I am a fan of cloud-based workstations but technically a laptop/desktop will work too. You probably want to run notebooks on the workstation too.
Depending on the needs of your project, you may need other components like a model hosting solution, specific support for feature engineering, or a distributed training system for DNNs. I think it is a good idea to start first with the foundational layers in any case.
**Laura Uzcátegui**
Hi 👋,
Will put my questions on this thread.
1. Is metaflow restricted to be used only with aws? Or does it have integration with other cloud providers?
**Laura Uzcátegui**
1. Until what point of having great performance and scalability is recommend it? Is there any trade-off or use case where you will say metaflow should not be used?
**Laura Uzcátegui**
1. I have worked previously in ML Platform team, it was fun.
Our infra/platform was based on collecting data across different services, serialised with avro and encrypt payloads to be send to Kafka.
Using Apache Flink, we will consume from Kafka, un-encrypt, do aggregations and send it to an encrypted blob store , so that ML Engineers could take it and use it for building models.
Questions:
* I have always thought about , if the blob store would make sense in case of big amounts of data, is this something could be done differently?
* How would you handle the trade off of having to do encryption of your data and reading it fast because you need to handle SLAs with respect to time to response and the overload of encryption itself?
**Ville Tuulos**
1. Coming soon! Metaflow will start supporting Kubernetes in a few weeks, which allow you to deploy it to other clouds (we tested it with Azure). We are actively working on to support e.g. Azure Object Storage natively.
**Ville Tuulos**
Could you rephrase your question about performance? I am not sure if I got it
**Ville Tuulos**
1. Re: when not Metaflow. Here are some good reasons for not using it:
* You use primarily JVM-based languages. Metaflow works well with Python, R, and extensions written in C/C++. There’s probably too much friction if you want to use it with Java/Scala/Clojure.
* Your use cases are all based on streaming data. Metaflow is quite batch-centric as of today. If you need _only_ online learning and low-latency data, Metaflow isn’t the right tool. However, Metaflow can work with with e.g. real-time predictions and streaming data with batch training, which is a common use case e.g. at Netflix.
* You have one specific use case that has very specific requirements, e.g. running ML on an embedded device. It is probably better to create a custom solution in that case.
**Ville Tuulos**
The list is not exhaustive 🙂
**Ville Tuulos**
1. Using a blob store could work well in a scenario you described. Netflix uses Kafka and Flink to populate data in S3 (a blob store) very actively. You may want to take a look at [Apache Iceberg](https://iceberg.apache.org/)
as a convenient metadata format which works with Spark and Presto/Trino for such use cases.
There are very high-throughput encrypting libraries available these days, so I doubt it would become a major bottleneck. The upcoming chapter 7 in the book will talk about high-performance data loading, so you can read it for inspiration.
**Laura Uzcátegui**
Thanks Ville!
Really good points there, previous team was using scala/java for the streaming part and orchestration with golang, as you said , for this use in particular it wouldn’t work.
Looking forward to read and learn from you as author of the book 🙌
**Dr Abdulrahman Baqais**
Hi Ville. Thank you for writing this most needed book. Few questions as I am going to set up a new DS team:
1) for a new DS lead, What are the skills needed in order for someone to design a good data science process or workflow if they can not use metaflow?
2) Can technical people : senior data scientists set up the process or it is better to be assigned to people with strategic mindset and process-oriented?
3) is CRISP still relevant and useful for data science projects?
4) Is this workflow applicable to deep learning as well?
5) Is there a checklist or a guidelines to know when Netflix metaflow is suitable to my team or not?
Thank you so much🙏 🙏 🙏 :thank\_you:
**Ville Tuulos**
`1)` The book advocates for a spiral approach like this, which is applicable to all projects whether to use Metaflow or not:
1. Start with a business problem - what do you need to deliver to make a positive impact in the business.
2. Do you have data that can be used to reach the desired outcome? Often the answer is “no” or “not quite”, in which case it makes sense to see how to improve the data situation first.
3. Assuming the data is there, how do you integrate the results to the surrounding business systems and processes, including human workflows, to achieve the desired impact? This is often a really deep question that requires lots of alignment and product management-type work.
4. Finally when you have a good grasp of the 1-2-3, start thinking about the modeling approach and how to build a process that allows data scientists to continuously improve the model. You can start with something super simple - [the first version doesn’t even have to use ML](https://eugeneyan.com/writing/first-rule-of-ml/)
.
Since you asked about skills, as a DS lead I’d prioritize my work and learning process in the order specified above, 1-2-3-4. Often the instinct is to focus on 4) - the modeling - but it is hard to even choose the best approach without knowing the problem domain well.
**Ville Tuulos**
`2)` I think it would be great if data scientists can at least help to define the process, naturally in collaboration with business stakeholders etc.
**Ville Tuulos**
`3)` I assume you mean this [CRISP](https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining)
. It is really close to the spiral model above, but it seems a bit waterfall-like: Understand->Model->Deploy, which puts too much emphasis on a static deliverable. Coming up with a process that allows you to iterate on models quickly is more important than any single model.
Providing the building blocks for such a process is the reason why you need Effective Data Science Infrastructure in the first place.
**Ville Tuulos**
`4)` yeah, DNNs are just models like any other models. I don’t think they change anything about the process, besides maybe putting even more emphasis on scalable data and compute layers.
**Ville Tuulos**
`5)` You can find [a non-technical checklist in the documentation](https://docs.metaflow.org/introduction/what-is-metaflow#should-i-use-metaflow)
. Whether it works well in your technical environment is a more nuanced question. I’m happy to follow up here if you want to know more technical details.
**Ville Tuulos**
great questions, btw! 🤗
**Timur Kamaliev**
Hi Ville Tuulos and thank you for being here.
I think my questions may be closer to the development. I usually notice that the position of an ML engineer often requires C++ background. But there are almost no mentions of Golang. It’s ok, if we deal with CV (and CUDA is here) or with some embedded system. But what about classic ML area? From my point of view, it’s quite strange, especially for all the relative advantages and comparable performance of Go in comparision with C++. All the experienced SW engineers I know, who are dealing with highload in production, switched on to the Go for 3-4 years ago. Do you think that it concerned with some infrastructure restrictions? Or DS/ML area is a couple of steps behind?
**Ville Tuulos**
I think Go is a great language but it has weaknesses when it comes to ML/DS use cases. It makes sense to implement low-level computational kernels in C or C++, since you want to control memory carefully and optimizing the last bit of performance matters. Due to its design, Go has exceptionally high overhead when calling C/C++ libraries ([the infamous CGo overhead](https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/)
), so it is actually harder to use Go with ML libraries than, say, Python. This issue applies equally to classic ML (think XGBoost or LibSVM) as it does to DNN.
In the future, there’s likely going to be a relatively small number of engineers building low-level libraries who will need to use C/C++ for performance reasons. In contrast, the vast majority of ML engineers and data scientists can use a high-level language like Python that delegates all heavy lifting to a low-level library like Tensorflow.
While Go is a superb language for systems software, I am not sure where it lands in the DS landscape: It is too slow for computational kernels and too low-level and hard-to-integrate-with for high-level work 🤔
**Timur Kamaliev**
My second question also about development. Let’s suppose that our ML service was developed on FastAPI/Flask and packed into a docker. And one day we faced with a performance problem in production due to an increased load. Assume that we have already done all the tricks with code optimization and so on, and now it’s time to make more serious steps. We have two ways:
* to configure the required number of docker containers and additionally write a special load balancer that will interact with these containers;
* or just take and rewrite the entire project in C++ (or another faster language).
So, which option would be more preferable in terms of infrastructure and costs? :thank\_you:
**Ville Tuulos**
Here’s how I understand your problem as a hierarchy of three levels:
`[ Microservice (Flask)
[ Query endpoint
[ ML Model ]
]
]`
What’s causing the slowness?
1. ML Model -> Producing inferences take too long, so it is a model architecture problem. If the model is e.g. a DNN model, you can use a project like [Apache TVM/OctoML](https://octoml.ai/)
to try to optimize it.
2. Query endpoint -> Since you are using Flask, I presume you are using Python. Maybe you are doing some non-trivial preprocessing of data in Python that’s slow. You might be able to refactor the code to be much faster (see the upcoming Chapter 5 in the book!) but in some cases it can get tricky, which might be a reason to consider another language.
3. Microservice -> Scaling microservices to handle an arbitrary queries-per-second is a common problem which is not specific to ML. Maybe you can use existing techniques to make it more scalable.
Chapter 5 also talks about the difference between scalability and performance. In this case, levels 1 and 2 might have performance issues and level 3 a scalability issue. Often it is easier to solve a scalability problem (just add more machines!) than it is to solve a performance problem (reimplement everything in C++), so I tend to bias towards solving the former, if possible.
**Timur Kamaliev**
Ville Tuulos Thanks for your detailed answers 👍
**Prasad Paravatha**
Hi Ville Tuulos, my question is about different Distributed training options successfully implemented.
To be more specific what are our options for Distributed training on Kubernetes vs Distributed training on non-Kubernetes?
**Ville Tuulos**
interesting question. It would be useful to understand what type of distributed training you have in mind: Do you want to train a huge model that is too big for any single machine to handle (model parallelism) or do you want to train a medium-size model that fits on a single box using a huge amount of data (data parallelism)? Or do you want to train many independent models e.g. with different parametrizations? There are different solutions for each use case.
For the non-Kubernetes case, you could look into a solution like [Sagemaker’s Distributed Training service](https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training.html)
. For K8S, [Determined.ai](https://www.determined.ai/blog/determined-on-kubernetes)
[is one option](https://www.determined.ai/blog/determined-on-kubernetes)
(it [works nicely with Metaflow](https://www.determined.ai/blog/metaflow-determined-integration)
too). Or you could roll-out your own solution using e.g. [Horovod](https://www.google.com/search?q=horovod+github&rlz=1C5CHFA_enUS921US921&oq=horovod&aqs=chrome.3.69i57j0i20i263i512l2j0i512l7.2851j0j7&sourceid=chrome&ie=UTF-8)
.
We are actively working on distributed training and K8S use cases - it is definitely a non-trivial problem, so happy to share more details if you are interested 🙂
**Prasad Paravatha**
Hi Ville Tuulos My 2nd question: Python and R are very popular in Data science and Machine Learning community
In your experience, What are some other programming languages which have potential to be suitable for DS/ML?
**Ville Tuulos**
When choosing a programming language for DS/ML, I’d consider
1. Existing library ecosystem. Unless you are in academia, you probably don’t want to implement modeling libraries from scratch.
2. Low-overhead interfacing with C/C++ libraries (more details in this thread [https://datatalks-club.slack.com/archives/C01H403LKG8/p1632733036461900](https://datatalks-club.slack.com/archives/C01H403LKG8/p1632733036461900)
)
3. Expressive, supports rapid iterations, and can handle even complex, messy business logic.
Obviously Python and R score really well on all the points, 1, 2, and 3. Julia is promising but its library ecosystem is still behind Python and R. JVM-based languages have a problem at least with 2, these days with 1 too. There are less mainstream languages that support 2 and 3 really well ([D](https://tech.nextroll.com/blog/data/2014/11/17/d-is-for-data-science.html)
, [Nim](https://benjamindlee.com/posts/2021/why-i-use-nim-instead-of-python-for-data-processing/)
etc) but they are sorely lacking on 1.
**Yasser**
Hello Ville Tuulos. Thank you for writing this great book. I have a question: What fundamentals and How much can I take for starting a data science infrastructure career ?
**Ville Tuulos**
thanks 🙇
You could approach a career in DS infra from different angles, depending on your personal interests and background. This may count as <#C01F53D373M|shameless-promotion> but I can answer your question in terms of the four “roles” we have open at [my DS infra company](https://outerbounds.co/workwithus)
(feel free to apply 😛) :
* _Systems engineer angle_: Learn about Computer Science, operating systems, distributed systems etc. The CS fundamentals are very useful when building infrastructure for ML/AI.
* _Cloud angle_: I believe that a vast majority of ML/AI workloads will be run in the cloud (as they already are). Systems like AWS are huge and complex enough that you could just focus on learning about VPCs, Placement Groups, Spot Instances, Auto-Scaling Groups, CI/CD systems, and various higher-level services - not to mention Kubernetes. Infra companies will be competing fiercely to hire you with this skillset.
* _Product design angle_: I also believe that the more our field advances, the more people will matter over machines. Often, optimizing the productivity of data scientists is more valuable than optimizing the performance of a piece of code. There’s a hugely important emerging field of [ML User Experience](https://www.meetup.com/MLUXmeetup/)
and [UX of ML tools themselves](https://future.a16z.com/the-case-for-developer-experience/)
.
* _AI/ML angle_: Getting exposed to various ML/AI problems in the real-life is the best way to understand pain points that infrastructure needs to solve. Knowing libraries like PyTorch or even Pandas inside-out allows you to understand how to improve them, or maybe build better libraries altogether. [Many people in this field](https://www.youtube.com/watch?v=gFEE3w7F0ww)
are here to scratch their own itch.
**Yasser**
Thanks a lot for your helpful detailed answer 🙏
**Alexey Grigorev**
What do you think of data versioning? Is it always worth the troubles?
**Ville Tuulos**
while it is a good idea, as long as commonly used data warehouses don’t support versioning as the first-class citizen, it is hard to claim that it is _always_ a good idea. It takes effort to implement it today.
While snapshotting bulk input data can be tedious, snapshotting and versioning smaller data like the internal state of workflows and especially metadata and metrics is _a really good idea_ and not too hard, which is why Metaflow does it out of the box.
**Ville Tuulos**
here’s an exploration of what a full data versioning solution could look like [https://www.dolthub.com/blog/2021-04-12-metaflow-dolt-integration/](https://www.dolthub.com/blog/2021-04-12-metaflow-dolt-integration/)
**Alexey Grigorev**
Also curious, did you have cases when you wished you had it, but you didn’t?
**Ville Tuulos**
all the time 🙂 For instance, Netflix has [a custom-built Fact Store](https://databricks.com/session/fact-store-scale-for-netflix-recommendations)
(aka a versioned datastore) for data points feeding into the recommendation system. However, it only supports that one use case well.
It would be great to have a similar solution for all use cases but building a universal fact store is hard, so even Netflix doesn’t have it. I am optimistic that it will become a default feature in all [modern data warehouses](https://docs.snowflake.com/en/user-guide/data-time-travel.html)
eventually, since cloud storage is so cheap.
**Ville Tuulos**
at small scale you can use tools like [dvc.org](http://dvc.org/)
and [pachyderm.com](http://pachyderm.com/)
but a challenge to any larger company is that your whole data infrastructure and governance policies should work well with versioned data, so it is hard to adopt point solutions
**WingCode**
Hi Ville Tuulos,
For an organisation starting to build their ML infrastructure, do you suggest adopting cutting edge tools from the start (ex: Kubernetes for training, inference) or adopt simpler tools (ex: Using airflow to schedule jobs in spark on YARN) then gradually move onto adopting cutting edge tools?
**Ville Tuulos**
there are of course many things to consider - what are the needs of the business / environment overall - but in general I’d recommend starting with managed systems, so you can maximize the time spent on ML and minimize the time spent on infrastructure.
This is the reason why Metaflow integrates with AWS Batch today, which is probably the easiest way to run compute in the cloud with minimal maintenance headache. For orchestration, AWS Step Functions plays a similar role. They provide a managed Airflow these days too.
**Ville Tuulos**
The main reason why things like Kubeflow exists, and why Metaflow is integrating with Kubernetes too, is that businesses have other, non-ML reasons to use Kubernetes and it is certainly convenient to use a single system that engineers know how to operate, instead of having a wholly separate underlying compute layer for ML.
In general there’s a lot of value in making sure that the ML platform integrates well with the surrounding infrastructure. If your ML projects live inside a walled garden, it can make it hard to access data from the outside world effectively and deploy ML to benefit the rest of the business.
Even if the business runs on Kubernetes, [I don’t think it should be necessary for data scientists to know Kubernetes (to quote Chip Huyen)](https://huyenchip.com/2021/09/13/data-science-infrastructure.html)
. It is more of an implementation detail.
**WingCode**
Thank you Ville for the answer 🙂
**Krzysztof Ograbek**
Hi Ville Tuulos. Thanks for being here!
A couple of weeks ago I deployed an ML model for the first time ever. I used Heroku. Are there any other platforms for deploying that are free and available? When does it make sense to pay for such platforms?
**Ville Tuulos**
congrats for your first deployment! 🎉
If you are testing and tinkering, you could take a look at [https://replit.com/site/hosting](https://replit.com/site/hosting)
which is possibly even a simpler approach than Heroku.
**Ville Tuulos**
for more serious use cases you can consider [https://cloud.google.com/appengine](https://cloud.google.com/appengine)
**Ville Tuulos**
unless you deploy complex deep neural networks that require e.g. GPUs for inference, you probably don’t need a dedicated ML hosting solution.
If you are curious what an ML-specific solution could look like, you can see e.g. [https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-deployment.html](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-deployment.html)
**Ville Tuulos**
non-ML specific hosting solutions tend to be quite cheap, so paying maybe $10/month just for the convenience of someone managing it for you seems like a good deal.
**Ville Tuulos**
I wouldn’t pay a lot of money unless the model produces a lot of money 🙂
**Krzysztof Ograbek**
Wow, thank you so much Ville Tuulos You wrote way more than I expected
**Wendy Mak**
Hi Ville Tuulos what are your opinions on using a platform independent tool such as Metaflow/MLFlow vs what the cloud providers already has (e.g. Vertex pipelines), assuming costs is not the deal breaker?
**Ville Tuulos**
great question! I like the idea of being able to use ML tools provided by the cloud. However, for the past many years these cloud-tools have been quite hard to use and constraining, which has motivated the development of systems like Metaflow, MLFlow and many others.
In other words, assuming that cost wasn’t an issue and the cloud tools provided an excellent user experience, using them seems like a good idea. Now, in reality, it seems that many companies find that either one or both of these assumptions not holding well.
**Ville Tuulos**
a nice thing is that it should be quite easy to evaluate them: Try implementing a real-world ML project (not just a tutorial) on the platform and see how it feels. If the platform works well for your needs and it integrates well with the surrounding business environment, that’s great!
**Ville Tuulos**
If the evaluation reveals any rough edges or missing features, then you can consider tools like Metaflow, MLFlow etc. which work well with the cloud, so you still get many of the same benefits, hopefully with a more pleasant user experience and more flexibility.
**Wendy Mak**
if there are things that don’t work so well, but some other aspects are nice, how do you decide at which point you’ll accept the time spent is enough and try another tool?
**Ville Tuulos**
that’s a tough one. I’d consider the migration cost too. If you have built N projects on a platform X but then find another platform Y better, Y needs to be much better to justify replatforming of the previous N projects. And, how many future projects M are going to be built on the new platform.
Really hard to give a generally applicable answer.
**Ville Tuulos**
I’d also emphasize thinking how the ML platform can leverage other tooling, talent, and existing expertise at the company (cf. [https://datatalks-club.slack.com/archives/C01H403LKG8/p1632933598496200?thread\_ts=1632893087.488600&cid=C01H403LKG8](https://datatalks-club.slack.com/archives/C01H403LKG8/p1632933598496200?thread_ts=1632893087.488600&cid=C01H403LKG8)
)
**Tim Becker**
Hi Ville Tuulos, thanks for the great book! After reading the first chapter of your book, I would also like to ask you a few questions.
**Tim Becker**
* How can I recognise incidental complexity and hot to avoid it?
**Ville Tuulos**
I :heart: the question! I can start by answering the other side: how do you recognize inherent complexity?
If something seems complex but the complexity is really not in your control, it is inherent complexity in your point of view. For instance, if you are analyzing a dataset that includes text written in Chinese, Arabic, Cyrillic, and Latin scripts, it seems complex but there’s nothing you can do about it - that’s the nature of the world.
On the other hand, if you are asked to develop a system to parse a JSON file but you decided to make it more generic by supporting XML, HTML, and YAML too, the extra complexity seems quite incidental and maybe hardly justifiable - you just decided to do it.
So how to avoid adding complexity deliberately? Keep things simple deliberately. This is the best talk I have seen about the topic [https://www.youtube.com/watch?v=LKtk3HCgTa8](https://www.youtube.com/watch?v=LKtk3HCgTa8)
(Simple Made Easy by Rich Hickey)
**Tim Becker**
* You mentioned that relying on pre-build solutions from cloud providers is not always cost affective. Could you elaborate on this? Which parts of, for example, AWS do you consider more and less cost affective?
**Ville Tuulos**
The AWS of today is a thick stack of services, building upon each other:
* At the bottom of the stack you have foundational services like S3, EC2, networking, security (IAM).
* On top of this, mid-tier, you have databases (RDS, DynamoDB, ElastiCache etc), container management systems (EKS, ECS), EMR, and bunch of other things like CloudWatch.
* At the top of the current stack you have high-level services like Sagemaker, QuickSight, WorkSpaces, all kinds of things.
Since most of AWS’ own high-level services are built on lower-level services, higher-level services can’t be less expensive than the low-level services they sit on top of. Often, they are more expensive, sometimes much more. For instance, Sagemaker and EMR charge you the cost of the underlying service (e.g. EC2) plus a margin. Some mid-tier services like ECS and AWS Batch don’t charge a margin on top of the underlying EC2 cost.
Whether a higher-level service is cost-effective depends on your specific situation. You should estimate the Total Cost of Ownership carefully. In contrast, lower-level services like EC2 and S3 are pretty much no-brainer cost-effective for most use cases (the jury is out on GPU instances…)
**Tim Becker**
* Could you give an example of when to use metaflow?
**Ville Tuulos**
definitely yes:
* You have many data scientists developing one or more ML projects using Python.
* You are using AWS already.
* You are experiencing any of the following symptoms:
◦ difficulties with scaling
◦ difficulties with deploying or managing workflows
◦ friction in collaboration
◦ want to get better at reproducibility, versioning, and tracking experiments.
maybe:
* You are using other clouds and/or Kubernetes (I’d love you to test our new K8S integration or talk about use cases on Azure/GCP 🙂 ).
* You are pretty happy with the current setup - no big pain points.
no:
[https://datatalks-club.slack.com/archives/C01H403LKG8/p1632727165458700?thread\_ts=1632726041.457100&cid=C01H403LKG8](https://datatalks-club.slack.com/archives/C01H403LKG8/p1632727165458700?thread_ts=1632726041.457100&cid=C01H403LKG8)
**Tim Becker**
* Do you know a way of measuring the quality of data science infrastructure and to determine bottlenecks that should be improved?
**Ville Tuulos**
here are some questions you can ask:
* how quickly does a new data scientist become productive with the platform and projects?
* can you hire and onboard domain experts who are not devops or software engineers and let them focus on being domain experts?
* how quickly can you iterate on new, experimental versions of end-to-end ML projects (not just the model)?
* does the platform fail randomly? Do data scientists spend a lot of time waiting compute/data/deployment to happen?
* can you test different modeling approaches easily and experiment with even bleeding-edge libraries and techniques?
* how easy it is to deploy new versions of models to production? is it scary? can anyone in the team do it confidently?
* can you A/B test models and workflows against the production version using live data before promoting them fully to production? Can you analyze the results of A/B tests confidently?
* how easily can you onboard new data in your projects, e.g. to test new features?
* do you know what’s happening? Do you know how models are performing?
* if something fails unexpectedly, can you quickly figure out the root cause, test a fix, and deploy it?
**Ville Tuulos**
then prioritize the questions based on your specific needs and address the most glaring pain points
**Tim Becker**
Ville Tuulos thank you so much for answers, Super useful 💯:thank\_you:
**luckylittle**
Warm regards Ville Tuulos - I have a simple question: Who is the target audience for your book? I briefly looked at the TOC and not entirely sure… When the book is finished, will there be any practical tips for DevOps engineers managing/designing data science platforms? Many thanks and enjoy your stay 🙏
**Ville Tuulos**
if you are a DevOps engineer designing a data science platform, you are very much in the target audience. Metaflow is only used as an example in the book, so you should find it useful even if you use another framework(s).
**Tino**
Hey Ville Tuulos 🙂 Thanks for taking the time. At what stage in companies would you recommend to focus on “effective” infrastructure? In many small startups they focus on delivery whereas in my opinion a solid foundation should be given at all time. Any thoughts on that or is it to individual?
**Ville Tuulos**
realistically it is probably not the first concern at a new startup. It becomes more relevant once you have a few people (data scientists / engineers) focusing on ML / data science
**Ville Tuulos**
I wouldn’t wait for too long though, especially with open-source projects that don’t require a huge upfront commitment. In a small startup you have often resourceful people who know that they could do everything by themselves, so there might be a feeling that “we don’t need anything fancy, we can just set up X, Y, Z quickly”.
Especially at a smaller company it is important to focus on the company’s core business and minimize the time spent on generic infrastructure.
**Krzysztof Ograbek**
Hi Ville Tuulos, thanks for answering my yesterday’s question. As mentioned, I just started deploying ML Models into production. For the next few weeks, my goal is to create small web apps based on flask and react with models making predictions in the backend. In the end, I want to deploy the apps. I’m comfortable with coding but everything else is new to me. So my here come my questions:
* How can your book help me?
**Ville Tuulos**
in your case, maybe the biggest help is to show how you can structure the training pipeline and get it running regularly, if you want to update your model automatically. It helps to set up the workflows in a way that makes debugging and further development easier if you keep working on your project.
**Krzysztof Ograbek**
* When should I start using Docker? Is it the sooner the better? How can Docker be beneficial?
**Ville Tuulos**
as we discussed in the thread above, these days you can deploy web apps without Docker, e.g. using [Repl.it](http://repl.it/)
/ Heroku / AppEngine and others. In this sense I don’t think you must use Docker.
If you anticipate that the web app will grow more complex over time and you want to invest energy in it, probably learning Docker is a good idea. It is relatively easy to get started with Docker.
**Ville Tuulos**
Docker is beneficial for two main reasons:
1. It allows you to package the whole execution environment, all libraries etc., in a single image.
2. The image can be executed in various environments.
**Krzysztof Ograbek**
* What are the topics I should learn first? Could you recommend any resources?
**Ville Tuulos**
can you tell more about your project? Do you want to mainly focus on the modeling side or the webapp side?
**Krzysztof Ograbek**
Hi Ville Tuulos and sorry for the late reply. My main goal is to train ML models and do something more than just evaluate them in a jupyter notebook or similar. This is why Im thinking about web apps. For now I’ve got two ideas how I’d like to bring more value:
* Educational purposes - let users interactively play with model and it’s input to see how it affects output
* Create some fun by playing some silly games, eg uploading an image to see if it’s a hot-dog.
So what is my main focus? I want to learn how to effectively deploy models. In my latest project I’m dealing with memory challenges. They’re completely new to me and I’m really stuck.
I hope any of this answers your question.
**Laura Uzcátegui**
Hi Ville Tuulos,
One last question before the round closes :)
In terms of constantly monitoring your models in terms of , for example, accuracy, precision, recall, rmse among all the other metrics for evaluation , I have few questions:
* How often those metrics should be evaluated ?
* What platform is used in terms of observability for evaluation metrics of a model in production?
* Would you say is fair to say if a metric presents a degradation under certain threshold to perform training or refresh of the model?
What resources would you recommend for learning more about evaluating metrics in production models?
**Ville Tuulos**
great question! There are many different kinds of monitoring / metrics evaluation involved in production ML.
Monitoring model metrics like precision, recall, RMSE etc. is a good baseline sanity check. Many tools like [Weights and Biases](https://wandb.ai/)
, Tensorboard, and of course standard visualization components from e.g. Scikit Learn can help. For more production-oriented monitoring, there are newer services like [arize.com](http://arize.com/)
and [monalabs.io](http://monalabs.io/)
(and many others) which can help.
For models that drive actual business outcomes in production, it is a good idea to focus on monitoring the KPIs that matter for the business rather than just model metrics. For instance, if your model predicts who will click ads, you should measure the actual Click Through Rates (or even better: Cost Per Click) and not just RMSE. For monitoring KPIs, any business analytics tool (Tableau, Mode Analytics etc) will work.
**Ville Tuulos**
re: refreshing models - in most cases it is a good idea to retrain the model frequently, independent of the metrics, backtest its performance with historical data to ensure that nothing is obviously wrong, and potentially A/B test in production if the use case is sensitive. If you retrain only when the performance degrades, you will suffer from data/concept drift and worsening performance until you hit the threshold. Instead of being reactive, you can be proactive and retrain continuously (e.g. once a day).
Measuring and plotting the delta in model performance day-over-day (or week-over-week etc) is a powerful way to catch issues when you roll out new models (and even when you don’t).
**Laura Uzcátegui**
Thank you so much for your answer and insights.
Really useful and I enjoyed learning from your experience.
Hope to see you around 😊
Do you have social media or a blog or a place where you share your knowledge ??
All the best Ville Tuulos
**Ville Tuulos**
you can find me any time at our Slack [https://datatalks-club.slack.com/archives/C01H403LKG8/p1633128052035000](https://datatalks-club.slack.com/archives/C01H403LKG8/p1633128052035000)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Transfer Learning in Action – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Transfer Learning in Action
---------------------------
#### by Dipanjan Sarkar, [Raghav Bali](https://datatalks.club/people/raghavbali.html)
##### The book of the week from 04 Oct 2021 to 08 Oct 2021
Transfer Learning in Action shows you how using pre-trained models can massively improve the accuracy and performance of your machine learning projects. Focused on the real-world applications of transfer learning, you’ll explore how to enhance everything from computer vision to natural language processing and beyond. Master hands-on techniques taken from the latest research, and discover how you can customize open source models for your specific needs.
* [Book's page](https://www.manning.com/books/transfer-learning-in-action)
* [Book's GitHub repository](https://github.com/dipanjanS/transfer-learning-in-action)
Questions and Answers
---------------------
**Kshitiz**
Hi Raghav Bali and Dipanjan,
Glad to have you here for answering our questions.
I have few questions to ask here -
1. Are there any prerequisites for applying transfer learning to get better performance? (For e.g. if the data is highly domain specific, I believe, it might not generate good results)
2. Has transfer learning in NLP reached the same performance compared to computer vision?
**Raghav Bali**
Kshitiz Full points for the first set of questions. Let me try and answer them:
1. The short answer is yes. Before you plan out for leveraging Transfer Learning, you need to keep a few things in mind, such as, how closely related are your source and target domains, how many labels/data samples do you have from the target domain, etc. These and a few other considerations play an important role to determine whether you should use TL or would there be any possibility of improved performance. Checkout chapter 1 from the livebook for a detailed discussion
2. Oh yes, this is my personal favourite. Computer Vision might have brought ML/DL into the limelight but it is the recent research and success of large scale NLP models that have had an even larger impact. Most importantly, the concepts of attention, self attention and the Transformer architectures have a very important role. These key concepts have not only improved the NLP domain but also provided a solid foundation for transfer learning. Even more importantly, these concepts though initially were applied only to NLP domain, are not being cross pollinated to Computer Vision, Audio etc as well.
Thanks for your questions again. Hope my response points you in the right direction 😃
**Alexey Grigorev**
Hi Dipanjan Sarkar, welcome 👋
**Alexey Grigorev**
I have a question for Dipanjan Sarkar. Do you describe in your book how to make a userpic like yours?
**Alexey Grigorev**
Does style transfer have anything to do with transfer learning?
**Tino**
Hello Dipanjan Sarkar, I know that transfer learning is a common concept in deep learning but I see it less in traditional algorithms. Some libraries offer convenient ways of using it e.g. for a logistic regression. What use cases do you know where something like this could be useful? And what do you think about using this for boosted trees as the transfer learning here is not well documented (from my experience).
**Raghav Bali**
Hi Tino
Interesting question and something we both are also exploring as well. The iterative nature and capacity of deep neural networks make them right candidates for leveraging transfer learning (see weight matrices).
Traditional learning algorithms have been studied for application of transfer learning as well (we share some further pointers to this as well, see: [https://arxiv.org/pdf/1812.11806.pdf](https://arxiv.org/pdf/1812.11806.pdf)
, [https://people.csail.mit.edu/lpk/papers/MarxRosensteinKaelblingDietterich-final.pdf](https://people.csail.mit.edu/lpk/papers/MarxRosensteinKaelblingDietterich-final.pdf)
)
You are correct, boosted trees such as XGBoost, CatBoost and the likes provide APIs to make use of warm start which enables online learning and transfer learning.
Dipanjan Sarkar, would you want to share more?
**WingCode**
Hi Raghav Bali & Dipanjan Sarkar ,
Could you suggest some popular pre-trained models and their use-cases for:
1. Video
2. Audio
3. Timeseries data
4. Recommendation systems
Example: NLP - GPT; Q&A answering, text generation, text similarity
**Raghav Bali**
Thanks WingCode, here’s my take (pretrained models for specific domains but there could be more as well):
1. Video:
a. Classification: movinet, tweening\_conv3d
2. Audio:
a. Audio Transcription: wave2vec, deepSpeech(2,3,etc), Conformer
b. Audio Classification: YamNet, TRILL,
c. Speech to Text: silero-stt (different languages)
3. NLP: BERT, XLNet, GPT-3, etc for a number of tasks such as classification, NER, etc
**WingCode**
Thank you Raghav for the answer 🙂
**ramreddy yasa**
Hi Raghav Bali and Dipanjan Sarkar
I have few questions:
1. Does transfer learning help if we have a dataset with images that are not similar to ImageNet datasets as most of the CNN models are trained on ImageNet datasets?
2. What is the best way to fine-tune the transfer learning model?
3. How do we initialize the weights? If we use the random weights, after proper training do we reach similar local minima when compared with model trained using pretrained weights?
**Raghav Bali**
Hey ramreddy yasa, thanks for your questions. Let me try and answer these briefly:
1. This is a really good question. You need to be careful about the characteristics between source and target datasets. If they are completely different, it is highly likely that transfer learning will have a negative impact
2. Best way to fine tune: well, this is a tricky and subjective question i believe. could you share some more details on what is the ask about? Typically we freeze till the penultimate layer and update the last layer itself
3. I am not sure if I understand the question but I suppose you are asking with respect to ResNet. The typical options are to either initialize the weights as:
a. random
b. pretrained weights file path
c. imagenet weights, typically bundled with frameworks such as TF, PT, etc
**ramreddy yasa**
Hi Raghav Bali
Thank you for answering the questions.
I have updated the 3rd question above and in 2nd question I just want to know
* how many epochs do we have to typically train a model by freezing it’s layers before training all its layers?
* Ex: If we train the network by freezing the layers for just 5-10 epochs then train whole network for epoch, will this approach give good results or should I train 1000 epochs by freezing the layers itself then train the whole network for another 1000 epochs?
In couple of articles I also saw people freezing till penultimate layer and train the network, later train the whole network. After few epochs, If there is no improvement in validation loss, now they freeze the bottom layers and train only the top layers. Does this really help?
**Doink**
Raghav Bali and Dipanjan Sarkar How to productionize models using Transfer learning to deal with milli-second throughput latencies?
**Raghav Bali**
Doink production related questions are very much environment dependent (by environment I am referring to budget constraints, infrastructure limitations, business SLAs, etc.). For milli-second throughput latencies, I would recommend checking quantization of models or making use of pre-trained models from TF\_Hub and the likes.
I know this is a very high level response. Let us discuss more if you have specific details around this one
**Doink**
Raghav Bali If you would like to share any experiences which you had on the above would love to hear some war like stories if there where any, especially when it came to post deployments.
**Evren Unal**
Hi Dipanjan Sarkar
Hi Raghav Bali
Are there any use case of using transfer learning for speech generation from text?
**Raghav Bali**
Evren Unal Good question. TTS or speech generation from text is an active area of study. Some of the popular models in this field for transfer learning are Tacotron, wave2vec (and its variants), deep voice, FastSpeech etc. We will be covering a few of these in the later parts of the book as well. Stay tuned 🙂
**Evren Unal**
It would be very nice.
Becouse in that field, there is not much book or tutorial for beginner level
People who want to generate their own speach engine need some reference about the topic.
**Yasser**
How can we handle transfer learning models which trained on images data with various data types such as text, signal, voice, etc.?
Can we use this concept to train data with data pre-trained with the same type of data?
**Raghav Bali**
Yasser are you referring to Multi-Modal architectures which can ingest inputs of different data types? I am afraid I would need more details to answer this one
**Yasser**
Thanks a lot Raghav Bali for writing this great book and I will wait your answer 🙂.
**Tim Becker**
Hello Raghav Bali and Dipanjan Sarkar, awesome topic for a book! It seems extremely practical and relevant.
**Tim Becker**
* How would you approach selecting the best pre-trained model from, e.g. Tensorflow hub, for a specific modeling task. There are so many potential models to choose from. Would you test several and compare results?
**Raghav Bali**
That’s actually a very valid question and challenge. There a few rules of thumb I typically follow:
1. Check the datasets on which the hub/pretrained model is trained upon. The higher the similarity/relevance to your problem, the higher the impact of transfer learning
2. The infrastructure requirements of the pretrained model. Unless you have access to on demand capacity and budget, there are tradeoffs to be made
3. The input dimensionality of the pretrained model. If your input is too large or too small for the pretrained model, it would be counter productive in most cases
**Tim Becker**
* How would you approach the problem of counting specific objects or object tracking? I imagine that it should be quite easy to select a pre-trained model to recognise certain object, but how to continue?
**Raghav Bali**
I am not quite sure I understood the question here. There are a number of pretrained object detection models to choose from, depending upon your requirements.
Counting and tracking are downstream tasks, once you are able to detect the required object(s) , counting or tracking them is fairly straightforward
**Tim Becker**
* Are there cases in which it is a bad idea to use transfer learning?
**Raghav Bali**
Yes, there are cases where transfer learning can have a negative effect. For instance (over simplified) image a model trained on fish species and you try to use this pretrained model for dog breed classification. More details in chapter 1 of our book. Checkout on MEAP
**Tim Becker**
* I haven’t heart anything concerning transfer learning for timeseries data or simple numeric data. Do you think it is promising?
**Raghav Bali**
That’s right, there are some works/research labs focusing on the time series class of problems from a transfer learning perspective but I haven’t personally seen a lot of breakthroughs over there. Off late there have been some papers in this space, especially in Time Series Classification space though I am yet to go through them in detail to comment.
Do share your experience on this. I will also circle back if I get anything
**Dr Abdulrahman Baqais**
Hi Raghav Bali :. Thank you for being here. Few questions:
1) Transfer learning are extensively in text and image kind of problems. Do you think transfer learning can work in other traditional or future domains: for example speech.
2) When it is advisable to start with transfer learning and when it is better to build your own solutions from scratch directly.
3) in a ModelOps era , do you see transfer learning has an advantage over traditional solutions?.
Thank you.
**Raghav Bali**
Hi Dr Abdulrahman Baqais
Thank you for your questions and patience.
1. Oh yes definitely. Transfer Learning is playing a huge role in the domains such as speech, video, Generative modeling etc. Some of the latest speech related models are enabling amazing use cases like edge device voice/speaker detection etc, all thanks to transfer learning
2. If you have a pretrained model trained on a similar dataset as yours, go ahead and use it. Building your own from scratch is typically motivated by scenarios where you have enough data, compute and time along with a very nieche domain where you wouldn’t find a pretrained model.
3. Absolutely, in the modelOps era, TL is not only helping you iterate faster but also achieve better results with lesser effort
**Timur Kamaliev**
Hi Raghav Bali and Dipanjan Sarkar. Thanks for doing that.
Just a couple of questions.
* Can transfer learning give satisfactory results on small datasets? And how the training process will be differ in this case? Could you please share some best practices from your background?
* And what often do you use - Pytorch or TF? :)
Many thanks!
**Raghav Bali**
Hi Timur Kamaliev
1. Yes, actually TL is highly impactful if your target dataset is relatively small. The training process isn’t different, you may try the pretrained model as a feature extractor with your own shallow classification head to achieve the desired results. See chapters 1 and 2 of the book or the repo for some easy to understand examples : [https://github.com/raghavbali/transfer-learning-in-action](https://github.com/raghavbali/transfer-learning-in-action)
2. Depending upon the setup, either is good though I myself prefer TF more than PT
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mastering Transformers – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Mastering Transformers
----------------------
#### by [Savaş Yıldırım](https://datatalks.club/people/savasyildirim.html)
, [Meysam Asgari-Chenaghlu](https://datatalks.club/people/meysamasgarichenaghlu.html)
##### The book of the week from 11 Oct 2021 to 15 Oct 2021
The book gives you an introduction to Transformers by showing you how to write your first hello-world program. You’ll then learn how a tokenizer works and how to train your own tokenizer. As you advance, you’ll explore the architecture of autoencoding models, such as BERT, and autoregressive models, such as GPT. You’ll see how to train and fine-tune models for a variety of natural language understanding (NLU) and natural language generation (NLG) problems, including text classification, token classification, and text representation. Furthermore, this book helps you to learn efficient models for challenging problems, such as long-context NLP tasks with limited computational capacity. You’ll also work with multilingual and cross-lingual problems, optimize models by monitoring their performance, and discover how to deconstruct these models for interpretability and explainability. Finally, you’ll be able to deploy your transformer models in a production environment.
* [Book's page](https://www.packtpub.com/product/mastering-transformers/9781801077651)
* [Amazon](https://www.amazon.com/Mastering-Transformers-advanced-processing-techniques/dp/1801077657/)
* [Book's GitHub repository](https://github.com/PacktPublishing/Mastering-Transformers)
Questions and Answers
---------------------
**Matthew Emerick**
Hello, Savas Yıldırım and Meysam Asgari-chenaghlu for doing this.
How long until you see transformers replaced with a new architecture or technique?
**Savas Yıldırım**
Hi Matthew Emerick
**Savas Yıldırım**
Two really tough questions 🙂 . Thanks a lot. I think we will continue with transformer architectures for 2-3 years. However, alternatives are being developed for many of its sub-parts. For example, the attention layer is the layer that creates the most complexity, and some sparsification is being employed for this memory and computational burden. It will sound like an advertisement, but we discussed them clearly in CH 08: The Efficient Transformer (pruning, quantization etc.) section.
In some studies, while the tokenization part is completely removed (see ByT5), in some studies they try to remove or change the attention part (FNet: Mixing Tokens with Fourier Transforms). It is said that only FF Neural Net will remain in the final stage. But it’s too early to say that they will be successful and will be used in the industry at scale.
I think I answered two problems in one shot :)
**Matthew Emerick**
Do you foresee a method to reduce the costs of using transformers?
**David Cox**
Very excited to see this book come through! Savas Yıldırım and Meysam Asgari-chenaghlu, I think it’s fun to explore everything transformers are doing so well at and the novel ways they are being applied. I’m always curious about the boundary conditions of various approaches. What are some things that transformers are not great at yet and alternative methods are recommended?
**Savas Yıldırım**
Thanks David Cox for the questions.
**Savas Yıldırım**
Let me try to say the first answer that comes to my mind. Maybe Meysam Asgari-chenaghlu will add something.
Attention can only work with fixed-length text strings, which is a really important limitation whereas we can cope with it with other traditional RNN like models and they are still in use. Well, the texts are needed to be split so that they can be a proper input in size for the Transformer. Likewise The document segmentation is the second issue, such as splitting a sentence from the middle, then we suddenly lost a significant amount of context. Even though Transformer XL-like model can handle it, we face such problems in the field.
**David Cox**
Thank you Savas Yıldırım!! I really appreciate this answer.
**xnot**
Hello Savas Yıldırım, Meysam Asgari-chenaghlu. What level of proficiency in ML / Engg is expected from the reader here?
**xnot**
What are some of the most creative uses of Transformers you’ve seen?
**Savas Yıldırım**
Hi xnot, thank you for the questions. Our expectation from the reader is that they should have at least experience in machine learning & AI and its culture and that they should have a good programming background.
**Savas Yıldırım**
It has many effective aspects. What surprises me the most and comes to my mind first is that it successfully processes two sentences, which is hardly handled with traditional approaches, at the same time and can even encode cross-sentence anaphoric (reference) relationships thanks to contextual word embedding, which in turn can solve many tough NLP problems with stunning quality. xnot
**Dr Abdulrahman Baqais**
Hi Savas Yıldırım and Meysam Asgari-chenaghlu.
1) Transformer was a breakthrough. Then a Reformer is proposed. What is ahead?
2) Transformer was adopted for computer vision. Do you think it will have the same success as in NLP.
**Savas Yıldırım**
Hi Dr Abdulrahman Baqais. Thank you for your challenging questions.
1) There are some ideas to radically change the attention mechanism and tokenization mechanism. I have seen many articles in this direction. But it will take time to reflect on the industry. Nowadays we see different models in addition to Reformer, which is mostly called efficient transformers, to name a few BigBird, Reformer, Performer, linformer, longformer, and so on… They mostly concern computation and memory efficiency of the architecture.
2)
It has been mentioned in a few articles that the transformer surpasses other architectures (CNN-based) in image processing and signal processing , but again, this may change over time.
There are models published for audio processing and CV on the HuggingFace Hub and they seem to be very successful. Please check!
**State Of The Art**
Hi Savas Yıldırım and Meysam Asgari-chenaghlu. i have few questions for you guys 🙂
1. What library is being used in the book? pytorch or tensorflow?
2. Do you think learning about transformers is enough like don’t we need knowledge about vectorizers, embeddings? like i see people not learning about languages or machine learning and they directly jump to advanced stuffs like applying Sota models or learning about sota models.
**Savas Yıldırım**
Hi State Of The Art
**Savas Yıldırım**
Thanks for the question
**Savas Yıldırım**
1. Transformer models discussed in the book can based on both libraries (framework) thanks to HuggingFace transformers library. In addition to these libraries, many other important ones have been utilized whenever needed, such as sentence-transformers, flairs, Bertviz etc.
**Savas Yıldırım**
1. It’s not enough 🙂 We are proceeding by putting the basic building blocks on top of each other. To develop a good model, it is necessary to know all the building blocks.
We need to be familiar (or even more) with deep learning approaches from a single Perceptron to GPT (175B parameters). Otherwise, We cannot produce new models and build effective SOTA models without understanding the transformation that AI has gone through.
But still, a lot of work can be done without knowing them in detail. However, there can be a blockage in creative and challenging problems.
**Savas Yıldırım**
Meanwhile, we have become able to solve hard problems that were called unsolvable in the past, with 2-3 lines of python code.
**Savas Yıldırım**
🙂
**State Of The Art**
Thank you for the amazing reply. 🙂
**Tim Becker**
Hi Savas Yıldırım, I am a total NLP and transformer newb. I hope you do not mind my simple questions.
**Savas Yıldırım**
Hi Tim Becker thank you for your interest.
**Savas Yıldırım**
1)To put it simply, This architecture effectively represents a sentence using contextual word embedding in a Feed-Forward Neural Net, and also allows transfer learning, allowing us to solve many NLP problems even with very little labeled dataset.
2) In transformers, we use self-attention mechanism when representing a word and sentence. In NLP we use the neighboring words around a word to represent both in traditional and deep learning framework. But, the attention mechanism selectively uses some surrounding words (using weights), not all neighboring words. We repeat this for each word and finally, create contextual word embedding and eventually sentence embedding.
3)
To start simply you can use pipeline module as follows:
`from transformers import pipeline`
summarization = pipeline(“summarization”)
summarization(yourtext)
If you want to get your hands dirty and have a labeled sentiment dataset in any domain, you can repeat our code with your project (fine-tuning phase) in GitHub
[https://github.com/PacktPublishing/Mastering-Transformers/blob/main/CH05/CH05a\_BERT\_fine-tuning.ipynb](https://github.com/PacktPublishing/Mastering-Transformers/blob/main/CH05/CH05a_BERT_fine-tuning.ipynb)
**Tim Becker**
* Could you explain the concept of what a transformer actually is? 😅
**Tim Becker**
* What is the attention mechanism?
**Tim Becker**
* Do you have a toy project in mind that would be good to get started with transformers?
**Max Payne**
Hi Savas Yıldırım,
Could you please elaborate on your sentence in Chapter 1 (Transformers section), where you were referring to tokenization schemes?
`universal text-compression scheme to prevent unseen tokens on the input side`
I mean what does tokenization have to do with compression here?
**Meysam Asgari-chenaghlu**
Hi Max Payne,
Actually it means that most of the tokenization schemes (white space and related ones) can not deal with OoV problem. But instead the BPE and related ones can deal with it. The main idea behind using subwords or byte-pairs comes from text compression. Finding the most common byte-pairs in huge data and then mapping them to smaller representations such as a single byte makes the compression more effective. The same idea is used in transformers tokenizers to make the tokenization more robust. I have seen some other articles that use similar method variations.
Refs:
Philip Gage, A New Algorithm for Data Compression. “Dr Dobbs Journal”
Sennrich, R., Haddow, B., & Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
**Max Payne**
Thanks. I was going through the book and found it a great read.
**Max Payne**
Please correct me if I am wrong.
In - say e.g. CNN - when we use Transfer Learning, like VGG16 on a cat-dog problem, we generally remove then add (or modify) the last layer _AND FREEZE ALL THE PREVIOUS LAYERS_. But while training transformers, on notebooks that I came across, I haven’t seen people freeze all but the last layers. In fact, they just train all the layers. Is there any wisdom behind this or is it technically incorrect (since the HF model has already been pre-trained on a corpus)? What would you recommend?
**Savas Yıldırım**
Hi Max Payne, Transformers are trained in an end-to-end fashion contrary to CNN architecture. We mostly add a task-specific thin layer on top of Transformers. Freezing would deteriorate the system’s performance. You can simply try by saying bert\_model.trainble=false or some specific layers 0..12. However please keep in mind that, each layer has its own characteristic. While some layers encode semantic information (mostly further ones), some layers (mostly initial layers) encode syntactic information. For example, some are experts in reference relations (he->John) . Therefore, you can determine the layers you will freeze according to the down task problem. Once again, freezing is not used much in regular transformer model training. It is worth discussing why.
**Timur Kamaliev**
Hi Savas Yıldırım and Meysam Asgari-chenaghlu! Thanks for your replies for the questions. And I also have one.
As I know today all State-of-the-Art language models based on transformers (like GPT, Megatron etc) are trying to improve their performance by increasing the number of parameters and the related with it computational powers. Do you think this trend will remain? Or maybe some other ways is have already developed?
Many thanks!
[https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing\[…\]orlds-largest-and-most-powerful-generative-language-model/](https://developer.nvidia.com/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/)
**Savas Yıldırım**
Hi Timur Kamaliev Thank you for your question. The contribution of such large models to the system in terms of success is around 2% at most. However, it is important for us that the models should be lighter and more accurate. It is very vital especially to use them on edge devices and other sorts of limited devices. Therefore, models such as efficient transformers are more focused on the speed and memory efficieny of the model.
Another important problem in Transformers is the input size (e.g. 512 tokens). Increasing it is more vital than increasing the parameters at the moment. So what we are looking for is to be able to work with longer inputs and produce lighter but successful models. That is, sounds like a parameter show to me.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# AI-Powered Search – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
AI-Powered Search
-----------------
#### by Trey Grainger, [Doug Turnbull](https://datatalks.club/people/dougturnbull.html)
, Max Irwin
##### The book of the week from 01 Nov 2021 to 05 Nov 2021
Great search is all about delivering the right results. Today’s search engines are expected to be smart, understanding the nuances of natural language queries, as well as each user’s preferences and context. AI-Powered Search teaches you the latest machine learning techniques to create search engines that continuously learn from your users and your content, to drive more domain-aware and intelligent search.
This book empowers you to create and deploy search engines that take advantage of user interactions and the hidden semantic relationships in your content to constantly get smarter and automatically deliver better, more relevant search experiences.
* [Book's page](https://www.manning.com/books/ai-powered-search)
* [Book's GitHub repository](https://github.com/treygrainger/ai-powered-search)
Questions and Answers
---------------------
**Matthew Emerick**
Hey, Doug Turnbull! Thanks for doing this!
The main thing I remember from my AI courses a couple decades ago is the A\* algorithm. Has that been unseated as the “ultimate” search algorithm? Will it ever be?
**Doug Turnbull**
oh ha! A\* is about graph search, no? This book is about natural language search 🙂 Of which I doubt there ever will be an ultimate algorithm given how diverse the space and domains are
**Doug Turnbull**
Hi all, I’m excited to be here. Hopefully I can drag fellow authors here a swell!
**WingCode**
Hi Doug Turnbull , good to have you here again 🙂
1. What are the commonly used “dials and knobs” used in search engine to fine-tune its behaviour? example: Synonym groups to handle domain level business jargons. Who usually controls these “dials and knobs”? Is it the data scientists, business team or someone else?
**Doug Turnbull**
Hey WingCode -
* Basically anything that defines the structure of the index and how it’s queried is open game. Maybe some major groupings?
◦ Stages of ranking from first pass retrieval and later reranking against different criteria / loss functions
◦ Synonyms, stemming, lematization, any kind of NLP between the content and the index (or query and querying the index)
◦ Any kind of statistic that might indicate quality (page rank, sales, clicks, etc)
◦ Really the limit is your imagination!
* I find its best if the domain expert manages direct synonyms, but the relevance/data team has to decide exactly how they interface into the main algorithm
**WingCode**
1. What are the characteristics of your dream search engine? example: For me personally, it is not using any of the facets or “sort by” options. The search engine knows my favorite color is red and usually I look out for the cheapest product out there.
**Doug Turnbull**
If by search engine you mean the underlying search index technology programmable to build a search solution, I want
* A math-oriented, not text-match-oriented, API (see Vespa’s ranking steps)
* An ability mix traditional sparse and dense vector indices for hybrid retrieval
* Doing all those things at high speed
* Declarative configuration, not programmatic configuration, so we can iterate on the search solution independent of the end application
* Built in ability to execute arbitrary python code at query and index time with the classic data science toolkit
**xnot**
Since you mentioned vespa, I’m curious if you would advise picking it over ES as a base search stack ? 🙂
**Doug Turnbull**
Probably yes these days, but I usually don’t recommend people go through extensive search rewrites just for the sake of the underlying index…
**WingCode**
Thank you Doug for the detailed answers
**xnot**
When is a good time to start thinking about investing in LTR capabilities in your search stack?
**Doug Turnbull**
I think you should always think about it, because the limiting factor is training data, and you would want good training data for a non-LTR solution anyway. Once you figure out the training data side the LTR optimization becomes “easy”
**xnot**
Great, does the book cover the event analytics side of gathering relevant training data?
**Doug Turnbull**
Yes! Very much so
**xnot**
Awesome
**xnot**
What are the low hanging fruits in AI powered search - stuff with the highest ROI in short amount of time
**Doug Turnbull**
Anything around query understanding. Can you classify queries into categories? Types of intents, etc based on simple click statistics?
**xnot**
Thanks! I’m looking forward to this part in the book. I have struggled with query understanding because of lack of click data. Simple methods like string matching lead to a lot of weird edge cases
**xnot**
That and spelling correction. I have tried the standard stuff like edit distance and metaphone based algos, but they still fall short of expectations
**xnot**
On a similar note, what are the things which will take a long time to give results, but in the end will be worth all the effort?
**Doug Turnbull**
This question is really hard as its so domain specific, IMO you really gotta work to spike ideas with an experiment before digging too deep. I’ve seen teams really invest heavily upfront, but not see payoff at the end. That’s a big thing to avoid
**xnot**
Quite a few of AI capabilities in search require decent amount of data. How do you deal with this if you are starting from scratch?
**Doug Turnbull**
Data as in clickstream data?
Well earlier I mentioned query understanding as an obvious win. But this classification can also be done through manual labelers (given a sufficiently well formulated task). Of course it breaks down as queries grow in complexity or domain specificity, but that’s a good start.
**xnot**
Ah yes, does the book cover a the proper ways to approach manual labelling? That space also seems to have exploded (so many tools!)
**Doug Turnbull**
Sadly we don’t cover this that much, but its a great topic!
**Adhi**
Hey Doug Turnbull. any thoughts on assessing and tackling position bias through empirical methods like Randpair vs theoretical methods built into LTR models?
**Adhi**
we found that the position biases calculated using [methods like EM](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46485.pdf)
didnt line up with what what we found with randpair - so wanted to know whats more typical in industry. thanks!
**Doug Turnbull**
I haven’t. messed with these, interesting!
Usually my approach is to debias the training data itself using a click model, so I’m not overly coupled to the LTR model itself and these sorts of assumptions…
**Doug Turnbull**
Because then you can take that training data and study it independent of any model, and decide whether it reflects reality
**Carsten Schnober**
In practice, search relevancy can be highly subjective, which makes results hard to evaluate and optimise for actual users. Do you think that “AI-powered search” is affected by this more/equally/less than “traditional” (keyword-based) search?
**Doug Turnbull**
yes it can be highly subjective. I think it means that you have to know, given a keyword, the many possible intents it could have and instead of ranking to one of those intents, you give them a mixture of them. Then as your confidence grows in their intent (through personalization? or just better knowledge about queries), you zero-in on one of the intents…
I wrote an articel about that! [https://opensourceconnections.com/blog/2019/09/05/diversity-vs-relevance/](https://opensourceconnections.com/blog/2019/09/05/diversity-vs-relevance/)
**Alexey Grigorev**
Speaking of labels, what do you think about clickstream data vs manual assessment (via platforms like mturk and similar)? What are the pros and cons for each? And how can we combine the two?
**Doug Turnbull**
clickstream data will be like implicit data, where users tell you what they wouldn’t say outloud - whether because they don’t want to say it out loud, or because they’re not conscious of it!
Mechanical turk is great to overcome cold start problems. But might not reflect the reality of your usecase/app. Especially if your app is domain specific
“Combining” is tough, rather I use them as different perspectives on the problem.
**Alexey Grigorev**
How do we know it’s time to add ML to our search pipeline?
**Alexey Grigorev**
And finally, what’s the easiest way of adding ML to our search pipeline? let’s say we already have search on our platform, and use solr or elasticsearch for indexing all the documents
**Doug Turnbull**
This is really tough, and depends so much on the org. Some teams become Lucene experts and get into the nitty gritty of modifying the guts of the search engine. These teams tend to be very engineering heavy and don’t mind doing this kind of work. This solution is nice because it turns your search engine into a “one stop shop”, without needing extra services, to solve your search problems.
But of course, if you’re more data scientist heavy, you’d prefer to work in python as much as possible 🙂 Such teams tend to build search services that front the search system. The nice thing about this is its not a single point of failure, you can fallover to Solr or Elasticsearch if the service becomes unavailable.
Of course, my dream solution would be a search service that lets me host and run the python stack as the query side, exposing to my python code the underlying data structures. Or something that lets me deploy tensorflow or other models into the engine. The closest things out there are Vespa or Jina from what I’ve seen…
**Doug Turnbull**
I’ll ask a question! What does your search, discovery, or recommendations platform look like at a high level? What do you like, dislike about it? (For example, do you just use Elasticsearch, or do you use Vespa fronted by a Java service? etc etc)
**Alexey Grigorev**
Is it a question for you Doug Turnbull or for us? 😆
**Doug Turnbull**
Oh for all of you sorry 😛
**Alexey Grigorev**
Maybe Cristian Javier Martinez can say a few words about OLX =)
**xnot**
The one I’m working on right now is pretty simple at a high level. LB-> Gateway -> Reco/Search Backend -> ES/Redis
**xnot**
Backend is written in Go. Using official lib of es to connect to it. Redis is used for caching of user historical features / recently watched items which we then use for reco
**xnot**
Don’t use any vector search engine right now. Content size is small enough that we got away with in memory ann index which gets built on service startup (this is used to serve reco)
**xnot**
What I don’t like - quality of query understanding + spelling correction components, lack of good quality labelled data so to build good classifiers, ES’s json based DSL is a pain, want to eventually decouple ANN from this system
**Rishabh Bhargava**
Doug Turnbull thanks for doing this!
I’m curious to understand how the best teams measure and understand performance of their search systems on an ongoing basis. What are the dashboards and alerts they’ll set up, and how do they use those to make incremental improvements to their search models?
**Doug Turnbull**
good question! It takes a lot of different effort and a deep appreciation of the pros and cons of different metrics:
* Human ratings, including judgments (evaluation of relevance of each result) and whole SERP evaluation (how ‘good’ does this search results page look)
* General search conversion rates over time (though these can be influenced by factors like checkout or product page design)
* Search CTR (understanding this is a combination of relevance, UX, perf). Another flaw here is users won’t click if they get their answer from the Search UX itself
* Roundtrip latency to the user and other performance metrics, like p90 latency, etc. Super critical and highly correlated with performance
* Best and worst performing queries -> a great product person can analyze to see what patterns you do well / do poorly with
* Typeahead success: after users clickthrough to a typeahead query suggestion, do they take a follow on action or was the click in the end perhaps not so great
* Content performance: what content does well / poorly in the search system? Are there areas where the content itself needs to be tuned to be more findable
Probably a ton others, but the really great teams, work really really hard here. It takes a lot of great data work to make use of these metrics!
**Rishabh Bhargava**
This is super interesting. Quick follow-up: how do teams quantify best and worst performing queries?
**Tim Becker**
Hi Doug Turnbull, interesting book! I have some basic questions for you.
**Tim Becker**
* Could you please explain what a search engine actually is?
**Doug Turnbull**
omg I have no idea. Some people use the term to mean just the technology that serves results from sparse (classic inverted) or dense vector (approximate nearest neighbor) indices. This is a piece of infrastructure
Other people use the term to refer to a full search solution that solves a specific problem. In the latter case lots of pieces of infra can be involved, not just the index-serving part but query understanding and rerannkig layers
**Tim Becker**
* What kind of ML models are usually applied to search problems?
**Doug Turnbull**
Classically GBDT (Gradient Boosted Decision Trees) have worked well and fast for ranking, as these play well with existing technologies. But increasingly, the problem can be distilled to similarity function modeled in a nearest-neighbor index. This similarity might be the result of an embedding generated from a deep learning or other model
**Tim Becker**
* What was the most exiting AI powered search problem you have been working on?
**Doug Turnbull**
Aside from Shopify, I got to work on the Elasticsearch Learning to Rank plugin that helps power Wikipedia search. Exciting from a tech, data, and impact perspective 🙂 Also very high scale!
**Tim Becker**
cool! Thank you for your answers 🙂
**Adhi**
Doug Turnbull any thoughts on the right way to combine filtering and nearest neighbor search? do we do the former first and then the latter or the other way around? is there a way to do both reliably?
**Doug Turnbull**
I actually don’t know 😅
With a faster NN index, I’d like to do the NN part first as it helps improve recall. Then filter out candidates. But I think combining the two is still an art, and very much an open area of research
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Blueprints for Text Analytics Using Python – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Blueprints for Text Analytics Using Python
------------------------------------------
#### by [Jens Albrecht](https://datatalks.club/people/jensalbrecht.html)
, [Sidharth Ramachandran](https://datatalks.club/people/sidharthramachandran.html)
, [Christian Winkler](https://datatalks.club/people/christianwinkler.html)
##### The book of the week from 18 Oct 2021 to 22 Oct 2021
Turning text into valuable information is essential for businesses looking to gain a competitive advantage. With recent improvements in natural language processing (NLP), users now have many options for solving complex challenges. But it’s not always clear which NLP tools or libraries would work for a business’s needs, or which techniques you should use and in what order.
This practical book provides data scientists and developers with blueprints for best practice solutions to common tasks in text analytics and natural language processing.
* [Book's page](https://www.oreilly.com/library/view/blueprints-for-text/9781492074076/)
* [Amazon](https://www.amazon.com/_/dp/149207408X)
* [Book's GitHub repository](https://github.com/blueprints-for-text-analytics-python/blueprints-text)
Questions and Answers
---------------------
**Asmita**
Hi, my question for the authors is - since there are many tools for NLP practices, how can we understand in depth working of those tools, is it also included in the blueprint?
**Asmita**
Also, in case of different spellings for the same words, and the use of characters such as -à,è,ü,ö and others is there a tool or method to compare and categorize the words into the same, without translating the words into the same language and then analysing?
**Jens Albrecht**
Hi Asmita, our book focuses on best practises how to work with the tools different NLP libraries provide. It still gives some theoretical background on the concepts used, though.
And yes, there is a blueprint for the treatment of different spellings. Just have a look at the notebook of Chapter 3 on Git and search for “character normalization”: [https://github.com/blueprints-for-text-analytics-python/blueprints-text/blob/master/ch04/Data\_Preparation.ipynb](https://github.com/blueprints-for-text-analytics-python/blueprints-text/blob/master/ch04/Data_Preparation.ipynb)
**Max Payne**
Does you book deal with ‘Personally identifiable information’? In either case, any thoughts on this field and tips if possible?
**Quynh Le**
Hi Jens Albrecht, Sidharth Ramachandran, and Christian Winkler thanks for writing the book! Could you share with us about the fields you see the most applications of NLP?
**Christian Winkler**
Good morning everybody!
**Christian Winkler**
Max Payne: We don’t explicitly address data privacy and anonymization. That could be part of the data acquisition process, but it is very complicated and depends on legislation in different countries. Pseudonymization might be a good idea to enable aggregation later. We have quite a few commercial projects where aggregation in the first step solves these issues - but that of course depends on the use case. Apart from data privacy, copyright law in different countries might also be something worth considering.
**Christian Winkler**
Quynh Le: Very interesting question, this probably depends on the field you are most interested in. If you concentrate on enterprise data, uncovering the structure of large document archives might be exciting. In the social sciences, you can avoid performing surveys using NLP and user-generated data. Apart from all these applications, the recent achievements in contextualization using transfer learning are really impressing.
**Asmita**
Hi Christian Winkler Jens Albrecht Sidharth Ramachandran, What is your opinion on automating NLP processes? Will automating them lead to some percentage of loss when compared to a human validating the process step by step?
**Nikhil Shrestha**
Hi Christian Winkler Jens Albrecht Sidharth Ramachandran, thank you writing this book.
Recently I have been working on some text analytics projects and all of it is very interesting.
So motivation for my question is that we usually see only one side of the problem we are facing. Either the negative side or only the positive side. Biggest example is when we talk about any sort of safety analysis we only look for accidents that have happened rather than researching about the positive aspect of the safety which includes campaigns, awareness drives and we never dig deeper into those aspect basically.
My question is if I want to layer the findings of my analysis over time say for example: over a period of time how did a specific topic say climate (as in terms of semantics, popularity, understanding etc basically negatives AND positives) have changed.
Here I am using climate but we could think of this problem in very wide spectrum say a product, software, book, organization etc.
Hence, is there a way that we could work around something like this?
Thank you in advance.
**Christian Winkler**
Thanks for your question, Nikhil Shrestha. You could use semantic embeddings like word2vec for supporting information retrieval. Another option would be to use something more full-fledged like txtai.
**Christian Winkler**
Good question, Asmita! I think it depends on the kind of task you are automating. Nowadays, language models based on transfer learning can achieve human performance (some are even superior to humans). However, irony and sarcasm is still difficult to interpret. In the unsupervised or semi-supervised regime, there is no real loss, it just makes interpretation easier. That’s also what we see in our daily projects - we are using AI more as a tool to supercharge humans.
**Nikhil Shrestha**
Thank you Christian Winkler for swift reply on the question, will check the resources and update you accordingly.🙂
**Christian Winkler**
Hope that helps Nikhil Shrestha and I haven’t misunderstood your question
**Nikhil Shrestha**
Yes Christian Winkler it pointed me to the direction which indeed will help
Thank you for that.
You totally understood the question 🙏🙂
**Christian Winkler**
We would be happy to answer further questions. Feel free to also ask questions about NLP in general - it does not have to be directly related to our book.
**Natasha john**
Hi everyone,
any suggestion for a book talking about self driving cars! 😁
**Alexey Grigorev**
| | |
| --- | --- |
| I think <#C01AXGTRESH | books> would be a better channel for this |
**Alexey Grigorev**
But if you already know a book about self driving cars and would like to invite the authors here to this channel, please let me know =)
**Natasha john**
Of course Alex, thanks for helping 🙏
**Maja**
Hello Christian Winkler Jens Albrecht Sidharth Ramachandran! 👋 Thank you for writing this book full of practical solutions. I just have to ask you for your opinion on transformers in NLP (BERT, RoBERTa GPT-3) advantages and disadvantages. Did you use them?
**Sidharth Ramachandran**
Thanks for the question Maja. We do make use of the BERT model in one of the chapters of the book. In fact I’m also seeing the use of transformer based models heavily in the industry. They work surprisingly well without much preprocessing. They are however not good across languages but here too there are variants that the community releases online that are helpful. The question of bias in these models must be tackled though and it depends on each use-case to determine how critical this is.
**Maja**
Thank you so much Sidharth Ramachandran for the answer. I’ll look into it!
**Christian Winkler**
Finetuning BERT and RoBERTa models often leads to quite similar results. A good starting point is the Huggingface page with all the models [https://huggingface.co/models](https://huggingface.co/models)
**Christian Winkler**
GPT-3 is a different story as it is a commercial model which is mainly used for generating text. Alternatively, you could take a look at GPT-J from EleutherAI.
These models are normally too large to train them on your own or even fine-tune them.
And they are still growing. Last week, NVIDIA and Microsoft announced the “Megatron-Turing Natural Language Generation model” (MT-NLG) with a whopping 530 billion parameters.
**Maja**
Thank you Christian Winkler so much for everything! I’m going to NVIDIA conference, start reading your book when I get it and I have started with Hugginggace.
**Asmita**
Thank you Jens Albrecht, Sidharth Ramachandran Christian Winkler For your time!
**Nikhil Shrestha**
Thank you Jens Albrecht Sidharth Ramachandran Christian Winkler for sharing the knowledge and clearing our doubts.
**Maja**
Thank you so much Jens Albrecht, Sidharth Ramachandran and Christian Winkler for all your guidance and fast replies to our questions!
**Quynh Le**
Thank you Jens Albrecht, Sidharth Ramachandran, Christian Winkler for coming to share about NLP!
**Tim Becker**
Good morning Jonathan Rioux, very interesting book! I have a few beginners questions.
**Mansi Parikh**
from another beginner, these were really nice to read so thanks for asking, Tim, and thanks for answering, Jonathan!
**Tim Becker**
* When should I start using Spark? I mean how large should my dataset be? Does it make sense to start using Spark, if my dataset still fit into memory, but I expect the size to increase?
**Jonathan Rioux**
Hi Tim!
This is an _excellent_ question 🙂 I don’t have a straight answer, but let me share with you the heuristics that I use when deciding for myself.
1. PySpark is getting much faster for single-node jobs, so you might be able to have acceptable performance with Spark on a single node right off the bat! See the following link about a discussion about this. [https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html](https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html)
2. Koalas was introduced in Spark 3 and merged into `pyspark.pandas` as of Spark 3.2. Now more than ever, you can convert Pandas code to PySpark with a lot less fuss. 🙂
**Jonathan Rioux**
1. For memory allocation, I try to have a cluster with enough memory to “store” my data and have enough room for computation. Data grows quite fast, and if you have a feel that the data source will grow (for instance, historical data), I find it easier to start with PySpark, knowing it’ll scale.
If you need a fast an loose rule for processing data (not counting ML applications), I would say that if you can’t get a single machine with 3-5x the RAM your data sits on, you probably want to reach for Spark, just for comfort.
**Tim Becker**
* How much worse is the performance with pySpark if it used on a small dataset.
**Jonathan Rioux**
I think I replied on your previous question 🙂 . The “Spark single-node performance tax” shrunk _dramatically_ since Spark 3.0 and even more since Spark 3.2.
[https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html](https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html)
In practice, I find that with very small data sets (a handful of hundred of rows) you will have much worse performance depending on the operations: that being said, it’s often a difference of 0.29 sec vs 0.85 sec which I am not too concerned about.
**Tim Becker**
* What are the advantages of using databriks?
**Jonathan Rioux**
Databricks has many things for itself!
1. Databricks provides proprietary performance improvements over open-source Spark so your jobs may run faster with no changes. I am especially excited about Photon ([https://databricks.com/product/photon](https://databricks.com/product/photon)
) which takes your Spark data transformation code through a new query engine.
2. The notebook experience out of the box is quite good (and I am saying this from the perspective of a person who doesn’t really like notebooks). I like being able to create ad-hoc charts from a result data frame and explore my data right from the same interface.
3. Databricks connect ([https://docs.databricks.com/dev-tools/databricks-connect.html](https://docs.databricks.com/dev-tools/databricks-connect.html)
) is the simplest (to me) way to connect my IDE on a remote cluster with the minimum amount of fuss. It can be a little capricious, but when writing the book, I’ve used much worse hacks to connect to a remote REPL with Spark enabled…
4. Databricks provides additional capabilities (Delta Lake for data warehousing, MLFlow of ML model/experiments management, etc.) which play well with the overall ecosystem.
5. The ecosystem is quite consistent around all three major cloud providers (AWS, Azure, GCP), which help if you’re moving around. :)
**Tim Becker**
* If we want to train ML models using pySpark, does the model have to support distributed training?
**Jonathan Rioux**
Spark’s ML model collection all work out of the box. They are all listed here: [https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html](https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html)
.
Some algorithms naturally lend themselves better to distributed computing and perform (runtime) much better than other. Random Forest for instance distributes super well, GradientBoosting a little less so.
On top of that, you can also use user-defined functions (UDF) to run single-node models in a distributed fashion (the model would run on a single node here). This allows for parallelizing hyper-parameter selection. I am considering to write an article/do a video on the topic as it is quite fun to do!
**Tim Becker**
thank you 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Analysis with Python and PySpark – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Analysis with Python and PySpark
-------------------------------------
#### by [Jonathan Rioux](https://datatalks.club/people/jonathanrioux.html)
##### The book of the week from 25 Oct 2021 to 29 Oct 2021
When it comes to data analytics, it pays to think big. PySpark blends the powerful Spark big data processing engine with the Python programming language to provide a data analysis platform that can scale up for nearly any task. Data Analysis with Python and PySpark is your guide to delivering successful Python-driven data projects.
Packed with relevant examples and essential techniques, this practical book teaches you to build lightning-fast pipelines for reporting, machine learning, and other data-centric tasks. No previous knowledge of Spark is required.
* [Book's page](https://www.manning.com/books/data-analysis-with-python-and-pyspark)
* [Book's GitHub repository](https://github.com/jonesberg/DataAnalysisWithPythonAndPySpark)
Questions and Answers
---------------------
**Tim Becker**
Good morning Jonathan Rioux, very interesting book! I have a few beginners questions.
**Mansi Parikh**
from another beginner, these were really nice to read so thanks for asking, Tim, and thanks for answering, Jonathan!
**Tim Becker**
* When should I start using Spark? I mean how large should my dataset be? Does it make sense to start using Spark, if my dataset still fit into memory, but I expect the size to increase?
**Jonathan Rioux**
Hi Tim!
This is an _excellent_ question 🙂 I don’t have a straight answer, but let me share with you the heuristics that I use when deciding for myself.
1. PySpark is getting much faster for single-node jobs, so you might be able to have acceptable performance with Spark on a single node right off the bat! See the following link about a discussion about this. [https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html](https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html)
2. Koalas was introduced in Spark 3 and merged into `pyspark.pandas` as of Spark 3.2. Now more than ever, you can convert Pandas code to PySpark with a lot less fuss. 🙂
**Jonathan Rioux**
1. For memory allocation, I try to have a cluster with enough memory to “store” my data and have enough room for computation. Data grows quite fast, and if you have a feel that the data source will grow (for instance, historical data), I find it easier to start with PySpark, knowing it’ll scale.
If you need a fast an loose rule for processing data (not counting ML applications), I would say that if you can’t get a single machine with 3-5x the RAM your data sits on, you probably want to reach for Spark, just for comfort.
**Tim Becker**
* How much worse is the performance with pySpark if it used on a small dataset.
**Jonathan Rioux**
I think I replied on your previous question 🙂 . The “Spark single-node performance tax” shrunk _dramatically_ since Spark 3.0 and even more since Spark 3.2.
[https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html](https://databricks.com/blog/2021/10/19/introducing-apache-spark-3-2.html)
In practice, I find that with very small data sets (a handful of hundred of rows) you will have much worse performance depending on the operations: that being said, it’s often a difference of 0.29 sec vs 0.85 sec which I am not too concerned about.
**Tim Becker**
* What are the advantages of using databriks?
**Jonathan Rioux**
Databricks has many things for itself!
1. Databricks provides proprietary performance improvements over open-source Spark so your jobs may run faster with no changes. I am especially excited about Photon ([https://databricks.com/product/photon](https://databricks.com/product/photon)
) which takes your Spark data transformation code through a new query engine.
2. The notebook experience out of the box is quite good (and I am saying this from the perspective of a person who doesn’t really like notebooks). I like being able to create ad-hoc charts from a result data frame and explore my data right from the same interface.
3. Databricks connect ([https://docs.databricks.com/dev-tools/databricks-connect.html](https://docs.databricks.com/dev-tools/databricks-connect.html)
) is the simplest (to me) way to connect my IDE on a remote cluster with the minimum amount of fuss. It can be a little capricious, but when writing the book, I’ve used much worse hacks to connect to a remote REPL with Spark enabled…
4. Databricks provides additional capabilities (Delta Lake for data warehousing, MLFlow of ML model/experiments management, etc.) which play well with the overall ecosystem.
5. The ecosystem is quite consistent around all three major cloud providers (AWS, Azure, GCP), which help if you’re moving around. :)
**Tim Becker**
* If we want to train ML models using pySpark, does the model have to support distributed training?
**Jonathan Rioux**
Spark’s ML model collection all work out of the box. They are all listed here: [https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html](https://spark.apache.org/docs/latest/api/python/reference/pyspark.ml.html)
.
Some algorithms naturally lend themselves better to distributed computing and perform (runtime) much better than other. Random Forest for instance distributes super well, GradientBoosting a little less so.
On top of that, you can also use user-defined functions (UDF) to run single-node models in a distributed fashion (the model would run on a single node here). This allows for parallelizing hyper-parameter selection. I am considering to write an article/do a video on the topic as it is quite fun to do!
**Tim Becker**
thank you 🙂
**Tim Becker**
Thank you 🙂
**adanai**
Hello Jonathan, just the kind of book I am interested to read further on!
I have been using Python (pandas/sklearn) for small-medium data related tasks (analysis / ML) and it is the only programming language I have some amount of exposure to.
In order to get started and work further with PySpark, do I need to have some prerequisites in understanding the spark architecture in its native implementation for working with big data?
Also, what is the approach to be taken to work with the scala/java implementation in case the switch has to be made?
**Jonathan Rioux**
Hello!
TL;DR: I wrote my book for someone like you! 🙂
You don’t have to use Java/Scala to use PySpark, but you are right in saying that the architecture and code approach can be different.
In Part 1 of my book, I discuss about how you can change your way on thinking about transformations vs. actions. Later, in Chapter 11, I discuss about narrow vs. wide transformations to help with understanding why the data needs to move from one node to another.
Everything that you’ve learned so far still carries to the big data world, fortunately 🙂 With my students and employees, the biggest “click” happens when they understand data locality (which data points need to be where in order for an operation to succeed) and they learn to read query plans (Chapter 11 as well) and structure code in a readable fashion.
If you really need to use Java, my now friend Jean-George Perrin wrote _Spark In Action_ from the same publisher and with a more data engineering point of view. You’ll find that the Spark API looks the same regardless of the language.
**José Bastos**
Hi Jonathan, books seems great. Here are my questions:
* What’s the data type you struggle the most? And the one you have more fun with?
* What is the most common error/assumption people do when using PySpark and general ML pipelines?
* Best tip to make training more efficient?
Thank you for your time!
**Jonathan Rioux**
Hi José!
_What’s the data type you struggle the most? And the one you have more fun with?_
My favourite by far is multi-dimensional/hierarchical/document data (think JSON). I don’t know why, but having that data model within the data frame (highly performance nested data frames) is _awesome_. It sometimes also feels like a puzzle to extract and work with the data the best way and I like a challenge 😄.
Binary data in Spark can be a little bit of a pain (think audio, video, images, etc.). Most of the work revolves around treating everything like an independent unit (you could even use a RDD for this). I like doing ML on unstructured data, but prepping in a distributed fashion can sometimes feel a little bit of a pain.
**Jonathan Rioux**
_What is the most common error/assumption people do when using PySpark and general ML pipelines?_
Easy 🙂 _Being too precious with their data pipelines/not planning and iterating_. Most data scientists I encountered are very hesitant about training a model. I usually go straight with a simple pipeline that takes the features ready to go and use a simple model. This
1. Serves me as an early benchmark.
2. Removes some of the magic of the model and helps me destress about metric anxiety.
3. Ensures I have a complete modelling pipeline working!
ML pipelines are super cool because you build the components independently of their application. It makes it easy to add/remove some as you go. Because of this, you are responsible to strike the right balance of planning and flexibility. I keep a notebook of what I want to do, what I expect as results, and I build my code as I go.
(Plus, if your model takes time, you can work on the next iteration while it fits!)
**Jonathan Rioux**
_Best tip to make training more efficient?_
Do you mean “speed up PySpark ML training”?
1. Read the SparkUI when working with ML model. It’ll help see the order of operations and review if you are spilling a lot of data.
2. Sometimes, you can get a _whole lot more performant training_ by saving the data frame in Parquet on disk and reading it again. It’s like a _super-cache_ that clears the previous operations.
3. _Use enough memory and compute._ Remember that Spark uses memory for both storage of the data and compute (+ the rest that makes a computer work). Don’t skimp on memory, even if it means using a smaller cluster for data transformation and then going all ham on ML.
**Asmita**
Hi Jonathan Rioux, I have been working on simple projects as of now. Will this book be a good start for learning about PySpark? Also, are there any specific hardware requirements for running?
I read your message about transfirmations vs actions, but is it useful for someone who is leaarning from scratch and has no idea abojt PySpark and its workings?
**Jonathan Rioux**
Hi Asmita!
If you know some Python, then my book will (I hope!) be useful yes. 🙂 It is to get started with PySpark.
I recommend a computer with at least 8GB of RAM (if you work locally) or access to a cloud provider Spark (there are some free offering, such as Databricks community edition).
Transformations vs. Actions is a pretty important concept in understanding how Spark processes data. It is explained right off the bat in Chapter 1, so you’re not expected to know anything about it first.
**Asmita**
That’s amazing! Thank you for answering my question!
**WingCode**
Hi Jonathan Rioux , thank you for doing this Q&A
1# How do you estimate the resources needed for a spark job? Is there a tool out there which looks at the volume of data with the spark code to determine the number of executors, cpu & memory needed? Or is it some rule of thumb and repeated iteration to tune out the best config?
**Jonathan Rioux**
Hi Wingcode!
Great questions!
#1: As far as I am concerned, no such thing exist at the time…
Size/volume of the data is important (both disk space and memory, because of potential spills), but also the type of operations you plan on doing. If the data can be logically represented by small-ish (single or a few nodes), your Spark code might need less resources because there will be less merry-ing around of the data.
As you write your own programs, and use the Spark UI to review the resources used by the cluster, you’ll be able to adjust as needed. I remember getting a little frustrated at first because it can look like a guessing game, but you end up building your own mental model for data usage.
As an example, for time series (~14 TB of data) using ML modelling and heavy feature engineering, I used comfortably 25 machines x 60 GB of RAM. Some spill, but it was good enough for me.
**WingCode**
2# In spark local mode vs cluster mode (YARN) what are the advantages in terms of my job performance? Other than:
a. Data locality. If my RDD partition and HDFS block is on same node hence no shuffle
b. Reliability. If my local executor is flaky, then my whole job performance/reliability suffers. YARN -> multiple machine hence flaky node can be mitigated.
c. Vertical scaling limit. If I need very high memory, CPU counts which the cloud provider doesn’t offer on single instance.
If I manually set num of partitions (via coalesce or repartition) in local mode I can make multiple tasks executing in parallel? Wouldn’t this be faster than cluster mode?
**Jonathan Rioux**
I think you hit the nail on the head. Spark in cluster mode (whether YARN or k8s, I think the other ones are not used much…), you gain the ability to scale beyond a single machine.
`coalesce()` is, to me, mostly useful when you want to limit the number of partitions (for instance, before writing to disk. For “short lived” data programs, I usually don’t use it much as Spark is pretty smart about reshuffling data in the best way possible. When using older version of Spark, it helped, but since Spark 3.X I’ve been pretty keen on relying on the default behaviour 🙂
The whole essence of Spark is its ability to scale horizontally. If you have a single beefy machine, you might yield better performance from another tool (although Spark single node is pretty speedy, see my previous answers).
**WingCode**
3# How significant is the overhead between using Scala for spark versus Python for spark (since Python data structures have to be converted to Scala datatypes) ? Is it worthwhile to convert frequently run Pyspark in production to Scala spark for performance & better resource utilisation?
**Jonathan Rioux**
When using RDD and regular Python UDF, you’ll pay a pretty significant performance tax from serialization/deserialization (serde) from Spark’s data model to something Python can consume.
The data frame API, even in Python, maps to JVM instructions and performance is quite similar to Spark in Java/Scala.
Now, with Pandas UDF and Arrow serialization, you can use Python/Pandas code within PySpark with minimal overhead. Python will not always be as fast as Java/Scala (Pandas/NumPy owe much of their speed to highly optimized low-level code), but the gap is narrowing.
**WingCode**
4# Do you think serverless Spark ([https://cloud.google.com/solutions/spark](https://cloud.google.com/solutions/spark)
) is going to be the next big thing for data analysis & ML similar to managed Kubernetes engine? Do most of the orgs in your experience, run private Spark clusters or use some cloud offering (DataStax, Databricks)
**Jonathan Rioux**
I find the expression “serverless Spark” quite hillarious 😄 because you always are concerned about the servers being used. Most serverless Spark offering rebrand the compute+memory units, which is counter productive as most Spark users understand RAM quantities pretty well.
I think that managed Spark is a pretty seducing option for data analysis and data science. Databricks is the dominating force here, but I’ve seen some great success/used other managed Spark product.
Looking at the link you provided, I think it’s a pretty seducing offering for the cloud provider because you give them the keys to scale for you. In practice, your IT organization might set pretty stiff limitations there. I also have not seen auto-provisioning that was so fast that I was convinced to move away from “provisioning the cluster myself”.
**WingCode**
Jonathan Rioux Thank you for all the super detailed answers. 🙂
**Jonathan Rioux**
My pleasure 🙂 Thanks for the excellent questions!
**Jonathan Rioux**
Hi everyone!
As for memory usage, especially when I get to scale a cluster, I use this image to remind me of the cluster memory vs. the available memory. On top of this, remember that Spark will spill to disk (which is slower, but not catastrophic).
When scaling a cluster, I found that most people try to skimp on memory. If you plan on using TBs of data, it’s not silly to have a lot of RAM (depending on the operations/transformation/modelling), you won’t have fun if you total 500GB of RAM across your cluster.
**Lavanya M K**
Hi Jonathan Rioux
1. Which is the best tool to use pyspark for creating eda and data visualization? I usually use zeppelin to do this.
2. Is there any difference in terms of resources utilisation or speed between using scala-spark and pyspark?
**Jonathan Rioux**
Hello Lavanya!
_#1_ : Within databricks, you can create charts from data frames pretty easily. It uses [plot.ly](http://plot.ly/)
in the back-end. PySpark also has the `summary()` and `describe()` methods you can use for diagnosing the data (at a high-level) within columns.
I haven’t used Zeppelin in a long time (and only for the notebook interface) but Jupyter (Spark open-source) or databricks notebook (databricks) should provide the same functionality.
**Jonathan Rioux**
_#2_ : With the data frame, PySpark and Spark are pretty similar, under the hood, PySpark will call the same JVM methods as Spark/Java. In some cases (for instance, when using the RDD or UDF), you will have differences (because you’re using plain Python/Pandas code and not the optimized Spark routines).
If you are more comfortable using Python, PySpark will serve you quite well!
**Lavanya M K**
1. How efficient is it to use spark UDFs to perform data processing?
**Jonathan Rioux**
I distinguish two types of UDF
_Python UDF_ are used with the `udf()` function/decorator. They merry the data from Spark to Python and then use Python to process the record. Those UDF are usually slower because of the data serialization/deserialization and also because Python can be slower than Scala/Java.
_Pandas UDF_ are used with the `pandas_udf()` function/decorator and are much faster than Python UDF. They use Arrow for converting the data from Spark to Pandas. Furthermore, Pandas can be very fast when using vectorized instructions.
UDF are useful when you need to do something that is hard to do with the PySpark API. See them as a specialized tool that unlock additional capabilities, not as a replacement for Spark’s core API.
In terms of efficiency, I read articles claiming that Pandas UDF can sometime be more efficient than Spark core data transformation API (!). I have not seen this in practice, usually witnessing a small slowdown (going through a few examples I wrote for fun — don’t take this as an official benchmark! — I see anything from 0.9-10.2x. I still use them quite a bit for my own data analysis and modelling. :)
**Jonathan Rioux**
Additionally: I often use UDF / Pandas UDF for promoting local Python/Pandas code to PySpark. It allows to scale the code (although more slowly) without needing a re-write. Bonus for me since I can use the old “promoted function” as test when rewriting in “native” Spark. 🙂
**Alexey Grigorev**
I remember a few years ago Apache Zeppelin was getting some traction as a nice and easy way to do analytics with Spark. But I haven’t heard about it for a while. Is it still used? What do you think about it?
**Alexey Grigorev**
I’ve seen in a thread that you mentioned that Jupyter has the same functionality. So it means people prefer to use it more often than Zeppelin?
**WingCode**
People in our org use both the ways (Zeppelin + Scala OSS Spark, Jupyter + PySpark) . Personally feel, Zeppelin is one of the best integrations out there if you want to use Scala + OSS Spark with in built viz, versioning, easy configuration of spark (mem, cpu, spark mode, reuse same interpreter across users, user access controls)
If you need such features from Spark + Jupyter, I think you need to manually configure so many things which is a pain.
**Jonathan Rioux**
I have not used Zeppelin in a very long time (~2 years) so my opinions are tainted a little.
Most cloud offering has Jupyter integration baked in which is ok. Databricks still has IMHO the best notebook integration. I think that Zeppelin/Livy had the option to work cross-languages for the longest time, but it’s not something I used extensively.
**Jonathan Rioux**
In the end, I think that comfort level and habit are quite important. There can be a little bit of tool fatigue at times 🙂 But WingCode is right that something configuration/configurability can be a pain.
**Lavanya M K**
Another drawback of zeppelin is it gets slow in rendering when creating visualisation
**WingCode**
Hi Jonathan Rioux,
I had few more questions 🙂
#1 Is there any advantages using HDFS 3.X over HDFS 2.X with Spark?
**Jonathan Rioux**
Good question!
I think that Spark uses Hadoop only for storage, so if Hadoop 3.0 is faster, all the better. In practice, I’ve used Spark mostly in a cloud setting (GCS, S3, Blob Storage, DBFS, BigQuery, etc.): it really boils down to where your data is and how you get Spark provisioned.
This might be a little out of my wheelhouse but I hope it answered your question!
**WingCode**
Yes answered my question 🙂 Thank you Jonathan
**WingCode**
#2 Do you recommend any other profiling tool for OOS spark jobs other than the information we get through spark Web UI ?
**Jonathan Rioux**
I’m not sure if it’s available standalone still anymore, but I was impressed by the work on the Spark UI the folks at data mechanics ([https://www.datamechanics.co/](https://www.datamechanics.co/)
) did. This is the only other product I’ve used and can comment on.
Update: [https://www.datamechanics.co/delight](https://www.datamechanics.co/delight)
← there you go, it’s over there :)
**WingCode**
This is a cool UI, unfortunately the dashboard server is not open-source. Thank you for the link Jonathan 🙂
**WingCode**
#3 Do you think users of Spark need to have some basic understanding of the internals to use it better compared to other data processing tools like Pandas, Numpy?
**Jonathan Rioux**
I think that every library user should read the documentation/data model of what they use 😉
Spark has a pretty straightforward data model abstraction (the data frame is easy to understand, yet very flexible). For starters, understanding this is enough.
As you go along, it’ll be insightful to gain deeper knowledge of how and where the data is processed, partitions, skew, etc. as it’ll help profile and reason about your programs. It’s not necessary (IMHO) at the start, but becomes important if you want to really grok Spark.
**Doink**
Alexey Grigorev I am seeing some wonderful discussions happening here, wondering if these discussions are saved in a separate document for future reference?
**Alexey Grigorev**
They are!
This is how it looks for previous books - [https://datatalks.club/books/20210517-grokking-deep-reinforcement-learning.html](https://datatalks.club/books/20210517-grokking-deep-reinforcement-learning.html)
**Doink**
[https://datatalks.club/books/20211011-mastering-transformers.html](https://datatalks.club/books/20211011-mastering-transformers.html)
here I didn’t see anything hence asked.
**Alexey Grigorev**
I’ll publish the answers eventually. It takes some time, that’s why I batch-process it =)
**Doink**
Oh okay got it thank you so much for this, is there a way we can contribute like in form of financial donations? You are really doing great work!!
**Alexey Grigorev**
Thanks for asking! Yes there’s a way to do it: [https://github.com/sponsors/alexeygrigorev](https://github.com/sponsors/alexeygrigorev)
**Wendy Mak**
Out of curiosity, is this automatable ?
**Alexey Grigorev**
I have a script that generates a yaml with messages. It automates like 90% of the work.
The part that’s not automated yet is
* Selecting the week
* Cleaning the responses a bit, like removing the announcements
* Putting the yaml to the website
**Wendy Mak**
now wondering if the rest of the 10% is doable with github actions 😂
**Alexey Grigorev**
They probably are, I’m sure it’s automatable. Maybe the second one is a bit tricky though
**Lalit Pagaria**
Second one good problem for zoomcamp 🙂
Anyway please for rest of the tasks (at least coding not ML) if you hands let me know
**Hironori Sakai**
Hi Jonathan Rioux . I have a question about dependency management for PySpark. When we need a library for a PySpark Job, then we may 1) install the library on all nodes or 2) submit all required library with `--py-files` option. I do not think that these options are realistic if the dependency is quite large. (e.g. install/update `spacy`. It requires around 30 libraries.)
My question is: what is the best approach to submitting a PySpark Job requiring a large dependency?
(If we use Scala, we do not have such a problem, because we can pack all needed packages in a JAR file.)
**Jonathan Rioux**
This is such a good question! `--py-files` IIRC does not work with libraries that have C/C++ code, such as `spacy` .
It boils down to which environment you are using. Are you on a cloud installation (databricks, EMR, Glue, HDInsights, Dataproc, etc.) ? If this is the case, you can specify and install dependencies at cluster creation time or runtime. This also has the advantage that you can “version” the dependencies of your cluster. I used dataproc (GCP) for a pretty significant project a few years ago and this is the route we took. Databricks has `dbutils` where you can install libraries on the cluster with a simple command from the notebook.
(Databricks, on top of this, has runtime versions that has predictible libraries and versions. I am a big fan of those, just to avoid thinking about which one to use 😄 see for example spacy 3.1.2 on databricks runtine 10.0ml: [https://docs.databricks.com/release-notes/runtime/10.0ml.html](https://docs.databricks.com/release-notes/runtime/10.0ml.html)
)
If you are “on-prem” or the cluster is not meant to be ephemeral, you need to be a little more careful with dependencies management. Again, this is product dependent (what do you use to manage your hardware provisioning) but I would assume that orchestration tools can help with this. I think that, in this case, you need to be more conservative with your dependencies to avoid some clash if multiple users are there.
This is one of the cases where I envy the JVM/jar world… 🙂
**Jonathan Rioux**
See `dbutils` here: [https://databricks.com/blog/2019/01/08/introducing-databricks-library-utilities-for-notebooks.html](https://databricks.com/blog/2019/01/08/introducing-databricks-library-utilities-for-notebooks.html)
**Jonathan Rioux**
Another example: installing dependencies on GCP dataproc.
[https://cloud.google.com/dataproc/docs/tutorials/python-configuration#image\_version\_20](https://cloud.google.com/dataproc/docs/tutorials/python-configuration#image_version_20)
**Hironori Sakai**
\> I would assume that orchestration tools can help with this.
Thanks for your answer! I did not have this viewpoint.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Generative AI with Python and TensorFlow 2 – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Generative AI with Python and TensorFlow 2
------------------------------------------
#### by Joseph Babcock, [Raghav Bali](https://datatalks.club/people/raghavbali.html)
##### The book of the week from 08 Nov 2021 to 12 Nov 2021
In this book, you’ll explore the evolution of generative models, from restricted Boltzmann machines and deep belief networks to VAEs and GANs. You’ll learn how to implement models yourself in TensorFlow and get to grips with the latest research on deep neural networks.
* [Book's page](https://www.packtpub.com/product/generative-ai-with-python-and-tensorflow-2/9781800200883)
* [Amazon](https://www.amazon.com/Generative-AI-Python-TensorFlow-Transformer/dp/1800200889)
* [Book's GitHub repository](https://github.com/PacktPublishing/Hands-On-Generative-AI-with-Python-and-TensorFlow-2)
Questions and Answers
---------------------
**Kshitiz**
Hi Raghav Bali and Joseph. Thanks for doing this. I am curious to know -
1. What problems Generative models can solve right now?
2. What are some latest developments in the field of Generative models?
3. With deepfakes being a think now, do you think the world has enough artillery to deal with it?
**Raghav Bali**
Hey Kshitiz, interesting and important questions… Here’s my take:
1. Generative models are being leveraged for a numbers applications currently. Some popular examples are: Painting/Art Generation (a number of neural artists are all the rage nowadays), FaceGeneration(check [thispersondoesnotexist.com](http://thispersondoesnotexist.com/)
) for stock photos, music generation for high quality background scores (free from copyright issues), style transfer for fun usecases (Snapchat/Instagram filters) , dataset augmentation (for classification usecases) and a number of other domains.
2. Recent Developments in the field of Generative Models: well this is quite a broad question but a number of amazing works are focusing on improving the output quality with far lesser infra and training time. GANs are being improved to be more stable , lean yet generate extremely high quality outputs. See the likes of StyleGAN3 from folks at NVIDIA. We cover quite a few of these recent architectures in our book as well.
3. Well, with any powerful technology, there is always a danger of it being used improperly. Researchers ans practitioners like us have a huge task at our hands to make people aware of stuff like deepfake. Apart from awareness, a number of research labs are focusing on ways of identification of fake(deep fake) content (though there is a long way to go for this). In short, tldr; we are not there but efforts are being made nonetheless
**Kshitiz**
Raghav Bali Thanks for the responses!
**Cam Buchanan**
Hi Raghav and Joseph, interesting stuff thanks for taking the time. I’m curious when making content in the creative field based on generative models, are there any methods you use to avoid “traps” like copyright infringement?
**Raghav Bali**
Hey Cam Buchanan
Apologies, looks like I somehow missed answering this very interesting and pertinent question.
Copyright infringement (and other aspects of law are quite varied and dependent on interpretation). But keeping the nuances aside, the following are some rules of thumb to keep in mind when generating content using models:
1. Generative Models are searching the training space at a very abstract level. There is a very highly chance the output would be derivative of the training space. Even solutions like Github’s autopilot and the likes have raised similar questions
2. Make use of available tools for a quick check. For instance, uploading content on soundcloud, youtube, etc, you should pay attention to the inbuilt copyright checks. If your content is getting flagged, go back to drawing board. But again, these tools are not always foolproof
3. Always mention a caveat or a disclaimer on how you generated this content and if someone claims their copyright, best to oblige or collaborate (if it is indeed the case)
**WingCode**
Hi Raghav Bali, Good to have you here again 🙂
I was wondering whether Generative AI can help in video summarisation within each frame of context. ex: In YouTube videos it is called video chapters ([https://techcrunch.com/2020/05/28/youtube-introduces-video-chapters-to-make-it-easier-to-navigate-through-longer-videos/](https://techcrunch.com/2020/05/28/youtube-introduces-video-chapters-to-make-it-easier-to-navigate-through-longer-videos/)
) where we have to manually create the window with it’s relevant text summary. This would be helpful for DTC since we do it manually 😄
Does it have a specific name in ML research where you generate a super short 2-5 words summary for long piece of video, text or audio?
**Raghav Bali**
Glad to have you back in this discussion WingCode. Excellent question n I can shamelessly admit that it took me down a rabbit hole. I am still looking to find more details on this (haven’t got too far yet), but here’s my take:
1. Video segmentation sounds like an apt name for it but fortunately or unfortunately it refers to segmenting objects within a given video frame, so we might have to get creative here. Let’s brain-storm till we find some papers detailing this?
**Raghav Bali**
1. contrary to your mention, it seems youtube started creating video chapters automatically using “ML”. The support pages do not detail much about it though. See here: [https://support.google.com/youtube/answer/9884579](https://support.google.com/youtube/answer/9884579)
Seems like the manual stuff for DTC can be managed through this feature? The documentation says that the service does the segmentation based on different text in the video to generate titles etc but I am pretty sure there is more here than meets the eye
**Raghav Bali**
1. Creating summary from a video frame sounds similar to the task of image captioning. There are a number of works which do this quite nicely, starting points are: [https://arxiv.org/abs/1411.4555](https://arxiv.org/abs/1411.4555)
, [https://arxiv.org/abs/1601.03896](https://arxiv.org/abs/1601.03896)
**WingCode**
Thank you Raghav Bali for digging up all the resource and the elaborate answers as usual 🙂
**Alexey Grigorev**
What’s the easiest way to generate an intro tune for a podcast? Asking for a friend
**Raghav Bali**
Well, Generative Models are to the rescue here (RNNs in particular). <#C01F53D373M|shameless-promotion> and plug, refer to my article here: [https://towardsdatascience.com/lstms-for-music-generation-8b65c9671d35](https://towardsdatascience.com/lstms-for-music-generation-8b65c9671d35)
The article also points to a few samples generated using the said architectures. The book explores it to a greater depth
**Lavanya M K**
Hi Raghav Bali asking this totally out of curiosity. Is it possible to create a sort of reverse subtitling from Generative AI?. Ex. Given text “a beautiful place”, the model has to generate a picture/video/art of a scenic place, something similar to how our mind generates.
If so, how are these models trained?
**Wendy Mak**
you might want to look at what openai has been doing with dall-e/clip etc (one blog post with runnalbe code [https://minimaxir.com/2021/08/vqgan-clip/](https://minimaxir.com/2021/08/vqgan-clip/)
)
**Raghav Bali**
Thanks for your question Lavanya M K and kudos to Wendy Mak for the perfectly crisp answer. DALL-E and CLIP are state of the art works and generate some really wonderful works of art.
**Raghav Bali**
Lavanya M K found something interesting on the lines of your question. I know its well past the AMA but who cares 😉
[https://blogs.nvidia.com/blog/2021/11/22/gaugan2-ai-art-demo/](https://blogs.nvidia.com/blog/2021/11/22/gaugan2-ai-art-demo/)
**Lavanya M K**
This looks super exciting. Thanks Raghav Bali for sharing.
**Lavanya M K**
Really interested to know how training is done for such models
**Raghav Bali**
Not answering to a question but Generative Models are becoming more and more common and creative nowadays. This recent art gallery exhibition by Emil Wallner (leading AI researcher with Google) takes Style Transfer to the next level, see here: [https://twitter.com/EmilWallner/status/1453050980438843397](https://twitter.com/EmilWallner/status/1453050980438843397)
**Raghav Bali**
Another interesting take on Generative AI
[https://twitter.com/\_joelsimon/status/1458507647515254785?t=rrN80ZIEweAX0ITBvmV7VQ&s=19](https://twitter.com/_joelsimon/status/1458507647515254785?t=rrN80ZIEweAX0ITBvmV7VQ&s=19)
**Tim Becker**
Hi Raghav Bali, thanks for being here again!
**Raghav Bali**
Hey Tim Becker nice to meet you 🙂
**Tim Becker**
Thank you for answering my questions 🙂
**Tim Becker**
* In your book, are you talking about data augmentation with generative AI? I am particularly interested in when this technique is useful and when not? And how beneficial is it? I guess, there are certain limits to this approach?
**Raghav Bali**
Unfortunately no. Data Augmentation using generative models is definitely a topic worth exploring but given that there weren’t many books introducing generative models and different nuances associated. Hence, we decided to focus the book on different types of generative models along with their different applications.
But yeah, outside the book using generative model for data augmentation is certainly gathering some steam. There are works by folks like Antoniou et. al. spearheading this space (important paper: [https://arxiv.org/abs/1711.04340](https://arxiv.org/abs/1711.04340)
)
My two cents on the topic (though I am pretty new to this aspect itself):
1. Highly beneficial if you have a training data space which is not so expansive/diverse (for the lack of better word) . This would help you generate samples for a robust model
2. Limits: certainly risky if you do not have a metric to understand the quality of your samples. For instance, I am working in the healthcare space and imagine a scenario where we would want to generate sample X-Rays for lung cancer. We would need very tough quality check to ensure that generated samples indeed make sense
**Tim Becker**
* Could you give an example of an adversarial learning paradigm?
**Raghav Bali**
This is something I want to explore for quite some time but haven’t been able to. Really interested to read and understand the poisoning strategies. Maybe we can catchup sometime soon to discuss more.
_Ask to the larger community here: any pointers/materials to get started?_
**Maja**
This is a very interesting research and [paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9483649)
about CryoGAN for me.
**Tim Becker**
* For you personally what is the most exciting topic in generative AI?
**Raghav Bali**
On a broader level, the whole concept of GAN is very exciting to me. Every new architecture and the ideas behind them simply amaze me.
From an application standpoint, I believe Music Generation and DeepFakes (though notorious) have great potential
**WingCode**
Hi Raghav Bali,
Have you come across any work on unsupervised labelled interpretable controls for GANs ? [https://www.youtube.com/watch?v=jdTICDa\_eAI](https://www.youtube.com/watch?v=jdTICDa_eAI)
In this video, they manually play around on each component to understand the changes it brings about but then it is manual and requires a human annotator to label each component.
**WingCode**
Digging up on the above paper, found reference to this paper [https://arxiv.org/abs/2002.03754](https://arxiv.org/abs/2002.03754)
which finds these manipulation axises unsupervised but I am not sure whether it generates a labeled text for that specific axis 🙂
**Raghav Bali**
Awesome week and equally Awesome set of questions. Thank you to all the participants and congratulations to all the winners. It’s been fun interacting and discussing stuff with you all.
Thanks Alexey Grigorev again for this wonderful platform. Would love to keep this discussion on going.
Cheers and keep exploring guys n girls
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Ace The Data Science Interview – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Ace The Data Science Interview
------------------------------
#### by [Kevin Huo](https://datatalks.club/people/kevinhuo.html)
, [Nick Singh](https://datatalks.club/people/nicksingh.html)
##### The book of the week from 15 Nov 2021 to 19 Nov 2021
This book is the best way to prepare for Data Science, Data Analyst, and Machine Learning interviews, so that you can land your dream job at FAANG, tech startups, or Wall Street.
* [Book's page](https://www.acethedatascienceinterview.com/)
* [Amazon](https://www.amazon.com/dp/0578973839)
Questions and Answers
---------------------
**Doink**
how to balance interview prep when interviewers are expecting SQL+Leetcode+ML Theory+ System Design and Github Projects as a way to gauge effectiveness of a candidate along with a full time job?
**Nick Singh**
It’s a lot, and can feel overwhelming. First things first — ask recruiters what you need to know, check this via Glassdoor, or better narrow down your search and only apply to most relevant titles. For example… System Design might be more for Data Engineer or ML Eng roles… not DS or Data Analyst. Similarly, ML Eng might not be tested on SQL but more LeetCode style question. This narrows down the space of things to know.
**Nick Singh**
But to get to the core of the question; balance. I have no idea, it’s deeply personal. I struggle to balance cooking healthy, working out, with my career stuff. Others have their own balancing act issues, balancing job search stuff with family friends and full time jobs. Ultimately comes to prioritization — there’s no easy answer!
**Doink**
Nick Singh yes the 2nd part of balancing job along with other commitments along with family how do you do that? Would love to know that also 🙂
Also to add a lot of them are also asking these product manager type of questions like how to improve Google Maps etc
**Kevin**
echoing what Nick said - focus on the particular job you want and what those job descriptions encompass. Product analytics roles will heavily focus on SQL/Product sense vs. ML eng which is coding & ML for example. Also depending on the company and role you will want either more breadth (FB product DS) or depth (Netflix ML research) so you should allocate your studying time accordingly
**Doink**
how to remember ML theory for long period of time?
**Nick Singh**
spaced repetition. I’ve been exposed to it, multiple times, over multiple years, that even if I don’t remember every detail I know what I can look up, or can easily refresh myself. No other way around it!
**Kevin**
Also it helps to practice the fundamentals - ML (like many technical topics) is not easy for rote memorization, and being able to pattern match among different use cases & fields is nice
**xnot**
What do you think is wrong about the current practices in tech companies when it comes to DS interviews?
**Nick Singh**
Coding questions often asked are too hard. When was the last time a DS needed to know about a tree structure or used a Linked List. Come to think of it… do even SWEs use Linked Lists? I think those leetcode style questions are already sorta dumb for evaluating SWE talent, and lazy thinking ported it over to DS.
**Kevin**
What Nick said also applies to hard technical questions in ML/Stats/Prob land for ML research / ML eng and quant roles (in finance) although it’s more relevant since for deeply technical roles you should know the fundamentals well (for example if you are doing ML research you should understand the math behind the algorithms)
**xnot**
What would an ideal interviewing approach look like?
**Nick Singh**
An early filtering round. Quick phone call about projects/background, throw in 1-2 stat/ML theory questions, and have them write a simple SQL query. Maybe of load some of this to a recruiter if possible. Then do 2 more technical rounds. First technical round is something more academic/textbook… aka a data manipulation/cleaning exercise, some more theoretical questions (what’s your favorite modeling technique and why… probe if they just call sci-kit learn or really know what they are doing and understand the fundamentals). … and then something more open-ended / how would you approach this real-world problem we are dealing with. Do they have business sense. Then a final behavioral interview to understand what this person is done, how fast they are growing, and whether they actually are interested in the job.
**Kevin**
Depends on the role - product analytics: SQL & product sense, ML eng: coding & system design & ML for example, and also the company (higher % of more senior ppl at Netflix than Facebook). More variance for startups since the hiring process is less defined. But yes what Nick outlined is generally good for a DS process
**Nikhil Shrestha**
Hello Kevin and Nick thank you for writing the book.
My questions are:
1. How to prepare theoretical topics for data science interview. As we see the job description keeps increasing every day. Even when you are confident about performing a concept practically how can we answer the theory question. Basic example: SVD is one question which I feel is vast, I know what SVD is but how to trim my answer to explain it to interviewer. You can even explain it using any other topic I want to understand the general approach
2. Now a days most of data science job description wants hands-on experience even if we know the concepts we don’t get the interview, resume gets selected but HR asks for hands on experience and people like me don’t have any. Hence how can we answer that question to HR person asking us. 😄
**Kevin**
1. one way I have found effective is to do mock interviews with friends or others - when you can explain deeply technical concepts in a succinct manner then you understand them quite well. also one tip for me personally is not memorizing any specific equations - if you focus just on rote textbook memorization, then when thrown a curveball, just like an ML algorithm you’ll have overfit your knowledge. 2) doing projects is super helpful here, to demonstrate that you are proactive, driven, and passionate about using data science for XYZ
**Nikhil Shrestha**
Thank you for the reply and i will follow these advices to improve myself. 🙂 🙏
**anas hasni**
Hello everyone, my first question is what to expect in terms of questions in roles related to deep learning (CV, NLP).
The second question is related to experience in deploying models to production. Is it really necessary to have this experience in order to get a position in industry especially with a research background. Thank you
**Kevin**
1. depends on your background (they will ask about your research experience if you have any) and then depending on research or eng will be more or less theoretical, 2) no since otherwise it’s a chicken & egg problem - that being said, the interviews are there for them to gain confidence that you understand the models well enough theoretically and also have some experience (whether through projects or research etc) to be a good fit
**anas hasni**
Thank you Kevin for your response. I really appreciate it 🙂
**Wendy Mak**
As an interviewer, what would be some good questions to ask (or red flags to look for) in order to decide whether someone would be a good fit for the team? (e.g. someone might be very good technically but a terrible team player in which case that might not be the person you want to hire)
**Kevin**
Might be some personal bias here but things I look for are: 1) motivation for joining company/team specifically, 2) where they see themselves in X years, 3) what gets them excited to do meaningful work everyday, 4) their ability to be a team player and questions that are in those areas may range from something like “tell me about a time you handled a conflict with a boss” (for 4) to “I saw you had XYZ on your resume as projects, how did you come about those?” (for 3)
**Wendy Mak**
As an interviewee, what are some questions you would ask your interviewer to gauge that this job is indeed a good choice for you and that you would enjoy your time at the company and be able to grow your skills?
**Kevin**
Again personal bias here but generally along the lines of what the company is looking for and how to grow within the company - examples: 1) what does the day-to-day look like? 2) how does the role change over time? where are the biggest areas of growth? 3) what is the company culture like? what about trade-offs (for example, speed of shipping new products vs. having highest quality)?
**serdar**
Does star method really work for the DS interviews?
**Nick Singh**
Absolutely! [STAR](https://careercenter.lehigh.edu/node/145)
is for behavioral questions.. I think of it as structured story-telling. And DS & ML roles def ask these types of questions!
**Kevin**
+1 on what Nick mentioned (he’s a verified pro with the method!)
**Doink**
How much of MLOps questions are asked for DS/MLE related roles for interviews?
**Kevin**
Depends on company and role again but those topics should only be asked if you have relevant experience/projects in real-life large-scale model deployment and or if the job description mentions MLOps specifically
**Doink**
How to master the skill of interviewing especially when one has to apply for multiple roles just to get a few interview calls?
**Kevin**
Practice makes perfect (generally) - just like an ML algorithm you get better with more data (interviews) 🙂 - also need to know when to focus on breadth vs. depth based on the roles
**WingCode**
Hi Kevin and Nick Singh,
Thank you for doing this.
How do you approach getting a DS job when the job application asks for Masters or PhD in Data Science?
**Kevin**
When most job descriptions have that job req, it (usually) isn’t a hard requirement, they just need a way to filter candidates. There are obviously exceptions, especially for deeply technical roles requiring specific knowledge (say ML research). My 2 cents here would be to try and do projects in the space to demonstrate drive and competency
**WingCode**
Thank you Kevin for the answer 🙂
**Maja**
Hello Kevin and Nick Singh👋 Thank you for being here with us and answering our questions. What are the advantages and qualities that we as an interviewee need to have that FAANG or Wall Street companies are looking for? From your experience what you have seen to be that crucial difference which makes you get the job in those companies?
**Kevin**
Definitely depends on company and role - from the behavioral side it’s a sense of motivation and drive and cultural fit, from the technical side it varies a ton but is generally related to either breadth or depth among a variety of topics. For example, FB product DS is about breadth and knowing SQL/product sense etc vs. Wall St quant is about stats/prob/coding/ML generally. Sorry it doesn’t answer the question but that is my observation, there is no silver bullet 🙂
**adanai**
Hello Nick and Kevin, thank you for doing the QnA!
For an individual transitioning to a role in Data Science and allied fields, are skills related to scalable tools required or is that something that can be picked up on the job? (Eg. Spark and similar tools)
**Kevin**
Can be picked up on the job for sure
**adanai**
Besides implementing projects, what is a good strategy to adopt to apply for jobs that prefer candidates with a Masters degree (when I do not have one)
**Kevin**
Generally those job descriptions are looking for lots of depth (specific knowledge) so good to focus on those areas. Projects are great as you mentioned, keeping up with state-of-the-art techniques also (if ML), and just generally being able to demonstrate that even if you have no direct job experience in the space, you can and will pick it up quickly
**adanai**
While preparing for interviews, should one also focus on learning to develop basic algorithms from scratch (that are generally available via popular packages) or is theory sufficient.
**Kevin**
Nice to do both, with a focus on the fundamentals (i.e. not memorizing and equations or specific syntax)
**adanai**
What are few points to keep in mind as a framework to be checked off before mentioning a project in the resume/portfolio
**Nick Singh**
great question! you should mention it, if you could talk intelligently about it. If you lack work experience… and just need to fill it up.. then def add more things. But don’t feel the need to over-stuff with random things… sometimes if something isn’t very special … but other things on the resume are… adding a simple neutral project detracts from the good stuf
**Kevin**
+1 to what Nick said - for me personally I only include it if I feel ready to be (potentially) grilled on the topic
**Doink**
How much time is required to prepare if your goal is to be proficient in Leetcode assuming 100-150 questions here, ML theory, ML System Design and SQL? Is 2-3 months of full time effort enough?
**Kevin**
Depends on the person and how much time spent per day - generally a few months of full time effort should be good enough to be quite proficient in any area although the more deeply technical the topic the harder. So should be an okay timeline roughly speaking
**Doink**
how much of an important signalling metric is Kaggle Competition rank? I know many say that Kaggle isn’t a good thing cause it applies kitchen sink of problem whereas in the real world it’s all about latency but then many companies are putting their internal problem as a kaggle competition.
**Kevin**
Not high signal unless you’ve won competitions imo
**Asmita**
Hi Kevin and Nick Singh, thank you for doing this! My question for you today is that generally the job descriptions are pretty vague. Should we prepare for all the required topics mentioned, or focus on the major ones? And how do we get to know what would be the major requirement in that particular role? Can you please specify what a person should be prepared with for an entry level position or internship in data science?
**Nick Singh**
Focus on the major ones, but generally most roles look for similar things. So one way to get a sense of what’s important is just to look at the same type of role… and across multiple companies you’ll get a sense of it. For example, for most Product Data Science roles you’ll see mentions of Exploratory Data Analysis, SQL, A/B Testing, and “supporting product/business teams” == biz acumen and product sense.
**Kevin**
+1 to what Nick said, for entry level it’s more about the fundamentals than anything too specific so be sure to brush up based on the relevant area (Product analytics is a lot of SQL & product sense for example, whereas ML eng is coding & systems design & ML)
**Quynh Le**
Hi Kevin and Nick, thanks for coming to share about your book! When preparing for interviews, how do you suggest a person approach learning different topics (programming, stats, machine learning, etc)? Should he/she learn all at the same time or each topic after each other? Also during interviews, how can an interviewee address some technical topics that he/she does not know or use yet but is aware of and could pick up later if got the job?
**Kevin**
Definitely depends on the person - with so many topics it can feel like a firehose (I personally like doing brief deep dives but then rotating among topics). If you get asked those topics you should just admit you don’t know / don’t have any experience in the space - in general interviewers shouldn’t press too hard for this as long as you don’t put random buzzwords on your resume that you can’t explain thoughtfully
**Allan**
Hi Nick and Kevin, thank to you both for taking the time here to answer questions here and share about your book, really appreciate it! How would you suggest that people in an existing technical role best approach the process of transitioning to DS/ML? Relatedly, If someone has relatively strong DS/ML knowledge but lacks actual DS/ML work experience, how to best overcome this issue? Side projects to build up a portfolio of demonstrable skills?
**Kevin**
Definitely side projects, and any research if relevant (for example, lots of ML hiring managers love research). Also helps to stay up to date with state-of-the-art and whats going on in open source world to demonstrate passion in the area
**Allan**
Yes, that’s what I had thought, and good point about the research component.
**ASHISH SONI**
Hi Nick Singh
What are the 5 best practices, a day before the ML interview??
**Allan**
What is the best approach to handle the reality that the state of the art methods in the field are progressing so fast, especially in the areas such as NLP? How to balance between preparing the “the basics” vs wanting to appear aware (if not proficient) of the newer developments?
**Kevin**
Good to know both and understanding tradeoffs if asked about them - still good to start with the basics, but if you’re applying to jobs in a specific space (say CV at self driving co’s) then it’s good to be able to keep up with and explain state-of-the-art techniques
**Allan**
Thanks!
**Amruta Ranade**
Hi Nick Singh and Kevin, firstly thankyou for writing this book!
I wanted to know is there a list of things noted that we should avoid doing during our interviews?
**Kevin**
Definitely - will let Nick chime in on the behavioral side but for the technical side some things include: 1) not assuming you know more than your interviewer especially if it’s specific in the context of the job, 2) pretending to know some technical topic then getting grilled on it and not being able to explain anything, 3) talking/coding/whiteboarding for way too long on a topic without checking in with the interviewer. Have been through all 3 personally :)
**Arthur c**
Hello everyone, first of all thanks to Nick Singh and Kevin for this important book, and to Alexey Grigorev for giving us the opportunity to ask questions to valuable authors.
Of course, this book is the product of a long and detailed study that takes a lot of time, but of course, there is a _\*summary of this work\*_ that is shaped in your minds.
So, what are your basic recommendations for candidates who work in the field of DS and who are at the stage of an interview, with a few items that have been filtered through all this work? (Technically and behaviorally)
**Kevin**
Will let Nick chime in on the behavioral side - on the technical side: 1) figure out whether you’re optimizing for breadth and depth since there’s an infinite among of knowledge in the spaces that make up “data science” 2) the more work experience you have, the more emphasis that will play in interviews (generally) and if you have less experience then projects are great, 3) fundamentals matter more than rote memorization especially for very technical topics like Stats & ML
**Arthur c**
Thanks Kevin definitely
useful summary.
**Tim Becker**
Hello Nick Singh and Kevin, thanks for being here. I would like to ask you what a cold email is?
**Tim Becker**
* What are the main differences between interviews for junior, senior and lead DS roles? Is your book suitable for all seniority groups or rather focused on one?
**Kevin**
Will let Nick chime in here - it should be suitable for all levels (we have problems ranging from “easy” to “hard” and across many companies. That being said, it has a lot of refreshers on topics, chooses specific topics, etc. and therefore is by no means a silver bullet for every person and use case
**Tim Becker**
* Often applicants will have a few days to work on a task that they will present to the interviewers. Since the time for the task is limited how would approach this and what would focus on?
**Kevin**
There’s a lot of variance on takehomes.. will let Nick add here but generally speaking my personal view is that it’s there to filter for candidates who truly want to work at the company since it’s generally not an easy task. So would only do it if you really want to work at the company, and would always timebox it (can always spend a long time but I personally prefer a quicker, mostly complete project than a completely perfect one since the marginal utility is lower for the latter imo)
**Tim Becker**
* What are questions interviewers want to hear from the applicants when it is their turn to ask questions?
**Kevin**
Will let Nick add on here but for me anything demonstrating curiosity for the role and the interviewer themselves, examples: 1) what brought you to XYZ company? 2) how do you see your role changing over time? 3) can you describe your day-to-day?
**Tim Becker**
* As an interviewer, what can you expect from recent graduates in terms of common sense and understanding the companies business. I had the impression that simple questions concerning the field the company is working in seem to be very hard for some graduates.
**Kevin**
there’s a good amount online about various companies so it’s just about doing your homework - yes generally new grads may have a worse grasp of business sense but that may change with technology adoption trends and a growing awareness
**Doink**
Nick Singh and Kevin I see that there are multiple roles like Data Analyst, Product Data Scientist, Data Scientist etc I am curious to know what are the required skills for each of the role like for Data Analyst it would be SQL,Excel,Pandas etc. Similarly what would it be for NLP Engineer, MLE, CV Engineer etc?
**Kevin**
depends on role and company but data analyst / product data scientist is usually focused on SQL / product sense whereas MLE (and NLP & CV engineer) is more coding / systems design / ML
**Gur Hevroni**
Nick Singh & Kevin thank you for taking the time to interact with the community!
1. Do you have tips or suggestions on how should one mentally prepare for the high rejection rate for these positions? That is, how to constructivy assess why you were rejected and how/what to improve?
2. Similarly, what are your suggestions for one to assess her/his level of preparedness, prior to submitting applications, or emailing recruiters?
**Gur Hevroni**
As a side note, your book was incredibly helpful for me in preparing for DS interviews - and yesterday I actually got my first job offer! :thank\_you:
**Kevin**
Congrats Gur Hevroni! Glad to know it helped 🙂 Regarding your questions: 1) its a numbers game and depending on the person it’s useful / not useful to try and analyze that metadata for takeaways (you can even try to apply some data science on the process if you interview at enough places haha), 2) doing mock interviews with others and feeling ready to be able to explain projects and various concepts are good indicators
**Gur Hevroni**
Thanks, Kevin! These are great tips!
**Allan**
How would you suggest creating/choosing a portfolio project that really stands out? How important are good portfolio projects in (a) getting the initial interview and (b) the interview process itself?
**Kevin**
Will let Nick elaborate here, but my 2 cents is that it’s important to just choose an area you’re truly interested in and go from there - DS/ML is a toolkit and can be applied to virtually every field. The importance in getting initial interviews and in the process itself varies with role/company but I find that for the former, it generally has a decent weight the less direct job experience you have, and for the latter it matters a lot in technical roles (ML especially)
**Allan**
Thanks Kevin
**Alex**
Hi there Kevin & Nick Singh, it’s a pleasure talking to you! Since your magnific book has probably been read by many people transition careers, here goes my (very specific) question:
For Data Science/Analysis positions that are not research-oriented, what weight would you give to business understanding as a skill? Say someone is trying to jump into data from a business background, do you think their potential transfer learning is higher since they get to know the business from a ROI point of view?
**Kevin**
The business sense is super useful and helpful for product analytics / product DS roles so high generally
**Prasad Paravatha**
Hi Kevin, Nick Singh Thanks in advance
* In addition to reading your book 😄, do you have any good recommendations for courses in DS and/or ML?
* What are the responsibilities of TPM (Technical Product Manager) or TPO (Technical Product Owner) at FAANG companies?
**Kevin**
1) I’ve looked at some of MIT OCW for technical topics and for ML there’s a lot out (Andrew Ng’s material for example) there so will need to come back to this. 2) they generally act as a technical PM so basically they interface with ML engineers/researchers etc. and hence very similar to regular PM except that they need to understand and speak the technical jargon
**Nikhil Shrestha**
Hello again Nick Singh and Kevin
Another question which I am interested in is what kind of projects do recruiters usually want to see in the resume ?
Online you find many projects ranging from basics to advance, domain specific etc.
But when it comes to selecting best projects to showcase in resume (say top 3 projects) which one should we select.
Is there some advice which you could give here ?
Also are there some link or resources which can help in getting some idea about “trending” projects which have better probability of catching an eye of recruiter.
**Varun Nayyar**
Hey Nick Singh and Kevin Kev, great to see you guys here I follow you on LinkedIn you both post great stuff. My questions would be what kind of projects would make you stand out of the crowd in relation with analyst-consulting based roles? Is it a good practice to research about the work already being done in a company you are applying to for a particular role? Does it come up as being a kiss up to recruiters?
Should I really work at my DSA if I am planning for DS roles only and not at MLOps or MLE profile? In case I find myself wanting to get into MLE, how much swe knowledge do I need to function as a good MLE? Do official certifications from tech giants like TF developer from Google or others like IBM, Nvidia really boost up placement chances in these companies? I was told that they do but still it would be better if you could throw some light on it.
One final question apart from academia.
Also how to deal with that overwhelming feeling when you know there’s a lot to cover(coding+stats+SQL+ML to name a few) but time is a constraint?
Sorry for throwing a lot haha
Cheers and all the best for your book. Looking forward to read it once it’s available here!!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Building Machine Learning Powered Applications – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Building Machine Learning Powered Applications
----------------------------------------------
#### by [Emmanuel Ameisen](https://datatalks.club/people/emmanuelameisen.html)
##### The book of the week from 22 Nov 2021 to 26 Nov 2021
Learn the skills necessary to design, build, and deploy applications powered by machine learning (ML). Through the course of this hands-on book, you’ll build an example ML-driven application from initial idea to deployed product. Data scientists, software engineers, and product managers — including experienced practitioners and novices alike — will learn the tools, best practices, and challenges involved in building a real-world ML application step by step.
* [Book's page](https://www.oreilly.com/library/view/building-machine-learning/9781492045106/)
* [Amazon](https://www.amazon.com/Building-Machine-Learning-Powered-Applications/dp/149204511X/)
* [Book's GitHub repository](https://github.com/hundredblocks/ml-powered-applications)
Questions and Answers
---------------------
**Álvaro Budría**
Hi Emmanuel Ameisen, I’m a DS intern just one week into my internship. For novices like me, what are the crucial basic good practices, skills and ways of approaching the role that will make someone like me stand out?
**Emmanuel Ameisen**
That really depends on your team, company and the goal of the internship! More generally when onboarding to a new company, here are a few things I would always recommend doing:
* Ask a lot of questions, especially at the start. Ask not only about subjects of interest, but meta-questions like “how can I find documentation by myself?” for example
* Setup introductory chats with many folks on the team (I would aim for at least one chat a day for the first few weeks) to get to know them. In that process, ask about their priorities and goals. This will let you know what matters to different parts of the org, and angle your work to match.
* Ask your manager and leaders what they think are the success and failure criteria ahead of time. Than, regularly check in to see whether you and them are in agreement on how everything is going
Last but not least: in an internship, you are also evaluating the company as a potential place to work. Don’t forget to take the time to explore and decide whether this would be the right full-time role for you.
**Álvaro Budría**
You give an angle to this question that I didn’t think much about. I’ll be applying all these tips. thanks!
**Nikhil Shrestha**
Hello Emmanuel Ameisen
Thank you for writing this book.
As we usually talk a lot about statistics and learn too many concepts, also on the other hand we come across senior lecturers n experts etc who also claim that you in real world you don’t use the theory but you use some tricks or things in practical way.
To give small example: when we learn precision recall we find different tricks like _P_recision = _P_ositive. The proportion of _TP_ from all _Positive labels_
But when solving in competition or real time we go vertical for Precision and Horizontal for recall
* My question is how does things actually work in real life ?
* How and when is statistics or other theoretical topics actually used in real time ?
Like I learnt from many instructors and webinars that they usually use practical statistics.
* But what does practical stats actually mean ?
**Emmanuel Ameisen**
Right, the body of theoretical knowledge that is taught at school is often much larger than what you will need in an applied role. This doesn’t mean it isn’t useful to learn all these concepts, but it is good to be aware of the practical realities of the job.
For statistics specifically, I’d say it depends on your role. Some companies hire statisticians which may use every tool in their arsenal. For Data Scientists and ML Engineers, you’ll most commonly use stats to:
* Understand model performance metrics, such as FPR/TPR, ROC curves, Brier scores, NDCG, etc
* Design and interpret results of applied experiments such as A/B test. That includes understanding confidence intervals, statistical significance, MDEs, etc.
* Understand model loss functions such as cross entropy loss and their impact on model training
**Nikhil Shrestha**
Thank you for the swift reply and yes I totally agree that theory is as important as knowing practicality. 🙂
**Dustin Coates**
Emmanuel Ameisen no question here, but a thank you. As a PM (“former” dev), your book was immensely valuable. Almost instantly, our team put into place the “do it manually first” rule in a way that we hadn’t before, and if that was the only thing we got out of it, it would have been useful enough, but I got much more. I’ve recommended your book to many others at my company.
**Emmanuel Ameisen**
I’m glad the book was useful and applicable to you and your team. It’s been amazing to hear which parts resonated with folks at different companies and roles. Thanks for the kind words and for recommending it to others! If you haven’t yet feel free to leave a review on Amazon if you have the time to let other PMs know to check it out, it really helps.
**Maja**
I agree 💯 Thank you for writing a great book on this topics and being here with us. One of my colleague loves your book and highly recommended it to all of us in the team. It was a huge help for him.
**WingCode**
Hi Emmanuel Ameisen,
How do you validate an DS idea to full blown product? Is doing a quick PoC and then followed by agile methodology to build incrementally and deploy work or do you suggest anything else from your experience?
**Emmanuel Ameisen**
Love this question. One thing I try to focus a lot on in the book and in real life is making iteration cycles as short as possible. When validating an idea, you’re often trying to do a couple things at once:
* Determine whether the product idea is useful before investing a lot of effort. For this, I recommend finding either a rule/heuristic or using humans to build a first version. Even if the performance isn’t close to what you think you can reach with ML, that will help you decide whether you need to work on this problem at all. Oftentimes at this point you will find a fundamental flaw in your product idea and rethink it
* Figure out whether you need ML at all. ML is great, but it has very high maintenance burden. Few models are ever “set it and forget it” (see googles paper about ML being the high interest credit card of technical debt). This is a different problem, but the advice above applies to it pretty directly
* Figure out whether ML can solve it. It may be the case that this is outside of the realm of what is doable today (“use ML to build a chatbot that can automate all of the work of a medical doctor” would be an example of something out of reach). For this, I recommend starting by carefully defining the inputs and outputs of your ML system (where is it going to live in your app? What information will it have access to? What kind of prediction will it make? How long will it have to make this prediction?). Once you have that, define what performance level will be your bar to ship this model. Base this on business outcomes! If you are building a recommencer system for a YouTube like service, you may for example decide that you’d only use a model that recommends at least one useful video to a user 80% of the time it displays recommendations. If the model will be used in a modal that shows three recommended videos, that means your top3 recall should be 80%.
Once you have those, you can start your iteration loop of model building and evaluating. Of course you may learn more as you go along and fine tune these goals, but doing the work ahead of time will save months of time wasted in dead ends
**WingCode**
How do you handle “pivots” in product development? For example: You start building your product around a particular framework later realise that there is a new superior framework out there. Do you have any steps which you follow to avoid these “pivots” in the future?
**Emmanuel Ameisen**
That’s a challenging one, and slightly outside the scope of my expertise. I’ll say that my experience is that pivots and refactors are always more costly than expected, so I generally require a strong motivation to buy in to one. Again, a bias for simplicity can help here as it makes swapping out any piece of your pipeline easier. It’s hard to say much more without a more specific situation
**WingCode**
Thank you Emmanuel for all of your elaborate answers 🙂
**Nikhil Shrestha**
Hello Emmanuel Ameisen again
We hear this a lot, if you want to get into the industry you must make projects on real life situations solving real life problem
I want to know how we approach towards solving one. Are there some pointers which you could tell in terms beginning a project, challenges and some tips to tackle them.
**Emmanuel Ameisen**
Practical experience is very worthwhile, so working on a project is valuable. My only guideline for picking a project is to pick something applicable and useful, and get your hands dirty. In my experience the main value of a side project for recruiting folks that are early in their career is to show that they can be productive. Often times I hear students get hung up on how much ml or rl or computer vision their project contains. As long as you can show that you know the required theory, it actually doesn’t need to be part of an applied project. I would consider them separate and chose a project by finding a use case you like, not shoe horning an ML application to it
**Nikhil Shrestha**
Thank you for the insights and I will use these in practical from now on.🙏🙂
**Nikhil Shrestha**
Also we study significance level and confidence level in theory class
As I learnt from some webinars and tutorials. Real world situation they are used extensively, before every project SL and CL are decided and using them you decide if model is generalized or not and also model evaluation.
E.g. criteria could be train-score < test-score AND test-score > CL
* My question is how far is this true. ?
* If it is true then what happens to the results ?
As I tried this technique in NLP and other ML data (toy data). I had to reduce my training accuracy to achieve this criteria.
* So 90-99 % accuracies are myths 🙆
**Emmanuel Ameisen**
In my experience, confidence interval are more often used for superiority and non inferiority tests (is this model better than that one) both offline and online (A/B test).
**Nikhil Shrestha**
Thank you
Totally understood and makes sense
**Álvaro Budría**
Nikhil Shrestha what do CL and SL mean?
**Nikhil Shrestha**
SL is significance level and CL is confidence level.
**Tim Becker**
Hi Emmanuel Ameisen, thanks for this really interesting book!
**Tim Becker**
* I was wondering which tech stack you are discussing in the book?
**Emmanuel Ameisen**
The book mostly uses the Python data science stack
**Tim Becker**
* Which deployment options do you cover in the book?
**Emmanuel Ameisen**
The book doesn’t focus too much on deployment infra. It discusses trade offs at a high level though, like synchronous and batch server side approaches vs edge deployments. If you want to learn about that specifically though, you’ll probably want to check out more specialized books
**Tim Becker**
* What is in your experience the most difficult part when building a ML powered application?
**Emmanuel Ameisen**
Understanding the user need and trade offs well enough to chose the correct technical approach. Being able to tell when the current approach is a dead end
**Tim Becker**
* What are the most common pitfalls during the process of building ML applications? And how can we avoid these?
**Emmanuel Ameisen**
Starting from the ML and deriving the application from it (“it would be cool to ship a sentence model, we could use it to autocomplete searches on our clothing website”) vs the opposite (“users often miss relevant categories, so we should build a classifier based on search queries that suggests relevant categories to browse”)
**Tim Becker**
thank you for answering my questions 🙂
**adanai**
Hello Emmanuel, thank you for doing the QnA!
How do we measure the outcome of the model in test after measuring the output during train/validation phase?
What are the common indications to look for when the model does not do well in the production/test phase even when the outputs showed the model was doing well in validation phase?
**Emmanuel Ameisen**
Regressions happen all the time between your validation set and production. They are commonly caused by mismatches in data generation, either on the filtering side (you trained on an unrepresentative distribution of samples) or the feature computation side (the way you compute some feature is different online/offline). The best way I’ve found to detect those mismatches is to rely on a shadow infrastructure. Shadow means that in production you send a copy of the request you send to the prod model to your candidate model, and log its output. Because you don’t action the prediction in any way, it is a safe approach that gives you information about production performance.
**adanai**
Shadow infra is something new to me and interesting, will read about it! Thank you
**Allan**
Hi Emmanuel, thanks for doing this and for the insightful answers you have provided this far. Sounds like a great book!
What would you say is the minimal realistic practical size for a team to take a new project from prototype to production?
**Emmanuel Ameisen**
Depends on the scale of the project and company! For internal tools, or at companies with strong infrastructure, a single IC can ship a model to prod. For companies that are smaller, or projects that are more complex, teams can get arbitrarily large. Think of the number of software, hardware, data and ML engineers are required to ship Tesla self driving, or OpenAI GPT-3
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Own Your Tech Career – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Own Your Tech Career
--------------------
#### by [Don Jones](https://datatalks.club/people/donjones.html)
##### The book of the week from 29 Nov 2021 to 03 Dec 2021
Own Your Tech Career: Soft skills for technologists is a guide to taking control of your professional life. It teaches you to approach your career with planning and purpose, always making active decisions towards your goals.
* [Book's page](https://www.manning.com/books/own-your-tech-career)
Questions and Answers
---------------------
**Nikhil Shrestha**
Hello Don Jones
Thank you for writing this book
I want to ask about communication.
As a data scientist we understand that our models and predictions will never be 100% bcoz we always deal with sample of data and not whole data (aka population in statistics)
When speaking to stakeholders or customer who understands only business (basically doesn’t understand these concepts) how will suggest to approach so that we could negotiate in terms of margin of error.
Say we ask for 20% margin of error but stakeholders or customers asks for 5% and now we want to bargain on this as we know 5% will be very difficult due to constraints present in that situation.
**Don Jones**
Teach them :). “I can do that, but here’s what would have to be true.” Or, “with the quality of data we have, that just isn’t possible.” Don’t feel obligated to say yes if you can’t truly says yes - but remember that negotiating is a two way street. If you give something you should get something in return.
**Don Jones**
I also suggest making sure you’re taking the time to understand their context. That is, they might understand “only business,” but that’s a really important perspective. Make sure you don’t understand “only tech”—that is, make sure you’ve taken the time to understand the business as well, so that you can help translate for them.
**Don Jones**
As a business leader, I’ll use your example to perhaps point out the difference. For me, someone coming to me with a 20% margin of error is basically coming to me with nothing. Emotionally, I feel I could throw darts at a board and get 20%; 20% margin of error is a 20% that the data is completely wrong and I’ll make entirely the wrong decision.
That’s where it can help for you to better understand the business side. “Sure, I can only get you +/- 20% on this data… but help me understand what your’e trying to prove or disprove. Maybe we can add some other data sets, and between their intersections, we can narrow this down for you.” They’re asking for 5% because they’d like some confidence in the data; when you offer 20%, you’re basically telling them to not be confident in the data. That doesn’t help drive a business decision - so what _would_ help drive a more valid business decision? Focus less on the data and more on the business outcomes being sought, and then look to how you can help support (or not) various aspects of that business decision. And if you just can’t… it’s fine to say so. “We don’t have the data to support this with any level of confidence one way or another. You’re going to have to go with your gut.” Sometimes, business leaders just need to rely on their experience instead of data they don’t have.
**Nikhil Shrestha**
Thank you Don Jones
I totally agree to this. Thank you for sharing this knowledge.
I can totally relate myself with context part you explained, as that’s what I have been doing but didn’t know how to apply in tech world. As it’s totally new field for me.
About the example discussed - 20%. If I understood correctly:
* Keep a range in mind rather than a point.
* Don’t disclose your margin before they do. Ask about how are they confident about the data.
* If you n team with little exploration that margin cannot be met. Negotiate by asking more data then in terms of margin ?
* Focus more on understanding stakeholder thoughts i.e. in terms of business, rather than restricting yourself to only data.
* Finally if we can’t figure out a way to achieve the results with the margins specified. Let them know.
Please correct me if I am wrong.
Main motive to learn about this part was to be able to communicate with people with only business understanding and get information from them in terms of tech requirements.
So, thank you for clarifying this 🙂🙏
**Don Jones**
Yeah, that’s pretty much it. Understand how the data is being used. 20% margin might be fine if the goal is to decide where to aim a small marketing campaign; it might not be if the goal is to decide where to invest half the company’s resources for the next year. If you don’t have enough data, either get more or just accept that you won’t be able to make a fully data-driven decision. It happens. Some things just aren’t always knowable!
**Wendy Mak**
Hi Don, what would you suggest as the essential things you should do/connections you should make when joining a new company to set yourself up for success?
**Don Jones**
I think reaching to peers out of your org silo. Ask if you can set up a 30min meeting to learn what they do, and what their teams do. It’s a great ice breaker and helps you understand the shape and function of the org more quickly. I love it when new hires ask me to have a call with them!
**Shankar Somayajula**
Thats a great open culture in the Org. Kudos.
How do they learn/come to know what you or anyone else has to offer? How does one get an idea of what areas/topics others are doing? Usually someone senior guides them towards experts… Search via Confluence, perhaps :-)
**Don Jones**
One recent new hire of ours just looked at the org chart (we use Workday, which constructs an org chart). She reached out to everyone at her level, and a lot of people a step above, and asked for 30 minutes. I’m sure her direct leader offered some advice (I do that for new hires, for example) as well. But it’s in the talking to me and others where they learn what’s everyone does, how the org fits together, and so on.
**Noa Tamir**
Hi Don Jones, I’m wondering if you have advice for minoritized people in tech, and how should they navigate they careers?
Would you give them the same advice as the majority or should they tackle their careers differently?
I know that I network differently, but I’m wondering if you have more specific advice from your experience and “perch”
Thanks so much for joining us for a Q&A this week!
**Don Jones**
I’m a white guy who just turned 50, so although I’m not heteronormative, most people don’t know at first glance that I’m not part of the usual club. That makes it harder for me to offer meaningful advice, because I don’t have experience to speak from. I can offer what I’ve seen others do, though.
First, I clearly see equality still as a major issue, even in companies that truthfully want to do better. Unconscious bias is very real, and people can’t change what they haven’t yet actively acknowledged. I think just knowing that kind of preps you for what you’re up against.
I think the advice in the book is universal - but for minoritized people, it’s still not enough. I’ve seen women, for example, who’ve managed to walk the fine line between reminding people of their accomplishments and bragging, and they’ve been super successful - it sucks that they have to continually point out what they’ve done, but it seems to be reality.
I’d suggest seeking out allies. I’ve tried to introduce myself to new hires and offer mentoring, advice, championship, whatever I can, so they know they’re welcome, they belong, and I’ll try to have their back if they need it. Seek that out - outright ask for it, if you can.
Such a huge topic, and as an industry one where we seem to perpetually do so poorly. I wish I could be more helpful.
**Noa Tamir**
Thanks, I appreciate the honesty and that no one has all the answers! Alleys have definitely been a huge part of my career so far. Very much looking forward to reading the other answers and your book 📚
**Doink**
Hi Don Jones, lets say you are in a meeting with your boss and you are in a disagreement on a certain idea related to work, lets say some architecture, how to make someone understand the point in a manner which is not implied rude, some people speaks in a very direct manner which makes people look impolite, although they are not trying to be but they just want to be clear about things.
**Don Jones**
I’d say you do have to learn to read the room.
I wrote fiction on the side, and when I tell a story to a Young Adult audience, it’s very different from writing for an older audience - even if it’s the same basic story. So you do need to understand your audience. If you need to put them at ease, step slowly into a conversation, etc.
Ask questions. Disagreements often result from a lack of shared context - ask questions that help you understand their context. What about their decision is driving it for them? Are there factors you haven’t considered? Rather than making statements, act as if you don’t have all the information and try to get them to explain.
And at the end of the day, it isn’t always your decision. Present your facts, explain how they fit the criteria as you understand it, and list your assumptions. At the end of it, accept the decision and move on. It isn’t personal (or shouldn’t be.)
**Don Jones**
I often find that it’s easier to come across as someone who doesn’t have all the facts and wants to learn them; it avoids putting the other person in an “inferior” position, and when you’re genuinely trying to learn, they’d have to be jerks to not help out, right? (And if they are, maybe you’re not at the right company for you.) I don’t mean asking challenging questions like, “can you explain why you feel that way.” I mean asking questions like, “So, I may not fully understand all the business criteria. Can you help me see how your way is hitting the right criteria? I’d like to be able to offer better recommendations in the future, and having your context on this will help me be better at that.”
**WingCode**
Hi Don Jones,
How would you approach role switch in your career? Example: Developer to Data Scientist role transition or ML Engineer to AI Researcher. How would you prepare for the job interviews? How would you utilise your network for this ? How will you validate your skills with respect to the industry in this new role which you want to work in?
**Don Jones**
That’s going to differ so much based on the exact role. I’d be very careful about reading the job description, try to speak to existing employees (hello LinkedIn!), and gather as much info as possible.
Certifications can provide a certain level of validation, but it’s usually minimal. You simply can’t “validate your skills” in most cases. You need to have confidence, and ideally have worked on some public or community projects so you have a portfolio to point to.
**WingCode**
How do you incorporate your side projects which involves cutting edge tech into your job? Example: You’re working at company A and company A only applies some simple rule based approaches. You have learnt deep learning and want to apply into your job at company A. How do you convince the stakeholders at the company to pursue your interests so that both of the parties are benefited ?
**Don Jones**
I’m not sure why company A would even want to, honestly. It’s not their role to provide your career with opportunities or to keep you interested. If you can articulate a business-focused, data-driven argument, I’d hope they would listen. But focus on business outcomes, investment cost, ROI, and business factors. But just because you’re interested isn’t a valid business factor. If you want to pursue something new, why does it have to be at work? What kind of community projects could you start or contribute to instead?
**WingCode**
Thank you Don Jones. This makes sense
**Rosona**
Hi Don Jones : I bill myself as a generalist, and have been warned this is a hurdle for my career development (at a big company) unless I want to be a manager. Do you have thoughts on the generalist/expert divide? Is it really as binary as specialist or “manager”?
**Don Jones**
It kinda is in all but small companies. There’s just a feeling that a generalist won’t be able to do as good of a job, and the fact that big companies like to know the whole shape of a person and how they fit into a specific role. It isn’t always morally right, but it’s how the world works. Consider working for smaller companies or startups, who tend to put more value on a few clever generalists in their ranks.
**Don Jones**
I’ll offer what’s hopefully a good example:
I’ve worked for companies big and small. One tech shop I worked in, for a big retailer, was maybe 12 people. Small! They were delighted to have me as a generalist. I did programming, ran an AS/400, ran the phone system - a lot. They didn’t need a lot of generalists, but having one or two was a great fit.
Another was huge - maybe 1k people. Big! They had very rigid job descriptions, and they needed people to stay in their lane so as to avoid disrupting others.
It’s very much about the company.
I also find that generalists have a lot of difficulty describing their business value. “I can do it all!” Is not terribly convincing or believable; it sounds like bragging. How could we put you on a project for six months and then move you? It’s why startups are often more open to generalists—especially if you’re good at standing up new teams and then making them autonomous and then stepping away. That takes generalists, but few articulate it in the sense of “what kinds of business outcomes does this create.” And even that’s more a management example, I guess.
Actually GOOD generalists aren’t that common, so companies don’t create positions for them, and so companies don’t want to hire them. I’m actually a VERY effective generalist… but I can use that to specialize into whatever field I need at the time. And that’s wound me up in a VP seat :).
**Rosona**
Thank you for the thorough and thoughtful answer. And the conclusion that, yeah, it tends to get you in a manager type role. :)
**Rosona**
Second question :). Don Jones
Question: what advice do you have for people who want to get into tech, have a math/science background, and don’t know where to start/what in the tech space interests them?
I’ve suggested podcasts, informational interviews, etc, but there are just so many roles, company sizes and types, etc. And maybe the person is burnt out and nothing really entices them. How do they find something sufficiently ok? :)
Background: I left academia for industry 4-5 years ago and have jumped into lots of short term roles to try them out and find my direction, which I’m settling into. I now have other former colleagues coming to ask me advice, and I kind of wish my transition had been more smooth and would like to help them achieve that (see, e.g. CJs interview last week with Alexey on transitioning).
**Don Jones**
Yeah, it’s hard for sure. I think talking to people is something that can help - heck, start a podcast and start interviewing people from different roles, asking them to articulate what it is about their jobs they do, they like, and they do not like. That way, both you and your listeners can start to understand those questions. But it does take a lot of exploring.
But I mean, you’re kind of asking the same question that a high school senior might be asking. And you took some of the same approach that a high school senior took. When you’re entering something new, you just don’t know any of those answers, so you have to experiment a little. That’s why I always encourage younger people to try a lot of different jobs, before they settle on one that will be their career, or even before going to college, and majoring in some thing. Despite the fact that you might be 30, or 40, or older, you’re at the same point in the career as they are.
**Don Jones**
I’m not sure if there’s a way to shortcut that apart from a lot of rigorous research and talking to people.
**Don Jones**
Even thinking about a roll like software developer… That is so different across different companies, different technology stocks, different industries, different languages, different everything. Unless you have spent some time and started to experiment, it can be really hard to figure out.
Now all that said, I will offer this: community projects can be one of the best ways to experiment without committing. If you think you might be interested in being a software developer, find a community project that speaks to you and try contributing to it. It’s just like having a job, only it doesn’t pay. So it’s a way to get a feel for the job, without having to go through the commitment of interviewing, applying, getting the job, and then finding out you were wrong. All across the entire technology industry, you will find community projects that are desperately in need of contributors, many of whom are doing really amazing work for the world. It’s a great way to get your feet wet, and a fantastic way to build a portfolio that a potential future employer could look at to learn more about your technical skills.
**Don Jones**
———————————
I should also point out that I run [https://ampere.club](https://ampere.club/)
, which is a free follow-on resource to the book. I write weekly articles (all free) and there’s an audio (“podcast”) version for most articles written this year and in the future. You’re welcome to drop by and see what you think. There’s an e-mail newsletter option, which sends each new article to your inbox, if you prefer.
———————————
**Matthew Emerick**
Hey, Don Jones. I greatly appreciate you doing this.
What general advice do you have for an older techie with a recent gap? I feel that my career is stalled.
**Don Jones**
I mean, what’s “stalled?”
Careers aren’t sharks that have to keep moving until they die. They’re like… benevolent kidnappers. They take you someplace and keep you there. So if your career is giving you the money and time you want, you’re there. You just need to stay there.
Apart from that, what problems do you solve in a business? Are those problems relevant today? Then no problem. Are they not? Well, you’ve got some learning to do - you need to solve a problem that companies need solved.
This isn’t just skills, like knowing the latest version of SQL server or whatever - it’s about being a solution that a company will pay money for. Think of yourself as a vendor. Who is your market? What do you do for that market?
**Don Jones**
So I ask that question of people a lot, and I’ll get replies like, “I’m a Microsoft Exchange admin.” That’s a job title, not a solution. What problem do Exchange admins solve? They keep servers running. They keep messaging systems running. Okay - that’s a solution, of sorts. Do companies need that solution?
Increasingly… no. They don’t. So that’s a problem. That means people who weren’t keeping their eye on the ball now solve a niche problem. So they’re gonna have to learn to solve a new problem—and find a way to convince potential employers of their ability.
They best way? _Get involved in community._ You get to learn new skills, solve common problems, and you create a public record of your expertise. A portfolio, something _way way way too few_ technologists take the time to do. But working on community and open-source projects is a _fantastic_ way to demonstrate the problems you solve. Yeah, you may need to learn new stuff… but I mean, that’s tech. People who don’t like to learn new stuff don’t work in tech, they work in lumber (haven’t had a new kind of tree for a minute). You’re in tech, so you can learn to solve problems, demonstrate that ability in public view, and leverage that into whatever you need your career to be doing for you.
**Rosona**
Don Jones can I quote your comparison of careers and (not) sharks? Brilliant analogy. Obviously would attribute you (was thinking a tiny LinkedIn post referring to the thread/your book, with said quote).
**Don Jones**
Absolutely. Please do!
**Sandhya G**
Don Jones I read this 🔼 response with interest. I’ve had the experience of being with a benevolent kidnapper. I had a good job with a name brand company, great pay and happiness. However, I never realized when I became a dinosaur.
“You just need to stay there” described above seems like a lot of work. Creating a good portfolio project and online presence takes a lot of time and effort, which, in addition to full time work takes time away from family, kids, friends, volunteering, chores, and just relaxation.
Tech, in a lot of ways, feels like a hamster wheel - Keep running faster and faster or fall off. This, I speak as I spend 3-4 hours every week upskilling. Why? The pay is good - but I do not want to repeat a scenario where my skills are outdated again.
Any advise for a jaded person here?
**Don Jones**
I mean, if you’re not into constant change… then tech isn’t a great field. It’s why old guys like me often move into management. Those skills last longer. I also hit a wall with being a tech practitioner, so I turned those years of experience into something different.
Yeah, keeping up with tech is a lotta work. For sure. And it never does stop. But that’s a lot of fields, right? Law. Medicine. Really any field that’s dynamic - and they mostly do pay well in exchange.
**Sandhya G**
Moving into management also means taking responsibility for outcomes. Maybe it’s a progression, but do you have any suggestions for thinking that way. I understand one can never be fully prepared.
**Don Jones**
Yeah… I mean, in good companies, entire teams are responsible for outcomes, right? Leaders are there to clear the way for their teams to do the actual work, to set a vision and facilitate creating a strategy. Good leaders recognize that they produce no work output and that they’re there to partner with their teams. If you’re an individual contributor and you don’t feel “responsible for outcomes” already, then you’re not being led very well. For me, the best leaders ar ones who can communicate what success looks like and help every team member _see themselves in that success,_ and show everyone, “hey, if we do these things, success will happen.” It’s not actually _hard,_ but it’s a distinct skill set.
**Oscar Baruffa**
Hi Don Jones thanks for the Q&A! I’ve found that making time to intentionally improve soft skills seems to be more difficult than for more technical skills, and I think it’s because “Improve my presentation skills” has a less definite outcome than “Learn X tool”. Do you have suggestions for how to frame the goal of learning these skills more concretely so that the allotted time for it doesn’t fall victim to more immediate and tangible demands.
**Don Jones**
Yeah!
One of the things I’ll be doing on [https://ampere.club](https://ampere.club/)
in 2022 is dedicating each month to a “skill theme,” and providing some actionable outcomes and tasks for each every week. So that can be a way to help.
But set yourself milestones. “Give a lunchtime presentation where I’m rated at least 3/5 by the audience,” for example—actionable, measurable, and contributes to improving a skill. I’d actually say “Learn X tool” isn’t all that great; “Learn to do X with Y tool” is far better, and if you think that way, then soft skills are just tools used to do actual things. Measure the things, not the tool.
**Oscar Baruffa**
Thank you, great suggestion!
**Lavanya M K**
Hi Don Jones thanks for doing this Q&A. My question is.
How will you evaluate a tech idea when you are starting a company. Say you are starting a quantum computing tech related company which might niche for the market, or might not give you the ROI right away. But you are really passionate about the tech.
**Don Jones**
”Passion” isn’t really a great reason to start a company. I know people say that all the time, but the best question is, “what problem am I solving, and what’s the best set of tools to solve that problem?” Then you can look at how many people have that problem, how much they might save by paying you to solve it for them, etc. Otherwise it’s a hobby, not a business. And hobbies are fine! But they’re hobbies.
**Asmita**
Hi Don Jones, thanks for doing the QnA session. We all know how important communication and being able to explain our work in simple words to layman is in the tech field. How would you suggest to enhance these skills with time?
**Don Jones**
First, practice, practice, practice. For example, start writing articles—even if you don’t publish them!—that explain complex technical concepts to a lay audience. Actually, that’d make a great blog.
Remember: teaching is nothing more than repackaging information for a specific audience. Teaching doesn’t create new information; it places existing information into someone else’s context.
So next, you have to learn more about other people’s context. Get to know your audience in general. What’s their background? What shared experiences do you have? What analogies tend to ring true with them? If you want to teach someone, the burden is on you to do the work to understand where they’re coming from, and to construct explanations that work for them.
In business, that also means taking on the work to better understand the business. Growing your business acumen, understanding business drivers and motivations, all of that is also very important. It’s a big part of what I do at [https://ampere.club](https://ampere.club/)
, in fact—focus on business acumen-building.
**Asmita**
Also, can you please suggest some ways for an individual to comfortably ask questions from colleagues or mentors….we tend to be scared that people will judge us based on our questions.
**Don Jones**
That’s a complex question, and it comes down to “fear of failure.” None of us wants to look stupid, and we believe that asking a question is admitting we don’t know something, which makes us look stupid.
Which is stupid. Nobody expects us to know everything.
If you don’t know something, _learn it._ Now… keep in mind that you colleagues also have jobs to do, and they don’t necessarily have time to answer a million questions. So make the effort and investment to self-educate as much as you can. Once you reach the end of that effort, you can ask informed, intelligent questions that show you’ve already made the effort—honestly, I think most of your colleagues will appreciate that. And if you work in an organization that doesn’t, maybe you should ask yourself why you work there.
**Álvaro Budría**
Hello Don, could you provide any tips to find out when one’s career is stalling and is time to move up the next step? I know it’s a bit vague but it’s hard to know sometimes if one is falling behind or progressing within one’s potential. Thanks 🙂
**Don Jones**
Well, I’ll refer you to the first chapter of the book. You need to decide what kind of life you want to live, and then determine what kind of career is needed to support that life. There was a previous similar question here about a “stalled” career.
From there, as I wrote at [https://ampere.club/what-problem-do-you-solve/](https://ampere.club/what-problem-do-you-solve/)
, it’s then about “what problem do you solve?” Your career is about you, but the jobs you hold are about the employers who pay for them. If you need a different job, then you need to look at the problems employers need solved, and then skill yourself to solve those problems. Think of yourself as a vendor: what do you bring to the table that employers are looking for?
A way to discover that is to browse job listings and look for common themes. Those are clearly areas of opportunity.
But your career is never “stalling.” Either its taking you toward _your_ success, or it isn’t. If it isn’t, you adjust your career plan. If it is, you continue executing your career plan. If you don’t know, or you don’t have a career plan, you read my book 🙂 And get a career plan put together.
**Álvaro Budría**
thnaks, I’ll definitely buy your book (unless I win this week) and spell out a plan
**Noa Tamir**
Hi Don Jones I’d be happy to hear your thoughts on
* lateral career moves in general. But to get it started, Is there a timing element to consider? or maybe specific moves you saw that worked well or didn’t? was it about domain combination?
* Moving from management to IC (and maybe doing it several times back and forth).. I am enjoying taking breaks from management, and refreshing my technical skills despite my love of management. Is this unique, or is it a career path pattern?
**Don Jones**
So, on laterals… not really. It’s no different than any other job move.
On the other… I mean, it depends on what you’re trying to get your career to DO for you, right? Going back and forth like that isn’t a sign of a focused career that has an outcome in mind, it’s a sign of someone just kind of following their nose. Imagine getting into your car, turning on the GPS, not entering a destination, and then hopping on and off the highway. Where are you going to wind up? Sure, you might great sights along the way, but eventually you run out of gas… where?
Your career isn’t thee to serve your interests of the moment. It’s there to create a life for you (work to live, not live to work). It can’t do that if there’s no plan.
People kinda act like “there are technical skills, and then you move into management.” Management is a set of skills. It’s a distinct career path. It’s HARD, when it’s done right. I’m not sure you can be the best manager your people deserve if it’s… a hobby that you set aside when you don’t want to do it anymore. If you’re in management and you still want to keep your hand in tech, find a deserving community project that needs you. Find a “Kids Who Code” charity and offer to be an instructor. There are lots of ways to “stay in tech” without making it a fractured career.
**Tim Becker**
Hi Don Jones thank you for all your answers so far. I was wondering if you have some advice on how to help improving a team from within. For example, motivating the team to commit to a certain set of minimum standards concerning code standard, PR, etc.. Also how would you motivate your colleagues to reach certain KPIs.
**Don Jones**
What motivates YOU?
Watch this: [https://www.youtube.com/watch?v=fW8amMCVAJQ](https://www.youtube.com/watch?v=fW8amMCVAJQ)
That’s “leadership from within.”
Your colleagues should WANT to reach those KPIs, because it’s what they get PAID to do. Where you can help is showing them that it’s achievable. Showing them that it matters. Showing them that it’s WORTH doing. That’s not going to work for everyone, and those people should consider leaving, but it will hopefully work for a lot of people. Show them WHY these things matter. be passionate about it—let them hear the excitement and conviction in your voice. Don’t lecture… _show._ Show the upsides of code standards… don’t just talk about it. That kind of thing.
**Tim Becker**
How to best collaborate with young highly motivated colleagues that are still missing a lot of the technical skills and with more senior colleagues that might have lost a bit of their drive?
**Don Jones**
Remind the senior people what it was like to be young. Remind them how many people took time out of their day to help THEM. They owe this to those people—paying it forward. We’re all just stupid monkeys on this planet; we need to help each other get along. Acknowledge the younger folks’ excitement and spirit—don’t beat it up. Life will beat them up plenty, you don’t need to help. Heck, maybe they can help raise the bar for everyone, in time—if their elders can set the right tone and give them the right support.
For the younger folks, let them know how interested you are in helping them. Let them know it’s okay to not know everything, and that it’s far more important to be a #DailyLearner. Let them know their questions are welcome. Let them know their perspectives are welcome—and that they need to seek context to understand why things are the way they are, but beyond that… they’re welcome to think about better ways that meet the same goals.
**Tim Becker**
thank you Don Jones I will definitely try to act more in this way
**Rosona**
Hi Don Jones! Are you on LinkedIn? I was hoping to follow you for awesome content and your name is a bit too common for me to quickly find you (figured I’m not the only one interested, so makes sense to post instead of dm).
**Don Jones**
Yup. /in/concentrateddon.
**Rosona**
Ok, one more question. Don Jones, you said in your answer above that your generalist nature did in the end land you in a VP role. Could you say something about that transition ? Are your generalist skills still your biggest asset, did they nicely power up your acquiring of management-related skills for such a leadership position?
**Don Jones**
So, I ran my own company for about 14 years. Being a generalist is 100% key in that scenario—you are the Finance team, the Sales team, the Production team, everything. I still think those generalist skills, as a VP, are my biggest asset—I can dive into a variety of topics, understand them quickly, and contribute meaningfully. It lets me interact and engage with a variety of teams in a variety of ways and really Get Stuff Done. Management skills are one of the things I’m a generalist at, and I’ve become more and more competent at those as I’ve worked in management—but I can also write a PowerShell script, when I need to ;).
**Don Jones**
You’re all welcome to follow/connect on LinkedIn ([https://LinkedIn.com/in/concentrateddon](https://linkedin.com/in/concentrateddon)
) and Twitter ([https://twitter.com/concentrateddon](https://twitter.com/concentrateddon)
), and I’m happy to take future questions (I watch Twitter the most, but please post publicly versus a DM so everyone can play along). And as I’ve mentioned, you can also find me at [https://ampere.club](https://ampere.club/)
, and there’s an “Ask” page you’re welcome to use anytime. For 2022, Ampere Club is going to have monthly career-skill themes, with small weekly activities designed to help you level-up in continuous, minor ways (like Agile for your career!). It’s all free, of course.
**Tim Becker**
thank you, really cool stuff!
**Don Jones**
I want to thank everyone for the fantastic questions! I’m going to genericize some of these and post longer answers, in the form of articles, at [https://ampere.club](https://ampere.club/)
, because I think your perspectives and questions reflect those of a much wider audience. Hopefully, my extended answers can even provide a little bit of extra value, and hopefully you’ll continue to pop by and ask questions!
**Raghav Bali**
Thanks Alexey Grigorev for yet another interesting week.
Question I have regarding the book:
“What according to you is a major drawback while using fastai as compared to TF or PT (apart from the fine grained control we have from these 2 frameworks)?”
Again kudos Mark Ryan for such a detailed book, the TOC looks quite comprehensive 👍
**Mark Ryan**
Hi Raghav Bali - fastai is a great platform, particularly for beginners. That being said, I see two noteworthy drawbacks: (1) compared to TF/Keras or vanilla PyTorch, there are not as many examples of fastai being deployed in production, (2) regressions. Compared to Keras, it is easier to hit regressions with the fastai platform (that is, something that used to work that stops working after a platform update), and you need to be prepared to work around regressions if you come across them in fastai. An example of a regression in fastai is the code to set the random seed, which stopped working as expected after an update. In addition to these drawbacks, the documentation for fastai is not as comprehensive as Keras documentation, but the fastai course forum [https://forums.fast.ai/](https://forums.fast.ai/)
helps to make up for that.
**Raghav Bali**
Awesome… Thanks for the detailed answer
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Deep Learning with fastai Cookbook – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Deep Learning with fastai Cookbook
----------------------------------
#### by [Mark Ryan](https://datatalks.club/people/markryan.html)
##### The book of the week from 06 Dec 2021 to 10 Dec 2021
The book begins by summarizing the value of fastai and showing you how to create a simple ‘hello world’ deep learning application with fastai. You’ll then learn how to use fastai for all four application areas that the framework explicitly supports: tabular data, text data (NLP), recommender systems, and vision data. As you advance, you’ll work through a series of practical examples that illustrate how to create real-world applications of each type. Next, you’ll learn how to deploy fastai models, including creating a simple web application that predicts what object is depicted in an image. The book wraps up with an overview of the advanced features of fastai.
* [Book's page](https://www.packtpub.com/product/deep-learning-with-fastai-cookbook/9781800208100)
* [Amazon](https://www.amazon.com/Deep-Learning-fastai-Cookbook-easy/dp/1800208103)
Questions and Answers
---------------------
**Raghav Bali**
Thanks Alexey Grigorev for yet another interesting week.
Question I have regarding the book:
“What according to you is a major drawback while using fastai as compared to TF or PT (apart from the fine grained control we have from these 2 frameworks)?”
Again kudos Mark Ryan for such a detailed book, the TOC looks quite comprehensive 👍
**Mark Ryan**
Hi Raghav Bali - fastai is a great platform, particularly for beginners. That being said, I see two noteworthy drawbacks: (1) compared to TF/Keras or vanilla PyTorch, there are not as many examples of fastai being deployed in production, (2) regressions. Compared to Keras, it is easier to hit regressions with the fastai platform (that is, something that used to work that stops working after a platform update), and you need to be prepared to work around regressions if you come across them in fastai. An example of a regression in fastai is the code to set the random seed, which stopped working as expected after an update. In addition to these drawbacks, the documentation for fastai is not as comprehensive as Keras documentation, but the fastai course forum [https://forums.fast.ai/](https://forums.fast.ai/)
helps to make up for that.
**Raghav Bali**
Awesome… Thanks for the detailed answer
**WingCode**
Hi Mark Ryan,
What is the advantage of using DataBunch or abstracted data types in fastai versus using pandas DataFrame & converting it to numpy for training the model?
**Mark Ryan**
Hi WingCode - the advantage of the abstracted data structures in fastai is that they are integrated into the deep learning workflow and make it easy, for example, to look at a sample batch or examine individual elements in a dataset. Simply put, they help make it easy to focus on the essential tasks of creating and training a deep learning model without having to write a lot of additional code.
**WingCode**
Thank you Mark for the answer 🙂
**John Trengrove**
Hi Mark Ryan does the book have examples for combining vision / tabular / text models together?
**Mark Ryan**
Hi John Trengrove - that is a great question. There are examples in the book of vision, tabular, text and recommender system models, but there aren’t examples that combine these data types.
**WingCode**
In the book’s TOC,
`Chapter 5
Training a recommender system on a small curated dataset
Training a recommender system on a large curated dataset`
How does [fast.ai](http://fast.ai/)
training differ for a recommender system on small curated dataset vs a large curated dataset? Do you use a collaborative filtering in the 1st one since it’s a small one and can fit into memory? For the 2nd one, do you use some content based approach?
**Mark Ryan**
Hi WingCode - the recommender system with the small dataset was an attempt to demonstrate an MVP of fastai’s recommender system support. The resulting model works but it is of limited practical value because the columns in the dataset don’t mean much to humans (it’s a movie rating dataset with no movie titles). Once the reader has been through that example, it motivates the second example with the large dataset (which is more than 10 times bigger than the small dataset and has a much more complex structure). The second example takes many more steps to get it to work, but the result is more satisfying - you see the recommender system making predictions about the rating a particular user will give for a movie that is generally thought to be excellent (L.A. Confidential) vs. a movie that is generally thought to be terrible (Showgirls).
**Mark Ryan**
The reason for doing an MVP first and then a bigger example is that some of the existing material on fastai doesn’t take the time to do an MVP and jumps right into the full-blown solution with a big, complex dataset. The problem with skipping the MVP is that it’s hard to figure out the reason for the additional steps for the big, complex dataset if you haven’t already grasped the minimum set of steps.
**WingCode**
Yes that makes sense Mark. Thank you again 🙂
**Allan**
Mark Ryan I think the approach of showing the MVP approach first makes a lot of sense, for the reasons you described. It also gives the reader a good example of how things might be done for a real-world problem or project they might be working on… start first with something simple and small to evaluate/validate the approach before moving on and investing the time on something more realistic and complex.
**Carlos Orjuela**
Hi Mark Ryan, how’s the Cookbook different compared to Jeremy’s book. What can we expect from it? Thanks
**Mark Ryan**
Hi Carlos Orjuela - the book by Jeremy Howard and Sylvain Gugger is a master work. It not only covers exhaustive details about fastai, it also takes the reader through many aspects of the theory of deep learning. It also includes a whole chapter on AI ethics. By contrast, my book is focused on using fastai to solve practical problems. It includes details on setting up an environment to use fastai and then has a chapter each on the 4 major application areas supported by fastai: tabular data, text data, recommender systems, and image data. In each of these chapters, you’re led through a working code example for a fastai curated dataset and then working through an example for a standalone dataset. My book wraps up with a chapter on model deployment and then a chapter on more advanced topics (including callbacks, augmented data, and more additional deployment tips).
**Mark Ryan**
So, Jeremy Howard’s book can teach you many general deep learning concepts and covers all kinds of nuances and details about fastai. It is also a complement to the deep learning for coders course [https://course.fast.ai/](https://course.fast.ai/)
. My book is focused on step-by-step application of fastai to solve practical problems.
**Carlos Orjuela**
Thanks Mark Ryan for the detailed response 🙂
**Haseeb Arshad**
HiMark Ryan!!! What is the best part of this book? Is this book good for beginners to start and learn about fastai? Thank you!
**Mark Ryan**
Hi Haseeb Arshad - thanks for your question. As for the best part of the book, it is probably not right for me to say so as the author, but if you look at the reviews on Amazon you will see some of the things that people liked about the book: [https://www.amazon.com/Deep-Learning-fastai-Cookbook-easy/dp/1800208103#customerReviews](https://www.amazon.com/Deep-Learning-fastai-Cookbook-easy/dp/1800208103#customerReviews)
. I do believe that the book is good for beginners to learn about fastai - it takes you from the very start of setting up an environment all the way to deploying a trained model.
**Carlos Orjuela**
Hi Mark Ryan, gathering from your previous answers and in particular from the drawbacks [Fast.ai](http://fast.ai/)
has in your opinion, the focus of your book helps to pave the way of showing more practical examples and deployment into production tips, is that a fair assumption? Thanks again
**Mark Ryan**
Hi Carlos Orjuela - the book covers practical examples and talks about ways to work around some of the drawbacks of fastai. It also describes the many strong points of fastai, such as (1) the way fastai provides comprehensive support for tabular datasets and (2) how fastai makes it easier than Keras to ensure that data goes through the same transformations when the model makes a prediction as the data went through when the model was trained. The book has a whole chapter on deployment. Deployment is a huge topic, so this chapter doesn’t cover all deployment options. In particular, many production deployments would be via a cloud platform like AWS, Google Cloud, or Azure, and the book does not explain how to do that.
**Carlos Orjuela**
Thanks again for your reply Mark Ryan
**Allan**
Thanks Mark Ryan for taking the time to answer questions here. From the table of contents it looks like a great book! Would you say that [fast.ai](http://fast.ai/)
is mainly appropriate for someone who wants to build practical deep learning models without necessarily needing to understand the underlying DL theory? If one wants to understand deep learning well, would you say, as I think Jeremey H posits, that by taking a top-down approach and becoming a practitioner first, that one is then in a better position to dive-in and understand the details and inner workings?
**Mark Ryan**
Hi Allan - I think it’s fair to say that Jeremy Howard’s philosophy about learning deep learning is that it’s best to start by doing. His course (and the fastai framework) emphasize getting something to work in code first and then go back and dig into the theory / math. He contrasts the fastai approach with the traditional approach to learning deep learning, which starts with abstractions and theory that can be offputting to somebody who is trying to use deep learning to solve a real-world problem. I think that Howard wants fastai to be used by people who don’t have an academic background in machine learning but do have deep subject area expertise and access to interesting data sets.
**Tim Becker**
Hi Mark Ryan, I was wondering if there is a difference between deploying fastai models and TF, Keras models? I can imagine that you might end up with large docker containers?
**Mark Ryan**
Hi Tim Becker - in the book I did a simple “from the ground up” web deployment of some fastai models using Flask. In this scenario, fastai was easier than Keras. To do the same kind of deployment with Keras for a tabular model, I needed to worry about the pipeline and I needed to load the model repeatedly. With fastai, the pipeline was taken care of automatically and I only had to load the model when the web page initially loaded, leading to a smoother experience.
**Tim Becker**
If you already know how to build models with keras, where would deliberately choose fastai?
**Mark Ryan**
The question of Keras vs. fastai is a good one. Keras definitely has some advantages over fastai, including a much larger user community and better documentation. If that’s the case, why would somebody who has already used Keras go through the effort of learning to use fastai? I can think of two reasons (a) gateway to PyTorch for somebody who has been working in the TF sphere. fastai provides a low-impact way to ease into PyTorch for somebody who is familiar with the TF ecosystem and wants to take a peek at the “other side”. (b) full-throttle support for tabular datasets. It’s certainly possible to create models trained on tabular data using Keras (I spent another book on that topic: [https://www.amazon.com/Deep-Learning-Structured-Data-Mark/dp/1617296724/ref=sr\_1\_1?keywords=deep+learning+with+structured+data&qid=1638993209&sr=8-1](https://www.amazon.com/Deep-Learning-Structured-Data-Mark/dp/1617296724/ref=sr_1_1?keywords=deep+learning+with+structured+data&qid=1638993209&sr=8-1)
) but fastai makes it much easier.
**Tim Becker**
Mark Ryan Thank you for your answers, very interesting! Could you elaborate a little bit what you mean by additional support for tabular data?
**Allan**
This is very helpful!
**Allan**
Mark Ryan I see from the TOC that there is a chapter on extended [fast.ai](http://fast.ai/)
and deployment features? Can you expand on what is covered there?
**Mark Ryan**
Hi Allan - there is a chapter in the book on deployment. This covers how to deploy fastai models trained on tabular and image datasets in a simple web application using the Python Flask library. These deployments include everything - the Flask module, the HTML and CSS files, and the trained models, as well as detailed instructions on how to set them up. The final chapter of the book gives additional detail on web deployment - for example, showing you how to show thumbnails of the images the model is making predictions on and showing you how to make a deployment on your local machine available to the external web. The final chapter also covers (a) fastai callbacks - these make the fastai training process efficient by avoiding non-productive training epochs and ensuring you exit the training run with the optimally trained model (b) fastai’s built-in support for augmenting image data (c) various features for getting additional details about models trained with fastai, such as confusion matrices and data points where the model did worst.
**Allan**
Thanks Mark Ryan , sounds like a very practical couple of chapters that adds a lot of value!
**Allan**
Also, a little bit off topic… but I was wondering what your take is, in general, on PyTorch vs Tensorflow - putting aside the fastai/keras layers that make each of them easier to use. It seems a lot of folks these days are moving toward and favoring PT… 🙂
**Mark Ryan**
Hi Allan - the conventional wisdom is that TF is used more by industry and PyTorch is used more by academics. The Stack Overflow 2021 survey backs this up: [https://insights.stackoverflow.com/survey/2021#section-most-popular-technologies-other-frameworks-and-libraries](https://insights.stackoverflow.com/survey/2021#section-most-popular-technologies-other-frameworks-and-libraries)
. I’d say that it’s hard to compare PyTorch and TF without bringing Keras into the discussion. After all, it is now the official high-level API for TF, and it’s used extensively, even by specialists. Search the Stack Overflow survey for Keras and you will see it, while fastai and Lightening (another high-level framework for PyTorch) are both missing. This could be because professional developers use PyTorch directly and don’t have use for a higher level framework, or it could be that developers working in industry are still predominantly on TF/Keras.
**Mark Ryan**
I can’t predict the future, but I think it would be a safe bet to say that while PyTorch will be adopted more by industry, TF/Keras isn’t going anywhere and will still be a significant part of the landscape in 10 years.
**Mark Ryan**
Hi - for anybody who is interested in more content on fastai, Keras (plus GPT-3, Codex, Rasa and other topics), please check out:
* my YouTube channel: [https://www.youtube.com/channel/UC33c02d4\_ssg0tGO3EXKG8g](https://www.youtube.com/channel/UC33c02d4_ssg0tGO3EXKG8g)
* my Medium blogs: [https://markryan-69718.medium.com/](https://markryan-69718.medium.com/)
**Allan**
Thanks for taking the time to answer questions here Mark Ryan !
**Mark Ryan**
Thanks Allan - I really enjoyed the chance to answer the questions this week. I appreciate the time that people took to share these great questions.
**Mark Ryan**
Congratulations to the winners, much thanks to Alexey Grigorev for hosting me here, and thanks to everybody who participated this week.
**Alexey Grigorev**
Are you working on another book right now? =) I bet you are and we can invite you again soon =)
**Mark Ryan**
Hi Alexey Grigorev - I’m not actively writing right now, but I have a couple of ideas in the works, so I look forward to coming back again to talk about the next book.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Mastering spaCy – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Mastering spaCy
---------------
#### by [Duygu Altinok](https://datatalks.club/people/duygualtinok.html)
##### The book of the week from 13 Dec 2021 to 17 Dec 2021
Build end-to-end industrial-strength NLP models using advanced morphological and syntactic features in spaCy to create real-world applications with ease.
spaCy is an industrial-grade, efficient NLP Python library. It offers various pre-trained models and ready-to-use features. Mastering spaCy provides you with end-to-end coverage of spaCy’s features and real-world applications.
By the end of this book, you’ll be able to confidently use spaCy, including its linguistic features, word vectors, and classifiers, to create your own NLP apps.
* [Book's page](https://www.packtpub.com/product/mastering-spacy/9781800563353)
* [Amazon](https://www.amazon.com/Mastering-spaCy-end-end-implementing/dp/1800563353)
Questions and Answers
---------------------
**Krzysztof Ograbek**
Wow, how great!! Thanks Duygu Altinok for doing this 🙂
**Krzysztof Ograbek**
My first question: What can you perform with spaCy but cannot with other NLP libraries (nltk, gensim, textblob, …)? Which features are unique?
**Duygu Altinok**
Thanks for the question!
First of all we want offer the users complete pipelines per each supported languages. In this aspect, NLTK and textblob are similar but gensim is different.
We include the following components:
* Sentence segmenter
* Tokenizer
* Lemmatizer
* Morphologizer
* NER
* dependency parser
* POS tagger
* Rule based matcher
* Entity linker
* Vectors & semantic similarity
* Tetxcat
* Spancat
**Duygu Altinok**
Problem with NLTK it’s not really suitable for industry-level usage. Some pretrained models are ancient but the main problem is the speed. I’d say it’s good for academical use, but not really suitable for production level NP.
Coming to unique components that you cannot find anywhere else I’d say:
* Morphologizer: This is a trainable component, calculate morphological features by looking at the word. one example: `hermosa-> singular, feminine`
* Rule based matchers are definitely unique and very useful components for extracting information based on patterns.
* Tok2vec: This component is really unique allows dependency parser, named entity recognizer and pos tagger to share a common NN. Also this layer generates word vectors for `oov` words, hence feature calculations of oov words don’t fail 😉 That’s a disadvantage in many libraries, misspelled words and some words that are not in the models vocab sometimes fails, sometimes work with the statistical models. We wanted to handle oovs in a proper way.
* Entity linker: This component disambiguates textual mentions (tagged as named entities) to unique identifiers, grounding the named entities into the real world via a KnowledgeBase
* Spancat: This is an uniquq and very useful component too, it can classify word spans. I’ll post an example project link.
**Duygu Altinok**
So each component is modifiable by your own training data, easy to use and really blazing fast.
Final remark is that we also have transformer-based pipelines. I consider it pretty uniquq as well 🙂
**Krzysztof Ograbek**
Oh yes! Rule-based matchers are awesome 🙂
**Duygu Altinok**
I really like playing with Matcher too, I enjoyed writing the Matcher chapter quite a lot. Also as Explosion we’ll soon publish some videos and I plan to make a Matcher video 🙂
**Doink**
what tasks can be done with Spacy more efficiently than a transformer?
**Duygu Altinok**
Thanks for the question:woman-raising-hand:
Here, paradigms are bit different. Transformers provide a state-of-art way of calculating context dependent word vectors and sentence representations. Hence if you feed your sentence to a transformer, you get a word vector for each token and a dense representation of your sentence.
If you want to do text classification, POS tagging or other downstream tasks with a Transformer
* You need to find an annotated corpus
* You need to train the Transformer by adding more layers on top (which should suit your task, seq2seq for NER, POS tagger or just softmax for text classification).
spaCy’s paradigm is different, we want to provide users with `pretrained` pipelines that can be usable immediately. This way you don’t worry about the training phase, directly start creating NLP applications. Here’s an example:
\`\`\`>>> import spacy
\>>> nlp = spacy.load(“en\_core\_web\_md”) #Load our pretrained model
\>>> doc = nlp(“I went there fast”)
\>>> for token in doc:
… token, token.pos\_
…
(I, ‘PRON’)
(went, ‘VERB’)
(there, ‘ADV’)
(fast, ‘ADV’)
Then do your stuff with the pos tags\`\`\`
==========================================
**Duygu Altinok**
Our pipelines are either based on word vectors or on Transformers. So we use Transformers for our downstream tasks too 😉
**WingCode**
Hi Duygu Altinok,
Does this book cover putting spacy into production?
**Duygu Altinok**
Moins, thanks for the question. There’s no dedicated section in my book. However, we aim our pipelines to be easily usable, hence integrating spaCy and pretrained models into your porject is not difficult at all. You can just add 2 lines to your project’s `requirements.txt` then you’re ready to go 🙂Here’s an example of what you can add:
`spacy>=3.0.0
[https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm-2.2.0](https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm-2.2.0)`
**Duygu Altinok**
This is quite pain free, usually with pretrained models
* DevOps guys need to download the model, store it on somewhere(usually) AWS
* Then put some S3 checkout lines to download the model to the project scripts
* and more work
Compared to that spaCy way is really painless😁
**WingCode**
Thank you Duygu Altinok for the answer 🙂
Follow up question, is there any aspects of spacy pipelines which makes it better than other framework pipelines (sklearn-pipeline, transformers pipeline etc) ?
**Álvaro Budría**
Hi Duygu, for students like me and people in academia, would you recommend spaCy over other alternatives such as Nltk? Thanks! 🙂
**Duygu Altinok**
Yes 🙂
Please refer to this previous answer: [https://datatalks-club.slack.com/archives/C01H403LKG8/p1639390846197000?thread\_ts=1639386552.194400&cid=C01H403LKG8](https://datatalks-club.slack.com/archives/C01H403LKG8/p1639390846197000?thread_ts=1639386552.194400&cid=C01H403LKG8)
**Álvaro Budría**
Oh I didn’t see that answer, thanks!
**Matthew Emerick**
Hey, Duygu Altinok. Thanks for doing this!
I have applied for many AI roles in the past year and read many NLP descriptions. I can barely recall just a few times when spaCy is mentioned. Why do you think the library is hardly mentioned when so many NLP applications are based on it?
**Duygu Altinok**
Many thanks for this interesting question:woman-raising-hand:
This is a point that is really curious to me, too. I know few companies where the whole application depends on spaCy but still no mention in the job add. Even if I see spaCy in the add, I see sth quite generic sth like this:
* familiarity with NLP libraries such as gensim, nltk, spacy
**Duygu Altinok**
I think the reason is copy-paste mania among HR. In the -3th company I worked for, HR was preparing the job adds and then show to us for corrections. Here’s one example of what he wrote down:
* terabytes of data (not true at all)
* state of art deep learning models (no DL, just some logistic regression)
* experience with spark (not used at all)
So, I asked him how he prepared this job add and I found out how most HRs prepare job adds: Look for some NLP job adds online and made a mix-n-match of those adds. This process converges to a very generic NLP job add at the end🧠
**Álvaro Budría**
Seems that technical people should get more involved in the hiring process!
**Matthew Emerick**
Is this book best for beginners, intermediate practitioners, or almost experts? Would college students interested in AI find this book useful?
**Duygu Altinok**
I included explanations of concepts along with examples, so this is a book I %100 recommend for beginners and intermediate level colleagues. Each chapter starts with explanations, include lots of code and ends with example applications.
For expert colleagues, I think they can find the all-hands-on chapters. Especially building a chatbot chapter would be interest to all NLP lovers. Rest of the book of course has teaching material as well, so I leave this decision to NLP experts themselves 🙂
**CJ**
I have spent an inordinate amount of my life explaining to people how to get information from text. Do you recommend starting with the easier bag of words, tf-idf models, or jump right into more complicated embeddings?
**Duygu Altinok**
I started from really the scratch because when I started (around 13 years ago) there was only tf-idf and bag of words 🙂 When word vectors came, I %100 understood the problem they wanted to solve and innovation they introduced. After that transformers came and the same, I appreciated the solutions that word vectors couldn’t offer.
I definitely saw the benefit of building up, when I’m vectorizing text I always know
* good sides and down sides of my approach
* shape of the vectors I generated, also the size they’ll take
* what sort of information my vectors encode
* why would my approach work
* for which cases my approach wouldn’t work and what can I do about it
So, I definitely recommend starting from scratch, also some tf-idf computations make students to warm up to the vector computations any way. However, young people are quite impatient and want to build cool stuff asap. I’m not %100sure they have the patience to practice from scratch 🙂 I’d say if this is a college class, I’d go ahead and teach it. If it’s a professional course for junior colleagues, I’d go over basics in 1 hr with some examples and then assign some reading material.
**Dmitriy Shvadskiy**
Hi Duygu Altinok Does the book cover training/finetuning Transformer as part of Spacy pipeline? If not what resources would you recommend on the topic. I was trying to integrate Spacy NER with Transformer few months back and it is not really obvious task
**Duygu Altinok**
OK, I covered the code of how to fine tune some components including NER and textcat.
If you want to fine tune a component, it’s no so difficult ; one just needs to provide some examples with labels. For your case, you can download a transformer-based model and then fine tune NER. You can see the example of how to provide examples from my book’s code: [https://github.com/PacktPublishing/Mastering-spaCy/blob/main/Chapter07/train\_on\_cord19.py](https://github.com/PacktPublishing/Mastering-spaCy/blob/main/Chapter07/train_on_cord19.py)
If you want to fine tune the transformer itself, it’s not an easy task because you need huge amounts of data. Transformers are giants, even with a huge corpus I noticed I only fined tuned maybe last 2 layers. If you want to go down this road then you need a spaCy project: [https://github.com/explosion/projects/tree/v3/pipelines/ner\_wikiner](https://github.com/explosion/projects/tree/v3/pipelines/ner_wikiner)
**WingCode**
Duygu Altinok
I see that spacy supports distributed training via ray. Is the text-preprocessing & inference pipelines also distributed?
**Duygu Altinok**
No, not supported due to Cython reasons.
**Krzysztof Ograbek**
Duygu Altinok Is it possible to perform stemming using spaCy? If yes, how? If not, why? I know there is lemmatization but I couldn’t find stemming
**Duygu Altinok**
Thanks for the question!
No, we don’t have a stemming component.
You can hunt down the `root` of a word either by lemmatization or stemming. Lemmas are more useful in general, so we prefer using lemmas.
**Evren Unal**
Duygu Altinok this question may be out of context.
I want to build an “arabic to turkish” dictionary.
I surmise that i need to use a stemmer for it.
Would you suggest a tool for arabic word stemming?
**Duygu Altinok**
Hey ho!
For this task, you need a lemmatizer, stemmer won’t be much of help.
I know two good Arabic lemmatizers, Farasa [https://farasa.qcri.org/lemmatization/](https://farasa.qcri.org/lemmatization/)
and Madamira [https://camel.abudhabi.nyu.edu/madamira/](https://camel.abudhabi.nyu.edu/madamira/)
If you need further help you can always open an issue at spaCy discussions :woman-raising-hand:
**Evren Unal**
Thank you very much🙂
**Noa Tamir**
Hi Duygu Altinok and thanks for joining us for the Q&A this week!
I’ve been using spaCy a bit recently to detect sentences in legal text. I was really impressed with the documentation and how easy it was to get started.
For now, I’m particularly interested in the more advanced chapters, as I can see myself needing to tweak the model to achieve better results and haven’t gotten into that yet.
Would you say the learning curve for the advanced chapters is as quick as getting started or does it get steeper as you progress? What other domains / resources would be useful to speed up mastery of the advanced chapters?
**Duygu Altinok**
I’d say no, you need spend more energy on advanced chapters 🙂 Advanced chapters uses information from previous chapters, so one needs to really melt what they learnt so far into one pot to digest the content. Still, putting knowledge together and at the end how code ends up is fun 😉
**Tim Becker**
Hi Duygu Altinok thanks for being here and answering our questions. I would like to ask:
**Duygu Altinok**
Thanks for the questions!
**Tim Becker**
* What project would you recommend to do if you want to get started with spaCy?
**Duygu Altinok**
Universe has a number of projects: [https://spacy.io/universe](https://spacy.io/universe)
All of the resources are great. I can recommend negspaCy, EpiTator and Rule-based Matcher Explorer for new comers 😉
**Tim Becker**
* Are there drawbacks of using spaCy in comparison to other tools?
**Duygu Altinok**
I don’t know any 🙂
**Tim Becker**
* What is new in spaCy 3.0? Do you cover the new features in your book?
**Duygu Altinok**
That’s a long list 🙂 I can recommend Ines’s video ([https://www.youtube.com/watch?v=BWhh3r6W-qE](https://www.youtube.com/watch?v=BWhh3r6W-qE)
) and this great page: Rule-based Matcher Explorer
My book is spaCy 3.x compatible. I focused on how to create applications withspaCy rather than spaCy inetrnals.
**Tim Becker**
thank you Duygu Altinok
**Allan**
Hello Duygu Altinok, appreciate your taking the time this week to answer questions!
For someone getting started in NLP, would learning it using Spacy be appropriate? Or is there too much performed automatically by the library/framework that one would miss out on some basics?
**Duygu Altinok**
Thanks for the question!
**Duygu Altinok**
I’d say it’s a good start because one needs to understand the linguistic concepts to grasp the spaCy concepts.
If I was a starter I’d do the following:
* Get started with Keras
* Read Jurafsky’s chapters 1-7
* Get started with spaCy
* Read Jurafsky chapters 12-14
* Become more accustomed to working with Keras, work on some easy/mid-level text classification projects
* Advance with spaCy
* Do some seq2seq models with Keras
* Read rest of Jurafsky
More advanced NLP colleagues (including me 🙂 ) designs and uses different types of seq2seq archtiectures for different tasks. At the very end while coding you should always have a clear mind, almost all text based models based on a linguistic concept. Without grasping the linguistic concepts, it’s easy to get lost. This is why I offer learning spaCy, while using the library you get a chance to work with different concepts of linguistics. Hope it works 🙂
**Duygu Altinok**
Forgot to give the Jurafsky book: [https://web.stanford.edu/~jurafsky/slp3/](https://web.stanford.edu/~jurafsky/slp3/)
**Allan**
Thanks Duygu Altinok !
**Allan**
Another question… how robust are the pre-trained pipelines? In what scenarios would one need to (re)train their own model as opposed to just using the pre-trained ones.
**Duygu Altinok**
Pretrained pipelines are quite good and usable (at least for German, English, French, Spanish and Portuguese I tried those ones myself 😁 )
Some of the projects you need your own NER definitely, but some of the projects you don’t need to if your design includes general entities such as location, person, time-date, money entities. Here’s a some example code for parsing restaurant reservation utterances (extracting ents and intent) : [https://github.com/PacktPublishing/Mastering-spaCy/tree/main/Chapter10](https://github.com/PacktPublishing/Mastering-spaCy/tree/main/Chapter10)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Machine Learning Engineering with Python – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Machine Learning Engineering with Python
----------------------------------------
#### by [Andrew McMahon](https://datatalks.club/people/andrewmcmahon.html)
##### The book of the week from 10 Jan 2022 to 21 Jan 2022
Machine Learning Engineering with Python takes a hands-on approach to help you get to grips with essential technical concepts, implementation patterns, and development methodologies to have you up and running in no time. You’ll begin by understanding key steps of the machine learning development life cycle before moving on to practical illustrations and getting to grips with building and deploying robust machine learning solutions. As you advance, you’ll explore how to create your own toolsets for training and deployment across all your projects in a consistent way. The book will also help you get hands-on with deployment architectures and discover methods for scaling up your solutions while building a solid understanding of how to use cloud-based tools effectively. Finally, you’ll work through examples to help you solve typical business problems.
* [Book's page](https://www.packtpub.com/product/machine-learning-engineering-with-python/9781801079259)
* [Amazon](https://www.amazon.co.uk/Machine-Learning-Engineering-Python-production-ebook/dp/B09CHHK2RJ)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Effective Pandas – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Effective Pandas
----------------
#### by [Matt Harrison](https://datatalks.club/people/mattharrison.html)
##### The book of the week from 31 Jan 2022 to 04 Feb 2022
Best practices for manipulating data with Pandas. This book will arm you with years of knowledge and experience that are condensed into an easy to follow format. Rather than taking months reading blogs and websites and searching mailing lists and groups, this book will teach you how to write good Pandas code.
* [Buy the book](https://store.metasnake.com/effective-pandas-book)
* [Announcement post](https://hairysun.com/announcing-effective-pandas.html)
Questions and Answers
---------------------
**Tim**
Kindly, I would like to know:
How this book is different from the other books that @Matt Harrison has written?
**Matt Harrison**
Hi Tim, Great question. My first Pandas book, _Learning the Pandas Library,_ was one of the first books on Pandas. After that book I had the chance to use Pandas even more, teaching and consulting with it. I had been planning on updating it after those experiences. In the meantime I was approached to do the second edition of the _Pandas Cookbook_. I had read the 1st edition and liked it. Though I added a few chapters to the cookbook, I also re-wrote almost all of the code. I think it is a great book. However, it is not “my” book. (edited)
**Dr Abdulrahman Baqais**
Thank you. Sounds an interesting book. I would like you ask few questions:
1) What do you mean effective? Time performance, readibility, best practise among other choices….
2) Some of the methods performed by Pandas can be done by other libraries such as : nupmy or sciket learn. For example:null impuatation or categorical varible conversion. Was there a comparison ?
3) Pandas is evolving and 1.0 already there. Do you think thaere is still a large room for improvement in a Pandas and in which areas?
4) When it is advisable to use Pandas? Any guirlines on the best cases where Pandas are preferred over R packages for example.
5) Where are the neck pains of pandas. For example transpose a dataframe is very slow comapring to transposing a numpy array.
Thank you so much and your expertise surely will help us.
**Matt Harrison**
Hi Dr Abdulrahman Baqais
1. Yes. You can read reviews. This book teaches a Pandas style that few teach, however it will completely change your Pandas code.
2. No. This is not an ML book. I was tempted to do an ML chapter but early reviewers advised against it. Perhaps in the next edition.
3. Great question. The Pandas API is HUGE. However, if you master a small subset, you can be really productive. Personally (in consulting, training, and using Pandas), I haven’t found post 1.x features to be used much. What is more interesting to me is optimization and the notion of the Pandas “API” being the standard interface for interacting with data in Python. (See libraries for scaleout, GPU, etc that all have a Pandas API).
4. I don’t use R in happiness or anger.😉 So I’m heavily biased for using Python. Use Pandas when you have “small” data. Use the Pandas API after that.
5. Great example. Numpy is excellent for matrices of like-typed data. You can use Pandas for that but as you mention things like transpose aren’t really meant for hetergenous data types. Index assignment is probably the biggest thing. It leads to confusion and bugs.
(edited)
**ouskä**
Hello, thank you very much for doing this. I would like to know:
1. Who is your target audience for this book?
2. Are there any prerequisites needed to get the most out of the book?
3. How is it Different than the _Pandas 1.x Cookbook (with Ted_ Petrou) ?
4. There are so many Python-based libraries out there which can be used for a variety of data science tasks. Where does pandas fit into this picture?
5. What are the most challenging lessons that you have learned while working on this book?
6. What advice would you have for beginners in data science/engineering? What things should they keep in mind while designing and developing their data science/engineering workflow?
7. Are there any specific resources which we could refer to, apart from this book of course?
Thank you.
**Matt Harrison**
ouskä, thanks for your questions.
1. The target audience is anyone who wants to write better pandas code.
2. Basic Python skills: Functions, loops. After that, I really encourage Lambdas and Comprehension constructs, and argument unpacking.
3. See my answer to Tim. Cookbook is a great book, however, it is not the book I want.
4. If you have tabular data that fits on your machine, Pandas should be at the top of your list. If it doesn’t fit on your machine consider the pandas “API” (spark, dask, modin, etc).
5. With respect to book authoring: books always take longer to write. (I wish it were out a year earlier). With respect to the content: finding real-world data to illustrate concepts is challenging. Also, the API is HUGE, considering what percent to cover is always a tradeoff.
6. Work on projects that interest you and you can learn from. Learn the basics of software engineering even if you don’t want to be a “programmer”. You are using a programming tool.
7. Watch my PyData talk for some of the big ideas from the book [https://www.youtube.com/watch?v=zgbUk90aQ6A&t=4604s](https://www.youtube.com/watch?v=zgbUk90aQ6A&t=4604s)
**Tim**
Matt Harrison Thank you very much for your response. Well noted. Second question if you don’t mind.
* There have been some errors that occur due to the latest version of pandas. Is there a section in your book handling this? eg performance warning, Future warning ‘your data frame is defragmented’
**Matt Harrison**
I intentionally write (and teach) Pandas in a way to avoid errors/warnings.
**Matt Harrison**
I do have some content on date and timezone conversion issues.
**Tim**
What is the pricing? Does it come on paperback only or there is a kindle version?
**Matt Harrison**
Paperback is on Amazon. Digital version at [https://store.metasnake.com](https://store.metasnake.com/)
**Tim**
Okay. Thank you very much Matt. I appreciate the time you took to answer my questions. It was nice chatting with you. I certainly hope I can get my hands on this book.
Good luck and stay safe.
**Philip Dießner**
Hi Matt Harrison, thanks for being with us!
1. What is your favorite functionality in Pandas that you would never want to miss? (If it is chaining, I also would be interested in #2 )
2. What do you think are the most common anti-patterns, which might trap people with experience in Python and starting with Pandas?
3. What part of the API are you using now very often that you picked up on far too late?
**Matt Harrison**
1. Great question. I would probably say the ease of grouping and aggregating or visualization.
2. inplace, index assignment, .apply, reading bad advice on the internet. 😉
3. Chaining. Not really an API, but a game-changer.
**Alexey Grigorev**
can you show an example of chaining?
**Matt Harrison**
See the get\_sales function in attached image
**Alexey Grigorev**
looks nice, thanks!
**Michael Harty**
inplace did seem like such a great idea when I first heard about it 😅
In reference to Alexey’s comment below about dplyr, I have barely used it, but many of my teammates use it, and they have shown me the basics. Not too long ago I thought. Wouldn’t it be nice if we could do this piping in pandas.
Turns out you can and it’s called method chaining. My python code has started to look a lot more similar to the dplyr pipes (and a lot better too) once I started learning how method chaining works
**Philip Dießner**
Matt Harrison Thanks for your answers! Regarding 2., as I have experienced the others but not this, could you explain the index assignment a bit or give an example?
**Matt Harrison**
Using index assignment makes it hard to chain because assignment doesn’t return a pandas object to chain on. Also, it exposes you to potential bugs because you might be working a view or a copy.
**Michael Harty**
What exactly do you mean by index assignment?
**Matt Harrison**
Updating a column by doing something like this. `df[col] = new_col`
You should be doing this instead`df.assign(col=new_col)`
**Alexey Grigorev**
What do you think about the convergence to this pandas API? Wouldn’t the world be better off if instead everyone used something similar to [dplyr](https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html)
?
**Matt Harrison**
The Python world (which is arguable 10-100X bigger than the R world) has converged on the Pandas API. I personally don’t think Pandas is the perfect API (certainly there are tough spots for beginners), but it is where we are at today.
I think it is a good thing in that an investment in learning pandas turns out to be a super-power when you need to move to Spark or Dask.
**Evren Unal**
Hi Matt Harrison
I would like to learn a tool which i can use to manipulate ndim array easly
Is Series enough for this job, or do i need to learn data Frames as well?
**Matt Harrison**
For n-dimensional arrays of the same type use numpy.
WRT pandas, +80% of the interface of Series and DataFrame is the same (one is applied to 1-dimension, the other 2-dimension)
**Evren Unal**
Thank you 👍
**Varun Nayyar**
Hey Matt Harrison welcome and congratulations on your latest Book.
It would be great if you could answer a few questions I have.
1. How superior is it to your Pandas cookbook from Packt?
2. Is the book for beginners or for people with some prior experience in pandas? Or indifferent to all levels?
3. How easy would it be for someone after this book to transition to PySpark for big data manipulation?
4. Why do you think pandas is a better choice for data manipulation when compared with other Libraries in python?
5. Would you say that this book is a reference book or does it require thorough reading to get the best out of it?
6. Is there some content dedicated to what after pandas? For example preprocessing and feature engineering for ML models?
Forgive me if there’s any overlap in the questions which you previously answered.
All the best for your book.
Hoping to grab a copy✨
**Matt Harrison**
1. Much superior 😉 (See other threads)
2. Anyone who wants to improve their Pandas. I have reviews from beginners and experience alike that this is changing how they think about Pandas.
3. PySpark recently merged the “Pandas API” 😉
4. It is better for “structured” or “tabular” data. Because it is built for that use case but also leverages NumPy so it is speedy and memory efficient.
5. There is a reference section at the end of each chapter. However, you can pick chapters that might be relevant to you.
6. It is focused on Pandas. There is an example of ML in the book, but it is not the focus.
Cheers!
**Bhaskar Sarma**
Hi Matt Harrison congratulations and thank you for deciding to answer our questions on this community.
I would like to ask three questions:
1. What is the presentation style in the book? Do you present through independent small examples through the whole book or, you start with small examples and then build up to a big project towards the end?
2. In one of the previous answers, you have told that “Use Pandas when you have small data. Use the Pandas API after that”. Where do you draw the line between small and big data (if you want to call it) ?
3. What is the difference between Pandas and Pandas API ? Asking as a beginner in Data Science and Pandas.
Thank you
**Matt Harrison**
1. There are “projects” scattered throughout the book. However, in contrast to many Pandas resources, I try to use real-world data throughout. So my code may be a little more complicated because it is not using canned data.
2. Small data (IMO) is data that will fit on a single\* machine.
3. Pandas implements the Pandas API. So does Dask and Spark and about a dozen other projects.
\`\* - Small varies as a single machine might have a few gigs to 100+gigs these days.
**William Jamir**
Congratulations on your book!
I would like to know more about your opinion on “inplace” operations, do you think there are valid cases to use it? (like memory consumption perhaps)
Also, does your book address best practices around inplace oprerations, or does it “completely” ban its use?
**Matt Harrison**
I recommend disregarding “inplace”. Don’t use it. In general it doesn’t do what you think it does and it prohibits chaining. There is a dialog among pandas devs to remove it completely.
Yes, my book has best practices which include “don’t use inplace”. Check out the reviews at [https://store.metasnake.com/effective-pandas-book](https://store.metasnake.com/effective-pandas-book)
to see how people are changing their code after going through the book.
**Michael Harty**
One of first things I noticed when trying your style of method chaining is that black code formatting does not agree with it. Have you found any formatters that work? Or in your opinion, is it best to not use one for pandas code?
**Matt Harrison**
I’ve thought about making “black panda”… (not enough time in the day).
Yes, black is horrible for properly chained pandas code. Right now I just format it manually. 😢
**Michael Harty**
oh yeah, “black panda” would be great
**A**
Hello Matt!
In a world where currently either most Data Scientists either strive for more core modelling skills or core engineering skills (mainly with respect to being more MLOps aware), how would you place skills like writing effective pandas code?
Also, what do you think is the biggest reason why people fail to write effective pandas code? Do you think is it because pandas is mainly used for exploratory work and thus, Data Scientists do not see much upside in learning it effectively or is this lack of awareness about what exactly is effective pandas?
Good luck with your book!
**Matt Harrison**
Pandas is useful for both exploring and creating pipelines. Generally the former is done in a very loose manner. The latter is done by software engineers. Many “scientists” claim they don’t want to be programmers and ignore best-practices.
Most of the examples of pandas code on blogs, videos, books, or otherwise push learners to use poor coding practices. I think that is because we are pretty early on with the library and crossovers from science to programming. Take heart, best practices are emerging though!
**Michael Harty**
I’m curious in your experience teaching pandas do you take a different approach based on your specific audience’s background? Im thinking there might be a different approach to a team of analysts that work largely in excel (like many of the engineers I work with) versus analysts who are more familiar with SQL but not Python and pandas. Thanks 🙏🏻
**Matt Harrison**
In my live courses I generally try to cater the content to the experience level of the attendees. Certainly I’ve seen that Excel experts prefer .pivot\_table while database gurus like .groupby. :woman-shrugging:
**Tim Becker**
Hi Matt Harrison, thanks for this interesting book! I would like to know what are the biggest anti-patterns in terms of computational speed. For example, if I remember correctly, iterrows seems to be very slow. Is there a situation in which you would recommend iterrow?
**Matt Harrison**
You want to stay in the “fast path” in pandas for numeric operations. Using .apply and .iterrows are generally bad for numeric performance. However, they are fine for string operations (though you may want to consider conversion to categoricals) as strings are already on the “slow path”.
I generally only find myself using .iterrows when I’m tweaking labels on a plot.
HTH
**Beau Waldrop**
Hi Matt Harrison, what’s something in pandas that is super useful that you feel doesn’t get used enough by a lot of people working in data?
**Matt Harrison**
Great question.
Many people don’t realize how easy it can be to visualize with pandas. Especially with an aggregation.
**Aruna Sri**
Hi Matt Harrison,
Congratulations on the release of the book, I am sure it will cover many techniques to help the ML community. Could you share your insights in reducing latency time with Pandas operations for real time ML applications especially computer vision.
Thanks
**Matt Harrison**
Hi Aruna, You want to stay in the “fast path” for numeric operations. Avoid .apply, .iterrows, etc. Use vectorized operations where possible.
**William Jamir**
For a team that doesn’t have much experience with pandas, what is your suggestion to better educate the team to write “idiomatic” pandas code (besides reading your book 🤓) ? Do you know any tools/linters that can help with this task?
**Matt Harrison**
Hi William, I’m not aware of linters for pandas, however I am very opinionated regarding best ways to educate a team.😉
In fact, I wrote a post about it … [https://www.metasnake.com/blog/learn-python-2021.html](https://www.metasnake.com/blog/learn-python-2021.html)
**Roy Jafari**
Hi Matt! I just ordered the book and looking forward to reading and improving my pandas’ coding and knowledge.
Obviously, you have a great depth of experience and expertise with programming and pandas and I am thankful that you are sharing that with us through this book. My question is exactly about this experience and expertise. How would you describe it? How many years and what type of experiences has led you to this exciting book?
**Matt Harrison**
Hi Roy, these days anyone can write a book. You just need to want to. I will say that many of the practices in the book came after years of using Pandas and deriving strong opinions.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# A Visual Introduction to Deep Learning – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
A Visual Introduction to Deep Learning
--------------------------------------
#### by [Meor Amer](https://datatalks.club/people/meoramer.html)
##### The book of the week from 14 Feb 2022 to 18 Feb 2022
Deep learning can be quite daunting to learn. With the abundance of learning resources in recent years has emerged another problem-information overload.
This book aims to compress this knowledge and make the subject approachable. By the end of this book, you will be able to build a visual intuition about deep learning and neural networks.
40% discount for DataTalks.Club: use promocode “DATATALKS40”. Valid till February 19.
* [Book's website](https://gumroad.com/a/63231091)
* [Book sample](https://bit.ly/34Seg9l)
Questions and Answers
---------------------
**Rui Ramos**
Meor Amer is this book available in epub format, or only pdf?
**Meor Amer**
Hi Ruiz. It’s only in PDF right now. Because the book contains mostly visuals, it might be tricky to have an EPUB format that can give a good reading experience. But TBH I haven’t explored it in great detail, will definitely take a closer look.
**Rui Ramos**
Thanks 👍
**Meor Amer**
Hello everyone, excited to be here! I’m happy to take any questions about the book, creating visuals, or anything in between! I’ll try to answer to my best ability.
In the meantime, here are:
1 - Further details about the book - [https://gumroad.com/a/63231091](https://gumroad.com/a/63231091)
2 - A 40-page sample of the book - [https://bit.ly/34Seg9l](https://bit.ly/34Seg9l)
**Alexey Grigorev**
Hi Meor, welcome 👋
**Meor Amer**
Hi Alexey, thanks for having me here.
**Tino**
Hello Meor Amer. Thanks for taking the time here 🙂 In times of the rise of Explainable AI, where do you think you books contributes the most in this area?
**Meor Amer**
Hi Tino. It’s my pleasure! Unfortunately, the book doesn’t cover Explainable AI. Where this book can contribute is by helping readers build a clear framework of understanding before diving deeper into further deep learning topics like Explainable AI.
**Tino**
Got it! Thanks 🙂
**Gur Hevroni**
Hi Meor Amer, I really like your illustrations posted on LinkedIn!
I’m fascinated by visualization and the art of data storytelling, and I was wondering what makes your book different from other illustrated DL intro books?
**Meor Amer**
Hi Gur, thanks for checking out my posts and for your kind words! Indeed, there are already a number of amazing illustrated DL books from top authors. Where I try to add value is making a book that helps the reader navigate the various concepts with the least friction. For example, the same dataset is used in all chapters so you have the same, consistent reference. And the math is kept to an absolute minimum because the main goal is to help the reader build a strong intuition first.
**Gur Hevroni**
Great! Thanks, Meor Amer.
I’m looking forward to seeing more of your work :)
**Dustin Coates**
Hi Meor Amer. I’m interested in what you mention in intro in the sample as your initiative to get into ML: how is ML applied to prosthetics, and what is that enabling for people?
**Meor Amer**
Hi Dustin. Yes, that’s what got me interested in this field. It’s been quite some time and the progress was much slower then. But now it’s really exciting with the emergence of companies like Elon Musk’s Neuralink. The use case possibilities are huge, stroke rehabilitation and amputee assistance being a couple of examples. Though we are still some way from practical and affordable solutions, it has definitely being given a boost in recent years.
**ouskä**
Hello Meor Amer, appreciate your time to answer some questions.. I would like to know:
* Who is your target audience for this book?
* Are there any prerequisites needed to get the most out of the book?
* What the Book Doesn’t Cover?
* What are the most challenging lessons that you have learned while working on this book?
* What advice would you have for beginners in machine learning / deep learning?
Thank you.
**Meor Amer**
Hi ouskä
Who is your target audience for this book?
People just beginning their journey into deep learning before going further into the technicals. Also leaders looking to understand deep learning from first principles.
Are there any prerequisites needed to get the most out of the book?
No pre-requisites. This book is beginner-friendly.
What the Book Doesn’t Cover?
Mathematical derivations, code examples, and further topics such as optimizers, regularization, embeddings, etc.
What are the most challenging lessons that you have learned while working on this book?
I’m big on ensuring a piece of material has an element of continuity and storyline. So the challenge was making sure all the technical concepts are covered while finding a way to make the reading enjoyable.
What advice would you have for beginners in machine learning / deep learning?
Start with hands-on projects, then reading the theory will make much more sense. The initial steps could be reusing existing code e.g. Kaggle notebooks, then modify and build upon what’s there.
**Benedict Neo**
Hi Meor Amer, what gave you inspiration to visualize deep learning concepts in the way you did in your book? Was it challenging to make those visualizations? What tools did you use?
**Meor Amer**
Hi Benedict. My inspiration is the amazing designer Jack Butcher (Visualize Value). Ironically, what he’d taught me is you don’t have to be good at designing (that’s me!) to be able to come up with impactful visuals. It’s more about sharing your perspectives than worrying about the aesthetics. It is still challenging, but through practice, it gets easier. As for the tool, I use Figma - highly recommended!
**Jing Li**
Hi Meor Amer, really impressive to me and looking forward to your more outstanding books! Here are my questions:
1. Beginners like me can easily understand it after reading with the visual intuition, however, after some days, some of the fundamentals may get vague in my mind, can we have some practice tips to consolidate the concept for readers?
2. Do you plan to simplify the mathematic algorithms and other complicated stuff like that in ML/DL by the way of visualization in the book?
3. What’s your next plan on the book, expanding the other knowledge domain related to ML/DL or go deeper of each part within ML/DL knowledge?
4. Do you have plan to spread this type or series of books to more areas, like country as China where has big potential and a huge number of readers who have great interests on ML/DL knowledge?
5. Can anyone get the fortune to get full copy If he can translate it into Chinese edition and try to boost its local publication? :)
**Meor Amer**
Hi Jing
**Meor Amer**
1. I can totally relate. That’s why sometime soon I’ll release the accompanying code used to run the examples in the book. And yes, further down the line I do have ideas for creating additional hands-on exercises.
2. Yes, that’s my intention with the book - to help readers build an intuition first (by way of visuals) using the minimum possible math and equations.
3. The next one I’m planning is A Visual Introduction to Machine Learning.
4. Absolutely, I’m always open to potential collaborations to help make deep learning accessible for more!
5. That’s an interesting prospect. Would love to explore this!
**Jing Li**
Hi Meor Amer, thanks for your response, looking forward to collaborate with you on the Chinese edition of the books🤝
**Alexey Grigorev**
How do you come up with visualization ideas?
**Meor Amer**
Hi Alexey. Here are some that have worked for me:
* As for any generic ML visuals, it helps to observe for phenomenons/things outside of ML/Tech. Because a lot of things around us happen via similar universal principles, we can take one concept from one area and apply it to another.
**Meor Amer**
* As for the book, first I built the storyline of how I want to book’s content to flow. From there, I listed down the key concepts of each page. And then I figured out the appropriate visuals to depict the idea.
**Alexey Grigorev**
What tools do you use for drawing?
**Meor Amer**
I’m using Figma and I really love it!
**Hafiz Muhammad Arslan**
Have you added concepts like receptive field and how it affects?
**Meor Amer**
Hi Hafiz. Unfortunately that topic is not covered in the book.
**Hafiz Muhammad Arslan**
👍
**Tim Becker**
Hi Meor Amer, I was wondering if your book also covers some intuition that helps to design and tune neural networks?
**Meor Amer**
Hi Tim. Yes, the book covers these concepts. For designing, it covers the tweaks required in the neural network to address four types of tasks: linear regression, non-linear regression, binary classification, and multi-class classification. For tuning, it covers the common hyperparameters that can be used to improve performance.
**Ali Rabeh**
Hey Meor Amer thank you for answering our questions.
1-How long did it take you to write the book? And which part of it was the most time intensive?
2-Where do you think the future of Deep learning is heading?
3-Is having knowledge in Machine learning necessary to understand and follow your book?
4-Do you have new visualization books ideas in your mind for future work?
5-How did your self learn deep learning?
**Meor Amer**
Hi Ali. You are most welcome.
1-It took me around 3 months of almost full-time work. The most time-intensive is the continuous structuring of what to cover vs. not to cover to ensure that the whole flow makes sense.
2-Less labels and multi-modal i.e. getting closer to how humans learn.
3-The book is written with the beginner in mind. So the good news is you don’t need any background in ML to read it!
4-Yes, I do. The next I have planned is A Visual Introduction to Machine Learning. It will be more challenging to write though because it’s broad whereas deep learning is specific.
5-Initially by working on domains I am familiar/interested in e.g. telecommunications, biomedical engineering. Combined that with theory learning like courses, books, etc.
**Roy Jafari**
Hi Meor Amer ! Thanks for answering questions about your book. I would like to ask your opinion on who would have the most gain from this book? An audience with no prior exposure to deep learning, or an audience who is comfortable with it, or somewhere in between? I’d appreciate if you could elaborate. Thank you in advance!
**Meor Amer**
Hi Roy. This book is beginner-friendly and so it’s fine for those without prior exposure to DL. If you are already comfortable with it, then probably you won’t gain as much new information. But perhaps the visuals can still help give different perspectives to what you already know.
**Meghana**
Meor Amer I have gone through your sample pdf. there you said your journey started at 2010 after your son born with a limb difference. how did AI helped in prosthesis?
**Meor Amer**
Hi Meghana, thanks for going through the sample. Yes, that’s how I got started. It’s been some time and I’d say the progress hasn’t been as fast as people would hope it to be. Viable consumer applications are still not there yet. Having said that, I’m excited that companies like Neuralink and Kernel are bringing these ideas more into the mainstream, which help to bring more talents into this field and accelerate the progress.
**Meor Amer**
Thanks Alexey for inviting me here. It’s been so much fun interacting with you all and I appreciate the time you took to ask questions and share comments.
This community is amazing with phenomenal growth, and I’m sure this is just the start!
BTW, I will keep the 40% book discount over the weekend if you are interested to get it. Use the discount code: DATATALKS40
**Meor Amer**
[https://gumroad.com/a/63231091](https://gumroad.com/a/63231091)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hands-On Data Preprocessing in Python – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Hands-On Data Preprocessing in Python
-------------------------------------
#### by [Roy Jafari](https://datatalks.club/people/royjafari.html)
##### The book of the week from 28 Feb 2022 to 04 Mar 2022
Data preprocessing is the first step in data visualization, data analytics, and machine learning, where data is prepared for analytics functions to get the best possible insights. Around 90% of the time spent on data analytics, data visualization, and machine learning projects is dedicated to performing data preprocessing.
This book will equip you with the optimum data preprocessing techniques from multiple perspectives. You’ll learn about different technical and analytical aspects of data preprocessing - data collection, data cleaning, data integration, data reduction, and data transformation - and get to grips with implementing them using the open source Python programming environment. This book will provide a comprehensive articulation of data preprocessing, its whys and hows, and help you identify opportunities where data analytics could lead to more effective decision making. It also demonstrates the role of data management systems and technologies for effective analytics and how to use APIs to pull data.
By the end of this Python data preprocessing book, you’ll be able to use Python to read, manipulate, and analyze data; perform data cleaning, integration, reduction, and transformation techniques; and handle outliers or missing values to effectively prepare data for analytic tools.
* [Book's page](https://packtpub.com/product/hands-on-data-preprocessing-in-python/9781801072137)
* [Buy on Amazon](https://www.amazon.com/Hands-Data-Preprocessing-Python-effectively-ebook/dp/B09F6R8V2L/ref=sr_1_1?dchild=1&keywords=Hands-On+Data+Preprocessing+in+Python&qid=1634537236&sr=8-1)
* [GitHub repository](https://github.com/PacktPublishing/Hands-On-Data-Preprocessing-in-Python)
Questions and Answers
---------------------
**GerryK**
Hi Roy Jafari, are the tools /techniques referred in the book applied for big data?
**Roy Jafari**
Hi GerryK! Great question. Thank you for asking it.
The answer is yes and no.
Yes, from the perspective that only python programming with the use of NumPy and pandas are covered in this book. As you know, being able to prepare, and analyze data using programing is the very first step in being able to deal with big data.
No, from the perspective that the book does not cover computational optimization, parallel processing, or big data module such as Spark.
**Carlos Orjuela**
Hi Roy Jafari, I saw part 2 is dedicated to prediction, classification models and clustering. How broad is the book’s coverage on these topics and to what extent can it be used in place of others? Thank you!
**Roy Jafari**
Hi Carlos Orjuela! That’s an excellent question. I am glad you’ve provided a chance for me to address that.
The second part of the book consisting of four chapters data visualization, prediction, classification, clustering analysis is not presented as a comprehensive introduction of these techniques. For instance, for classification and prediction, hyperparameter tuning which is a very important part of learning classification is not covered.
Then what are the goals of this chapter? This content is provided to support the third part of the book: the preprocessing. One needs to know how the data will be used at the analytic stage for them to be able to effectively preprocess it. The simplest example is that if one needs to use clustering algorithms on their data, and if one understands that the clustering algorithm works by calculating the distances between the data objects, the requirement of normalizing the data before applying clustering algorithms will naturally come to them.
**Carlos Orjuela**
Great explanation. Thanks Roy Jafari!
**Roy Jafari**
Thank you Francis Terence Amit and Alexey Grigorev for the introduction and the opportunity. Also, I would like to thank GerryK and Carlos Orjuela for starting us off with great questions. Thank you!
I am very excited to be here and for this opportunity to answer questions. Please don’t hold back any questions. Fire away!
Looking forward to answering your questions.
**Clifton Baldwin**
Hi Roy Jafari. One of my biggest problems with new projects is often just getting the data read in. Data is often collected by researchers using multiple Excel files, Word files, or various text files, often with no naming convention for the columns. A majority of my time is just getting the data so I can analyze it. I realize every situation is unique, but does the book raise awareness to these problems?
**Roy Jafari**
Hi Clifton Baldwin! Thank you for your excellent question.
The book covers similar situations such as this both as a level 1 data cleaning and also as Data Integration.
The book’s approach to data integration is to introduce the readers to the most common challenges of data integration. Using examples, 6 challenges are introduced and techniques on how to deal with them are presented. One of the challenges is called “Unwise Data Collection” and is exactly referring to the situations such as the ones you are describing here.
Here is an excerpt from the book that raises awareness about these issues.
_Chapter 9, Data Cleaning Level 1, Example 1 – unwise data collection_:
“From time to time, you might come across sources of data that are not collected and recorded in the best possible way. These situations happen when the data collection has been done by someone or a group of people who have not had enough skillset in database management. Regardless of why these situations might have happened, you are given access to a source of data that requires significant pre-processing before it is in one standard data structure.
For instance, imagine you have been hired by an election campaign to use the power of data to help move the needle. Omid was hired just before you, and he knows a lot about the political aspects of the election but not much about data and data analytics. You are assigned to join Omid and help process what he is tasked with. In your first meeting, you realize the task is to analyze the speeches made by the 45th President of the United States, Donald Trump. To bring you up to speed, he smiles and tells you that he has completed the data collection and all that needs to be done is the analysis now; he shows you the folder on his computer that has a text file (.txt) for every of Donal Trump’s speeches made in 2019 and 2020. ….
After seeing the folder you instantly realize that a big task of data preprocessing remains to be done before considering any analytics. In the interest of building a good working relationship with Omid, you don’t directly go to the point that a huge data preprocessing needs to be done and instead comment on the aspect of his data collection that has been great and can be used as a cornerstone for data preprocessing. You mention that it is great that the naming of these files follows a predictable order. The order is that city names come first then comes the name of the month in three letters and then comes the day in one or two digits and the year in four digits.
As you are well-versed with Pandas DataFrame you suggest that the data be processed into one DataFrame and Omid, eager to learn, accepts enthusiastically.
The following presents the three steps that you can do to process the data into a DataFrame.”
**Senthilkumar Gopal**
Hi Roy Jafari love the idea behind a book focused on pre-processing and data preparation. Some of the questions that we can benefit from your thoughts.
1. What do you reckon are the key roadblocks that data scientists face in their EDA
2. What are your thoughts on imputes and what role does that play in preprocessing
3. Common standards and pipeline for data processing for ML
4. Growth opportunities in data preparation space - For ex: companies provide cleaned data for financial and stock analysis
**Roy Jafari**
Hi Senthilkumar Gopal! These are great quesitons. I will answer them one by one.
**Roy Jafari**
_Q1 - What do you reckon are the key roadblocks that data scientists face in their EDA?_ That’s an important and a big question. I will list three roadblocks that I believe have the most impact on the effectiveness of data scientists.
First, I think data scientists could hugely benefit from optimizing their database and programming skills. There is a huge difference between a code that works just now, and a code that is optimized, and will work again and again and again in the future. Just like, there is a huge difference between a solution that gets static data and output model insights just for that static data, and a solution that automatically gets data from the source, preprocess it and then outputs the insight; the first one must be run manually, the second one can be deployed into a live dashboard or be used as an ML solution for automatic decision making. If one needs to improve their programming skills regarding data preprocessing and analytics. this book is a great resource.
Second, I think data scientists need to start moving away from the Kaggle model where the dataset is provided for them to apply their models. I believe data scientists need to start empowering themselves to be the ones that design those datasets, collect the data for them and preprocess them for the best possible performance. It is more convenient to think it is the job of data engineers to do those things, but I disagree. For data pipelining indeed data scientists need the expertise of data engineers, however, the data scientists must be central in the discussion of what should be pipelined and how it should be preprocessed.
Lastly, I am convinced data scientists should look more often to data preprocessing as a way to improve their solutions as opposed to tuning their models or comparing models. The positive impact of a delicately done data transformation and data massaging can sometimes look like magic, however, it is the results of experience and understanding of different data preprocessing that leads to significant improvements. I think data scientists should invest in their mathematical skills to benefit more from tools such as functional data analytics and feature extractions. Both Functional data analytics and feature extraction are extensively covered in the book.
**Roy Jafari**
_Q2 - What are your thoughts on imputes and what role does that play in preprocessing?_ I am not sure what you mean by imputes. I guess, you mean imputing missing values? I will answer assuming that is what you meant. Please let me know if that was not the case.
Imputing missing values is indeed a topic that is covered in Chapter 11 Data Cleaning Level III- Missing Values, Outliers, and Errors. Imputing missing values is one of the four possible strategies that one might use when they face missing values. These strategies are “Leave as missing”, “drop attributes”, “drop data objects”, “impute values.” Furthermore, four types of methods are presented for imputing values: impute with general central tendency, impute with specific population central tendency, impute with an interpolated value, and impute with estimated value using regression analysis.
The book also covers how to choose what strategy, and what methods are appropriate. It will involve understanding the type of missing values (MCAR, MAR, MNAR), how the data will be used at the analytics stage, and the type of data.
One thing that I try to instill in the readers’ mind that might be different is that when imputing we are not trying to accurately predict what the value could have been, we are trying to impute a value with two goals: 1) to be able to keep as much real data as possible, 2) to add the least amount of bias due to our imputes.
**Roy Jafari**
_Q3 - Common standards and pipeline for data processing for ML?_ I am not aware of any standards for data pip liens or data processing as that is not my area of experience and expertise. I think a data engineer could answer this question better.
I would say that there should be some guidelines for effective data pre-processing. I have outlined these in the book. For instance, the biggest one is clarity on the structure of the final dataset you will be using for the analysis. You would need to know exactly what each row of the dataset you will be creating represents before embarking on data preprocessing.
**Roy Jafari**
_Q4: Growth opportunities in data preparation space - For ex: companies provide cleaned data for financial and stock analysis?_ I think the best way to grow in this space is through actual experience. Resisting the temptation to do the data cleaning, integration, and preprocessing in excel and doing them either while pooling the data or afterward using scalable python modules such as NumPy and pandas, however small your datasets are. You will start learning that data preprocessing for each project is like a puzzle that needs to be solved. There are many ways to go about the same data preprocessing, however, some of them will be more scalable, will use less computation, and will teach you new things as well.
As far as other growth opportunities, I don’t believe there is a community that focuses on data preparation and data preprocessing. I do have a YouTube channel for which I create videos regarding data preprocessing among other things. But I think, a data preprocessing learning community can provide many benefits to its members. I would be interested to join and contribute for sure.
Nothing else comes to mind. I hope this was helpful.
**Senthilkumar Gopal**
Roy Jafari Thanks so much for taking the time to respond and the thoughtful responses..
**jeromy**
Hi Roy Jafari, what are, in your opinion, typical mistakes or anti-pattern that people make during the data preprocessing phase (for example using pandas)? And what are the best way to resolve those mistakes?
**Roy Jafari**
Hi jeromy! Thanks for your question.
This is a great question and can have a very extensive answer. I’ll try to keep it short by just mentioning the three biggest ones that come to mind.
The first one is very basic but is very easy to happen if data preprocessing has not happened methodically. There are times that whoever has created the data has used numerical codes for purely categorical attributes. It is a huge mistake to use those values as numbers in your models. One of the first steps of data preprocessing is to recognize the type of each attribute (nominal, ordinal, or numerical), and based on the type of attributes the way they can be included in each analytic might be different.
A second mistake that comes to mind is the assumption you need to prepare a dataset for a problem, use that preprocessed dataset for any tools, models, or algorithm, and that’s not the case. Each algorithm and method uses the data differently and data preprocessing must be done specifically for that algorithm. For instance, while outliers are huge issues for regression models, they are not a big deal for an MLP (artificial neural network) model. If the dataset prepared for regression in which we’ve removed the outlier is also used for MLP, we’ve made a mistake because barring computational complexity, MLP could have used the information in the outlier to improve itself.
The third is not checking the practicality of independent attributes for predicting the dependent attributes. I have seen time and again that a model reaches high accuracy but the model is not usable because the value of the independent attributes will be known exactly when we need to know the dependent attribute or very close to it. Checking the practicality of the independent attribute is very important.
I know I said three, but here is the fourth one. This can happen to data scientists that don’t have up-to-date programming skills and they might use looping to perform data preprocessing on their models. Not only looping is computationally more expensive, but also it is very time-consuming to write, maintain and update the code. My recommendation is for a data scientist to learn about NumPy vectorization and broadcasting first and any data manipulation they have they should try to use that strategy for its implementation as it is the fastest approach. If that is not possible, my second recommendation would be to apply functions to their data as opposed to doing loops. To keep the code fast and scalable, it is also important to learn what to avoid when applying a function to keep the code scalable.
**jeromy**
Thank you for the exhaustive and clear answer!
**Carlos Orjuela**
I love that fourth point Roy Jafari, is vectorization discussed in the book?
**Roy Jafari**
Thanks for your positive feedback Carlos Orjuela. The answer is no, the book does not cover vectorization and broadcasting but it does cover mapping/applying functions.
**Carlos Orjuela**
Roy Jafari, do you know of any robust packages that helps to speed up EDA? As some task might be repetitive. I heard of some but never actually got to use any so I’d like to know your take on it
**Roy Jafari**
Thanks for your question Carlos Orjuela.
Apart from matplotlib, seaborn, and statmodels modules, which are great, I haven’t come across any other outstanding package. If you any know any others I would love to know about them.
**Carlos Orjuela**
Thanks again Roy Jafari. One that comes to mind is Pandas profiling but I’ve never used it tbh
**Roy Jafari**
Thank you Francis Terence Amit for featuring my book and also, thanks everyone for their excellent question.
Here is a 6 minute vide tour of the book: [https://youtu.be/iUHAFPucYZU](https://youtu.be/iUHAFPucYZU)
Just in case there were more unanswered question.
Cheers and happy learning, everyone!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Engineering with Apache Spark, Delta Lake, and Lakehouse – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Engineering with Apache Spark, Delta Lake, and Lakehouse
-------------------------------------------------------------
#### by [Manoj Kukreja](https://datatalks.club/people/manojkukreja.html)
##### The book of the week from 14 Mar 2022 to 18 Mar 2022
In the world of ever-changing data and schemas, it is important to build data pipelines that can auto-adjust to changes. This book will help you build scalable data platforms that managers, data scientists, and data analysts can rely on.
Starting with an introduction to data engineering, along with its key concepts and architectures, this book will show you how to use Microsoft Azure Cloud services effectively for data engineering. You’ll cover data lake design patterns and the different stages through which the data needs to flow in a typical data lake. Once you’ve explored the main features of Delta Lake to build data lakes with fast performance and governance in mind, you’ll advance to implementing the lambda architecture using Delta Lake. Packed with practical examples and code snippets, this book takes you through real-world examples based on production scenarios faced by the author in his 10 years of experience working with big data. Finally, you’ll cover data lake deployment strategies that play an important role in provisioning the cloud resources and deploying the data pipelines in a repeatable and continuous way.
By the end of this data engineering book, you’ll know how to effectively deal with ever-changing data and create scalable data pipelines to streamline data science, ML, and artificial intelligence (AI) tasks.
* [Book's page](https://www.packtpub.com/product/data-engineering-with-apache-spark-delta-lake-and-lakehouse/9781801077743)
* [Buy on Amazon](https://www.amazon.com/Engineering-Apache-Spark-Delta-Lakehouse/dp/1801077746/ref=sr_1_1?dchild=1&keywords=Data+Engineering+with+Apache+Spark%2C+Delta+Lake%2C+and+Lakehouse&qid=1634832283&sr=8-1)
* [GitHub repository](https://github.com/PacktPublishing/Data-Engineering-with-Apache-Spark-Delta-Lake-and-Lakehouse)
Questions and Answers
---------------------
**Isaac Toluwani**
Hi Manoj Kukreja . Congratulations on your book
Does this book also cover in detail how Data Lakes differ from Data warehouses and use cases for either one or both of them
**Manoj Kukreja**
Hi Isaac Toluwani Over several years, data warehouses have been the de-facto standard for OLAP use cases, until the challenges around the volume, variety and velocity of data took over. Last few years, we have been in an era where analytics has moved over to data lakes. But this has created a unique challenge which I refer to as the “The power struggle” in my book. Moving from data warehouses to data lakes has forced us to sacrifice a few good things such as ACID compliance, indexing and caching. And that’s precisely why the modern Lakehouse architecture is taking over. My book not only promotes the new architecture but also explains how it is different from other architectures like lambda and kappa.
**Isaac Toluwani**
Nice.. Thanks
**Tony Gunawan**
Hi Manoj Kukreja.. From the book description it says that the book explains more detail about Microsoft Azure Cloud for data engineering purpose. How about people that do not using the cloud service, instead using other ones like GCP or AWS? Could your examples and codes in this book be implemented also in other cloud service provider? Thank you and congrats for your book.
**Manoj Kukreja**
Hi Tony Gunawan Moreover, all cloud vendors perform data processing using open-source frameworks like Spark, Hadoop, and Kafka but are packaged into services under different names. My book tries to teach the readers about big data concepts rather than enforce a particular technology or cloud service. Most of the examples particularly Databricks Spark can be run on the cloud platform of your choice.
**Tony Gunawan**
Cool. Thanks for answering, Manoj Kukreja
**Anand**
\> Hi Manoj Kukreja - Congratulation on your book. From the description of your book, it would provide steps to deploy the data pipelines in a repeatable and continuous way. May I infer that it would cover data-ops as well?Would it provide the steps to implement the data-ops using Azure ?Would it also cover similar steps using cloud agnostic way?
**Manoj Kukreja**
Hi Anand Towards the end of my book, Chapter 11 “Infrastructure Provisioning” and chapter 12 “Continuous Integration and Deployment (CI/CD) of Data Pipelines” are covered in great detail. Since this book is centered around Azure, I have used services like ARM templates and Azure DevOps. Having said that these services use the same deployments principles as any other cloud agnostic tool such as Terraform or Ansible.
**A**
Hello Manoj Kukreja 👋🏻
Asking questions that I am personally struggling with at this point of time. Hope to get your viewpoints on the same.
1. Do you consider Infra first or Use case first as sort of a chicken and egg problem? Should organizations hack together some models and define some business use cases and then move on and create an DE backed infrastructure to support them or should the first and foremost thing should be to create an Infrastructure of consistent and reliant data and then working on Data Science use cases?
2. With more and more push towards data compliance, what role do you think DE or Solution architects should play to ensure that Data availability should not be a bottleneck for a Data Scientist?
I might have some more questions which I’ll try to get your opinions on. Hope thats alright. Would love to hear other peoples opinion as well.
**Manoj Kukreja**
Hi A
Answer to Q1
Back in 2011, I and my team were one of the early adopters of big data. In my experience, starting a data science practice without a proper data engineering back-end is a mistake that several organizations have made in the past and unfortunately have paid dearly for it. Data scientists are not the best data engineers but are forced to perform that work in their absence, in many cases they don’t get proper time to do the work they are originally supposed to be doing. In my opinion, these days it is mandatory to have a solid DE back-end that enforces proper governance, master-data-management, and data sharing techniques like data mesh.
Answer to Q2
Data engineers play a very huge role in ensuring data compliance. Newer regulations such as GDPR and CCPA are very strict about enforcement. Once again having a DE practice that takes care of data security, standardization, quality, and cataloging is becoming a huge necessity for any organization that dreams of adopting effective data science principles. Instead of being a bottleneck, it ends up making the job of data scientist easier. It lets them focus on what they do best.
**juan manuel franzante**
Hello Manoj Kukreja. Thanks for the opportunity to get your amazing book free. I read the topics and I understand that you propose the delta lake architecture.
1. Which technic of modeling data is the more accurate for this kind of architecture? For example: Data Vault 2.0 , Canonical tables, Kimball, OBT, a mix of technics etc.
2. In which layer of the architecture would you apply this technique? Do you consider it necessary to order and modeling data starting in the bronze layer?
Thanks for sharing the knowledge and experience. Regards
**Manoj Kukreja**
juan manuel franzante I don’t think it is advisable to rank the modeling techniques based on accuracy. Overall, I can say that the use of Kimball model is generally preferred and widespread. The modeling techniques are usually applied at the gold layer. However, in some cases particularly related to denormalization, you may end up implementing them at the silver layer as well. The bronze layer represents the state of data in the shape or form that it was delivered or ingested from sources. There are many reasons why you should not model data in the bronze layer of a lakehouse:
* Having the exact state of data is important for auditing
* You may need to replay data in the future in which case you may need the pre-existing state
* Format of data in bronze is usually a mixture, applying a particular modeling technique is technically not even possible
**juan manuel franzante**
But Kimball is based on oldest paradigm when the storage and compute were expensive in my opinion. I think that its important have a technic of modeling that offer more advantages for the current Cloud paradigm like Data Vault. I’m totally agree with apply these in the silver layer or gold. Thanks for answer!
**Manoj Kukreja**
I do agree that its the oldest paradigm, and that is precisely why good practices and differs from reality. Most times on projects you are forced to do things a certain way based on what the customer is comfortable supporting and/or has skills for.
**Tim Becker**
Hi Manoj Kukreja, thanks for this really interesting book and for the opportunity to ask questions.
**Tim Becker**
* What is a delta lake?
**Tim Becker**
* In your book, do you cover an end-to-end project?
**Tim Becker**
* Do you cover streaming and batch processing of data?
**Tim Becker**
* Why did you choose Spark and Azure?
**Manoj Kukreja**
Tim Becker What is a delta lake?
Delta lake is a new framework that works over Spark to provide some useful features to data lake such as ACID transactions, time travel, schema evolution, etc.
In your book, do you cover an end-to-end project?
My book covers an end-to-end project for an online electronics retailer that wants to streamline its inventory, shipping, finance, and marketing operations using analytics.
Do you cover streaming and batch processing of data?
The project covers both streaming and batch operations.
Why did you choose Spark and Azure?
Azure is one of the most prevalent cloud platforms for big data storage and computing operations. Similarly, Spark is the most widely used distributed compute platform. Serious big data operations using Spark can most effectively be supported using highly scalable platforms like Azure.
How do you implement monitoring in practice?
In the book, I have pretty much relied on Azure monitor. However, I would recommend using Datadog for enterprise monitoring.
What kinds od tools do you use to automate it and what are best practices and common mistakes?
Azure DevOps and ARM templates
I am currently implementing monitoring for ML models and I believe there could be a lot of similarities. Do you have any advice?
I have many times used Prometheus (Kubernetes service monitoring) to collect metrics from endpoints.
**Tim Becker**
thanks a lot 🙂 I will look into it
**Tim Becker**
* How do you implement monitoring in practice? What kinds od tools do you use to automate it and what are best practices and common mistakes? I am currently implementing monitoring for ML models and I believe there could be a lot of similarities. Do you have any advice?
**Philip Dießner**
Hello Manoj Kukreja, Congrats on your book!
Depending on the size of a company/project (and the types of data to be aggregated), a data or delta lake seems to possibly introduce a lot complexity than needed, especially when considering very small use cases. Do you give recommendations in the book on when to extend from e.g. a DWH to a data lake? Or would you always start with a data lake(house) architecture to take advantage of the scalability?
**Manoj Kukreja**
Thanks Philip Dießner You have a valid point. The problem is not visible and can be easily resolved if the data is small. Overall. it is not about whether Delta Lake introduces complexity or not, it’s about can we survive without it. In a normal DWH/database environment, atomic updates to row data are easily possible. Data lakes are file/object stores, there is no concept of atomicity. This means every change (CDC) is delivered to you as a duplicate row. In days before delta lake, whenever we got CDC we had to run compute-intensive operations (sometimes lasting hours depending on the size of data) to deduplicate data. All of that has disappeared when delta lake was introduced. In many respects, it has proven to be a life savior for data engineers.
**Manoj Kukreja**
There are legit cases where the complexities of a data lake are unwanted. But hose cases are typically that have a limited volume, variety and almost non-existent velocity of data.
**Philip Dießner**
Thanks for your answer and sharing your experience.
**Manoj Kukreja**
Thanks everyone for the great questions, please don’t hesitate to ask more in the future.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Serverless Analytics with Amazon Athena – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Serverless Analytics with Amazon Athena
---------------------------------------
#### by [Anthony Virtuoso](https://datatalks.club/people/anthonyvirtuoso.html)
##### The book of the week from 28 Mar 2022 to 01 Apr 2022
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using SQL, without needing to manage any infrastructure.
This book begins with an overview of the serverless analytics experience offered by Athena and teaches you how to build and tune an S3 Data Lake using Athena, including how to structure your tables using open-source file formats like Parquet. You’ll learn how to build, secure, and connect to a data lake with Athena and Lake Formation. Next, you’ll cover key tasks such as ad hoc data analysis, working with ETL pipelines, monitoring and alerting KPI breaches using CloudWatch Metrics, running customizable connectors with AWS Lambda, and more. Moving on, you’ll work through easy integrations, troubleshooting and tuning common Athena issues, and the most common reasons for query failure. You will also review tips to help diagnose and correct failing queries in your pursuit of operational excellence. Finally, you’ll explore advanced concepts such as Athena Query Federation and Athena ML to generate powerful insights without needing to touch a single server.
By the end of this book, you’ll be able to build and use a data lake with Amazon Athena to add data-driven features to your app and perform the kind of ad hoc data analysis that often precedes many of today’s ML modeling exercises.
* [Book's page](https://www.packtpub.com/product/serverless-analytics-with-amazon-athena/9781800562349)
* [Buy on Amazon](https://www.amazon.com/Serverless-Analytics-Amazon-Athena-semi-structured/dp/1800562349/ref=sr_1_1?dchild=1&keywords=Serverless+Analytics+with+Amazon+Athena&qid=1634629990&sr=8-1)
* [GitHub repository](https://github.com/PacktPublishing/Serverless-Analytics-with-Amazon-Athena)
Questions and Answers
---------------------
**adanai**
Hello Anthony, thank you for the QnA! I am relatively new to AWS and working to specialize in Analytics+ML in AWS.
What level of experience in AWS is expected when starting with this book?
**Anthony Virtuoso (Author)**
as long as you have an AWS account, basic SQL knowledge, and know how to modify IAM policies you can do everything in this book. We tried to really make it accessible to beginners and intermediate folks alike. Only one or two chapters at the end are a bit more advanced and do require comfort with developing java code.
**adanai**
What does an ETL pipeline look like? What are the components in it and why does Athena primitively not require one(or avoids it)?
**Anthony Virtuoso (Author)**
In this case we typically mean that you do not need to run jobs/queries that read literally ALL your data and then rearrange it or pre-join it with other data to make it either (a) small enough (b) in the right format or (c) denormalized across data sources. These techniques can still be helpful with Athena, but Athena does its best to still make data that is no well organized, optimized, or formated using best practices accessible without all the upfront work/maintenance of ETL
**adanai**
Would I, as a book reader, need to incur costs in executing the examples in this book?
**Anthony Virtuoso (Author)**
Yes, we typically mention it in the chapters but while authoring the book we spent < $30 total on AWS fees and keep in mind we had to run through the exercises 5 or 6 times to refine them.
**adanai**
I recently started learning about and working with Tableau (small data - 100K records). All I did was click and drop elements and write a few queries for calculated fields. The results seemed satisfactory.
What are the ways in which the Athena integration make working with Tableau better? Is it the scale that makes the difference?
**Anthony Virtuoso (Author)**
Aside from the visualizations that you get with Tableu I think youll find the experience with Athena is pretty similar with one key difference being that if you ever need to scale up from 100k records to 100Billion or somewhere in between Athena will handle that just as well and you might start to see other differentiation in terms of price and performance as you scale up. But id agree that at 100k records, probably even Excel can used for a bunch of quick analysis.
**Rui Ramos**
Hi, reinforcing adanai question. What would be the level of expertise required in AWS to start on this book ? Also does the book contains any use-cases for the usage of this service ? Are there practical examples to tryout ?
**Anthony Virtuoso (Author)**
We tried to include practical examples in every chapter in the form of exercises. There are also lots of stories about cases we’ve seen people use Athena for X or Y. As for AWS knowledge, the pre-requiste is pretty small, just that you (a) have an AWS Account (b) know some basic SQL (c) are able to modify IAM policies in your AWS account since each chapter requires different levels of access (though you could just do the entire thing with admin access if the account your using isn’t a production account or is a ‘burner’ as we say).
**Matias Rebolledo Dezerega**
Hi Anthony Virtuoso (Author)! :blob\_wave: Now i’m starting work with AWS and my questions is about the pipelines, can you make a ETL in Athena or just work with a ELT like in google BigQuery?
The books has examples of making a datalake loading data from different sources (like API, connectors, etc)? Thanks!!
**Anthony Virtuoso (Author)**
There is a chapter in the book with some basic examples on how to create what many would call an ETL pipeline but if your needs include chains of jobs (probably anything more than 3 or 4 jobs). I’d recommend looking at Glue ETL as it has the concept of scheduled jobs as well as a dependency modeler that can run chains of jobs based on triggers like data arriving in S3, completion of another job, or a schedule.
**Matias Rebolledo Dezerega**
Thanks!
**Anthony Virtuoso (Author)**
Great questions! keep em coming, ill check back in later today.
**Tim Becker**
Hi Anthony Virtuoso (Author), really interesting topic! I was wondering who AWS charges for Athena and Lake Formation?
**Anthony Virtuoso (Author)**
what do you mean by who?
**Tim Becker**
ah sorry, I meant how, sometimes my autocorrect behaves strangely
**Tim Becker**
Could you please explain the difference between Athena and redshift (for beginners)?
**Anthony Virtuoso (Author)**
Redshift targets datawharehouse usecases while Athena is more geared towards ad hoc analysis without ETL. Keep in mind this is an extremely reductive explaination and the two products can both do many of the same things just with different price/performance and strengths.
**Tim Becker**
Is it possible to use terraform or cloudformation to setup the data lake? If possible, would you recommend it?
**Denis L.**
Hi Anthony Virtuoso (Author), thanks for answering the questions. Once ACID transactions are out of public preview for Athena and Lake Formation, do you see it potentially replacing Redshift (+ Spectrum)? It seems there are 2 competitive solutions that are converging. What is your take on that?
**Anthony Virtuoso (Author)**
it won’t replace Redshift + Spectrum as those services do a lot more than just ACID. They have a different performance profile, cost, and SQL feature set.
**Anthony Virtuoso (Author)**
I do think youll see more overlap as time goes on since AWS is striving to make it easy for customers to move between and/or use a combination of services as one seemless offering since each it fit for a specific purpose. For example, can you use a hammer to break concrete? Yes, but a sledge hammer would be btter. and You can use a sledgehammer to drive nails but a hammer would be better.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Interpretable Machine Learning – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Interpretable Machine Learning
------------------------------
#### by [Christoph Molnar](https://datatalks.club/people/christophmolnar.html)
##### The book of the week from 11 Apr 2022 to 15 Apr 2022
Machine learning has great potential for improving products, processes and research. But computers usually do not explain their predictions which is a barrier to the adoption of machine learning. This book is about making machine learning models and their decisions interpretable.
After exploring the concepts of interpretability, you will learn about simple, interpretable models such as decision trees, decision rules and linear regression. The focus of the book is on model-agnostic methods for interpreting black box models such as feature importance and accumulated local effects, and explaining individual predictions with Shapley values and LIME. In addition, the book presents methods specific to deep neural networks.
All interpretation methods are explained in depth and discussed critically. How do they work under the hood? What are their strengths and weaknesses? How can their outputs be interpreted? This book will enable you to select and correctly apply the interpretation method that is most suitable for your machine learning project. Reading the book is recommended for machine learning practitioners, data scientists, statisticians, and anyone else interested in making machine learning models interpretable.
* [Book's website](https://christophm.github.io/interpretable-ml-book/)
* [Buy on Amazon](https://www.amazon.com/Interpretable-Machine-Learning-Making-Explainable/dp/B09TMWHVB4/ref=sr_1_1?qid=1647836815&refinements=p_27%3AChristoph+Molnar&s=books&sr=1-1&text=Christoph+Molnar)
* [GitHub repository](https://github.com/christophM/interpretable-ml-book)
Questions and Answers
---------------------
**Daniel**
Hi Christoph Molnar, thank you very much for doing this. I would like to know:
* Who is the target audience for this book?
* Meanwhile there are several approaches available to explain AI algorithms. I guess there is no free lunch and it depends on the use case, but out of your experience: Do you have any personal favorite among all the tools/ techniques? Maybe one that tends to be used most often?
* With the rise of bigger and more and more complex models it’s probably getting even harder and harder to explain these black box models. What do you think are the most challenging things with regard to explainable AI in the future? Any ideas how to solve it and where this finally leads us?
* Is there any specific advice you have for beginners in ML with regards to explainable AI?
Many thanks in advance!
Cheers
Daniel
**Christoph Molnar**
Hi Daniel, thanks for these questions.
1. Interpretable Machine Learning is for everyone who wants to learn how we can explain machine learning models. I know that many data scientists who build predictive models themselves are readers of the book. Then there are students and teachers of machine learning, but also technical managers and many other people.
2. Really depends on what you want to do. Shapley/SHAP is a good all-rounder for both explaining individual predictions, but also summarizing what the model does in general. If you want to know feature importance, the Permutation Feature Importance is a good go-to approach.
3. We have many interpretation tools that are model agnostic, so your models becoming more complex is not necessarily the biggest issue. One of the challenges I see is in adapting the interpretability to the target audience: People trained in understanding linear regression models might easily understand any explanation in the form of a weighted sum. But others would have a hard time. Another challenge: For some interpretation methods it’s not 100% clear how we should interpret them, or if they even do what we want them to. For example, saliency maps for neural networks are quite problematic.
4. I think explainable AI can be a great tool, especially when you want to learn ML. Explainable AI tools can give you some more intuition about the workings of machine learning. For example, you should see the effect of regularization in feature effects and importance. Feature effect curves will reveal differences in how different models model relationships between features and the target: A feature effect curve for an SVM looks completely different from a decision tree. For beginners, I believe it makes sense to invest some time also in explainability.
**Daniel**
Thank you! :)
**Matthew Emerick**
Hey, Christoph Molnar! We really appreciate you doing this.
Does choosing an interpretable algorithm add a layer of algorithmic complexity to a model?
**Christoph Molnar**
It can. That depends on how your chosen interpretation approach works. So-called post-hoc or model-agnostic methods are applied after the model was trained. So it does not add to the model prediction or training complexity. You can read more about model-agnostic [in this chapter of my book](https://christophm.github.io/interpretable-ml-book/agnostic.html)
Other interpretation tools can modify how the model is trained. For example, adding monotonicity constraints. Those methods may increase the algorithmic complexity
**Kwang Yang Chia**
Hi Christoph Molnar! Thank you so much for doing this. I would like to know:
1. What industries would greatly benefit from explainable AI more so than others?
2. Considering the popularity of deep learning (and therefore the popularity of black-box models), where would explainable AI fit in Data Science?
3. What are the typical methods that make black box methods more interpretable?
**Christoph Molnar**
1. Good question. As a rule of thumb I would say that interpretability is required when only making predictions is not enough, but you also want to learn from your model about the world. Anything science-related. Imagine you build a prediction model for whether some crop will grow on a certain location. Maybe the model is really good at prediction. But you might also want to know what the important factors were, understand what makes a plant thrive and what factors limits its growth. Marketing is another example: A churn model is much more useful when you have an idea why the customers are turning the backs on your company.
2. As a data scientists you can be in the position to convince others (like management) that your model is robust and brings benefits. I know some places which would not accept black box models, since they don’t trust such opaque models. Interpretability can close this gap, and make black-box models more appealing for many business cases.
3. A big question to ask, I could copy the content of my book here 😄. But the short answer: Shapley values/SHAP and LIME are popular methods that can explain individual predictions. LIME and SHAP are quite flexible and can produce explanations for tabular data, text data and image data. Then we have methods that can explain the average behavior of a machine learning model. The permutation feature importance can tell you for example how important a feature was for the correct prediction or classification (on average for the entire dataset). If you want to know the effect that a feature has on the prediction Some data scientists would not use black box models in the first place, but instead use model that are already interpretable: Generalized additive models, decision rules and decision trees, …
**Kwang Yang Chia**
Thank you for your answers! Very informative, and I learned something new. 😄
Just one more question - Are there any developments that seek to improve the interpretability of current models (e.g. anything that helps with the interpretability of Deep Learning models, for example?)
**Christoph Molnar**
There are many, and it’s difficult to keep an overview here. Here just one example: [https://proceedings.neurips.cc/paper/2015/file/ced556cd9f9c0c8315cfbe0744a3baf0-Paper.pdf](https://proceedings.neurips.cc/paper/2015/file/ced556cd9f9c0c8315cfbe0744a3baf0-Paper.pdf)
Deep Convolutional Inverse Graphics Network -> This paper proposes to disentangle image representations within a neural network. Instead of allowing any neuron to learn anything, they try to force individual neurons to be responsible for separate attributes like pose or light.
**Christoph Molnar**
Disentanglement in general is a big topic for increasing deep learning interpretability.
**Kwang Yang Chia**
Thank you for the insight! Really appreciate it 😄
**Jasmin Classen**
Hi Christoph Molnar , thanks so much for doing this. My question(s) would be:
From your experience, what are the most common Use Cases where you would use explainable ML in practice? What audiences of your model could benefit from this? E.g. gaining trust from technical vs non-technical stakeholders, for your own / team-internal projects to understand your own model, anything else? Maybe you have some concrete examples when you used explainable ML in practice?
**Christoph Molnar**
A common use case is also model debugging: With interpretation techniques, you can quickly see if something goes completely wrong. Like one of the features being, unexpectedly, one of the most important features could hint at something like target leakage. I used interpretability for checking if my model makes sense, get ideas for feature engineering, and so on.
For example, I once built a classifier for financial transactions. It was an interpretable model, so we could check against domain expertise whether the decision rules within that model made sense. This was very reassuring to see, before we gave the model into production 😁
**Jasmin Classen**
Cool, thanks so much! Have you ever used any of the methods to help stakeholders understand your models?
**Christoph Molnar**
I’ve used random forest feature importance and partial dependence plots.
None of the newer methods because I have moved on to doing research. I know of many practitioners who regularly use Shapley values, permutation feature importance, LIME and so on.
**Jasmin Classen**
Thanks so much, really interesting!
**Vladimir Finkelshtein**
Often, when I train my first models, the most important feature is usually the one, which I later discover drifted in distribution the most between training and validation sets 😞 So it is a nice way to know what you are overfitting to.
**Vladimir Finkelshtein**
Your book contains many established and somewhat acceptable methods in the industry like LIME, SHAP and counterfactuals, etc. Are there some approaches that you see as promising and would include in the 3rd edition of your book?
**Christoph Molnar**
One topic I wanted to add is eep learning-specific attention mechanisms.
Then mostly I will have to update many of the old chapter, because a lot of great research is coming out scrutinizing these more established methods.
**Vladimir Finkelshtein**
How do the interpretability algorithms handle interdependencies in the data? For example, if I trained logistic regression and included features `X` and `X***3*` (which might make sense for certain use case), it does not help me to know what would have happened if I increased `X` and decreased `*X***3`. What if I trained a model where I am not aware of interdependencies between features, but they exist and are more complicated?
**Christoph Molnar**
Poorly. Most methods handle interdependencies poorly.
And the interdependencies problem is both a technical but also a deeper problem. Interpretation often means isolating a single feature, and studying how changing this feature changes the prediction.
But with dependencies, does it even make sense to study the feature in isolation?
That’s why I would recommend to study the interdependencies before using interpretability methods. You can have a look at the correlations between the features, but also use more complex dependence measures such as HSIC.
In the case of your example, you could technically wrap this transformation where you compute `x**3` and make it part of the model. Then you could study how increasing `x` changes the prediction through both logistic regression terms.
**Vladimir Finkelshtein**
Our company in industrial data science (metallurgy) has quite a few use cases where it makes sense to explicitly study one feature in isolation, where almost all of the features in the model are very heavily interdependent. There is academic research about these interdependencies for decades and only small subset of them is understood. Once you consider hundreds of features, there is basically no way to know what’s going on. We know for fact that everything is correlated with almost everything, but the relations are very complex to model, so recreating your suggestion with `x**3` is unfortunately out of question 😞
**Integralytic Team**
Christoph Molnar Thank you for sharing your book this week. What would you recommend with regard to explaining NLP models, particularly using Spark NLP and Spark ML? We are trying to understand how to explain which words or word combinations influence NLP predictions.
**Christoph Molnar**
I think NLP is the topic that got the least attention in my book 😅 (pun intended)
Technically, LIME and SHAP work. In the case of text, they both work by similar mechanisms: For a given text and classification, they create snippets of this text with some of the words missing, then get the model prediction / classification score. Changes in the prediction are then attributed to the individual words.
If you are using a deep learning approach with some attention mechanism you could also look at visualizing the weights (although some say that attention make poor explanations).
**Christoph Molnar**
For Shapley values there seems to be a Spark version: [https://pypi.org/project/shparkley/](https://pypi.org/project/shparkley/)
I haven’t tested it.
**Integralytic Team**
Thank you!
**Clara B.**
What is in your eyes the most neglected area in machine learning interpretability research or the area with still the highest potential?
**Christoph Molnar**
Studying how users interpret the results and what the best interpretation output for a given audience is.
These real-world studies are difficult to do. There are some human studies, but they are usually conducted with Amazon mechanical turk or with surveys of students.
**Christoph Molnar**
And the tasks are quite artificial, so it’s difficult to translate results into how people would understand the black-box interpretations in a real application.
**Clara B.**
Will your book be translated to other languages?
**Christoph Molnar**
Interpretable Machine Learning has translations into Chinese, Japanese, Spanish, Bahasa Indonesia, Korean, and Vietnamese. See her: [https://christophm.github.io/interpretable-ml-book/translations.html](https://christophm.github.io/interpretable-ml-book/translations.html)
**Max Payne**
Hi, does your book touch upon ExplainableAI (i.e. deep learning/NN instead of ML)
If not, how different would the two be?
**Christoph Molnar**
It does! First, all model-agnostic methods can also be used for deep learning as well by.
And an entire part of the book is dedicated to interpreting deep learning approaches: [https://christophm.github.io/interpretable-ml-book/neural-networks.html](https://christophm.github.io/interpretable-ml-book/neural-networks.html)
This part covers:
* Visualizing learned features (feature visualization)
* Pixel attribution aka saliency maps
* Concept detection (TCAV)
* Adversarial examples
* Influential instances
**build-failing**
Hello Christoph Molnar, it would be great if you could consider answering the following questions regarding interpretability of deep learning models:
1. Looking at things from a deep learning for computer vision standpoint, the go-to techniques for explainability are: CAM, and the improved GradCAM, GradCAM++.
2. Does the book go over these topics as well? Or would you be covering these topics in the next edition?
3. Recently, there have been significant research efforts on the use of causal attribution to addressing explainability. I would like to know your thoughts on the same.
4. On the ML front, does the book cover simple yet intuitive techniques such as partial dependency plots–on the various features so that we can make a more informed decision on the set of features that the model is actually using in its decision-making process.
Thank you in advance.
**Christoph Molnar**
1. and 2. Grad-CAM and similar appraoches are covered in this chapter: [https://christophm.github.io/interpretable-ml-book/pixel-attribution.html](https://christophm.github.io/interpretable-ml-book/pixel-attribution.html)
2. I haven’t covered causal attributions so far. My take at the moment is: You have to approach causality already when you fit your model. Meaning you need a causal model when you want to have a causal interpretation.
3. The book covers many of the simpler approaches:
PDP: [https://christophm.github.io/interpretable-ml-book/pdp.html](https://christophm.github.io/interpretable-ml-book/pdp.html)
Permutation Feature Importance: [https://christophm.github.io/interpretable-ml-book/feature-importance.html](https://christophm.github.io/interpretable-ml-book/feature-importance.html)
**build-failing**
Thank you very much for your detailed reply, and for pointing me to these references.
**CJ**
As model risk becomes increasingly important to businesses, interpretability will likely need to be more incorporated into business practice. Does your book address the various use cases for interpretability?
**Christoph Molnar**
The book is more centered around the methods rather than specific use-cased. Some motivation for why we need interpretability can be found here: [https://christophm.github.io/interpretable-ml-book/interpretability-importance.html](https://christophm.github.io/interpretable-ml-book/interpretability-importance.html)
**naaavI**
What is your approach to ensembles results explanation? Especially with different approaches inside? Christoph Molnar
**Christoph Molnar**
For ensembles you can also use model-agnostic methods. You view your ensemble as one model: Input data comes in and you get the prediction. Then you can still calculate, for example, permutation feature importance or explain individual predictions with Shapley values.
With model-agnostic interpretation methods it does not matter how many models are inside the ensembles or how complex they are.
**naaavI**
Do you really uncover unsulervised learning algorythms results interpretation for absolute beginners?
**Christoph Molnar**
Not sure I interpret the question correctly. Is your question whether beginners can understand the results of interpretation methods?
**naaavI**
Yeah, a kind of.
**Christoph Molnar**
I do think there needs to be some training how to interpret the outputs correctly. For example what it means if permutation feature importance is zero. Or how to interpret Shapley values.
**naaavI**
What i your definition of explanation ?: i mean - explain how the figures are calculated , or how to interpret results in different business cases?
**Christoph Molnar**
The word “explanation” is a bit fuzzy and used differently by different people. In the context of explaining black box models, it’s often used as “explaining” how the model made a prediction. And explanation is often equated to attributing the prediction to the input features.
**Tim Becker**
Hi Christoph Molnar, could you provide us with some intuition on how SHAP and LIME methods work and point out the differences between the methods? Are there situations when you would select one over the other?
**Christoph Molnar**
Intuition LIME: We can explain a single prediction by approximating the complex prediction function with a simpler model, like a linear regression model. Idea is: We fit this linear model only with data that is very close to the data point that we want to explain.
Intuition Shapley: We want to fairly attribute the prediction for a data point to the individual feature values of this data point. Shapley values tries out different combination of feature values to determine how much each of the feature values contributed towards the prediction.
Similarity between SHAP and LIME: Both can be expressed as linear models. By choosing a certain kernel for LIME, you can get the same results as for Shapley values. But in practice, LIME uses other kernels to weight the data points in the neighboorhood.
Which to use: I think that Shapley values has a more solid theory. LIME has the big unsolved problem how to define what the neighbourhood of a data point is.
**Vladimir Finkelshtein**
Machine learning algorithms are always ranked/judged/hyped by their performance on some benchmarks datasets. On the other hand, with explainability algorithms, there is an example of its output, but I have not seen a systematic evaluation.
Is there some idea of a metric for explainability algorithms? Like, we give models list of problems, where we already know what the explanations should be and rank the models? Perhaps, one can create synthetic problems, where we know causality?
**Christoph Molnar**
There are various metrics that can be used to assess how well explainability methods work:
**Christoph Molnar**
Fidelity: Some explanation methods like LIME build a prediction model themselves. With fidelity you can measure how close the predictions of your explanations are to your model predictions.
Sparsity: You can count how many features were used for your explanation. Shorter explanations should be easier to understand for users.
**Christoph Molnar**
Then there are human- or task oriented metrics:
* How well can a user predict what the model will do, when the user is allowed to see explanations for the model?
* How much faster are users with their task when they are also given expllanations
**Christoph Molnar**
But the main problem: There is no ground truth as we have with supervised machine learning.
I would say, interpretable machine learning / explainable AI is more like descriptive statistics: For example, you can compute either the mean or the median. But nobody tells you whether the mean or the median is the correct way to describe your distribution. Similarly, the interpretation computes different “descriptive statistics” of your model. Our goal as researchers is to make it clear what the correct interpretation is.
**Arkadiusz Modzelewski**
Christoph Molnar First of all, thank you for the opportunity to ask questions and for a great book. I currently work as a machine learning specialist, but I also often think about recruiting for a PhD. One topic that I am very interested in is precisely explainable artificial intelligence. Would you perhaps have any tips for someone who wants to go for a PhD? Or do you see any areas in explainable and interpretable artificial intelligence that are particularly worth looking into as part of a research path? I see now that authors answer questions until Thursday, but maybe there will be an exception, so I try to ask :D
**Christoph Molnar**
I have seen so many labs starting to work on interpretability. So the field is becoming much more crowded.
I would not work on another paper that tries to improve Shapley values or that introduces another saliency map algorithm.
I think the most valuable is to bring more rigor to the field: Analyze what the limitations of existing methods are. Make sure that we better understand when Shapley values or other methods fail or may lead to a misleading interpretation. But to be honest, it can be harder to publish such papers, since it’s often easier to just invent some new method, unfortunately.
**Arkadiusz Modzelewski**
I recently read this article: Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead ([https://arxiv.org/abs/1811.10154](https://arxiv.org/abs/1811.10154)
) The author seems to have a rather strongly critical approach to explaining black box methods. Are you perhaps familiar with this article and could you let us know what your opinion is on the subject?
**Christoph Molnar**
I’ve read the paper. I don’t agree with all points, but I do agree that we should not use black box models for high stakes decisions.
I wouldn’t say that what the authors call “inherently interpretable” models are the solution. They can also fail and are often not as interpretable as one thinks. I would even g say that for many of these high stake decisions we should not use ML at all, like bail decisions.
**Arkadiusz Modzelewski**
Christoph Molnar Thanks a lot for your answers!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Artificial Intelligence with Python – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Artificial Intelligence with Python
-----------------------------------
#### by [Prateek Joshi](https://datatalks.club/people/prateekjoshi.html)
##### The book of the week from 09 May 2022 to 13 May 2022
Artificial Intelligence with Python, Second Edition is an updated and expanded version of the bestselling guide to artificial intelligence using the latest version of Python 3.x. Not only does it provide you an introduction to artificial intelligence, this new edition goes further by giving you the tools you need to explore the amazing world of intelligent apps and create your own applications.
This edition also includes seven new chapters on more advanced concepts of Artificial Intelligence, including fundamental use cases of AI; machine learning data pipelines; feature selection and feature engineering; AI on the cloud; the basics of chatbots; RNNs and DL models; and AI and Big Data.
Finally, this new edition explores various real-world scenarios and teaches you how to apply relevant AI algorithms to a wide swath of problems, starting with the most basic AI concepts and progressively building from there to solve more difficult challenges so that by the end, you will have gained a solid understanding of, and when best to use, these many artificial intelligence techniques.
* [Book's page](https://www.packtpub.com/product/artificial-intelligence-with-python-second-edition/9781839219535)
* [Buy on Amazon](https://www.amazon.com/Artificial-Intelligence-Python-intelligent-TensorFlow/dp/183921953X)
* [GitHub repository](https://github.com/PacktPublishing/Artificial-Intelligence-with-Python-Second-Edition)
Questions and Answers
---------------------
**Dr Abdulrahman Baqais**
Thank you Prateek Joshi on writing this book. The book includes some interesting chapters about Heuristics, GA and gaming in AI which is very helpful. The book seems comprehensive and targeted for python developers which is great.
What python libraries is used for GA and Heuristics?
Also, in comparison to matlab for GA and heuristic, Does python provide advantages?
In logic programming where lisp and prolog are dominating, In which areas python can give cutting edges?
Thank you again for this comprehensive book that address other parts of AI rather than ML.
**Prateek Joshi**
Dr Abdulrahman Baqais Thank you for the kind words and the question.
With regards to Python libraries, I’ve experimented with simpleai for Heuristics and DEAP for GA. They provide most of the functions you’d need.
With regards Python vs Matlab, I prefer Python because it’s more production ready. You can build an application and deploy it without having to redo the work. It’s lightweight compared to Matlab (which matters a lot when you’re deploying). Plus the Python community is huge, which means you can get answers to your questions relatively easily.
For logic programming, the big advantage of Python is interoperability. When you’re working on a tool that’s in production, different parts of the product can use different frameworks. Python plays well with all the cloud tools, so plugging in your tool into the larger product becomes easy. Plus you get access to many people who are constantly building and experimenting in Python. You don’t have to reinvent the wheel for many underlying methods/functions you end up using.
**Dr Abdulrahman Baqais**
Interesting….seems python is close to be a universal programming language for AI…..one programming language to work on all AI projects.
**Alexey Grigorev**
Hi Prateek!
You’ve written many books, more than 10. How did your manage to do that and stay sane? 😄
**Prateek Joshi**
Alexey Grigorev I aim to stay consistent with my writing schedule. The act of showing up everyday helped develop my writing muscle in my early days. This allowed me get the work done on time. By after 13 books, I had nothing new left to say. So decided to take a break. 🙂
**Allan**
Hi Prateek, thanks for taking the time here to answer questions. What advice would you give to someone with many years of experience in database technology that is looking to switch to data science / machine learning.
**Prateek Joshi**
Allan: Thank you for the question. Given your years of experience in database technology, you already have a leg up here. Being proficient in data wrangling is a great skill to have.
If you want to switch to DS/ML, my recommendation is to get familiar with basic ML concepts and start doing projects.
* Pick a domain that interests you e.g. search engines, healthcare, image recognition, ecommerce, fintech, recommender systems.
* Once you do, look for problems that seem interesting e.g. how to identify at-risk patients or how to identify repeat buyers on an ecommerce site
* There are a large number of open source datasets available. You can use them to train your model.
* The goal is to start from a real problem and build a tool that addresses the problem in a practical way.
Once you finish it, I’d also recommend you write a brief blog post (500-700 words) explaining what you did. This will help you strengthen your own understanding.
As you continue to do this over time, you can develop a good understanding of how to DS / ML to solve real world problems. Hope this helps.
**Allan**
Thanks Prateek, that is very helpful!
**Sushant Mittal**
Hi Prateek, I had the exact same question as Allan. Thanks for your advice. 👍
**Tim Becker**
Hi Prateek Joshi, thanks for answering questions 🙂
* Why was it time for a second edition of your book?
* For what kind of problems would you suggest logic programming?
* What is the most difficult part of building a startup? What is your advice if someone wants to do it?
**Prateek Joshi**
Tim Becker Thank you for the questions 🙂 Here are my thoughts:
1. Publishing the second edition was mainly a function of demand. I received many positive responses from readers for the first edition. Plus they gave a lot of good feedback. This led to us publishing the second edition. It gave us a chance to incorporate all the feedback and add new chapters as well.
2. Logic programming is a programming paradigm (like objected-oriented or functional programming). It looks for solutions using facts and rules. Once we specify a goal, the solver comes up with a tree that constitutes the search space to solve the problem. We can provide raw input and then ask questions about the missing pieces. Here are a few examples where we can use logic programming: matching mathematical expressions, validating prime numbers, inferring relationships in a family tree, analyzing geography, solving puzzles
3. With regards to building a startup, the most difficult part depends on the stage you’re at. It can go from: talking to potential users/customers to see if your product needs to exist in this world –> fundraising –> getting the first 10 paying customers –> demonstrating product market fit –> recruiting early teammates –> fundraising again –> scaling up revenue –> recruiting leaders who can build out different functions (sales, marketing, product).
There are many other parts that can pop up during the journey as well.
My advice to someone who wants to do it would be to to just get started. The first step would be to reach out to as many potential users/customers and understand how they get their work done today. Your way of getting that work done should 10X better than status quo. That’s how you know there’s a real gap.
**Tim Becker**
thank you 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Natural Language Processing with Transformers – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Natural Language Processing with Transformers
---------------------------------------------
#### by [Leandro von Werra](https://datatalks.club/people/leandrovonwerra.html)
, [Lewis Tunstall](https://datatalks.club/people/lewistunstall.html)
, [Thomas Wolf](https://datatalks.club/people/thomaswolf.html)
##### The book of the week from 25 Apr 2022 to 29 Apr 2022
Since their introduction in 2017, transformers have quickly become the dominant architecture for achieving state-of-the-art results on a variety of natural language processing tasks. If you’re a data scientist or coder, this practical book shows you how to train and scale these large models using Hugging Face Transformers, a Python-based deep learning library.
Transformers have been used to write realistic news stories, improve Google Search queries, and even create chatbots that tell corny jokes. In this guide, authors Lewis Tunstall, Leandro von Werra, and Thomas Wolf, among the creators of Hugging Face Transformers, use a hands-on approach to teach you how transformers work and how to integrate them in your applications. You’ll quickly learn a variety of tasks they can help you solve.
* Build, debug, and optimize transformer models for core NLP tasks, such as text classification, named entity recognition, and question answering
* Learn how transformers can be used for cross-lingual transfer learning
* Apply transformers in real-world scenarios where labeled data is scarce
* Make transformer models efficient for deployment using techniques such as distillation, pruning, and quantization
* Train transformers from scratch and learn how to scale to multiple GPUs and distributed environments
* [Book's page](https://www.oreilly.com/library/view/natural-language-processing/9781098103231/)
* [Buy on Amazon](https://www.amazon.com/_/dp/1098103246?tag=oreilly20-20)
* [GitHub repository](https://github.com/nlp-with-transformers)
Questions and Answers
---------------------
**leandro von werra**
Hi everyone, looking forward to answering all of your questions!
**Lewis Tunstall**
G’day everyone, it’s great to be here 🤗 ! Looking forward to your questions too 🙂
**Igor Dal Bo**
Hi guys, thanks for taking the time to answer our questions 🙂
* what is the knowledge level needed for this book?
* which role do transformers play in topic modelling?
**Lewis Tunstall**
Hi Igor, thanks for your questions!
1. The book assumes some familiarity with deep learning and PyTorch, so we recommend reading it after working through e.g. the [fast.ai](http://fast.ai/)
course / book.
2. Are you referring to unsupervised topic modelling, e.g. cluster my texts into a set of topics? For this, there’s a neat library called BERTopic ([https://github.com/MaartenGr/BERTopic](https://github.com/MaartenGr/BERTopic)
) that get’s quite good results in my experience. The alternative is to use `sentence-transformers` to embed your documents ([https://sbert.net/](https://sbert.net/)
) and then apply clustering on the embeddings - this too gives quite good results.
**Igor Dal Bo**
thanks Lewis Tunstall for your answers 🙂
* I had in mind taking that course for a long while, your suggestion add another point to it 😄
* that is exactly what I wanted to know, thanks a lot for pointing me out to the right resources. So far what I have been working on was a supervised approach, but defining a topic dictionary manually requires a lot of time and knowledge..
**Gonzalo Ancochea Blanco**
Hi!! Thanks Leandro, Lewis and Thomas for sharing some of your time!
Through your QA chapter I became very interested in Haystack. Retrieval+extraction/generation is very relevant for our team’s needs and I’ve seen they have more nodes but we also need flexibility to add any other NLP components to the pipeline (e.g. NER, topic modelling, sentiment…)
Do you have any experience with building custom Haystack nodes and using the framework to implement full pipelines, and if so what is your take on it? Any alternatives? Thanks again 😄
**Lewis Tunstall**
Hi Gonzalo, thanks for your question!
I believe that `haystack` did a major overhaul of their `Pipeline` abstraction in v1 (which unfortunately came out after our book was in production), and my understanding is that it now supports several NLP tasks like summarization (i.e. summarize the retrieved docs): [https://haystack.deepset.ai/reference/pipelines#searchsummarizationpipeline](https://haystack.deepset.ai/reference/pipelines#searchsummarizationpipeline)
For a completely custom pipeline, I’d suggest joining their Slack group: [https://haystack.deepset.ai/community/join](https://haystack.deepset.ai/community/join)
They’re really responsive and helpful for these kind of questions!
**Gonzalo Ancochea Blanco**
Thanks a ton Lewis Tunstall I will check out the source code of Pipeline and definitely join the Slack group!
**Alexander Seifert**
So awesome, good to have you guys here! My first question would be: Being both NLP practitioner _and_ book publisher I know that (1) you often can’t include everything you want into a book, and (2) the field of NLP moves faster than the printing presses. Were there things you had to cut that you would have liked to include, and what advances in the field that came about since you handed off the manuscript make you excited to have included in the next edition of the book if there was one?
**leandro von werra**
When we started working on the book multi-modality was not really a thing, yet. Since than CV, Audio, Tabular applications etc. were also conquered by transformers. We were able to give a brief overview of what exists in the last chapter but obviously it would have been nice to include them in more detail and show how to train them. Maybe v2 of the book needs to be renamed to `Machine Learning with Transformers` 🙂
It helped a lot to work closely with the maintainers of `transformers` and `datasets` to plan a bit around upcoming features or features that would soon be deprecated. E.g the streaming API used in the `Training transformers from scratch` chapter was finalized at the same time as we wrote the chapter.
**Alexander Seifert**
I heard that you wrote the entire manuscript inside Jupyter notebooks. How was your experience with that toolchain for writing a whole book, and what did O’Reilly think about that? I imagine it’s still somewhat unusual.
**Lewis Tunstall**
We used the fastdoc library ([https://github.com/fastai/fastdoc](https://github.com/fastai/fastdoc)
) to handle all the conversion from notebooks to Asciidoc (which is O’Reilly’s preferred format).
Overall we found the experience was rather smooth, and we were lucky that O’Reilly already had experience from the fastai book 🙂
We heard from the production editors that they’re getting more submissions in this format, so I think notebooks might become the norm for these kind of publications - it really makes writing a breeze when everything is in one place!
**leandro von werra**
One point to addd here is that the O’Reilly team was happy to keep the notebook -> asciidoc conversion pipeline in place until the very last reviewing process which kept the notebooks in sync with the formatting/copy-editing changes.
So it was really nice of them production team to adapt to this new workflow. As a nice side effect the whole book is nicely git version controlled 🙂
**Max Payne**
Hi, great project and book indeed,
I have a question regarding explainability, which is gaining a lot of attention. With such a large model/training data, is it possible for these APIs to ‘explain’ their decision?
**Lewis Tunstall**
Hi Max, thanks for the question! As far as I know, explainability of transformers is still an open research problem and one that we didn’t have space to cover in the book.
One of the best examples I know of is Jay Alammar’s `ecco` library ([https://github.com/jalammar/ecco](https://github.com/jalammar/ecco)
) which uses various visualisation techniques to understand which factors lead to a specific prediction. It doesn’t give you an explanation “for free”, since you have to do some work to interpret these results, but I think it might be interesting to you
**Max Payne**
Thanks alot
**Max Payne**
Also, regarding explainability, don’t you think that the paper ‘Pretrained Transformers as Universal Computation Engines’ sort of makes it even harder, since it claims something like there is something special in the architecture that makes it generalizable and hence making it more difficult to justify the outputs?
**Alexander Seifert**
I am working on a human-in-the-loop NER system, where I’d like to use the prediction scores as a proxy for correctness. As far as I understand, this is called “uncertainty estimation” and it is not something that transformer models are providing out-of-the-box. Is there content on this problem in the book, and if not, are there any good resources you would recommend?
**leandro von werra**
So if by prediction scores you mean the model’s output probabilities then yu can actually get them fairly easy from the `transformer` models: For each token the model will return the logits and by applying a SoftMax function you’ll get each classes probability.
In the book we show how to get the logits from the model for NER so the only thing that’s missing to get probabilities is applying SoftMax. Hope that helps!
**Alexander Seifert**
the problem is that the softmax will indeed squish the logits into a probability distribution (formally), but this distribution is unfortunately poorly calibrated to the correctness likelihood: [https://arxiv.org/abs/1706.04599](https://arxiv.org/abs/1706.04599)
**Hugo**
Thanks for this wonderful book! I am looking forward to reading it. Writing a book is a very time consuming task so I have two general questions: 1) how long did it take you to do it? 2) what was the most challenging section of the book to write?
**leandro von werra**
1. It took as roughly 1.5 years to write the book. Since we did it in our spare time this means that we mostly worked on it on weekends 🙂
2. For me working on the few labels and pretraining from scratch were the most challenging. The few labels chapter required a lot of experimentation and research to first figure out what works well. The pretraining chapter was one of my first contacts with distributed training and in addition we used a lot of features of the hugging face ecosystem that were brand new.
Maybe the others have other chapters they found challenging 🙂
**Adisorn**
Hi, I am a beginner in this field. I heard from the above comment that you use Jupyter notebooks to write your code. Can I follow your code step by step if I use Collab?
**leandro von werra**
Yes you can! See the table with Colab links in the book’s repository:
[https://github.com/nlp-with-transformers/notebooks](https://github.com/nlp-with-transformers/notebooks)
**Adisorn**
In case my interesting NLP task is the Text summarization. Is there a connected issue to this topic?
**leandro von werra**
You could look at chapter 6 where we show how to train a transformer model on text summarization. All the book’s code is freely available in the repository (but without the text from the book, so no explanations).
**Amruta Ranade**
Hi leandro von werra, the concept of transformers is new to me and am learning about their architectures and functions currently. I was interested in knowing how transfer learning can be adopted with the transformers and are there any specific applications that you have covered in this book?
**leandro von werra**
Hi Amruta Ranade, yes transfer learning can be applied to transformers and is in fact the driving force behind their success. Except for the last chapter where we train a model from scratch all chapters show how to apply transfer learning to tune a pretrained model on a specific task such as classification, NER, QA, or summarization.
**Amruta Ranade**
Thankyou for the information ! Sounds interesting to me. I am definitely going to buy your book! 😊
**Sitao Zhang**
Congrats HF team! !
This is Sitao from the J&J team, really glad to see you guys published this wonderful book and been awarded as the <#C01H403LKG8|book-of-the-week>.👏🎉
Just a glance of the book description, I’m curious about:
1. Whats the major difference between this book and the HF documentations online?
2. Since I have been leveraging the transformer for a while, which part/parts of the book do you mostly recommend for people who have some experience?
3. Not sure whether the HF BigScience will be cover here? I know the BigScience is super recent, maybe the similar concepts will be discussed in the book?
Thank you guys!👍
**leandro von werra**
Hi Siato, thanks for being here!
1. What we try in the book is to be much more practical than the documentation, which is why each chapter aims to solve a realistic use-case. So it involves more steps like preparing your data or doing some error analysis of your model.
2. In general the later chapters in the book are mode advanced (e.g. making transformers efficient for prod, few labels or pretraining from scratch). Almost every chapter also tackles a different task (classification, ner, qa, summarization etc.). So you could also pick a task that you haven’t encountered before.
3. Unfortunately, we don’t go into detail of BigScience. The very last chapter looks a bit ahead especially at the scaling trend of transformers and multimodal transformers.
**Lalit Pagaria**
Thanks team for conducting this QnA.
Have following query -
HF being a leader in practical deployment of transformers model and super heading collaborative research initiative like Big Science. What are practical pipelines (pre-processing) must before giving input to transformers (production setup)?
**leandro von werra**
So I think this can be divided into two categories:
* data cleaning: before you train your model you want to make sure that the data is as clean as possible. for BigScience we spent a lot of time cleaning and filtering the data. this includes: removing duplicates, deleting html code from websites and so on. whatever you do to your training data here you should also do to the inputs in production. always clean your data 🙂
* tokenization: there are a few steps that besides splitting the string into tokens that the tokenizer takes care of such as unicode normalization or lower casing all strings. as long as you use the same tokenizer in prod as you used during training you should be fine.
hope that answers your question! 🤗
**Lalit Pagaria**
Yes it answers partially. But I was looking for a deeper answer. 🙂
More of practical example
**leandro von werra**
I guess the deeper answer depends on your use-case 🙂 To work as well in production as on your local test set the most important thing is to make sure the data preprocessing steps are the same and the test set reflects the production data.
The second aspect is the performance on the test set: data cleaning is usually very important here but again which particular steps are necessary depends on your use-case and dataset.
**Lalit Pagaria**
In low resource setup. Is multi-head single transformers deployment is more suited or serially connecting multiple specific transformers? For example if we need translation + classification + summarization on same given input.
**leandro von werra**
I experimented in the past with creating a single model for three classification tasks and thus creating a model with three heads. the motivation was more to have just one large model in production instead of three and we didn’t see a significant performance improvement. Also this was the same task type just three different classification criteria.
A model that works well on a wide range of tasks at the same time is T5. Maybe you can tune it for the tasks you are trying to cover.
Alternatively one thing that can work well in low resource setting is to use models that are already trained on a e.g. summarization task instead of the raw pretrained models. We do this in the QA chapter: we use a model that has been trained on SQuAD and train it a bit more on the Amazon dataset which gives better results.
Which (if any) of these suggestion works depends a lot on what you are actual use-case looks like. My general advice is to build a good evaluation set that captures your use-case and lets you easily evaluate your models. Start creating a few simple baselines and then iterate quickly on different ideas. Without the evaluation pipeline the iterations will be slow.
**Lalit Pagaria**
Thank you leandro von werra
Yes trial and error is the way forward
**A**
Huggingface has done a tremendous job in making transformer based models accessible to a lot of people. As a result, trying these SOTA models is no longer difficult and has enabled organisations of all scales make use of them
1. How do you feel about the widespread applicability of these models? Are there any cool use-cases in maybe some obscure domain where transformers are working there magic and which you’d like to share?
2. The popularity and ease of use of Huggingface means even non-DS background folks also are able to use the power of BERT. Most of the times this leads to models which are not built taking into bias and other components of responsible-ai into consideration. What do you think can be done to tackle this (if it needs to be tackled) ?
Thanks for the book! Definitely need to read this.
**leandro von werra**
Hi A, thanks for being here!
1. I am personally very excited about ML in science. I think any tool that enables iterate faster in science will have a huge impact. One of my favourite application here is [AlphaFold](https://www.nature.com/articles/s41586-021-03819-2)
from DeepMind. Predicting how proteins fold is a critical task in chemistry and biology to make progress and until AlphaFold the only way to get good predictions was through very expensive and long experiments that could take weeks or month. With AlphaFold the same researchers can now iterate through 100s of molecules in a day which changes the game completely.
2. I would even argue that most biased models that make headlines these days were deployed by DS folks 🙂 I am not an expert here and maybe Thomas Wolf has more to share. My two cents are that thinking about and investigation bias and impact of a deployed model should become a core task of DS lifecycle. This requires documenting on what exactly and how the model was trained and rigorous testing of how it performs in different situations. There is a lot of work going on in [BigScience](https://bigscience.huggingface.co/)
for example where people work on showing how a large language model can be built and released responsibly.
**Mischa Ungermann**
Hi guys, thanks for taking the time for this Q&A!
I was working with mostly CV topics for the last years and now I am curious to get an insight into this world of NLP everyone is talking about. For this I was wondering what resources you would recommend as an entry point to the field? I could imagine a certain book will be high on the list, and probably also the Hugging Face course, but maybe you know of more gems?
**leandro von werra**
Indeed our book is NLP beginner friendly and the main requirement are the core ML concepts (what’s a train test split, how to build and train a neural network, standard metrics etc). So if your are familiar with CV that won’t be an issue 🙂
I know a lot of people got into NLP with Chris Mannings excellent lecture series: [https://web.stanford.edu/class/cs224n/](https://web.stanford.edu/class/cs224n/)
. Also the popular fastai course covers some NLP: [https://course.fast.ai/videos/?lesson=8](https://course.fast.ai/videos/?lesson=8)
**leandro von werra**
So many great questions so far! Thank you and keep them coming! 🤗
**Max Payne**
It’s the first time I have seen a book (predominantly targeting a specific API) and also addressing the topic of ‘Text Generation’ (i.e. beam search, greedy decoding etc.). How does HuggingFace deal with these methods - I mean do we as a user have a choice in selecting the method? or the model has a default setting? (Sorry if this question is answered in the book)
**leandro von werra**
Yes, we go into that in more detail, but to give you the short summary here:
Every model in the `transformers` library that is able to do text generation has a `.generate()` method. This method is loaded with options to do all sorts of generation strategies: greedy, beam search, sampling (with top-k or nucleus), repetition penalties etc. You can also define your own stopping criteria or processors to be applied.
[see here](https://huggingface.co/docs/transformers/v4.18.0/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate)
Patrick von Platen wrote a great article about that function and how to use it:
[https://huggingface.co/blog/how-to-generate](https://huggingface.co/blog/how-to-generate)
**Max Payne**
Awesome. Thank you!
**Marcello La Rocca**
Hi! Congrats on the book, it looks awesome!
Question: where would you say it’s the main advantage in using Huggingface’s tranformers vs say GPT-3? What could be a disadvantage?
**leandro von werra**
There are a number of advantages:
* you can host the modals on your own infrastructure - for many companies sending data to an outside API can be complicated, especially if the data is sensitive
* you have a lot more flexibility to train the models and iterate through different training setups. you can also customize your architecture to your use-case
* dopending on your setup hosting your own models can be much cheaper, especially if you have heavy workloads. Nils Reimers wrote a thread about this: [https://twitter.com/nils\_reimers/status/1487014195568775173](https://twitter.com/nils_reimers/status/1487014195568775173)
**Marcello La Rocca**
Thanks Leandro! Indeed, that totally makes sense.
Also thanks for the link! 🙏
**Marcello La Rocca**
2nd question (partially related, but also more about the book content): what’s the hardest challenge in scaling transformers?
**leandro von werra**
oh there are many, hard to pinpoint a single hardest 🙃 just to list a few:
* | | |
| --- | --- |
| engineering: training such models requires a large distributed infrastructure with s of GPUs. you need some resilience when some of them crash and you need to make sure they are optimally used. also at that scale you can be haunted by many numerical instabilities. in BigScience we are training such a model in the open and you can read the chronicles of the experiments: [https://github.com/bigscience-workshop/bigscience](https://github.com/bigscience-workshop/bigscience) |
* dataset: training these beadts requires a lot of data (~TBs of text). at the same time you need to make sure that the data is clean which gets harder at that scale. you can‘t just look at it
* release: how can one responsibly release such models. there is a lot of work going on about for example licensing.
maybe Thomas Wolf has some more insights here :)
**Marcello La Rocca**
Ah, interesting! How can you deal with the numerical instabilities? I imagine they also propagate and get larger and larger 🤔
I’m looking forward to reading more in your book 🙂
**Marcello La Rocca**
Also what do you think about “sparsity-aware inference engines”?
**Marcello La Rocca**
3rd one: Do you think that transformers will be possibly applied beyond text/vision/speech? Maybe reasoning?
**leandro von werra**
Currently it seems that transformers are penetrating almost every field in ML. on the topic of reasoning you might find this recent paper by Google interesting: [https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903)
**Marcello La Rocca**
Wow, that’s impressive!!! Thanks a lot for the link
58.1% on GSM8K is impressive (though also a reminder of long path ahead!)
**Marcello La Rocca**
(thanks a lot!!!)
**Evren Unal**
Hi.
Thank you participating this event.
I would like to build a language translator some time in the future.
Q1) Can i use transformer technology to built it?
Q2) if so, did you give enough information in your book to build it?
**leandro von werra**
1. definitley. there are for example ~1000 translation models already on the huggingface hub! see: [https://huggingface.co/models?pipeline\_tag=translation&sort=downloads](https://huggingface.co/models?pipeline_tag=translation&sort=downloads)
. you can either use them out of the box or further tune them on your specific data!
2. we don‘t go into details on translation specifically, but we show summarization in more detail. conceptually the task is very similar (one input text and an output text) and we also show and explain BLEU which is often used in translation.
**Evren Unal**
thank you very much 👍
**armin zirak**
Hi all!
Thanks for the great book!
Q: If I want to use transformers for numeric data instead of text, how can I do that? Do you think feeding the numbers as text does the job in the best way or is there any specific solution?
**Lewis Tunstall**
Hi Armin, thanks for your question! There do exist transformers like SAINT for numerical features like the ones you would find in tabular data.
However, it seems that gradient boosting remains the state-of-the-art and it might be a while before transfer learning is truly cracked for tabular data (see plot).
There’s a very nice survey of these models here: [https://arxiv.org/abs/2110.01889](https://arxiv.org/abs/2110.01889)
**Allan**
Hi, thanks for taking the time to answer questions here! Looking forward to the book!
Does the book address issues related to deploying and using transformers in production settings?
**Lewis Tunstall**
Hi Allan, yes Chapter 8 covers various techniques related to optimisation including:
* Knowledge distillation
* Quantization
* Graph optimization with ONNX Runtime
* Pruning (although at the time of writing it wasn’t easy to prune transformers effectively)
The end result is figure like this which shows how you can reduce the latency of a BERT model by ~7x 🙂
**Allan**
Thanks!
**Shantanu Ladhwe**
Hey, thanks for the amazing book. I am yet reading, and really liked the explanation of transformers there.
Question: How different would be Google’s next AI Architecture- Pathways from the Transformer architecture?
**leandro von werra**
Hi Shantanu Ladhwe, not an expert on Pathways but as far as I understand it is mainly a vision towards multi-tasking, multi-modality, and maybe a switch to sparsity (see Jeff Dean’s article [here](https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/)
). This vision could be (and will likely be) realized with a Transformer architecture. There are already multi-tasking (e.g. T0pp), multi-modality (e.g. CLIP, Perceiver), and sparse (e.g. SwitchTransformer) transformer models out there.
The Pathways Language Model ([PaLM](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html)
) uses a classical transformer decoder architecture and the main innovation is around infrastructure to train the model efficiently on 6000 TPUs.
**Max Payne**
I wish I could use a Transformer here to generate questions (and hence increase my chances of getting the book) :-)
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Everyday ML Questions – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Everyday ML Questions
---------------------
#### by [Santiago Valdarrama](https://datatalks.club/people/svpino.html)
, [Vladimir Haltakov](https://datatalks.club/people/vladimirhaltakov.html)
##### The book of the week from 16 May 2022 to 20 May 2022
A different way to learn
This book aims to teach you something new, one question at a time.
There are 20,000 + 1 machine learning and data science books out there. That’s great, but we wanted something different.
In April, we started publishing one machine learning question every day. A quick story with a problem and a few possible answers. Something quick, practical, and fun that you could solve in a few seconds. In just 30 days the site attracted more than 80,000 visitors and every day over 1,500 people answer the question and it keeps growing.
Many reached out asking for previous questions.
This book is a compendium of the 30 questions we published online from April 1 to April 30, 2022.
* [Book's page](https://gumroad.com/a/850109555/yjbkr)
* [Bnomial subscription](https://gumroad.com/a/313238643/alehb)
Questions and Answers
---------------------
**Alexey Grigorev**
Hi Santiago and Vladimir! Welcome!
How did you come up with the idea of bnomial? Which problem did you see with other similar websites and blog posts?
**Santiago**
Thanks for inviting us!
I’m always been fascinated by multi-choice questions because they are an excellent way to exercise your brain.
When I’m studying something new, I need to find ways to practice. Sometimes building projects is the way, but sometimes that’s not an option.
So we decided to build a site that would help people with their machine learning and data science journeys. Not to overwhelm people and make them sit through a long exam, but to go at it in different way: a little bit every day.
Bnomial tries really hard to do what I want for my life: to build a learning habit.
**Santiago**
Our goal is not to compete with blog posts, tutorials, videos, or books. Those are great!
Our goal is to become a different component of your learning journey. One that’s fun, doesn’t take a long time, and will stick with you for a long time.
**Alexey Grigorev**
Have you thought of doing something like duolingo eventually? I use it for learning German
**Santiago**
This is an interesting question.
Here is where we are headed in the short term: _creating a scoring system that encourages people to show up and answer more questions_.
Letting you know where you stand, and helping you track your improvements over time is key.
Right after that, we’ll see where we want to take this. We are taking it one step at a time.
**Alexey Grigorev**
By the way, that’s what duolingo does as well - they really built-in gamification into learning and did it pretty well. If you haven’t used it previously, check it out
**Santiago**
Yeah, I’ve used it. My daughter learns French through it.
**Shaksham Kapoor**
Santiago & Vladimir Haltakov Thank you both for such a unique initiative and for writing the first volume of this book 🙂 🙌
I have a few questions for you:-
* The questions that you post on your website, for instance, today’s question, it is a mix of an algorithm taught in AI + another taught in Data Science. So, this mix-and-match thing, is it something that you intend to do? Trying to show similarities and differences between concepts from similar fields.
* Curiosity - how do you come up with these questions? Are these some sort of interview questions or something that you come up with while studying?
* Are you planning to cover the recent advancements in NLP, CV etc., too via these questions?
All the best for your future endeavors 👍
**Santiago**
Hey Shaksham Kapoor!
1. You’ll see purely theoretical and practical questions. They range from probabilities and statistics, to algorithms, machine learning, data analysis, game theory and pretty much anything under the Artificial Intelligence umbrella. This is on purpose: our goal is not to tackle one specialized narrow area, but to find ways to increase the knowledge we all have.
2. Some of these are problems we deal with every day. Some are the result of what we read or conversations we have with others.
3. Absolutely! If it’s useful, there’s a place for it here.
**Shaksham Kapoor**
Thank you for your response!!
Another question, we have certifications for data science like “AWS Certified Machine Learning – Specialty”, any plans to cover questions asked in these exams in the future?
PS: of course not just focused on AWS but in general if anyone wants to prepare for a certification exam.
\> I like the way the questions are posted, like storytelling, so it helps connect things quickly + we don’t forget it easily 🙂
**Santiago**
I’d love to cover some of those questions—at least the core learnings they are trying to evaluate!
**Shaksham Kapoor**
That would really be helpful. I know these certifications do have suggested learning paths to help prepare for the exams, but they are more focused around their ecosystem and how it is used for performing data science.
As you mention, if there is a possibility to cover generic things that any of these certifications evaluate on, then it’s worth looking at 🙂
**Santiago**
definitely!
**Alexey Grigorev**
What were the easiest and the most difficult questions so far?
**Santiago**
The hardest question so far was the one we published on April 19. It was about the Softmax activation function.
The easiest question was the one we published on April 30. It was about the definition of hyperparameter tuning.
**Alexey Grigorev**
What was the question on april 19? maybe you could share the pages from the book with it? really curious about it
**Santiago**
Here is the question:
**Alexey Grigorev**
Hmm I initially wanted to say it’s one, but softmax doesn’t get the input to the network. It gets the output of the previous dense layer.
It also doesn’t return the index or the max value, so 2 and 4 are also our.
The closest is 3, but it doesn’t sort the probabilities…
So which one is it? =)
**Santiago**
2 and 4 don’t say that softmax returns that thing. They say that softmax is a version of the function that returns that thing.
**Santiago**
Basically, Option 2 says that Softmax is a smooth version of argmax
Option 4 says that Softmax is a smooth version of max.
**Alexey Grigorev**
I guess then it’s probably that this value is max… then it’s option 4?
But I understand now why it was the most difficult one =)
Thanks for sharing it!
**Santiago**
Here is part of the answer 🙂
**Alexey Grigorev**
Nice, thanks!
**Alexey Grigorev**
How many people got this one right?
**Santiago**
Can’t tell. We were’t storing individual data back then. We were only tracking correct choices (every question has up to 4 correct choices.)
**Santiago**
So I can tell you the overall performance, but not how many people got it correctly.
**Santiago**
But yeah, it’s been by far the most difficult question. Very tricky + dense wording = HARD
**Alexey Grigorev**
How much time do you spend preparing questions and answers? Is it something you do daily or you do it once per week?
**Alexey Grigorev**
And also curious to know how you work together on this. How does the process look like?
**Santiago**
I write one question every day and review and schedule another question.
Vladimir does the first round of reviews to everything I write. When he finishes, I let it sit for a couple of weeks before I do the final review.
Leaving some time between the writing and the editing session is key to get a much stronger question.
In summary, we write a question, then they go through a couple editing sessions before they get scheduled.
**Vladimir Haltakov**
When I write questions I try to write a couple at once. However, I have to admit recently Santiago is writing almost all of them…
One interesting thing is that we do this fully asynchronously. We have a GitHub repo and we use the issues. One issue is one question. The one who writes it assigns it to the other one for review. We use several labels to track where the question is in the pipeline.
**adanai**
I enjoy the case-study based exercises, thank you for doing this 😄
The questions in some way make one feel that they are planning a decision than just giving out an answer, it’s nice!
Would you….consider ….doing a freemium model for students? It would be helpful!
**Santiago**
Hey Adanai, answering the daily question is completely free and it will stay that way.
We are charging money for the book with every monthly question, but that’s a byproduct of what we really want to accomplish: get people to show up every day and learn something new.
**Merve Unlu**
Hello! Thank you Santiago and Vladimir Haltakov for this interesting and fun way of learning. The detailed explanation about the answers is very helpful.
I am not sure if there is one, but maybe a discussion forum could be useful. Could other people contribute to publish/generate questions?
**Santiago**
Thanks, Merve!
We are planning to enable a Discord server so people can get together and discuss individual questions.
Regarding collaborations, we are taking it very slow because we aren’t sure how to compensate people yet. Right now, I’m working with a person that reached out to me to try and explore how collaborations would look like. Based on the experience, we will open it up to more people in the future.
(The main issue with collaborations is compensation and rights. We are using this question as comercial content, so we need to ensure we retain the rights to the questions. That’s why we want to compensate those who contribute.)
**Merve Unlu**
Thank you for the answer.
**Gur Hevroni**
Hi Santiago & Vladimir Haltakov, I have a non-technical question.
The concept of daily problems is a great choice (very much like the awesome [https://thedailybyte.dev/](https://thedailybyte.dev/)
), and your mission statement is aligned with that idea (by mentioning the overwhelming amount of machine learning and data science books out there).
But in essence, when I look at your subscription plans, it seems that your subscribers are getting a ML-DS book every month…
Is there a more creative way to keep your subscribers engaged (and paying) instead of overwhelming them with more books?
I don’t mean to sound sarcastic or patronizing, I’m truly curious!
**Santiago**
This is a great question. Just for now, the subscription will get people every question we publish, but we will not stop there.
We have several ideas to improve that subscription over time. Without promising much, I can tell you that there’s a lot of value in a community of machine learning and data scientist professionals, so we are thinking on ways we can tap on that and connect companies with individuals.
**Vladimir Haltakov**
I like to think about the books as a reference. The goal is to have people coming to the website every day and answering the questions. As Santiago mentioned, we are working on a kind of a leaderboard to gamify the process a little bit 🙂
The book is if you want to go through the old questions. Maybe you had a problem at work and remembered that you saw the question on Bnomial and want to check it again and follow the reference link. Or maybe you are preparing for an interview and want to test your knowledge on past questions? Then you need the book.
**Santiago**
By the way, Gur Hevroni I didn’t know about TheDailyByte. Seems to be pretty similar.
**Alexey Grigorev**
Where and how do you get inspiration? It’s quite challenging to come up with a question and the explanation for the answer every day
**Santiago**
That’s my super power 🙂
I draw inspiration all around me, from the problems I face, the books I read, the conversations I have.
**Alexey Grigorev**
I guess this is what happens when you tweet multiple times per day for more than a year? 😃
**Shaksham Kapoor**
Do you think it would be helpful to include the concept of ELI5 in your current model? For instance, currently, the questions assume that anyone attempting it has background in AI (in general); however, that is not always the case.
Therefore, it would be helpful if in future, you can explain the important concepts from stats, probability, ML, DL etc. in an ELI5 manner. I believe it will be very helpful, since it is difficult to find ELI5 examples of such concepts. Of course, for a few of these, you can find ELI5 explanations, but those are scattered across the web. So, a common place for anything like this would definitely be an extremely useful resource.
**Vladimir Haltakov**
I think we already kind of do this sometimes. There are certain questions on more basic concepts and the explanation gives you more background about.
Like for example, questions about the bias and variance trade-off.
**Gregor**
Are you in any way affiliated with Lindsey Martin? She posts the #DailyML questions (as far as I know from r/dailymachinelearning). It seems like your questions are more of case-study-type (from what I see at [today.bnomial.com](http://today.bnomial.com/)
) while hers are very short statements. I still like the overall concepts as people start interacting with each other, discussion the questions and edge cases and even seemingly simple questions often offer some new insight.
**Vladimir Haltakov**
I didn’t know about Lindsey Martin and #DailyML - thanks for sharing! It certainly looks very interesting and I’ll check it out in more detail.
I couldn’t find the subreddit you mentioned, is it maybe r/learnmachinelearning?
And regarding discussions - I totally agree! This is something we have on the roadmap - a community where people can discuss about the questions.
**Chin Wee Chok**
Vladimir Haltakov I found on LinkedIn [https://www.linkedin.com/posts/lindseym1\_dailyml-activity-6933281680547868672-e\_9L](https://www.linkedin.com/posts/lindseym1_dailyml-activity-6933281680547868672-e_9L)
**Arsen Poghosyan**
I just love the picture and the font of the cover, it’s so different from the covers of other ML related books! Did any of you guys (Santiago and Vladimir Haltakov) participate in making this picture and in designing the book?
**Vladimir Haltakov**
Thanks 🙂 The image is in fact AI-generated using VQGAN + CLIP. I wrote more how this is done here: [https://twitter.com/haltakov/status/1455982555610636291](https://twitter.com/haltakov/status/1455982555610636291)
After that Santiago did some more processing to get the final look 🙂
**Arsen Poghosyan**
Thanks for the link! AI-generated picture for the book about AI, that totally makes sense 🙂
**Vladimir Haltakov**
Yeah, that was the goal 🙂
**Allan**
Thanks for answering questions here. Are all of the daily questions multiple choice format? That seems like one of the challenging parts in creating these questions - taking complex concepts and preparing answers choices that are at just the right level of difficult for learning purposes. Also, love the provided recommended reading links in the solution section!
**Vladimir Haltakov**
Thank you! Yes, creating this type of question is more difficult than the usual trivia questions, but they teach you much more.
They are all multiple choice for now (and usually more than one choice can be correct), but we also have an idea how we can extend them a bit so you will have to write some simple code.
**Santiago**
Interesting. We are not affiliated with her, but I’ll check her work out!
**Allan**
Thanks!
**Sandhya G**
Awesome concept. Wordle for ML. I think the website/ app + chat works better for this than a book format. The information is just right-sized.
I like the setup of today’s question, Riley’s speed dating match. It’s fun, light hearted.
One thing is that the UI needs a bit more clarity. Once I submitted the answers, it was not clear from the > and \* are the ones that I chose.
**Santiago**
You are right.
Funny story: a few weeks into this, we decided to change the way the colors and symbols work. For some reason, I talk myself into believing that the new way was better.
We released it. People hated it so much that we reverted back a few hours later.
I do agree with you: we need to think about it a little bit more carefully.
**Tim Becker**
Very interesting Santiago and Vladimir Haltakov. Do you attempt to switch between more theory and more case-study related questions? Also, do you try to evenly cover different areas of ML or is the selection based on your experience/preference or just randomly?
**Santiago**
For now, it’s mostly random, although I try to ensure that questions covering similar topics don’t go out close to each other.
We have a little bit of everything: there are purely theoretical questions, and other more practical ones. We are planning to keep it that way with a slight bias towards practicality.
And undoubtedly, the flavor of the questions highly depends on our own knowledge and experience. For example, I don’t have time-series analysis experience, so it’s hard for me to write about that.
**Prateek Joshi**
Santiago and Vladimir Haltakov: Kudos on building bnomial. Love the concept. The questions are engaging. What’s the future plan for this as it continues to grow? Would this be a full fledged learning platform for ML? Also will you be expanding into other types of content to support the learning needs of different types of ML students?
**Santiago**
Hey Prateek! Thanks for the comment!
For now, we are focused on bringing a community of people that want to learn while answering these practical questions.
**Santiago**
My focus is 100% on helping them build a new habit. Something they can do every day and make improvements without even noticing.
**Prateek Joshi**
Santiago Thank you
**Batul Bombaywala**
Santiago Vladimir Haltakov great website - its a fun way to learn!! My observation is that i just dont remember to go to the website everyday - is it possible for you guys to send email/notification everyday?
**Vladimir Haltakov**
Oh, we have this feature already 🙂
You can click LOGIN in the menu of the main page and enter your email. We will send you the new question every day then 🙂
**Santiago**
This might help:
**Santiago**
But point well taken: we need to make sure this is more obvious.
**Arsen Poghosyan**
I have registered 2 days ago and receive daily just one email with the question, no more and no less. To me everything is very convenient 🙂
**Batul Bombaywala**
oh, i missed this! thank you!
**Santiago**
No worries. I need to do a better job at making sure that’s obvious.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Practical Fairness – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Practical Fairness
------------------
#### by [Nielsen Aileen](https://datatalks.club/people/nielsenaileen.html)
##### The book of the week from 23 May 2022 to 27 May 2022
Fairness is becoming a paramount consideration for data scientists. Mounting evidence indicates that the widespread deployment of machine learning and AI in business and government is reproducing the same biases we’re trying to fight in the real world. But what does fairness mean when it comes to code? This practical book covers basic concerns related to data security and privacy to help data and AI professionals use code that’s fair and free of bias.
Many realistic best practices are emerging at all steps along the data pipeline today, from data selection and preprocessing to closed model audits. Author Aileen Nielsen guides you through technical, legal, and ethical aspects of making code fair and secure, while highlighting up-to-date academic research and ongoing legal developments related to fairness and algorithms.
* Identify potential bias and discrimination in data science models
* Use preventive measures to minimize bias when developing data modeling pipelines
* Understand what data pipeline components implicate security and privacy concerns
* Write data processing and modeling code that implements best practices for fairness
* Recognize the complex interrelationships between fairness, privacy, and data security created by the use of machine learning models
* Apply normative and legal concepts relevant to evaluating the fairness of machine learning models.
* [Book's page](https://www.oreilly.com/library/view/practical-fairness/9781492075721/)
* [Buy on Amazon](https://www.amazon.com/Practical-Fairness-Achieving-Secure-Models/dp/1492075736)
* [GitHub repository](https://github.com/PracticalFairness/BookRepo)
Questions and Answers
---------------------
**Matthew Emerick**
Is 100% fairness even possible? If not, what is an acceptable metric?
**Aileen Nielsen**
100% fairness is not possible by any definition of 100% fairness. You’re not going to have a metric that 100% of people find fair. You’re not going to have a model that will be guaranteed fair in 100% of the cases. This is for a few reasons, but a big one is that there are many plausible and even compelling definitions of fairness. In many realistic scenarios, it’s not possible to satisfy all such cases.
**Aileen Nielsen**
Re: acceptable metric. This will depend on a host of factors, including which kinds of mistakes are most worrying, whether the stakes are high or low, whether there are historical reasons to not just go for formal equality but to even favor equality-enhancing measures, and so on. So for example, I would say that in the case of a life-saving medical application (high stakes), the right fairness metric might not be the same as the case of legal price discrimination (that is where retailers might price goods differently based on their belief that some customers are willing to pay more for the same good).
**Matthew Emerick**
Is 100% secure data possible (on the technical side)?
**Aileen Nielsen**
This is also a no. I am not a data security expert, but based on what I do know, there are lots of “known unknowns” (such as the extent of joinable data already out in the world, or the extent of vulnerability of a given ML model to certain adversarial attacks). There are also presumably unknown unknowns. So yeah, no computer security expert is going to promise 100% data security. This is especially true because humans ultimately have some form of access at some point. For example, even in end-to-end encrypted applications, there are easy data leaks, such as a friend screenshotting and sharing pictures of conversations.
**Matthew Emerick**
Is there any trade off between fairness and security?
**Aileen Nielsen**
This is an interesting one. I am going to assume that when people ask about fairness, they are largely referring to the specific fairness concerns of equality and anti-discrimination. In such a case, empirically trade offs have been observed. For example, I have seen papers (not difficult to find if you look) where training models to be more robust to adversarial attacks tends to enhance unfairness as indicated by a variety of fairness metrics (such as differential error rates between groups).
**Aileen Nielsen**
In general, with most technical systems, it’s safe to assume there are trade-offs almost always (unless proven otherwise). _However_ these trade-offs can sometimes be minuscule and not of true concern even if they do exist.
**Dinara Kanarina**
Hello Aileen Nielsen!
Thank you for your time! The topic your book covers is definetely in demand.
May I ask some questions that are not only technical:
* What is your view on changing the existing language in order to shift systematical thinking towards more acceptable and fair? (For example, in some languages that have gender distinctiveness, people try to add new endings specific ending to female professionals etc.)
* Do we always try to identify bias and create models by excluding it or sometimes it’s important to keep it? When then?
**Aileen Nielsen**
In answer to your first question, I am all for making language more inclusive (and precise). So I find the idea of modernizing language use, grammar, etc, to be more inclusive to be a great idea. I also think it’s good to call out language that has long been accepted but that actually perpetuates inequality. For example, it’s still common for people to say things like “so easy your grandma could do it” or “how would you explain this to your grandma?”. “Grandma” (but not grandpa) tends to be the stand-in for a presumed doddering old person. This is both sexist and ageist, so I try to call this out.
**Aileen Nielsen**
On the other hand, I’m not sure it’s always productive to be the “thought police” and if people don’t feel that they can speak freely without getting attacked, that can lead to its own problems. So yes, in general I am all for more inclusive language, but I am also for a welcoming environment for all and taking care not to become the “thought police”
**Aileen Nielsen**
But these are my own personal opinions and not related to the topic of coding for fairness so don’t take these as anything more than my own thoughts 🙂
**Aileen Nielsen**
Regarding your second question, whether bias should always be identified and removed, in theory I would say yes. A perfect world would be one where we can get rid of all bias. However, in practice, I think for the moment it’s best to focus on high stakes applications where the real costs to humans of bias are quite high. So I am far more worried about bias potentially baked in to ML models built for policing or healthcare or education than I am about ML models built for retail or gaming. (Also see my earlier question that what constitutes fairness or bias has many definitions, so that it won’t always be possible for everyone to agree on whether bias has been removed).
**shaolang**
hi, Aileen Nielsen
As biasness are often detected in production (and usually with material impact):
* Are there ways to catch them before that?
* Would the model(s) need to be explainable?
Also, are models that use online learning susceptible to biasness and even insecurity over time?
One last question: is fairness and security equally applicable to unsupervised learning? If so, how?
Thanks!
**Aileen Nielsen**
Re: q1: In my opinion, bias is detected in production models for two reasons. (1) No work has been done to debias/audit that particular model (surprisingly common) and (2) Some work has been done, but the people identifying the bias may have a different definition of bias or a different threshold for establishing bias.
But yes there are absolutely ways to audit a model for bias before it is rolled out into production. There are in fact many ways to do so (and more than one way can be used). For starters, you can preprocess the data to remove bias you identify in the data, you can in process by adding fairness awareness to your training process, or you can post process to modify model outputs to make them more conformant to a fairness definition. You don’t have to do just one of these things….you can do them all.
**Aileen Nielsen**
Re: question of explainability. Do models need to be explainable? This is a hot topic, and I certainly think that explanation can be appropriate and even necessary in many domains. On the other hand, I sometimes think explainability is overemphasized in some discussions. For example, we should also recognize that in human decision making there is a lot that goes unexplained (or where explanations can be pretextual rather than true). Therefore I am a big believer in explainability generally but I do not see this as necessary or even desirable in all cases. In fact, sometimes explanations can be more confusing than helpful.
**Aileen Nielsen**
Regarding your questions about online learning and unsupervised learning. Yes, there are absolutely security and fairness concerns here as well. These elements of ML are less used in industry, so that’s one reason I focus on them less. Also there way to make such practices more fair and robust are also even less clear than in the case of supervised ML. However, a good example of a real world scenario where such a model existed and rapidly went bad is of course Microsoft’s Tay chatbot: [https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist](https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist)
**shaolang**
Woah, thanks for such thoughtful responses! Sorry for such late reply, ‘cos I had some urgent family matters to attend to.
**Prateek Joshi**
Aileen Nielsen: Thank you for being here. Fairness of AI systems is an important topic. What metrics can be used to measure the fairness level of an AI system? What needs to be standardized between two AI systems before we can compare their fairness levels (so that it’s apples-to-apples comparison)? And what levers are available to influence the fairness level of an AI system?
**Aileen Nielsen**
Good questions. Regarding metrics, there are SO many fairness metrics. A good starting point to get a sense of some commonly used metrics but also to understand some of the nuances and social influences is Arvind Narayanan’s tutorial on this topic: [https://fairmlbook.org/tutorial2.html](https://fairmlbook.org/tutorial2.html)
(_21 fairness definitions and their politics)_
**Aileen Nielsen**
Also I think your second question about apples to apples comparisons is super interesting. I would say that models trained for different purposes are not comparable. For example, is there any sense in which I can say that a particular medical AI application is fairer than some ML credit scoring algorithm? I think not. For example, the consequences of making a mistake are quite different (e.g. death from wrong treatment versus not getting a loan won’t kill you). Also the historical considerations may be different (for example, there is evidence that minorities continue to be discriminated against in the case of credit….but I think this is less problematic for women. On the other hand in the case of medicine there is substantial evidence that minorities but also women face substantial discrimination in the quality of medical care they receive).
**Aileen Nielsen**
So I think it’s an interesting question but I’m not sure any simple apples to apples comparison is achievable.
**Prateek Joshi**
Aileen Nielsen Thank you for the answers.
**Aileen Nielsen**
Hi everyone! Thanks a lot for your questions. I’ll be answering in individual threads to the original poster of each question.
**Tim Becker**
Hi Aileen Nielsen, thanks for your time.
* In your opinion, what are the biggest security risks a data scientist should be aware of?
* Is there a general procedure on how to test if my models are discriminating?
**Aileen Nielsen**
Re q1: I think a lack of precautions regarding protecting identifying information is the biggest problem I have seen in my experiences in a variety of organizations. This presents itself in a variety of ways. Sometimes identifying information is gratuitously made available when people are modeling even though it is in no way necessary. Relatedly, information is not segmented to minimize what is available to any given user (privacy by design). Finally, most data scientists I have worked with do not seem to be aware or take seriously the danger of reidentification even when identifying information has been putatively removed.
**Aileen Nielsen**
Re q2: There are many general procedures but there is no one single procedure. Two good resources to get you started are two toolkits. (1) [http://aequitas.dssg.io](http://aequitas.dssg.io/)
and (2) [https://aif360.mybluemix.net](https://aif360.mybluemix.net/)
**Aileen Nielsen**
I discuss both of these toolkits in my book, but you can also easily learn them through the many tutorial examples they give.
**Tim Becker**
thank you :)
**Bengsoon**
Hi Aileen Nielsen thanks so much for writing this book and for your time here. I’m quite new to AI fairness and I still find it hard to wrap my head around its framework / toolkits.
In particular, I notice the topic of AI fairness is mostly wrapped around data/models that involve users/groups of people etc. Personally, I work mostly around data from machinery sensors and engineering data in the energy sector, a lot of which do not have direct implications to any particular groups of people. Will AI fairness still be applicable to my data and models, or am I amiss in even asking such question? In general, how do I know if AI fairness applicable to my work / domain?
**Aileen Nielsen**
Absolutely! sensor data is actually incredibly sensitive from a privacy perspective, and I’m a firm believer that privacy is a core pillar of algorithmic fairness
**Aileen Nielsen**
Consider for example this law review article that describes why energy usage is in fact quite sensitive and should be protected: [https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=3667125](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3667125)
**Aileen Nielsen**
Also to the extent that you view fairness through an equality lens, I can imagine many ways in which sensor data in the energy sector may have unequal impacts. For example are communities of color disproportionately subjected to highly granular data collection of energy?
**Bengsoon**
awesome thank you so much for your input. A lot to chew on for sure, but nevertheless a very essential and much needed topic for a time such as this!
**Bengsoon**
Also apologize for creating another thread, but I have a different question: for all the AI fairness metrics that are available out there, what is the ground basis for fairness? How does one define “fairness”?
**Aileen Nielsen**
There is no single definition of “fairness”. In fact, this is something you’ll find varies between cultures and between individuals
**Aileen Nielsen**
Given my legal training, I tend to focus on the most fundamental definitions of fairness, which tend to relate to antidiscrimination (equality) and to fundamental rights (fair process). In these cases, there are decades of legal guidance, and so I tend to think a good start is to - at the least - make sure that algorithmic fairness in domains regulated by law is at least meeting the same standards that have been in place for decades
**Aileen Nielsen**
but more generally the question of what is fair is not a question anyone can answer definitively
**Aileen Nielsen**
therefore for productive ML work I think the best is to consider the fairness considerations of a particular domain and then to think about what fairness metric makes sense for that domain
**Bengsoon**
okay that’s pretty clear - sounds like one needs to evaluate the context of the region and cultures etc.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# AI-Powered Business Intelligence – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
AI-Powered Business Intelligence
--------------------------------
#### by [Tobias Zwingmann](https://datatalks.club/people/tobiaszwingmann.html)
##### The book of the week from 06 Jun 2022 to 10 Jun 2022
Use business intelligence to power corporate growth, increase efficiency, and improve corporate decision-making. With this practical book, you’ll explore the most relevant AI use cases for BI, including improved forecasting, automated classification, and AI-powered recommendations. And you’ll learn how to draw insights from unstructured data sources like text, image, and voice audio files.
Author Tobias Zwingmann, senior data scientist and cofounder of Germany-based AI startup RAPYD.AI, helps BI, business, and data analysts understand high-impact areas of predictive and prescriptive analytics. You’ll learn how to leverage popular AI-as-a-service and AutoML platforms to ship enterprise-grade proof of concepts without the help of software engineers or data scientists.
* Learn how AI can generate business impact in BI environments
* Use AutoML for automated classification and improved forecasting
* Implement recommendation services to support decision-making
* Draw insights from text data at scale with NLP services
* Extract information from documents and images with computer vision services
* Make voice audio files accessible for reporting with AI transcription services
* Build interactive user frontends for AI-powered dashboard prototypes
* Implement an end-to-end case study for building an AI-powered customer analytics dashboard
* [Book's page](https://www.oreilly.com/library/view/ai-powered-business-intelligence/9781098111465/)
* [Buy on Amazon](https://www.amazon.com/AI-Powered-Business-Intelligence-Improving-Forecasts/dp/1098111478)
Questions and Answers
---------------------
**Tobias Zwingmann**
Hello everyone! Happy to join this discussion and excited for your questions!
**Ruddi Rodriguez**
Hello my name is Ruddi Rodriguez . I would like to ask you about security issues in platform like H2O for autoML , and in general all cloud service like plotly for Dashoboards , or for example prefect for workflow control . How do you think that could be the best wait to introduce these tools in a company that use customer data ? Thanks
**Tobias Zwingmann**
Hey Ruddi Rodriguez ! Thanks for your question. Enterprise ML platforms like AzureAuto ML, Sagemaker, Vertex AI, Datarobot, H2O etc. generally have a high security level compared to most of your own custom solutions. GDPR is still a roadblock for many companies in the the EU. I like to use these platforms for rapid prototyping because 99% of the more traditional businesses don’t have the resources to build and maintain their own custom ML stack. They need to focus on use cases and execution instead.
**Ruddi Rodriguez**
Hi Tobias , thank you for the answer , I have in mind that perhaps scaling and encoder , numerical and categorical data locally before to use a service in the cloud can be an option . Then the data that si exchange with the cloud does not contains data that can be tracked back.
**Tobias Zwingmann**
Good approach to anonymize data first before you send it to the cloud!
**Ognen**
Is the purpose of the book to teach non-data scientists how to use black box solutions by pointing and clicking?
**Tobias Zwingmann**
Haha Ognen! No, it’s for people who are well versed with business intelligence (including the underlying data) and who want to improve their reports and dashboards with the help of some data science techniques. Nocode tools help them to use these toys and build fist prototypes without the need of looping in a data scientist or turning into a data scientist themselves.
**JaimeRV**
Hi Tobias, one question regarding organizations that are not data driven and do not necessarily see how AI-Powered BI could help them. How could they decide if it makes sense for them to spend time/resources in learning about AI-Powered BI?
I am thinking about bigger organizations who could intuitively see some value on some dashboards but might be skeptic to invest AI/ML
**Tobias Zwingmann**
Hi JaimeRV - great question! Prototyping is the way to go. In my experience, building a quick PoC in a small team (3-5 people) helps a lot as it demonstrates value even to non-technical management and get buy-in. It definitely helps if the management follows some top-level data or digitalization strategy as you can potentially link your prototypes/pocs to that which makes everything much easier.
**JaimeRV**
Thanks for the answer Tobias Zwingmann!
**Ruddi Rodriguez**
Hi Tobias I have another question actually several , why Azure and not AWS for the book? a second one: with tools like H2o where you can train for free your model and from my point of view with a simpler interface , do you think that the future will go in the direction of platforms like H2o?
**Tobias Zwingmann**
Hi Ruddi Rodriguez, I chose Azure because I find it easier to use than AWS. Also, many companies build on the Microsoft stack for their BI (eg PowerBI) so I found Azure more natural for them. H2O is also a great platform. But even here, once you move to their enterprise MLOps platform you have to pay. Most non-tech companies are much better off taking an MLOps platform from the shelf and using it instead of trying to build with open source tools and reinvent the wheel imho.
**Ruddi Rodriguez**
Thanks for the answer, I see Power BI gives to Azure an advantage in many cases. So, if I understood well if I use AWS it lacks a tool integrated like power BI, sorry for the naive question I have zero experience using these tools I do everything locally mainly because of security restrictions. We are not even allowed to use power BI.
**Ruddi Rodriguez**
well “we” , I am the only one
**Tobias Zwingmann**
AWS doesn’t have a native BI tool as far as I know. But it’s very easy to build a model with eg Sagemaker, deploy it as an HTTP endpoint and query this endpoint from within PowerBI / PowerQuery. That’s why I have not focused on the “native” integration between Azure and PowerBI in the book, but explained how to bring those tools together with a little scripting in Python or R.
**Tim Becker**
Hi Tobias Zwingmann, thanks for being here and for answering our questions!
* Could you give us some example of where AI-powered analytics had a great business impact?
* What is an AI-powered analytics dashboard?
**Tobias Zwingmann**
Hi Tim Becker - glad to be here and thanks for your questions!
1- I had an aha moment once when I realized how dashboard users where using natural language queries very intuitively to get the metrics they wanted from a dashboard. It works really well. The technology has become mature enough and user have been trained to this kind of data retrieval over years because it works similar like a google search. I think key is when you really start with the basics and thus empower the masses. I have seen computer vision applications for turning unstructured data into structured data also deliver great results (eg document analysis)
**Tobias Zwingmann**
2- An AI-powered analytics dashboard for me is when you use AI to improve your analytics/reporting experience throughout multiple stages (descriptive, diagnostic, predictive and prescriptive analytics)
**Tim Becker**
thank you 🙂
**Tim Becker**
Concerning question 1, do I understand correctly that the user would for example type “monthly sales” or “daily imports November” into a field and then get figures with corresponding data?
**Tobias Zwingmann**
exactly! Even more complex queries work pretty well meanwhile thanks to the advancements in NLP. eg “sales in the US over time” or “daily active users january vs february as bar chart”
**Tobias Zwingmann**
you can combine that with a custom business glossary so the tool speaks the language of your analysts which really makes it a powerful technology
**Ruddi Rodriguez**
The first example indeed is very powerful. But I think not many companies use are not using such tools in their daily tasks. Many companies are tight to Excel or even worst to delivering reports. Do you have any thoughts on why too many companies, including big banks, are still far behind in terms of AI? And why are they still struggling to change from power BI to tableau to excel and back without a good strategy to develop or adopt solutions as you mentioned?
**Tobias Zwingmann**
Technology adoption always happens in phases.
Why did people still use typewriters when the could use computers?
Why did people still send letters when they could send an email?
Why do people still use email if they could use slack?
Sooner or later, new technologies will replace old, but it’s hard to predict which and when. We humans are crazy creatures 😃
**Tobias Zwingmann**
But some general principles for enterprise AI adoption:
* must be triggered from top to bottom. No leadership buy-in, means no AI strategy
* you have to come up with a use case roadmap and tie that into a vision
* you need to sell that vision to the employees
* you have to acknowledge that culture eats strategy for breakfast
**Ruddi Rodriguez**
👍
**Allan**
Hi Tobias Zwingmann , would this book be appropriate for who has a background in data science and ML, but not necessarily BI?
**Tobias Zwingmann**
Hi Allan, they won’t probably find it as valuable as the other way around. But if you’re looking for use case inspiration to apply your data science skills to BI (and especially PowerBI) it might still be useful for you.
**Allan**
Thanks Tobias, appreciate your taking the time to answer questions here!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Python Machine Learning By Example – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Python Machine Learning By Example
----------------------------------
#### by [Hayden Liu](https://datatalks.club/people/haydenliu.html)
##### The book of the week from 20 Jun 2022 to 24 Jun 2022
Python Machine Learning By Example, Third Edition serves as a comprehensive gateway into the world of machine learning (ML).
With six new chapters, on topics including movie recommendation engine development with Naïve Bayes, recognizing faces with support vector machine, predicting stock prices with artificial neural networks, categorizing images of clothing with convolutional neural networks, predicting with sequences using recurring neural networks, and leveraging reinforcement learning for making decisions, the book has been considerably updated for the latest enterprise requirements.
At the same time, this book provides actionable insights on the key fundamentals of ML with Python programming. Hayden applies his expertise to demonstrate implementations of algorithms in Python, both from scratch and with libraries.
Each chapter walks through an industry-adopted application. With the help of realistic examples, you will gain an understanding of the mechanics of ML techniques in areas such as exploratory data analysis, feature engineering, classification, regression, clustering, and NLP.
By the end of this ML Python book, you will have gained a broad picture of the ML ecosystem and will be well-versed in the best practices of applying ML techniques to solve problems.
* [Book's page](https://www.packtpub.com/product/python-machine-learning-by-example-third-edition/9781800209718)
* [Buy on Amazon](https://www.amazon.com/Python-Machine-Learning-Example-scikit-learn-dp-1800209711/dp/1800209711/ref=dp_ob_title_bk)
* [GitHub repository](https://github.com/PacktPublishing/Building-Machine-Learning-Systems-with-Python-Third-edition)
Questions and Answers
---------------------
**Ramsi Kalia**
Hayden Liu If you’re still new to ML and DL (like myself), Is it better to know TensorFlow and Pytorch or to focus on just one? And which one would you say is easier to implement?
**Hayden Liu**
Glad you ask Ramsi. There is no definite answer to this. However, for ML or DL beginners, PyTorch has an easier learning curve as it is more Pythonic and debug-friendly and more straightforward when it comes to handling data. On the other hand, TensorFlow could indicate a steeper learning curve because of the low-level implementations of the neural network structure. Although TensorFlow has the high-level Keras API, which makes it easy to get started learning the basic concepts, PyTorch has a good trade-off between being easy to use and being more easily customizable than Keras.
**Hayden Liu**
So picking either one is fine, don’t need to focus both
**Ramsi Kalia**
Thanks Hayden Liu! I had started with Pytorch some months back, did a couple of projects and moved onto TensorFlow (my internship projects were all in TensorFlow), I’ve been thinking if I need to go back and improve my skills in Pytorch , glad to hear that juggling both early on is not a requirement!
Thanks again :)
**JaimeRV**
Hayden Liu thanks for presenting your book! When it comes to ML, for me at least, the more practical the book the better. Most of the people I know started with “Hands on Machine Learning …” from A.Geron which is also very practical. Could you compare your book to this one? Maybe some pros/cons or why people should read one or the other? Or both?
**JaimeRV**
Hayden Liu? 🙂
**James Gough**
Hi Hayden Liu. Given the practical focus of your book, and as someone with very little theoretical knowledge of ML, would you recommend complementing this book with a more theoretical intro book? I suppose I’m worried about falling into the trap of knowing how to implement certain ML algorithms and not the ‘why’ behind it all.
I think it’s a tricky equation balancing the need to ‘get your hands dirty’ and move quickly, with also having solid fundamental (mathematical?) knowledge to underpin it. I was interested in hearing your thoughts on this as I’m someone with plenty of Python engineering experience but very new to ML. Thank you :)
**Hayden Liu**
Hi James. Thanks for your question. A lot of readers wonder the same. You can dive into the book with little theoretical knowledge of ml. As each chapter of the book explains the mechanism ML theory and models at the beginning, with the help of examples and codes (instead of pure math), followed by implementations (from scratch and/or with ML packages) and applications. As long as you know Python, and follow the recommended reading list (basic stats and probability) in chapter 1, you are good to go.
**Philip Dießner**
Hello Hayden Liu, thanks for being here. You are presenting industry-adopted examples in your book. Do you also talk about how these models are deployed/served and used by clients?
**Hayden Liu**
Thanks Philip for the question. There are sections on setting up / running model training and inferencing in GPU, data processing, model training/inferencing on Spark.
Are you referring to serving in API (such as Flask, REST)? Due to the page limit, we decided to keep the book focused on ml thoery and implementations, and applications. But I think you could find tons of one-pager resources online, and super easy to follow
**cactusmkt**
Hello Hayden Liu, thanks for coming to answer our questions about your book. Is there an order for the project examples on your book that you recommend for a ML beginner?
**Hayden Liu**
Hey. thanks for asking. It’s a great question btw. You can simply follow the chapters. as they are in the order of difficulty and complexity.
**Allan**
Hi Hayden Liu , thanks for taking the time to answer questions here! Sounds like a great book. Does it also touch on issues related to deploying models in production?
**Hayden Liu**
Thanks Allan for the question. There are sections on setting up / running model training and inferencing in GPU, data processing, model training/inferencing on Spark.
Regarding deploying production, due to the page limit, we decided to keep the book focused on ml thoery and implementations, and applications. And in fact, it is rare that an ML book covers it, as it is really stack specific, environment specific. But I think you could find tons of one-pager resources online, and super easy to follow.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Designing Machine Learning Systems – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Designing Machine Learning Systems
----------------------------------
#### by [Chip Huyen](https://datatalks.club/people/chiphuyen.html)
##### The book of the week from 27 Jun 2022 to 01 Jul 2022
Machine learning systems are both complex and unique. Complex because they consist of many different components and involve many different stakeholders. Unique because they’re data dependent, with data varying wildly from one use case to the next. In this book, you’ll learn a holistic approach to designing ML systems that are reliable, scalable, maintainable, and adaptive to changing environments and business requirements.
Author Chip Huyen, co-founder of Claypot AI, considers each design decision–such as how to process and create training data, which features to use, how often to retrain models, and what to monitor–in the context of how it can help your system as a whole achieve its objectives. The iterative framework in this book uses actual case studies backed by ample references.
This book will help you tackle scenarios such as:
* Engineering data and choosing the right metrics to solve a business problem
* Automating the process for continually developing, evaluating, deploying, and updating models
* Developing a monitoring system to quickly detect and address issues your models might encounter in production
* Architecting an ML platform that serves across use cases
* Developing responsible ML systems
* [Book's page](https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/)
* [Buy on Amazon](https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969)
* [GitHub repository](https://github.com/chiphuyen/machine-learning-systems-design)
Questions and Answers
---------------------
**Michal**
Hello, what do you think about end to end ML solutions? Tools that will help you do more than just one thing (for instance: training, monitoring, and serving). Are we going this way or specialized libraries (that do one thing exceptionally) for each task are the future?
**Michal**
Also, one more thing. Do you cover testing ML models before putting it in production in your book?
Thanks for coming here and answering some questions!
**Chip Huyen**
yes!! half of chapter 6 and half of chapter 9!
**Dr Abdulrahman Baqais**
Hi, Is this book targeting data scientists or machine learning engineers more closely?
**Chip Huyen**
depends on how you define data scientist / ML engineer. they’re defined differently at different orgs
this book discusses ml systems as whole, so i believe it has a lot of relevant points for both DS and MLEs
**Dr Abdulrahman Baqais**
Can the design be applied to deep learning , NLP, recommendation systems?
**Chip Huyen**
yes
**Dr Abdulrahman Baqais**
Does the book explore preML-modeling stages such as Data Engineering? Modern data stack?
**Chip Huyen**
yes, chapter 3 discuss data engineering
**Dr Abdulrahman Baqais**
Does it explore ML systems in the cloud? For embedded deviced? or at the edge?
**Chip Huyen**
yes, that’s what chapter 7 is about
**Shaksham Kapoor**
First of all, I would like to thank you Chip Huyen for all that you have done (and still do) for the ML community 🙂 🙏 I learn something new almost every time I read your blog.
My question to you is - _what do you think are the bare essentials that a data scientist need to know?_
\> _Context_:
\> I have seen poorly defined JDs where the expectations from a data scientist is to literally know anything and everything between ML and DevOps. The issue that I see with such roles is that these are separate fields, and getting a mastery over either of them is challenging (let alone both) given the rapid pace in which the fields evolve. I am a core data science person and I am good at it; however, I know that there are still a lot of things in data science (alone) that I need to learn to become a complete data scientist.
\>
\> Having said that, I also want to gain experience in other components of ML lifecycle, but where do you draw the line when you say “this much” of DevOps is sufficient for a data scientist to know.
**Chip Huyen**
thanks Shaksham Kapoor for your kind words!
**Chip Huyen**
agree that JDs can be ambiguous. at the same time, different orgs have different challenges. 2 data scientists at 2 different companies, even same company on different teams, can do very different things.
i’d work at it backwards. figure out what teams you want to join, read their tech blogs / look at their JDs / talk to people on their team and figure out what problems they care about
**Shaksham Kapoor**
I agree. I have observed the following in my experience:-
* Startups (any stage) - they require an all-rounder, who knows anything and everything.
* Mid-size firms (>1000 to <=10000) - there is a split. Some require an all-rounder, while others don’t.
* Large-size firms (>=10000) - they don’t need an all-rounder but an expert. I believe the main reason for this is the presence of separate teams (sometimes departments) that takes care of different components of ML, so one can get a chance to really focus on core ML stuff.
**Dan Gurin**
What do you perceive as the biggest challenges / opportunities on the horizon for the future of machine learning?
**Chip Huyen**
real-time ML!
**Jeanine Harb**
Hello Chip Huyen! Thank you for this book, can’t wait to read it!
From what I’ve seen in the field, a lot of ML projects have a hard time getting into production. In your opinion, what are the biggest hurdles and how can companies/organizations overcome them?
**Chip Huyen**
nice to meet you Jeanine! i think the biggest hurdle is that companies don’t invest enough into infra to enable data scientists to do their jobs. ml production is largely an infra problem.
hiring infra engineers doesn’t solve it though. infra engineers will need to work closely with data scientists to understand their workflows. not all companies have their communication channels set up to enable cross-functional team communication.
**Sergio Rozada**
Hi Chip Huyen! It’ll be a pleasure to read your book, seems really insightful!! I would love to hear your opinion about batch vs online inference. Thank you very much!
**Chip Huyen**
i have 10 pages in my book discussing batch vs. online prediction alone so not sure how to respond to this in a message 😅
**Contato Rupp**
Hi Chip Huyen! In your opinion, which aspect of ML system designs is particularly hard to iterate on? and when should teams think about a complete redesign of their systems?
**Chip Huyen**
devops track metrics such as:
* change failure rate: how often does new update fail?
* time to detection: how long does it take to detect a problem
* time to response: after a problem is detected, how long does it take to address it
we should track similar metrics in mlops. how often does new models fail? how long does it take to discover new model failure? how long does it take to update models / debug the systems?
teams should think about redesign whenever those metrics fail short of their expectations.
and if they can’t track those metrics, they should definitely consider redesign 🙂
**Shaksham Kapoor**
Another question for you Chip Huyen - how do you keep yourself updated with the breath and depth of techniques that are available/upcoming in data science?
**Chip Huyen**
by working with people smarter than me!
**adanai**
Hi Chip, thank you for the work you do and the QnA! I hope to earn an in-person interaction opportunity with you someday!
Q: What are some strategies for data collection to approach problems. What is the role of data ethics in the process?
**Chip Huyen**
great question. data ethics is a nuanced topic that i don’t think i can discuss in one message. chapter 4 covers data collection [https://github.com/chiphuyen/dmls-book/blob/main/ToC.pdf](https://github.com/chiphuyen/dmls-book/blob/main/ToC.pdf)
**adanai**
Q: What are few critical points to keep in mind for scalability of the system? Should scalability be planned for in the initial design of the ML system or can it be planned in future iterations when the need arises?
**Chip Huyen**
depends on when you anticipate the scaling issues to arise: a week from now or a year from now?
i’m a big fan of scalability, but i’m not a fan of premature optimization [https://wiki.c2.com/?PrematureOptimization](https://wiki.c2.com/?PrematureOptimization)
**adanai**
Thank you for the insight! I agree; it is more important to get things running than implement the system to perfection.
**adanai**
| | | | |
| --- | --- | --- | --- |
| Q: For someone trying to dive deeper into ML, how should they pick a specific branch to attain mastery. I am interested in a number of branches in ML (NLP | CV | Time series | …), but unable to choose one specifically. Do you suggest a framework for the same? |
**Chip Huyen**
you know the venn diagram that says to choose the intersection of:
1. what you’re interested in
2. what the world needs
3. what you’re good at
it sounds like all branches of ML are pretty needed right now (though time series and tabular data might be a little bit underrated compared to CV & NLP)
i’d choose based on what problems interest me and what i’m good at
**adanai**
I think it’s called `Ikigai` . Will keep this framework in mind going further on
**Chip Huyen**
good luck!
**Ashish Lalchandani**
Hello Chip Huyen! Thanks for being here! My question is :
1. As someone who is looking to switch from data engineering to MLOps, what are the skills required for it and what is the best way to prepare for interviews? By focusing more on projects?
2. How to determine the right amount of MLOps for a project?
**Chip Huyen**
1. yes, real-world projects. but it can be difficult if you don’t already work at any company already.
2. the right amount is whatever is needed to achieve the project’s goals!
i also wrote a free book on interviews – hope that helps! [https://github.com/chiphuyen/ml-interviews-book](https://github.com/chiphuyen/ml-interviews-book)
**Ashish Lalchandani**
Oohh great, you have written on interviews too, awesome! Thanks for contributing so much to the community, much appreciated!
**Sergio Rozada**
Hello Chip Huyen, another question here, can you share any insight into how to make scalable systems with large models (e.g. transformers)? Thank you very much!!
**Chip Huyen**
hi Sergio, thanks for your question! i’d need more context on:
* how your system fails at scale (e.g. higher latency, higher cost)
* where does your system fail when scaling (e.g. if it’s high latency at inference, is it due to network latency or model latency), etc….
**Sergio Rozada**
Hi Chip, thanks your answer, for the sake of clarity:
* You just hit the point! In our case, we struggle a lot to improve in terms of latency keeping the costs controlled. We’re running on a Kubernetes cluster in GCP (managed by us).
* Our main latencies are model latencies.
**Utku Savaş**
Hi Chip Huyen, how should we perform monitoring on computer vision related projects? Thank you.
**Chip Huyen**
this would be a looooong answer 😭 maybe we can start with: what problems do you have with monitoring CV projects rn?
**Utku Savaş**
Hi, I can check model results on UI but I want to monitor whether data drifting(or similar problems) occurs or not on our projects. Which programs or approachs should I use to notice data drifting before it happens on image data?
**Tim Becker**
Hello Chip Huyen, thanks for being here. I would like to know:
* What are frequent mistakes when designing ML systems and how to avoid them?
* What kind of considerations should I take in mind when setting up automated re-training? How often should it be done? Only if the quality of the model decreases?
* How do you document your ML projects?
**Chip Huyen**
hi Tim, thanks for your questions!
i wish there were shorter answers to those questions, but they are complicated 😭 i have a pretty long section in my book on the myths of ml deployment and half of a chapter on question 2 (which includes a section discussing how often to update models). here’s the summary
Eugene Yan has a lot more thoughts on documenting ML projects 👀
**Warrie Warrie**
Hello Chip Huyen. Thanks for investing time here.
1. As an academic researcher and application engineer, what areas in ML system design have you observed to be more disparity between the 2 industries?
2. How critical is software engineering skill in designing an ML system?
**Chip Huyen**
hii Warrie, i gave a talk on 1 here [https://www.youtube.com/watch?v=c\_AUuTuPA5k&ab\_channel=StanfordMLSysSeminars](https://www.youtube.com/watch?v=c_AUuTuPA5k&ab_channel=StanfordMLSysSeminars)
engineering is critical for designing ml systems
**Javier**
🤯
**Max Payne**
Hi,
How different is the design process for traditional ML vs DL?
**Chip Huyen**
i think the difference is less between ML vs. DL but between different projects, goals, and constraints
**adanai**
\> _Q:_ lets say you are building a model with X technique which is in production but now you see that Y technique which is state of the art outperforms better than X during development do you remove model X and replace with model Y?
Hello Chip, this question was earlier asked by a fellow member in another channel of the DTC workspace. I am interested to learn of your opinion
Source (with suggestions from other members): [https://datatalks-club.slack.com/archives/C01BQDWEAHW/p1654315552971539](https://datatalks-club.slack.com/archives/C01BQDWEAHW/p1654315552971539)
cc: Doink
**Chip Huyen**
i’d agree with the first answer there: run experiments (both online and offline) to compare the 2 models on the metrics you care about (not just overall performance metrics but can also be other metrics like latency, inference cost, interpretability, how easy it is to update each model, how likely each model’s performance will improve over time with more data)
chapter 6 as a pretty long section on the framework for comparing 2 different models. here are 6 key points:
1. _Avoid the state-of-the-art trap_
2. _Start with the simplest models_
3. _Avoid human biases in selecting models_
4. _Evaluate good performance now versus good performance later_
5. _Evaluate trade-offs_
6. _Understand your model’s assumptions_
**Ramsi Kalia**
Chip Huyen Hi Chip, thanks for answering questions here!,
I am wondering how re-usable ML systems are in your opinion? With every new problem statement, the dataset, features, splits, training etc. changes right? I am trying to understand how much value can be generated from building entire systems around ML projects?
I understand that for large industries like Uber etc. if the models are being used daily and retrained very often it would make a lot of sense.
But for smaller businesses, where models are retrained infrequently and there is a larger focus on trying out different techniques and newer models, how should we justify the extra time and effort spent on building systems? And what would the benefit be?
Appreciate any insight you can offer!
Thanks again for your time!
**Chip Huyen**
i think you answered your own question!
100% agree with you that how much to invest into a reusable ML platform depends on how much you need it (not just today but also in 6 months).
it sounds like you’ve spent a lot of thinking about it. curious what your approach for this i?
**Ramsi Kalia**
Hi Chip Huyen, I don’t really have a plan of action atm,
I’m working at a startup in the automotive space (we just won the NASSCOM Gamechangers Award in the transport & logistics category , company is Carscan) but we don’t really have full fledged systems for retraining.
There is active learning in place for the primary damage detection model, but even for that we’ve been trying out different methods to improve performance (e.g. SAHI, Autoassign, PAANET etc.)
For edge deployment models, up till now we just trained a bunch of classifiers (cos they’re faster than object detection) and moved them to s3 and work from there.
We’ve had numerous discussions on how to streamline the edge deployment of models for the webapp and we just seem to be going in circles lol.
The last sprint was focussed in edge deployment and we tried out Hydranet, multihead models, mobilenet ssd, multilabel classification to name a few.
With regards to code reusability and experiment/model tracking, I am giving a training session to my team on DVC this coming Monday, (still figuring it out for myself),
however, for designing systems, I think we’re still lost.
An argument could be made in favor if the benefits of taking such a thing on were greater than what I seem to be understanding atm.,
Have you worked with startups before?
Do you have any case studies you’d be open to sharing?
**Sandhya G**
Chip Huyen,this is an exciting book. How do we determine which problems are worth exploring if ML is a viable solution? Sometimes, ML might add more cost/ complexity than using an expert to solve the problem. For example, experts scan ground scan data to determine where to drill for oil. Using ML here maybe difficult as 1. we do not have a lot of training data 2. Experts maybe using their intuition which may be hard to codify. What is a framework for answering if ML is a viable route with decent chance of success given the cost to develop it. Also, any framework to have ML assisted workflows? Thanks!
**Chip Huyen**
whoa that does sound like an interesting challenge. how are you doing about approaching this?
**Sandhya G**
In my workplace, this is mostly based on instinct (experience). Do a pilot, about 6 months, see where we get. We also go for a lot of non ML components for the solution (data management and access, for example) so that customers get guaranteed value.
However I’ve seen this in my learning too. I’d start off with something in mind, but fail to produce a good model. Since this is for learning, I do not know if it is because I am not doing it right or if the problem is not amenable.
**Dustin Coates**
Chip Huyen thank you for doing this. I’m coming at this from a PM perspective, and one question we have is: how do you show incremental added value in early days of ML projects?
**Chip Huyen**
oooh this would be a fun discussion. happy to set up a call to discuss more!
**Bharat**
Chip Huyen thanks for doing this! My initial 2 questions:
* What’s the latest industry trend with respect to ML system design? I have heard (and read blogs) that there is increasing online learning & retraining, is this true?
* What would you recommend as the first set of skill(s) to pick up for a Software Engineer transitioning to an ML Engineer apart from just the ML Theory basics?
**Chip Huyen**
hi Bharat thanks for stopping by!
1. i’m very big on online prediction and continual learning, so perhaps i’m biased, but i’ve talked to a LOT of companies interested in online learning & retraining!
2. hmm this is a hard question as the answer depends on who’s asking. for me, i love databases and devops!
**Gur Hevroni**
Hi Chip Huyen! Great to have you here 🙂 Your book sounds super interesting!
Do you cover in the book the things you should consider before you start to building/designing anything? E.g. how to approach a problem, how to validate your understanding of the situation and use cases, etc.
Looking forward to learn more about your book 📖
**Chip Huyen**
Hi Gur, nice to meet you and thanks for your kind word!!
Yep that’s what the first 2 chapters are about!
**James Gough**
Hi Chip Huyen I have what I think is an easy or at least mundane question for you (or anyone else if they know). 🙂
I want to buy your book but do you know how compatible your book is with Kindle? I know O’Reilly’s subscription service doesn’t provide Kindle-compatible ebooks so wondered if there’s been much QA for the Kindle version. I’ve bought some programming books on Kindle before and the formatting isn’t always the best. Thank you.
**Chip Huyen**
ooh i’d love to know that too. if anyone does know please lmk. i’ve only looked at the book sample on kindle and it seems fine!!!?
**Ashish Lalchandani**
Hi Chip Huyen, another question - What are the best practices for ML in production, and as someone who is looking to switch from data engineering to MLOps, what are the bad practices to avoid?
**Chip Huyen**
Hi Ashish, nice to meet you!
Sooo best / bad practices are going to be a loong conversation – I think it might take a book 😛
Short answer: i wish we have better engineering best practices in MLOps!
**Ashish Lalchandani**
Thanks Chip Huyen..waiting for you future books then🤪 hehe
**Chip Huyen**
haha it’s part of this book 😛
**Ashish Lalchandani**
Oh same book, nice! Will have to wait till i get my copy then😅 🙌
**Daniel**
Hi Chip Huyen, thanks for taking the time and providing the offer to answer questions. I’d like to know
* What’s the approach of the book: rather theoretical explanations to get an overview over the topic or a practical one with hands-on code/ best practice-examples to go through by yourself?
* Which chapter was the most difficult to write and why?
* Just out of curiosity: team PyTorch or Keras/Tensorflow?
Thanks!
**Chip Huyen**
Hi Daniel!
* I think it’s practical with examples but very little code.
* They’re all difficult in different ways, but Chapter 3 (data systems), 8 (distribution shift / monitoring), 10 (ml platform) took the longest time!
* PyTorch & JAX!
**JC**
Hi Chip Huyen, thank you for taking the time to share your opinions. I’d like to ask about the best practices on improving the ML/DL reference time. And what can we do/consider ahead of time in order to improve the inference time when designing the ML/DL system? Thank you a lot!!
**Chip Huyen**
Hi JC, quantization seems to be the most common method today!
**Gagan M**
Can we also add Knowledge Distillation if we are dealing with DL?
**Chip Huyen**
yes! in the Model Compression section in the book i discussed the pros and cons of distillation vs. other compression methods like quantization
**Philip Dießner**
Hello Chip Huyen, as is evident by the multitude of questions (Thanks for answering all of them!) your book is definitely hitting a nerve with the community.
Can you talk a bit about how your writing process worked?
What was your motivation in finding the breadth and width of the content?
And as all is now done, would you do it again (e.g. writing another book)?
**Chip Huyen**
hi Philip, nice to meet you!
the writing process was an iterative one over 4 years with a ton of feedback and restructuring!
i’d love to write another book one day, but not sure the topic yet. would love to hear if you have any topics you’d be interested in learning more about!
**Doink**
Decentralized Machine Learning?
**Allan**
Hi Chip Huyen! Thanks so much for taking the time to answer questions here.
This book sounds super interesting and relevant! I know some of your blog postings have been about real-time ML. To what degree is this covered in the book? (Seems like Ch9 deals with it, at least in part).
**Chip Huyen**
Hi Allan, you’re right that i’m very excited about real-time ML 😄 in the book, real-time ML appears in multiple parts:
* data engineering (batch vs. streaming)
* deployment / prediction service (batch vs. online prediction)
* real-time observability
* continual learning
**Allan**
Thanks Chip, yes it seems like real-time ML is really something wanted/needed in industry today. Looking forward to learning more about it! 😀
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Grokking Streaming Systems – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Grokking Streaming Systems
--------------------------
#### by [Josh Fischer](https://datatalks.club/people/joshfischer.html)
, [Ning Wang](https://datatalks.club/people/ningwang.html)
##### The book of the week from 04 Jul 2022 to 08 Jul 2022
A friendly, framework-agnostic tutorial that will help you grok how streaming systems work—and how to build your own!
In Grokking Streaming Systems you will learn how to:
* Implement and troubleshoot streaming systems
* Design streaming systems for complex functionalities
* Assess parallelization requirements
* Spot networking bottlenecks and resolve back pressure
* Group data for high-performance systems
* Handle delayed events in real-time systems
Grokking Streaming Systems is a simple guide to the complex concepts behind streaming systems. This friendly and framework-agnostic tutorial teaches you how to handle real-time events, and even design and build your own streaming job that’s a perfect fit for your needs. Each new idea is carefully explained with diagrams, clear examples, and fun dialogue between perplexed personalities!
* [Book's page](https://www.manning.com/books/grokking-streaming-systems)
* [Buy on Amazon](https://www.amazon.com/Grokking-Streaming-Systems-Real-time-processing/dp/1617297305)
* [GitHub repository](https://github.com/nwangtw/GrokkingStreamingSystems)
Questions and Answers
---------------------
**Ramsi Kalia**
Hi Josh Fischer and Ning Wang, thanks for answering questions here!
If I wanted to build an active learning pipeline (for deep learning models), would streaming systems work better than batch systems?
If I have multiple different modes of input, working on different active learning models, will streaming systems help ?
Thanks for your time!
**Josh Fischer**
each have their trade offs. Do you have any requirements that state your pipeline needs to be realtime?
**Ramsi Kalia**
So, actually I’m very interested in working in Industry 4.0 and trying to think of situations where streaming systems are employed.
And the question I asked was way too vague, but I’m wondering that if I was working in predictive maintenance, or logistics and handling, would streaming systems be better than batch?
I know this question is still too vague but that’s because I’m not really able to grasp at what point you decide that you need stream data instead of ignoring it, besides maybe Google live traffic updates, lol,
Could you suggest some resources for the same?
I would really appreciate it! tia!
**Josh Fischer**
I think you should use a streaming system only after you’ve failed with other methods of processing data due to issues in latency or that _real-time_ or requirement. Stream processing is great for pulling data in and taking action on it as soon as it is created, but then you run design decisions/issues that will affect how accurate or complete you data is. One of these issues that may come up is called “Delivery Semantics”
**Josh Fischer**
How’s that for a non answer? 😄 . I think Ning Wang has good knowledge on this topic.
**Ning Wang**
It is an interesting question. i don’t have a direct answer either. My feeling is that it really depends on the real use case. Functionality wise, Streaming and batch are very similar to each other. The difference is that with streaming, events can be processed one by one or with a window, so the system could be more flexible and has lower latency. However, batch could be more efficient when the data amount is very high. Therefore, one key question you need to answer first is: low latency, or high throughput, which one is critical for you.
Another factor is what Josh mentioned: delivery semantics. Batch can be more friendly if you need accurate result, because failure handling could be easier with batch as the data is normally stored somewhere and easy to replay, instead of flowing through the system.
I was in an ML lab in college but I am not familiar with deep learning at all. For old school pattern recognition systems, my personal feeling is that batch might be more efficient for training side, but streaming system can be helpful to improve latency and flexibility for the prediction side if they are necessary. But I don’t know if it makes sense for deep learning systems.
**Mario Tormo**
I’ve just discovered thanks to the title that “to grok” actually exists and it has a beautiful meaning (according to the Merriam-Webster dictionary “to understand profoundly and intuitively”). How did you find this word?!
**Josh Fischer**
Hi Mario,
Thank you for the questions. Manning Publications has a “Grokking” series that they use for foundational books to answer questions like “why do we even do this” in relation to technologies. While working with Bert Bates, our writing coach, he helped us decided which style of book we wanted to write. We choose the Grokking Series as we started writing it almost 4 years ago and a lot of people were still learning “what is a data stream?” and “why/how would I use one?” We also didn’t want to write a book about a specific streaming framework as they come and go so fast. So, we wrote this book in hopes of a longer shelf life as it will teach others the foundation (and more advanced) concepts to start using any streaming framework.
**Josh Fischer**
Grok came from the book “Stranger in a Strange Land”: [https://www.amazon.com/Stranger-Strange-Land-Robert-Heinlein/dp/0441790348](https://www.amazon.com/Stranger-Strange-Land-Robert-Heinlein/dp/0441790348)
**Josh Fischer**
Here is a little info on Bert Bates, he is a phenomenal teacher and person : [https://g.co/kgs/eQczQG](https://g.co/kgs/eQczQG)
**Mario Tormo**
Heinlein?! That’s so cool! I’ve read a lot from him but never Stranger in a Strange Land.
I didn’t know about the grokking series, but there are a lot of grokking books. I must admit I was only acquainted with the “regional dress customs” books, that have actually also a very beautiful covers.
Bert looks like he knows what he is doing, specially with Java 🙂 Where does he teach?
**Josh Fischer**
He is a self employed contractor. He is the person who coaches authors behind all the Manning and O’Reilly books. He also trains Horses with his wife, Kathy Sierra. Quite the interesting combination of skills, wouldn’t you say?
**Mario Tormo**
Woow! That is a really complete profile. It sounds amazing
**Mario Tormo**
To which kind of reader is the book aimed, with which prerequisites?
**Josh Fischer**
We wrote this book for developer with 2 - 3 years of experience, who is looking for direction on which path they should take in their career.
**Mario Tormo**
What is necessary for not getting lost with the book? What is not necessary but could help to get better profit from the book?
**Ning Wang**
The code for the book (for the basic concepts in the first a few chapters) requires Java 8 to compile/run (on Mac, Linux and Windows). So Java background (beginner level) can be helpful for you to play with the demo and understand the code better. Streaming systems are for data processing, so any data processing related knowledge could be helpful. Our goal is to explain the basic concepts in streaming systems, so I hope any level of developers should be able to follow with interests of realtime data processing.
**Mario Tormo**
Thanks for your answer! 🙂
**Josh Fischer**
I agree with all things Ning said, except we support Java 11 at this time, not 8
**Ning Wang**
Thx for the correction! My memory lags a lot these days.
**Mario Tormo**
The cover is beautiful, specially if you compare it with the technical book cover standards nowadays. Is the rest of the book illustrated like the cover? In case not, what’s the style?
**Josh Fischer**
The cover is one of my favorite parts of the book. We don’t match the style of the cover, but we did create many diagrams to help teach. I’ll send some in the comments.
**Josh Fischer**
**Josh Fischer**
**Josh Fischer**
**Mario Tormo**
Well, these are also very nice diagrams.
**Carise Fernandez**
The book looks so delightful! Looking at the chapter previews, I like that it’s really focused on how streaming systems work, without injecting a particular library/framework/etc. Was that a challenge when you were writing the book?
**Josh Fischer**
You are correct. We (attempted) to do the best we could to explain the fundamental concepts behind each streaming system without referencing any of framework. It was quite the challenge. There were several instances where we had to go digging around in different code bases (heron, flink, storm, etc) to make sure we were explaining the concepts generally enough that it would make sense for most. This was about the most challenging part of the book for me. Ning, my Co-Author, even went above and beyond to build our own simple streaming framework to help us teach while we wrote the book.
**Josh Fischer**
[https://github.com/nwangtw/GrokkingStreamingSystems](https://github.com/nwangtw/GrokkingStreamingSystems)
**Carise Fernandez**
Yes, I saw the repo and was very impressed that no cloud account or special 3rd party frameworks were required. Kudos to you!
**Carise Fernandez**
(I read a different streaming systems book awhile ago and while it was helpful to think of the batch/streaming abstractions in terms of a particular framework, I also felt like I didn’t _really_ understand what was going on)
**Josh Fischer**
The Kudos goes to Ning. He wrote most of this, he is a phenomenal technologist. I can understand the feeling of not knowing what is going on too. These frameworks are extremely complex at times.
**Carise Fernandez**
Thanks again, both to you and Ning for this book. Looking forward to reading it 🙂
**Ning Wang**
Thanks! 😄 To be fair, We borrowed a lot of basic logic from [Apache Heron](https://heron.apache.org/)
(we are both contributors of the project).
**Ning Wang**
And you are totally right. It is kinda important to understand what’s really going on in order to build and maintain data processing systems.
**Carise Fernandez**
Nice, I will check it out 🙂 to be honest, there are a lot of Apache streaming/batch engines/frameworks/etc that makes me wonder if Apache could one day put out a mega cheat sheet for them 😂
**Ning Wang**
Yeah. Different data processing frameworks have different goals and trade-offs. It is interesting to compare them and (maybe) build a cheat sheet. There are also many in-house systems when there are special needs I feel.
**GerryK**
Hi Josh, the book looks super interesting and the diagrams too. Is there a reference on streaming tools or protocols? Like mqtt, spark, kafka?
**Josh Fischer**
In this book we stay completely agnostic to any framework out there today. It is meant to show people why and when they would use something like Kafka, instead of how.
**Josh Fischer**
Our targeted reader is a developer with a few years experience who is looking for the next stack of technology they want to learn. This answers the questions for streaming frameworks without people getting bogged down in framework specific code. Well that was our hope, at least.
**Ning Wang**
Adding one more thing, We hope the existing systems like Kafka and spark make more sense to readers after reading the book.
**Mario Tormo**
It also makes sure the book is going to stay fresh longer than the tools. We all have books about software that doesn’t exist anymore… XD
**GerryK**
Thanks for your answer both! Makes sense!
**Josh Fischer**
Ning Wang
**Ning Wang**
Thanks! Josh Fischer
**Abbas Akkasi**
Hi, could you please introduce me a good,short, and efficient book to learn PySpark ML?
**Ning Wang**
Sorry I don’t have a good recommendation. Do you have any? Josh Fischer
**Josh Fischer**
I do not, sorry.
**Philip Dießner**
Hello Josh Fischer and Ning Wang, thanks for being here! Nice to see such a book teaching the concepts not tied to any framework.
How did you come to the decision to write the book in the way it is? How much work was it timewise?
**Josh Fischer**
Thank you for the kind words. We decided to write a framework agnostic book so we would have a book that would last longer than individual frameworks. We are hoping to teach people the fundamentals to learn how to plan, predict, and diagnose the state of most streaming systems as opposed to being tied to one technology implementation.
**Ning Wang**
Timewise, it did take more time I feel as we need to step back and think about the fundamentals, especially we are both first time writers so it takes a lot of time to learn how to write a book as well. We got a lot of helps from Manning editors,
**Ning Wang**
at the very beginning we were thinking of using a framework but it is hard to avoid being a reference book which is not interesting.
**Philip Dießner**
Do you touch upon unfied batch/streaming architectures or do you think one can apply learnings from your book when building something like this?
**Josh Fischer**
We touch on both batch and streaming architectures in this book.
**Ning Wang**
yeah we touch a little at the beginning about the architectures and then focus on streaming. many concepts are similar in batch.
**Ning Wang**
we don’t touch `unfied` architectures though.
**Alexey Grigorev**
Sometimes when a team/companies discovers the benefits of streaming, they want to migrate all the microservices to streaming.
Do you think it’s a good idea, and if it’s not, how to push back?
**Ning Wang**
It really depends on the use cases. Streaming systems have typical use cases (like processing messages stored in a pubsub system), and microservices have theirs too (like taking requests and then sending back responses). They are also not exclusive to each other, like a streaming system might rely on some microservices. For example, a streaming system can process message and write data to a k-v store which is a microservice. while streaming (and batch) systems are powerful, they have many limitations: complicated since there are more moving pieces (hence hard to maintain/optimize), microservice are (ideally) very staightforward and much easier to optimize. I sometimes feel each component in a streaming system is like a microservice, or a proxy to some microservice.
**Josh Fischer**
I agree with all things Ning said, “it depends” 😄
**Josh Fischer**
Another factor to take into consider is how the data flows through a system. For example, do we need that request/response model like we get with REST or will we expect once we send data somewhere that something else downstream will take action on it? This is a question I often find myself thinking about.
**Alexey Grigorev**
How do I know if I need this request/response model? Often it seems that I do, but with a bit of redesigning it turns out that I don’t
**Alexey Grigorev**
But on the other hand, that redesign might overcomplicate things
**Ning Wang**
Totally. Request/response model is straightforward to understand and operate, and this is the major reason to follow the model. Hence the model could be a good starting point at least for many cases. To change to (could be partially) streaming model, you need to have compelling reason about what you gain by the extra complication and make sure the ROI is reasonable/desirable. One typical reason might be that the process in a microservice becomes more and more complicated and harder to maintain, and streaming could help to break the process into multiple stages and make it more maintainable although the architecture is more complicated.
**Josh Fischer**
I Agree Ning. I think a development team should try with other traditional methods of moving data around request / response, batch, etc before moving to streaming. This way they have an understanding of their current pain points and a baseline to measure progress from as they make the transition to a streaming system
**Alexey Grigorev**
Also how to best select if I should deploy my microservice as a web service or as a streaming application? What are the pros and cons for each?
**Ning Wang**
My 2 cents: if it is necessary for the data to be processed in multiple stages, streaming system might make sense. Otherwise, microservice is likely to be more efficient and much more maintainable. Again, microservices and streaming systems are not exclusive to each other. Even when multiple stages are necessary, it is likely that a combination of microservices and data processing systems could be a better option.
**Ning Wang**
For example, a web server might be used to accept user data and store into a queue; and a data processing system can subscribe the data and process them to get the final results.
**Josh Fischer**
Again, I think this goes back to the idea of “how does the data flow through my architecture?” or “Is it a one way path from start to finish or am I expecting that request/response handshake?”
**Carise Fernandez**
Maybe it’s getting ahead of everything, but do you have suggestions for next books on streaming systems, after we read Grokking Streaming systems?
**Josh Fischer**
I would check out Pulsar in Action by David Kjerrumgard
**Carise Fernandez**
Thank you!
**Carise Fernandez**
What’s it like being an Apache committer? Do you work on it full time? Always have been interested in working on open source long term, but intimidated too!
**Ning Wang**
It is totally different between learning something interesting and implementing it. 🙂. also working on open source projects is a chance to work with different people from very different background instead of your team mates. Definitely not full time for me. And TBH, I am quite busy at Amplitude, and I sadly couldn’t spend enough time on Heron these days. 😞
**Josh Fischer**
Working on open source had been the most fulfilling work I’ve done. Heron was my first open source project I’ve worked on. It’s not my full time job. I also have been involved as much as of late because if job, family, and a startup I’m building.
I can see how it’s intimidating, but I can promise you that if you are willing to take the chance and do the work to solve some problems for a community they will be greatful for your contributions. If you stay around long enough people will start to look at you for guidance.
**Carise Fernandez**
Definitely a dream of mine to work on an open source project that benefits the software development community. I’ve worked on open source in the past, but it was not really something like a software that a community depended on. Thanks for sharing your experiences!
**Ashish Lalchandani**
Hello Josh Fischer and Ning Wang, thank you for being here! My question is, what advantages do streaming system provide while deploying ML/DL models, as compared to deploying them on local device and sending data to server? Let’s say a company has face recognition model for its employees. Wouldn’t it be cheaper to just deploy that model on local device and setup a server containing information about the employees, instead of using streaming system(assuming we are using AWS Lambda and Kinesis for streaming data) ?
**Ning Wang**
Yeah, it could be cheaper to deploy data and code directly to the device. Some potential (since it may or may not be better) benefits are: 1. the code runs on the device is pretty simple and stable since it is only responsible for collecting data. when there are more models/data, you don’t need to redeploy the software too often or worry about the device performance (so the device cost could be lower when there are a lot of them). 2. you can allocate less resource for the simple models and more for the expensive models to achieve the performance you need, because now they don’t run on the devices directly.
**Ashish Lalchandani**
Okay, that makes sense. Thank you for clarifying!
**Josh Fischer**
I think Ning answered this question well 💯
**Ashish Lalchandani**
Yes he did! All the best for your book Josh Fischer Ning Wang!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Hands-On Data Analysis with Pandas - Second Edition – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Hands-On Data Analysis with Pandas - Second Edition
---------------------------------------------------
#### by [Stefanie Molin](https://datatalks.club/people/stefaniemolin.html)
##### The book of the week from 18 Jul 2022 to 22 Jul 2022
Data analysis has become an essential skill in a variety of domains where knowing how to work with data and extract insights can generate significant value. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with the Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn.
Using real-world datasets, you will learn how to use the pandas library to perform data wrangling to reshape, clean, and aggregate your data. Then, you will learn how to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. In the concluding chapters, you will explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data.
This updated edition will equip you with the skills you need to use pandas 1.x to efficiently perform various data manipulation tasks, reliably reproduce analyses, and visualize your data for effective decision making—valuable knowledge that can be applied across multiple domains.
* [Book's page](https://www.packtpub.com/product/hands-on-data-analysis-with-pandas-second-edition/9781800563452)
* [Buy on Amazon](https://www.amazon.com/Hands-Data-Analysis-Pandas-visualization-ebook/dp/B08R67H7F5)
* [GitHub repository](https://github.com/stefmolin/Hands-On-Data-Analysis-with-Pandas-2nd-edition)
Questions and Answers
---------------------
**Alexey Grigorev**
Hi Stefanie! Which datasets did you use in the book? And do you know some good datasets that you wanted to use in the book but at the end didn’t?
**Stefanie Molin**
Hi Alexey – there are many datasets in the book: earthquakes, weather, stock prices, planets, and wine. For some of these datasets, I also cover how to use APIs to collect the data.
**Stefanie Molin**
The UCI Machine Learning Repository ([https://archive.ics.uci.edu/ml/datasets.php](https://archive.ics.uci.edu/ml/datasets.php)
) is a great spot to find public datasets, as are government websites (I provide lots of sources in chapter 12). Narrowing it down ultimately came down to what would be interesting to write about that would also make for good examples.
**Stefanie Molin**
When I designed my workshops ([https://github.com/stefmolin?tab=repositories&q=workshop&type=source&language=&sort=](https://github.com/stefmolin?tab=repositories&q=workshop&type=source&language=&sort=)
), I wanted to use different datasets (e.g., taxi rides and meteorites), so people would have new examples.
**José María Vargas Mendoza**
Hello, Stefanie! In your experience, what methods have proven to be effective in spear-heading the transition from the “ol’ reliable” Excel to Python for analytics? A lot of small to medium businesses (specially those who are already established in the market for some time) tend to rely on Excel a lot, and the analysts who try to introduce Pandas, Dask, or other tools available to the process, usually get blocked by the stakeholders.
**Stefanie Molin**
Automation, automation, automation. In my experience, framing the move from Excel (or similar tools) to coding as freeing up time for new analyses is key. Are there frequent requests for stakeholders that can be automated? Automate those and work on something that you wouldn’t have had time for if you needed to use Excel – the stakeholders will see the additional value being provided.
**José María Vargas Mendoza**
Thanks for your answer! Indeed, there are quite a number of processes that have the potential for being automated in the classic “analytics with Excel” workflow. Hopefully, as more and more analysts adopt Python and other tools as their go-to for day-to-day analytics, businesses of all sizes start implementing changes to accommodate these newer, more efficient workflows in their operations.
**Sergio Rozada**
Hi Stefanie!! Thanks for being here. Can you share some insights in the key Pandas aspects that we have to look closely when deploying anything involving Pandas in pro?
**Stefanie Molin**
Hi Sergio – The first thing to consider is whether you actually need pandas in what you are deploying. While pandas makes it very easy to explore your data and figure out what kind of processing you need, it has some overhead, which can be mitigated by converting your logic into standard Python objects once you have thoroughly explored and identified what needs to be done.
Secondly, if you must use pandas in production, it’s important to consider how you are using it. Are you immediately removing columns/rows once they are no longer of use? Are you converting strings to categories? Are you performing operations in place, where possible? Are you using chaining? If not, you are probably making multiple copies of that data in memory for each operation, which is going to be costly.
**serdar**
Hi. Excuse my ignorance i am new to ML but dont you think we might really need to work big data when it comes to real world problems? Feel like we really need to use pyspark iso pandas for better performance.
**Stefanie Molin**
I’m a firm believer that you need to use the best tool for the job. Sometimes that’s pandas and sometimes it isn’t. You will encounter numerous datasets that aren’t big data, on which you can comfortably use pandas without the need for PySpark – it’s important not to prematurely optimize.
Also, keep in mind that ML is not exclusive to big data by any means. Consider a linear regression to find the relationship between price and number of goods sold: do you need millions of points to calculate that? Definitely not – you could get an answer much quicker and just as accurate with significantly fewer data points.
**serdar**
Thank you very much for the clarification.
**Ashish Lalchandani**
Hi Stefanie! Thanks for being here! I always struggle with EDA whenever working on a dataset. I see many people posting their code on kaggle, and everyone performs EDA in different manner. My question is, is there any standard way to perform EDA, like step 1 -> do this, step 2 -> do that, etc.?
**Stefanie Molin**
Hi Ashish. Unfortunately not. People will approach this differently. Different datasets may require you to approach things differently, as well. The emphasis is really on the “exploratory” part – you will move around between cleaning, wrangling, and visualizing the data as you explore, but you are free to explore how you see fit.
**Ashish Lalchandani**
Okay so cleaning, wrangling and visualizing is a must, and other than that we are free to play with data however we like. Got it, thanks!
**Tyrone Li**
Hi Stefanie! It was great meeting you at ODSC East this year! I’m fairly new to Python so I’m curious: what were some of the key methods that helped you to transition from learning how to use Python to eventually becoming an expert with a textbook of your own? For instance, do you make a habit of perusing other textbooks? Or perhaps you spend a good bit of time interacting with other practitioners in the field and their githubs? Many people learn how to code but writing your own book is absolutely next level, so I’m curious how that happened.
**Stefanie Molin**
Hi Tyrone – nice to see you here! While learning how to use Python, I did a lot of LeetCode (and similar sites) and practice projects. I perused some books to get started with the Python data science stack, but the biggest help was using the libraries for projects both at work and at home. Going through the docs and trying out what’s there on datasets I was working on helped me more than looking at the examples in the docs. For me a big part is having a use case or project, identifying a tool, and then learning how to make it work.
As far as the book, Packt (my publisher) reached out to me about writing it. From the books and documentation I had read while learning, I had lots of ideas of how I would approach it differently, and I knew writing it would make me even more knowledgeable on the topic. Teaching is one of the best ways to learn since it forces you to fill in the gaps in your knowledge – the things that you know how to use but can’t explain the how/why behind them. The same goes for my workshops: I learned plenty of things while creating them as well. For me, it’s important to always be learning something.
**Carlos Orjuela**
Hi Stefanie Molin , thanks for taking the time to answer our questions…. Does the book deal with working through Pandas error messages or do you have any suggestions on this? I’m not expert and I’ve found sometimes a bit difficult to try to understand what’s going on
**Stefanie Molin**
Hi Carlos - Beyond a specific error related to not using loc/iloc, it does not. However, this is a skill that isn’t exclusive to pandas code, so it’s definitely important to hone it. Python 3.11 is going to have better error messages that should make this easier, but we will still need to have this skill.
What I do when I’m faced with an error message is first look at the type of error, the error message, and also the stack trace to find the line where the exception was triggered. With that information, I work on debugging the code: what does the data look like just before the code that raised the exception was called? Understanding the format that different functions expect their parameters will help you narrow down where the issue could be, so if you aren’t too familiar with that code, check out the docs. Googling the error is also helpful if you can’t see why it is happening. Another useful strategy is to break down everything into simple operations so instead of chaining multiple calls, run each one separately to see exactly where it is failing.
**Tim Becker**
Hi Stefanie Molin, thanks for being here. I am wondering what is new in the second edition? Also, in your opinion, what are the most useful pandas features that are rarely used?
**Stefanie Molin**
Hi Tim – for the second edition, all code examples have been updated for newer versions of the libraries used. The second edition also features new/revised examples highlighting new features. For pandas in particular, the first edition uses a much older version than what is currently available (pre 1.0), and this edition brings the content up to date with the latest version (1.x). You can look through the pandas release notes to get an idea of all the changes that have happened since the version of pandas used in the first edition (0.23.4). In addition, there are significant changes to the content of some chapters, while others have new and improved examples and/or datasets.
**Stefanie Molin**
As for your second question, I’ve often seen casual users not using assign() or not being aware that pandas has plotting functionality. Many people also don’t realize that you can resample by month but use the start date of the month instead of the end date just by changing the frequency to MS instead of just S (always check the docs before trying to hack something together). Another would be crosstabs, which are a very powerful feature that people seem to forget about.
**Tim Becker**
Thank you! 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Data Analytics Initiatives – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Data Analytics Initiatives
--------------------------
#### by [Ondřej Kubera](https://datatalks.club/people/ondrejkubera.html)
, [David Bednar](https://datatalks.club/people/davidbednar.html)
, [Ondřej Bothe](https://datatalks.club/people/ondrejbothe.html)
, [Martin Potančok](https://datatalks.club/people/martinpotancok.html)
##### The book of the week from 01 Aug 2022 to 05 Aug 2022
The categorisation of analytical projects could help to simplify complexity reasonably and, at the same time, clarify the critical aspects of analytical initiatives. But how can this complex work be categorized? What makes it so complex?
Data Analytics Initiatives: Managing Analytics for Success emphasizes that each analytics project is different. At the same time, analytics projects have many common aspects, and these features make them unique compared to other projects. Describing these commonalities helps to develop a conceptual understanding of analytical work. However, features specific to each initiative affects the entire analytics project lifecycle. Neglecting them by trying to use general approaches without tailoring them to each project can lead to failure.
In addition to examining typical characteristics of the analytics project and how to categorise them, the book looks at specific types of projects, provides a high-level assessment of their characteristics from a risk perspective, and comments on the most common problems or challenges. The book also presents examples of questions that could be asked of relevant people to analyse an analytics project. These questions help to position properly the project and to find commonalities and general project challenges.
* [Book's page](https://www.routledge.com/Data-Analytics-Initiatives-Managing-Analytics-for-Success/Bothe-Kubera-Bednar-Potancok-Novotny/p/book/9781032208510#)
* [Buy on Amazon](https://www.amazon.com/Data-Analytics-Initiatives-Managing-Success-ebook/dp/B09YWQW9HN)
Questions and Answers
---------------------
**ASHISH SONI**
Hi guys Ondrej Kubera David Bednar Ondrej Bothe Martin Potančok
I hope you are doing great 🙂
I had a Two part question
1. What is the ambition of the book, in terms of the reader?
2. Which characteristics of the reader, you wanted to target -> to improve/ benefit/should become better at, after finishing the book?
**Ondrej Kubera**
ASHISH SONI Thanks a lot for your question. For the first part - I would say the ambition is to uncover specifics and complexity of data analytics initiatives, in a structured and holistic manner. To answer the second portion - after reading the book, the reader should have better understanding of analytical work and its various types and understand the complexity and typical challenges. As a result he or she can better set up the analytics initiative for success and avoid or be ready for typical challenges. Did I answer your question? 🙂
**ASHISH SONI**
Yes Ondrej Kubera Thank you 🙂
**Tim Becker**
Hi everyone, thanks for being here! I would like to know what are the most common reasons for failure in analytics projects and are the reasons really that different depending on the type of project? Could you maybe provide an example?
**Ondrej Bothe**
Hi Tim Becker, thank you for your question.
There could be many different reasons, why the project could fail. May be more preciously, why the insight is not delivered or used. This is caused by the complexity of an analytical work itself. In the book, we are trying to describe the Framework, how to deal with the complexity - how to describe it and understand the consequences and interdependencies. We believe that this approach will limit misunderstanding and miscommunication and allow us to define and focus on the most critical aspect of the work. The fact, that there is a common agreement and understanding of the critical area is itself limiting the probability of failure.
For example, one project could fail, as we are not able to ingest/integrate data in the current IT ecosystem because of GDPR. Another could fail because we are not able to deploy an analytical model into production with proper operational support. Another could be considered a failure because the team was not able to find a model with “good enough” analytical results or design a report, that satisfies the requirements of consumers. Potentially, it could be all together as well. Did I answer your question?
**Tim Becker**
Ondrej Bothe yes, thank you very much. It is clear!
**Dr Abdulrahman Baqais**
Thank you for the book. I have few questions:
1) How importance do you think to implement a strategy for analytic projects selection?
2)Is this strategy should be handled by technical team during implementation or business team during initiation or it could be hybrid.
Thank you.
**Ondrej Bothe**
Hi,
thank you for your question.
Regarding the first part: Project selection is potentially the follow-up process. First, we need to understand the type of the analytical project and the work, that need to happen to deliver successfully (from the data, IT and stakeholder perspective). Such consistency across the projects could help us to compare the initiatives one with another and decide, where to start. Also, the evaluation of business benefits needs to be considered (a lot connected with analytical maturity).
Regarding the second part: Both IT (technical team) and business need to work closely together (potentially focusing on different areas of the project component). It is important to establish the evaluation as an ongoing process as analytical initiatives are developing over time, so the importance of different aspects is changing continuously.
**Dr Abdulrahman Baqais**
Also who benefits the most of this strategy:
startups, SME with limited budget or bug corporate who might have alot of projects in a pipeline.
**Ondrej Kubera**
Thanks a lot for your questions! The framework we are describing in the book can be applied to all analytical initiatives and we hope anyone can learn and apply the learning to their data analytics projects regardless the size of the company. Biggest benefits it will probably have for folks from large companies, or corporations with higher complexity of analytical, technology and data landscape respectively.
**Cyril de Catheu**
Hey Ondrej Kubera David Bednar Ondrej Bothe Martin Potančok,
Thanks for sharing the book release.
Does the book approach analytics project as integration projects (eg plugging Google Analytics and make use of it) or as internal platform business/engineering effort? (eg build an analytic stack with open source technologies). Or both aspects are discussed?
Also, it is always good to learn from failures. Were you able to get real life analytics project post-mortems into the book?
**Ondrej Kubera**
Hi Cyril de Catheu , thanks for your question! We look at data analytics projects as any initiative for which the goal is to improve decision processes by bringing insights from data (create a new insight, automatize insight, reduce the time needed to gain insight,…). Coming back to examples you mentioned - yes, in our perspective data analytics project can a be a large internal analytical platform on one hand but also small analytics component as part of software integration initiative. We dedicate portion of the book to categorization of the types of the projects and explaining different challenges associated with each of them.
Regarding the post-mortems - we can’t disclose the details of projects we worked on with our customers in the past, but we tried to share a lot of examples of potential failures throughout the text of the book. And yes, we experienced good amount of them. Honestly the another subtitle of the book could be “what can go wrong and how to avoid it” 🙂
Does it help? 🙂
**Cyril de Catheu**
\> what can go wrong and how to avoid it
hahaha
I like the approach and the clear definition of “analytics project”.
Helps a lot thanks Ondrej Kubera!
**Ash Smith**
Have you ever dealt with a project where there is no clear owner of a company data strategy which then means ownership in analytics projects becomes a huge point of concern? e.g. whos supporting the report once its completed or a model is developed but data scientist move on and models gather dust. How would you address this and does the book assist in complex project like this?
**Ondrej Bothe**
Hi Ash Smith, thank you.
It looks like you have already read some chapters from the book and using the example from them :-) For sure it happened many times… It is difficult to help to address it - the book is trying to highlight and categorize such challenges as you stated, and bring them into the context (as there could be many others). In case you are aware of the risks at the beginning of the project, it is much easier to set up a proper expectation and so manage the delivery in time (it is interesting, that types of challenges are changing in time, as the analytical initiative is moving ahead). Did it help?
**Ash Smith**
haha Ondrej Bothe I actually havn’t yet. Just quoted some big pain points i’m dealing with right now 😄 but thanks for responding.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Essential Math for Data Science – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Essential Math for Data Science
-------------------------------
#### by [Thomas Nield](https://datatalks.club/people/thomasnield.html)
##### The book of the week from 29 Aug 2022 to 02 Sep 2022
Master the math needed to excel in data science, machine learning, and statistics. In this book author Thomas Nield guides you through areas like calculus, probability, linear algebra, and statistics and how they apply to techniques like linear regression, logistic regression, and neural networks. Along the way you’ll also gain practical insights into the state of data science and how to use those insights to maximize your career.
Learn how to:
* Use Python code and libraries like SymPy, NumPy, and scikit-learn to explore essential mathematical concepts like calculus, linear algebra, statistics, and machine learning
* Understand techniques like linear regression, logistic regression, and neural networks in plain English, with minimal mathematical notation and jargon
* Perform descriptive statistics and hypothesis testing on a dataset to interpret p-values and statistical significance
* Manipulate vectors and matrices and perform matrix decomposition
* Integrate and build upon incremental knowledge of calculus, probability, statistics, and linear algebra, and apply it to regression models including neural networks
* Navigate practically through a data science career and avoid common pitfalls, assumptions, and biases while tuning your skill set to stand out in the job market
* [Book's page](https://www.oreilly.com/library/view/essential-math-for/9781098102920/)
* [Buy on Amazon](https://www.amazon.com/Essential-Math-Data-Science-Fundamental/dp/1098102932)
Questions and Answers
---------------------
**adanai**
What are some concepts you’d recommend reading over after completing the book?
**adanai**
Do you feel that learning to build the algorithms from scratch is important or using the tools is enough? eg. Implementation of gradient descent using libraries vs using numpy to build from scratch
**Thomas Nield**
Like I posted in previous answer to Kevin’s post above, I believe it’s necessary to build from scratch at least once to have strategic value and insight into how these black boxes work. It really depends on how seriously you want to become a subject matter expert on a given topic, and of course you have to prioritize what you learn.
in my book, I teach how to do gradient descent and stochastic gradient descent from scratch for linear regression, logistic regression, and neural networks. For the first two of those I show how to do that without NumPy!
What you study after the book is dependent on your goals and what problems interest you. in the last chapter I emphasize it’s truly a “choose your own adventure” based on what you want to pursue! It wouldn’t be right for me to presume what every reader should pursue after the book, because I have a feeling many readers will react differently to what’s next for them.
**David J.**
Hi Thomas Nield, thank you for taking the time to answer questions about your book! My question is, what is something you learned later on in your data science journey that you wish you knew earlier on?
**Thomas Nield**
Oh boy, the way I describe my book to friends and colleagues is “it’s what I wished I knew 12 years ago before data science and AI became a thing.”
I think the biggest lesson I’ve learned is to ask questions when nobody else is. We live in a strange corporate culture where people don’t ask questions, especially the “elephant in the room” ones that people are afraid to ask (maybe out of fear they are missing something, or appearing ignorant?) I actually give this a name in my book, the Jabberwocky Effect.
What I find surreal is asking questions, especially the uncomfortable ones that challenge popular assumptions, really took me for a ride. It frustrated some folks, but I got relieved gratitude from others simply by cutting through narratives and seeking grounded information. I now teach at University of Southern California advising government, military, and aerospace agencies on artificial intelligence system safety… simply by asking questions. I now wrote a book on a subject I’d never think I was qualified to write, simply by asking questions.
Ted Lasso said it best, “Be curious, not judgmental.” You don’t have to be confrontational or rock the boat. Just be curious, tell people you’re confused, and you need help understanding. At worst you learn something new and show you aren’t afraid to admit you don’t know something. At best, you bring clarity to glaring problems that aren’t being acknowledged or addressed.
**David J.**
Thank you for the thoughtful response! I am definitely someone who has struggled with the Jabberwocky Effect for the reasons you stated (fear of missing something/appearing ignorant), but I think I’ve gotten better at trusting myself in knowing when I haven’t properly reflected on a question and when I think my question/confusion is legitimate. Sometimes that’s hard to gauge though, so when in doubt I try to embrace question-asking and follow Ted Lasso’s advice 🙂
**Christian**
Hi Thomas Nield, thanks for sharing your thoughs with us.
What do you think are the best ways to convert our reading activity on essential or fundamental Math principles for Machine Learning to expertise in solving real world issues?
**Thomas Nield**
I get this question in some flavor a lot. It’s easy to feel like a hammer looking for nails, which is pretty common for those who study machine learning. You can always create your own self-study projects, on public datasets or toy datasets you create (which I like doing as controlled experiments). But if you have a job that has you doing less glamorous tasks with data and isn’t providing you opportunities to use machine learning, try to take a problem-first approach. What problems does your employer have? And once you’ve identified that, try not to bludgeon the problem with machine learning but rather look at what other solutions are out there: linear programming, optimization, heuristics, metaheuristics… pairing the right solution to a problem is an invaluable skill. I think half the value of knowing machine learning is just simply recognizing what it doesn’t do, and confused employers can benefit from that kind of knowledge expert.
The other route is to explicitly pursue roles that are machine learning geared, like ML Engineer rather than a generic data scientist.
**Christian**
Thanks a lot 🙏
**Ricky McMaster**
Hi Thomas Nield thanks a lot for doing this! Along with the main areas covered in the book of calculus, probability, linear algebra, and statistics, do you address (or do you have any thoughts on) other areas such as Optimization (which is of course highly related to linear algebra and calculus)?
The reason I ask is that I have sometimes seen candidates or colleagues approach a task by devising complex machine learning models, when in fact all that was really required was to solve a non-linear equation, for example.
Update:
I see that you’ve already mentioned this in your last answer to Christian , so my question would be then be - what do you advise those who do not come with significant academic grounding in mathematics in choosing the appropriate technique for a task (like the scenario above)?
**Thomas Nield**
Glad you discovered my answer to your first question. I do talk about optimization a bit in my book, and I even throw a section on linear programming in the appendix. I would have loved to include a chapter on metaheuristics and optimization, like the traveling salesman problem. But I had a 350 page book already so I just mention these other techniques.
What’s funny about academics and people with PhDs is they often are hyper specialized in their own rabbit holes. Many are much better writing papers on the HOW rather than the WHY. They can fill a whole whiteboard full of Greek symbols and impressive math equations, but handwave over the silliness it is an algorithm to separate pictures of hot dogs from dachshunds wearing hot dog costumes. What value does this create? So you have to take the academic gravitas with a grain of salt, but there are exceptionally talented academics of course who do valuable work. There are a few I look up to.
A Cherry-picked example though is a secondhand account from a colleague who went to a predictive modeling conference. A PhD candidate created an elaborate model trying to pinpoint factors causing a major airport to have sudden delay problems. During his presentation full of theories, data, and regression models, my colleague said “the airport closed a terminal for construction this year! Your model is not at all accounting for this!” He reacted sheepishly and carried on, even though a quick Google search nullifies his entire project as the delay cause was well known.
The question then becomes how do you stand out and have a skill set most academics do not have? I think the best answer to that is to have more awareness of business context and what’s practical, not having deep specialization in one topic with little insight on the overlap with the real world. I hate to sound like I’m advocating becoming a “jack of trades, master of none” but there is so much information and complexity out there. Special interests are selling silver bullets to get more funding and investors. Somebody has to be the one to know how to pair the right solution the right problem, and question experts who can’t see beyond their own field.
**Ricky McMaster**
Wow thanks a lot for the considered response! That’s a great and memorable anecdote about making sure you keep practical context and current available knowledge in mind.
So in other words…. there’s just as much danger in applying too much academic theory as not enough, possibly more so?!
I suppose from the point of view of practical statistics, this would tie in somewhat about checking your assumptions (and indeed your biases) at the outset of the project. And of course Occam’s Razor is always good to keep in mind - actually Alexey, the creator of this group, has something similar at the beginning of his Machine Learning book. Namely, ask yourself whether an ML model is actually relevant to the current task?
**Thomas Nield**
Exactly 💯
**Evren Unal**
Hi Thomas Nield,
When I was looking at your book\`s content I realized that it is very concise.
How did you choose the content of the book?kilicdaroglu
**Thomas Nield**
Choosing content wasn’t easy. There are certainly topics I wish I could have included such as how to build simulations as well as optimization algorithms in more depth. But I made machine learning the end goal of the book, and to get there I guided readers through foundational topics like linear algebra, calculus, and statistics which then feed into linear regression, logistic regression, and neural networks. The “build upon” approach worked quite nicely, and areas I couldn’t get to like optimization could at least get called out as other areas to explore, and I provide tons of resources throughout the book to learn more. I made a diligent effort as well to tie in real world examples and insights, as well as pitfalls to watch out for.
The last chapter covers career advice and my polite rant on the state of data science. I give advice on how to navigate a field that’s largely been co-opted by many interests due to poor definition on what is and is not data science, and how to thrive and avoid career pitfalls. That was a fun chapter to write and one I think will help most readers. Based on the questions I’m getting on this channel, I am getting further affirmation that’s the case.
**Evren Unal**
Thank you very much
I hope your experience give good insight to readers.
**Matthew Emerick**
Thank you for doing this, Thomas Nield. We appreciate your time.
Is your book geared more toward newcomers to the mathematical side of ML? Or is it better suited for those who once learned the subjects at university and are trying to relearn it and dig a bit deeper?
**Matthew Emerick**
Should the learner pick up some basic ML first, or start with the math?
**Matthew Emerick**
What is your opinion of learning through methods such as your book and YouTube videos versus taking classes?
**Thomas Nield**
This book is definitely geared more towards newcomers and beginners, who at least has some understanding of high school algebra. no other knowledge is assumed including machine learning. That being said I think a lot of folks who have been dabbling in data science and machine learning will learn something new from the book. I pack in a lot of lesser-known knowledge that I wish more people knew going into the field, including brief insights into how self-driving cars work and how the application of ML matters in terms of hazard and risk.
I think learning is all about pursuing good information regardless of the medium. Classes, videos, books… I have found many good instructors across all these media and I share them in my book. It’s not so much which medium is best but rather the instructor is good and knowledgeable, has experience, and is able to explain things that click with learners.
**Alena Kniazeva**
Hi Thomas Nield, thanks for sharing your expertise in math and data science.
How do you think, what is more valuable in data science: mathematical or programming background? I mean who is more likely to succeed in data science:
1. experienced programmer, if he’ll learn math concepts and take some data science course
2. an investigator or university professor with a strong knowledge of math, if he’ll learn Python and also take some data science course?
**Thomas Nield**
I talk about this extensively in my book, and you’ll probably not be surprised by my answer based on some previous answers I gave to other questions ; ) I think the the experienced programmer is going to do better in a majority of data science job listings out there, because most tasks in data science are unglamorous data wrangling and moving it from one place to another. Then there is a growing awkward need to put models in production, and a programmer is already going to know how to do this well. This is 95-99% of _useful_ data science work.
There are exceptions for some roles the PhD would do better, if the role is hyperspecialized and requires said PhD. Maybe a high profile role at a big tech company would require that kind credential as well, and I’m guessing if they are qualified for that role they already know how to code. If there isn’t programming involved a role would probably be more advisory than coding. These are just my observations though and are somewhat anecdotal, but I’ve seen this pattern from what other people have shared with me too.
**Alena Kniazeva**
Thank you for a thorough answer. It is very very useful to hear position, that is based on real practical experience :thank\_you:
**onyeka okonji**
Hi Thomas Nield how important is an understanding of Maths for a non-research career in Deep Learning with a focus on computer vision.
Secondly, how well do you think one needs to master the Maths of DS. Say on a scale of 10.
**Thomas Nield**
I think there is definitely some mathematical proficiency needed, especially on the linear algebra and basic calculus front (and yes, my book covers both). There are definitely rabbit holes knowing every minute detail on how TF works. But conceptually knowing gradient descent, matrices, vectors, tensors, and mathematical functions is largely unavoidable if you want a productive understanding of TF. Chapter 7 teaches how to build a neural network from scratch too.
For data science in general, data science can mean something different to each organization. But at minimum I would be familiar with statistics and hypothesis testing. It’s just as important to have comfort and proficiency working with data (SQL, pandas) and programming in general from my experiences.
**Thomas Nield**
Best thing you can do is to learn what’s relevant for your job and to always find the right tool for the problem, not the other way around! Math may be involved, it may not
**onyeka okonji**
Thank you Thomas Nield i hope I get lucky. I’ll definitely want to read the book.
**onyeka okonji**
I should ask, does the book cover statistics and probability too?
**Thomas Nield**
onyeka okonji yup! Each of those topics get their own chapters.
**Alexey Grigorev**
Animals for O’Reilly books always seem a bit random - but why mice? 😃
I know you probably didn’t have any control over it, but maybe you have a theory how the cover is related to the content? 😅
**Bhupendrasinh Thakre**
Read somewhere (maybe rumor) that animals in their books are about to extinct or needs attention. Probably these are not mice 🐁
**Rafael Socorro**
[https://www.oreilly.com/content/a-short-history-of-the-oreilly-animals/](https://www.oreilly.com/content/a-short-history-of-the-oreilly-animals/)
**Thomas Nield**
Last I checked, the cover artist chooses the animal. My first book (Getting Started with SQL) had a natterjack toad. This one had mice. One thing that seems to be consistent somewhat is animals are thematically consistent somewhat based on their taxonomy. Reptiles are used for data books? Cats for Java? Rodents for data science? It’s a mystery…
**Diogo Telheiro do Nascimento**
Hey Thomas Nield! Nowadays it is getting much easier to try ML models and check if it fits to your data (especially Classic ML approaches). It is commonly done almost like searching for your sneakers size.
My question is: How does Math should be used in this context of model selection?
**Thomas Nield**
What a question. I could give you the conventional answer that you should choose the model that fits best to the test dataset and ROC/AUC (I cover this ad nauseum in my book) or produces the highest R2.
But I caution a lot in the book that math and data does not capture context. Just because your test dataset or validation dataset scores well or you found a set of hyper parameters that give a result, it does not mean your model is at all connected to reality. The data (inevitably) is biased, the model (inevitably) has assumptions. The hyperparameters are easily P-hacked. A magic math formula is not going to quantify any of that or capture those qualitative issues that only a human in the loop can solve. Computers are incapable of discerning correlation from causation, detecting bias in data, or having any notion of ground truth in higher dimensional problems.
This is why I tell people to be analysis-driven, not data-driven.
**Diogo Telheiro do Nascimento**
Thomas Nield, thank you a lot for answering my question. This is by far the best answer I have ever had in this subject. 🤯
**Sandhya G**
Thomas Nield i was wondering about the reasoning for including SciKit Learn in a chapter on Neural Networks. Thanks!
**Thomas Nield**
Good question, I didn’t want to inflict another library on readers when they worked with a few already throughout the book (numpy, scipy, sympy, sklearn). It was basic and simple to just use what sklearn already provided, following the previous API patterns from previous chapters. Also a majority of that chapter focuses on building a neural network from scratch using NumPy, including backpropagation and Stochastic gradient descent. TF or PyTorch were mentioned but not given a tangential focus from the purpose of that chapter: just getting insight on how neural networks actually work.
**cactusmkt**
Hi Thomas Nield, thanks for coming to talk about maths in data science! How do you suggest data professionals to navigate through maths concepts and applications in data science? Is there a way to know which ones are must know amongst different paths in data science: analyst, data scientist, machine learning engineer, and so on? If I don’t use some concepts often in my work, it’s very difficult to remember them. Any suggestions would be helpful, thanks!
**Thomas Nield**
I talk about this A TON in the final chapter of the book, which is career advice paired with my polite rant on the state of data science. What seems to be widespread is employers jump on the data science wagon but there doesn’t seem to be a good definition that everyone can agree on. This is problematic for anyone working in the field, because there’s no scope or restrictions on what is and is not data science. Why this happens is due to organizational politics, and I expand on this in my book. I also provide a sensible and practical definition: a data scientist is a software engineer with proficiency in statistics, machine learning, and optimization.
My advice is to learn what solves the immediate problems in front of you. Learn some foundational building blocks which my book attempts to share, but don’t bias towards a specific tool out of FOMO. The greatest solutions to everyday problems are often obscure and sensible, not making news headlines : )
**Ricky McMaster**
Thomas Nield I guess from the above that you might have something to say about communication barriers between business and data/tech? This has certainly been a recurring issue for me, and I presume many others.
**Thomas Nield**
Those barriers exist as always yes, but I also think there are larger systemic cultural issues that are unique to the current corporate climate. The barriers to what is and is not data science has regressed to a point anyone who touches data can call themselves a data scientist. I think this is largely because middle managers under pressure to check the “data science” box have every incentive to rebrand their existing analysts, SQL developers, and Excel jockeys as data scientists.
But this isn’t just due to ignorance, but rather the wrong incentives that are put in place. High dollar management consultancies tell their F500 clients that sentient AI is just around the corner (not true) and this further enables them to sell more services to make the organization “AI-ready.” Management then grasps at straws trying to get talent as stated earlier and they haven’t even tangibly defined what they are trying to achieve…
And that’s another thing to consider! One has to weigh the gold rush that machine learning has created, and you have to look at the people selling shovels: consultants, cloud vendors, media outlets, GPU vendors, people who hold stock in these companies… You also have to observe the speculative markets pushing stories about AI to increase stock valuations. Slight tangent, I find it well-timed that Elon Musk used the Twitter fiasco as a vehicle to sell his Tesla stock, shortly after admitting that full-self driving is harder than he thought, and it would devastate Tesla’s stock valuation if it didn’t pan out. Maybe coincidence, maybe not. But it certainly was convenient he could signal he wasn’t selling Tesla stock out of concerns for the company’s future, but rather he wanted to buy Twitter for free speech activism reasons and then blame it for its well-known bot problem.
But I digress. Pulling back, there are a lot of valuable career paths in becoming proficient with data and computer science techniques to work with it. The challenge is navigating the current corporate culture that’s highly vulnerable to narratives and financial incentives that distort expectations on what’s possible, as well as what’s actually useful. I talk about this a lot in the final chapter.
**David J.**
Thomas Nield Super interesting. A somewhat digressive question related to your Tesla digression: are there any blogs/publications that you read on a regular basis (daily/weekly) to keep up-to-date on data and data-adjacent news and perspectives?
**Thomas Nield**
David J. For vehicle automation, I like Autonocast. Great podcast. Two of my favorite episodes.
[http://www.autonocast.com/blog/2020/3/27/181-stefan-seltz-axmacher-on-the-end-of-starsky-and-the-future-of-autonomy?format=amp](http://www.autonocast.com/blog/2020/3/27/181-stefan-seltz-axmacher-on-the-end-of-starsky-and-the-future-of-autonomy?format=amp)
**Thomas Nield**
[http://www.autonocast.com/blog/2020/4/1/182-nancy-post-of-john-deere?format=amp](http://www.autonocast.com/blog/2020/4/1/182-nancy-post-of-john-deere?format=amp)
**Thomas Nield**
For staying up to speed, what’s funny is I spend a lot of time studying classical statistics and finding how much has been forgotten by contemporary practices. I try to understand how that progression happened, and more often than not it’s a little scary how messy things have become.
I read a lot of books (Aurelian Geron’s on Machine Learning is fantastic) but synthesize information from a lot of disparate sources, from research papers to YouTube videos. For data-adjacent I read WSJ and nonfiction books. Most importantly I do targeted research and try to pursue information relevant to what I’m interested in, while occasionally chasing my tail with what the cool kids are talking about this week. I need to dive into transformers at some point…
**Ricky McMaster**
Very interesting (Tesla etc.). For sure it would benefit the public and indeed our field if stats and data were better understood - indeed that’s why How to Lie With Statistics was written, among others. I just hadn’t considered that there were now such huge vested interests against it.
**Thomas Nield**
Companies don’t like sharing their safety data because of confidentiality and competitive secrets as well. Major frustration with safety world. It usually takes a high profile accident to make any strides in getting insight, but this struggle is still ongoing.
[https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf](https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf)
**Kevin**
Hi Thomas Nield I remembered from my time at college that my math teacher used to say that probably we would never perform matrix multiplication manually and we should learn the tools (at college maple) and understand what we were doing instead remembering how to perform those operation manually.
What is your opinion about it?
**Thomas Nield**
Good question. I say this in the last chapter of my book, but what you learn has to be prioritized. I can’t tell you how a regular expression compiles but I am very good at using them. I have no reason to go down that rabbit hole unless my job suddenly needs me to become a subject matter expert on the ins and outs of regular expressions… and that’s the key determining factor!
But for things like machine learning, and if you want to practice machine learning, it is beneficial to attempt building a linear regression, logistic regression, and neural network from scratch at least once. And yes my book covers this! This requires some matrix multiplication, but by doing this exercise you can speak to the libraries you use with more insight and subject matter expertise. That’s not just invaluable but arguably necessary.
So it really depends on how much knowledge authority you need on a subject, and whether doing a deep dive into the black box has strategic value. Machine learning is a topic that very few people actually understand and yet are making strategic decisions on, so it might be a liability to just have a black box understanding and nothing more.
**Kevin**
Thomas Nield thanks for the answer!
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Fundamentals of Data Engineering – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Fundamentals of Data Engineering
--------------------------------
#### by [Joe Reis](https://datatalks.club/people/joereis.html)
, [Matthew Housley](https://datatalks.club/people/matthewhousley.html)
##### The book of the week from 15 Aug 2022 to 19 Aug 2022
Data engineering has grown rapidly in the past decade, leaving many software engineers, data scientists, and analysts looking for a comprehensive view of this practice. With this practical book, you’ll learn how to plan and build systems to serve the needs of your organization and customers by evaluating the best technologies available through the framework of the data engineering lifecycle.
Authors Joe Reis and Matt Housley walk you through the data engineering lifecycle and show you how to stitch together a variety of cloud technologies to serve the needs of downstream data consumers. You’ll understand how to apply the concepts of data generation, ingestion, orchestration, transformation, storage, and governance that are critical in any data environment regardless of the underlying technology.
This book will help you:
* Get a concise overview of the entire data engineering landscape
* Assess data engineering problems using an end-to-end data framework of best practices
* Cut through marketing hype when choosing data technologies, architecture, and processes
* Use the data engineering lifecycle to design and build a robust architecture
* Incorporate data governance and security across the data engineering lifecycle
* [Book's page](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/)
* [Buy on Amazon](https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302)
Questions and Answers
---------------------
**Jasmin Classen**
Hi Joseph Reis and @matthew housley, thanks so much for answering questions here. I’d be curious about the following things:
1. How would you describe the target audience of this book? (e.g. Software Engineers as well as Data Scientists, beginner experience with Data Engineering vs. more advanced)
2. I’m very interested in the „cut through the marketing hype“ part of the book. How would you sum up this section exactly, how do you yourselves decide which technologies would be worth trying in an enterprise context?
3. Does this book touch upon Data Engineering in context of providing data for Data Science projects?
Thank you again and have a nice day!
**Joseph Reis**
Hey Jasmin,
1. Another way to answer this question are learning outcomes - the reader will come away with a good understanding of the foundations of data engineering, which they can apply in their respective disciplines (SWE, DS/DA, data engineering, etc)
2. To determine the best technologies worth trying in an enterprise context, please read Chapter 4 🙂
3. Yep, it provides a holistic context for providing data for DS projects. This is covered throughout the book, via the Data Engineering Lifecycle
**Matthew Housley**
Jasmin Classen One thing I’d add to Joe’s response to 2 above: study classes of technologies rather than single technologies. For example, instead of just looking at Apache Kafka, try to understand streaming event platforms as a class (Kafka, Pulsar, Kinesis, Pub/Sub.)
**Daniel**
Hi Joseph Reis and Matthew Housley,
thanks for doing this. I’d be interested in …
* How do you stay up-to-date with the breadth and depth of techniques that are available/upcoming in data science?
* From your experience: What are frequent mistakes when developing robust data systems?
* Any cloud provider you prefer/ would recommend (if you had to choose a single one)?
Many thanks in advance!
Cheers
Daniel
**Joseph Reis**
Your first question - Staying up to date is tricky, as there’s so much going on. I personally read a ton of newsletters, articles, and whitepapers on a weekly basis. For example, on most weekends, I’ve got 50+ articles/papers queued on my ipad to read. In general, I try to focus 80% of my time on areas in my peripheral area of expertise, with another 20% on outliers or stuff that might impact the field I work in.
**Matthew Housley**
My follow on to what Joe said about staying up to date is related to my response to Jasmin Classen’s question about technologies. Try to connect that dots between different technology developments. Try to understand common engineering threads between current technology announcements and other technologies that you are familiar with.
**Matthew Housley**
Regarding frequent mistakes when developing robust data systems:
We frequently see companies “rolling their own” stacks, when they could rely more on off the shelf solutions. (Note that off the shelf includes open source.) This leads to excessive complexity, maintenance overhead, massive tech debt and slow ongoing delivery.
**Matthew Housley**
Small startups will often point to Google or Facebook as examples of why you should build your own technology from the ground up, but this ignores the fact that many off the shelf technologies didn’t exist when these companies were trying to solve problems.
**Joseph Reis**
re: cloud providers - they’re all great 😉
**Matthew Housley**
Build a custom solution when you discover a problem that is not solved by something already in the ecosystem.
**Joseph Reis**
\> Small startups will often point to Google or Facebook as examples of why you should build your own technology from the ground up, but this ignores the fact that many off the shelf technologies didn’t exist when these companies were trying to solve problems.
**Joseph Reis**
We sometimes call this “cargo-cult data engineering”
**Cesar Garcia**
Is that the reason for the proliferation of endless orchestrators in the ecosystem?
**Matthew Housley**
Haha, that’s part of it.
**Matthew Housley**
We’ve seen several startups create their own orchestrators. (I’m even aware of an orchestrator written in Clojure…)
**Matthew Housley**
Although I think that orchestration is so central to data engineering that there are legitimately many different visions. Unfortunately, there’s only so much mindshare available, so the small players end up on the sidelines.
**Roman Zabolotin**
Hey Joseph Reis and @Matt Housley,
Thank you for sharing your ideas with us in your book.
Can you give a piece of advice to ones, who stand in data engineering path.
What practice project would be best to advance in this path? Can you give us some sketch examples?
**Joseph Reis**
There’s no single right answer to this question, but the approach I’ve seen fail over and over is doing projects that you think will land you a job. It might, but what did you learn from the project?
The best practice project is the one you’ll be interested in working on after the “homework” is over. Meaning, pick something that truly interests you.
**Matthew Housley**
Also, decide what kind of job you’re looking for. Are you trying to work for a Silicon Valley giant? A rapidly growing startup? A large enterprise?
**Matthew Housley**
For the latter two, focus on cloud skills, both data specific ones and general cloud infrastructure skills. Consider working on cloud certifications to stand out.
**Ricky McMaster**
Hi Joseph Reis and Matthew Housley thanks so much for doing this!
In your recent appearance on the Data Engineering Podcast you spoke of the need for a new conversation on data modelling. A reevaluation of dimensional modelling was recently offered (in brief) by Holistics’ [Analytics Setup Guidebook](https://www.holistics.io/books/setup-analytics/)
; given how much has changed technologically since the ubiquity of cloud data warehouses (which even the more recent additions to the canon such as Agile Data Warehouse Design do not consider), could you imagine a new (or modified) set of modelling principles usefully stretching to book-length?
Also, can you cite any interesting discussions of the topic? For example, Maxime Beauchemin’s contention that slowly changing dimensions could be considered obsolete given the cheap columnar storage provided by cloud DWH’s, which allows options like daily partitioned dimensions that were previously considered anti-patterns?
**Joseph Reis**
These are all good ideas. Modeling needs to incorporate much more than batch-paradigms. I’m personally excited on data modeling for streaming and event data, among other innovations.
Data Vault is pretty dope too.
**Joseph Reis**
Graph is another area I’m excited about, and I think there’s quite a bit to borrow from graph models with respect to traditional relational data modeling.
**Matthew Housley**
I think there are a lot of ideas out there on the present and future of data modeling. However, so far I haven’t seen someone commit to a new global vision of data modeling principles that can be adapted to current database and data lake technologies. That could potentially be a great book (or multiple books), but would require a huge amount of work.
**Ricky McMaster**
Thanks for the responses guys. So far I haven’t needed to do much with streaming data but modelling for that sounds interesting, along with graph - will check that out. I guess the neo4j O’Reilly book would be a good place to start?
Admittedly I’ve only really used the Kimball approach so I don’t really know what Data Vault could offer in addition.
Ok so a new book would be difficult but definitely useful then - here’s hoping that happens 🤞
**Ricky McMaster**
Sorry Joseph Reis and Matthew Housley, a follow-up on this one: I’ve been using the Kimball approach exclusively as it seemed best suited to what was necessary (and I liked the iterative approach in comparison to Inmon). However, it seems Inmon is personally quite persuaded by Lakehouse…
In any case, you mention Data Vault above: would you say any of the three main dimensional modelling approaches lend themselves best to the cloud DWH landscape?
**Joseph Reis**
There might be some confusion with how these various modeling techniques are described. Inmon originated the concept of the data warehouse (and later extending this to the data lakehouse). The data warehouse as defined by Inmon uses data marts to present data for various business use cases. These data marts might be modeled using Kimball’s dimensional modeling techniques.
Inmon defines a data warehouse as “a subject-orientated, integrated, time variant, non-volatile collection of data in support of management’s decision-making process.
Kimball took the notion of dimensional modeling, and applied it to a data mart only. He called this a data warehouse, which caused decades of confusion and infighting about the meaning of a data warehouse (I define it as Bill Inmon originally did).
Data Vault is considered the son of the data warehouse (according to Inmon). It’s a way of organizing your data from various sources in a very coherent way. You can add Kimball star schema on top of the Data Vault.
There’s not a one-size fits all answer to the cloud DWH, sadly, since can also go for wide, quasi-denormalized tables too, incorporating nested data structures.
As I mentioned before…confusing, isn’t it?
**Ricky McMaster**
Hmmmm yeah maybe my perception of Inmon is not in fact the reality 😉 … I need to research him and Data Vault for sure.
**Joseph Reis**
welcome to a giant can of worms 🙂
**Ricky McMaster**
Hahahahaha thanks 🙂
**Ricky McMaster**
\> Data Vault is considered the son of the data warehouse (according to Inmon). It’s a way of organizing your data from various sources in a very coherent way. You can add Kimball star schema on top of the Data Vault.
Very good to know, thanks a lot. This is exactly the sort of thing I wanted to find out.
**Joseph Reis**
sadly, this stuff is often buried underneath hundreds of pages of text in various books
**Ricky McMaster**
Your hours (weeks?) of pain on our behalf is much appreciated
**Joseph Reis**
months and years and decades :)
**Ricky McMaster**
Yes I can imagine… based on that, I guess even some stuff that’s very dated in technical terms, e.g. The DWH ETL Toolkit, might still have useful ideas/techniques?
**Joseph Reis**
The underlying techniques are still very valid
**Ricky McMaster**
Very good to know
**Matthew Housley**
One problem with traditional data modeling techniques is that the emphasize normalization, but absolute adherence to normalization doesn’t really make sense for today’s data and databases.
**Matthew Housley**
For example, third normal form (3NF) prohibits arrays and other forms of nesting. With the rise of NoSQL, trying to remove all JSON nesting in analytics data really doesn’t make sense. In addition, modern columnar database offer extraordinary performance on nested data and arrays; data engineers are foolish not to take advantage of these capabilities.
**Matthew Housley**
On the opposite end of the spectrum, I’ve frequently seen people advocate for denormalizing everything. Does this mean that you should try to put all of your data into one huge table? It’s a non-sensical prescription.
**Matthew Housley**
My suggestion is that data engineers learn traditional approaches to modeling, then make adjustments based on the data their handling, the use case, and the technology involved.
**Matthew Housley**
Normalization should be viewed not as a strict set of rules, but as a knob we can turn to suit the problem at hand.
**Matthew Housley**
For example, data that is naturally nested should potentially remain that way in tables for analytics, ML, etc. On the other hand, data that’s reused in many places should potentially live in its own table and get joined into other tables as required.
**Cesar Garcia**
Hey Joseph Reis and Matt Housing! Thanks for creating this book on Data Engineering. These are my questions for you:
* What are the areas less developed in the Data Engineering ecosystem?
* Do you envision that Data Engineering practices could pervade other areas besides big corps, like Open Government Data?
* What would be the biggest knowledge gaps for someone with sysadmin background to work as Data Engineer?
Thanks for your time!
César
**Joseph Reis**
Less developed areas in the DE ecosystem - collaboration with upstream/downstream stakeholders is a big mess right now, the immaturity of skills and competencies of data engineering teams is something that holds back the full potential of data engineering best practices. That’s hopefully where a book like FoDE can help bridge the skills gap.
**Matthew Housley**
I would love to see data engineering principles applied to open government data, but this is not something easily achieved given that government IT tends to be way behind corporate IT, and even further behind startups and tech companies. In the US, a huge initiative at the federal level would be required to make this happen.
**Cesar Garcia**
Thanks for your responses! I am getting ready some publications regarding data engineering principles applications to improve open government data quality.
I totally agree about the inertia in public administration so I am proposing a complementary approach: could citizens/civic initiatives collaboratively construct a quality checking infrastructure as some sort of check and balances system?
To offer a concrete example, I am envisioning some kind of shared Meltano infrastructure, that automatically extracts data from Open Government Data Portals and then triggers a quality check using Great Expectations. These Great Expectations rules could be versioned and shared across different cities, states, countries, etc. These checks could also trigger some metrics on response times, data drift, etc.
If you’d like to know more about this topic, please let me know so I can send you the links when published
**Joseph Reis**
seems reasonable
**Ning Wang**
It would be great if there is a unified governance system. Maybe finally smart contract has a suitable use case that can benefit everyone in the world. 🙂
**Nakul Bajaj**
Hi Joseph Reis and [Matthew](https://datatalks.club/people/matthewhousley.html)
,
Thanks for creating the book.
I wanted to ask when building data ingestion pipelines and or data ingestion patterns, what design principles can be utilised when creating pipelines and design patterns?
I also wanted to ask, if there are frameworks to help choose between open source pipeline orchestrators on the basis of their deployment, maintenance vs managed services. Will the book cover those?
Finally. What is the role of the data quality framework in data engineering? Also any tips on best practices for ELT and ETL? And which one is better for modern data warehouses such as snowflake or Big query?
**Joseph Reis**
\> I wanted to ask when building data ingestion pipelines and or data ingestion patterns, what design principles can be utilised when creating pipelines and design patterns?
**Joseph Reis**
Think in terms of push/pull and sync/async. Ensure reliability of payloads in terms of schedule and structure.
**Joseph Reis**
\> I also wanted to ask, if there are frameworks to help choose between open source pipeline orchestrators on the basis of their deployment, maintenance vs managed services. Will the book cover those?
**Joseph Reis**
yes, we cover this extensively in Chapter 4 - choosing the right technologies.
**Joseph Reis**
\> Finally. What is the role of the data quality framework in data engineering? Also any tips on best practices for ELT and ETL? And which one is better for modern data warehouses such as snowflake or Big query?
**Joseph Reis**
Data quality is part of data management, one of the undercurrents of the Data Engineering Lifecycle. Definitely incorporate data quality into your workflows.
As for ELT vs ETL for Snowflake or BQ, both can handle either one. We don’t pick a side with ELT vs ETL, and suggest using the best approach for the job.
**Matthew Housley**
Regarding data ingestion: use off-the-shelf ingestion tools when they exist. Writing custom ingestion code is often a waste of time if someone else has already done this work. You’ll find that you still spend a lot of time on ingestion pipelines where there are no existing solutions.
**Matthew Housley**
In the long term, I would like to see data providers move toward a “data sharing” paradigm where they land data in object storage or a cloud columnar database. This would free up a lot of time for data engineers to focus on higher value tasks. Google is a leader in this area, and we’ve recently seen companies like Stripe move in this direction.
**Joseph Reis**
And Snowflake
**Matthew Housley**
Regarding open source pipeline orchestrators: this area is changing extremely fast. I would suggest watching the current market leaders (Airflow, Dagster, Prefect) and talking to other practitioners. There will likely be interesting new entrants in 2022 and 2023, though I would be careful about being an early adopter.
**Joseph Reis**
Cron 😉
**Joseph Reis**
jk
**Matthew Housley**
😱
**Ricky McMaster**
Yeah this was something I was thinking about asking - what you both reckon about the data platforms like Snowflake and Databricks. Now I know.
**Nakul Bajaj**
Thanks so much for the reply Joseph Reis and Matthew Housley..
Love airflow, used it as a managed service mostly.
Started testing out Prefect..
Any serverless orchestrators or serverless frameworks suggested? And your thoughts on these compared to deployments..
**Matthew Housley**
Thank you!
**Joseph Reis**
Howdy Matt!
**Matthew Housley**
I just joined the channel.
**Rosona**
Hey Joseph Reis and Matthew Housley . From the “look in the book” preview, this looks like a bird’s eye view, readable like a book book instead of a “bam, here’s a GitHub repo, get your hands dirty” book. I’m here to ask if that changes mid book or if I’ve understood the vibe and writing style correctly.
Also, thanks for coming here to answer questions.
**Joseph Reis**
Part 1 sets the tone. Part 2 gets decently technical, but in an approachable way.
There are no github repos or code exercises.
**Matthew Housley**
We essentially wrote a book to compliment other technical resources. Data engineering is so vast that we felt like we had to take bird’s eye view. But we would suggest reading this book in conjunction with general engineering books (e.g. Designing Data Intensive Applications) and specific technology books based on the stack you’re working with. (E.g. Learning Spark or Stream Processing with Apache Flink).
**Rosona**
Excellent! Thanks for the very thorough answers.
**Gur Hevroni**
Hi Joseph Reis and Matthew Housley, thanks for writing the book and for taking questions here!
I have a couple of questions:
1. Do you cover in the book what would a data engineer role entails in different organizational settings? (corporate, startup, etc.). For example, I’m curious to know what would be the first order of business for the first data engineer hire in a 50-100 employees startup, compared to that joining one of the big-tech companies.
2. What are the set of skills required to become a successful data engineer? Would it be more similar to the skill set of a software engineer? a data scientist? a combination of both/others?
I’m looking forward to learn more about your book!
**Joseph Reis**
1. Yes, we discuss this quite thoroughly throughout the book. As the end of each chapter, there’s a “who you’ll work with” section. We also cover early vs mature companies, and how they should work with DE
2. There’s not a single answer to the question. A successful DE is the one who can keep the business and stakeholders happy :)
**Laura**
Hi Joseph Reis and Matthew Housley, nice to meet you here. I really enjoyed listening to your Super Data Science podcast episode. 👋
I have a questions as well:
What’s your advice for a Data Scientist trying to get more professional experience with cloud technologies? Doing some hobby projects and watching YouTube tutorials is a nice start, but I feel that I am still missing a lot of knowledge on how to actually run things in production. Do you have any advice?
**Joseph Reis**
Nothing like digging into cloud products, tinkering, and breaking things 🙂
**Joseph Reis**
On another note, to get a comprehensive view of a cloud (AWS, GCP, Azure), you might want to consider getting one of their dedicated certs. While some people might bash cloud certs, we think they’re useful for providing a baseline context and skills for that particular cloud. There’s what you think you know about a cloud, and then there’s the way the cloud wants you to use it. These aren’t always clear, and a cert can help clarify these things.
**Matthew Housley**
I second that, and I’ll add that most of the clouds have data engineering and machine learning specific certs. For example: Google Cloud Platform Certified Professional Data Engineer or the Google Machine Learning Engineer certifications.
**Joseph Reis**
We’re answering questions now. Feel free to drop new questions. We’ll answer until 5pm MT
**Cesar Garcia**
This is side question but… why are there almost no scholar references about data engineering? In the first chapter you talk about the story of data engineering (coming from DBs to Big Data to current situation) and the term is quite new. But, are academics still using BigData terms? Are there people outside O’Reilly / Packt writing about this topic? Is everything moving so fast that people don’t have time to publish about it?
**Joseph Reis**
Academics tend to be quite behind industry (we also teach at a university, so we know of which we speak). Data engineering is still a relatively new field, so it will take some time for the term and practices to solidify.
That said, read ch. 11 if you really want to be confused 🙂
**Cesar Garcia**
I will do it for sure! Thanks for the tip!
**Matthew Housley**
The mandate of academia is much more focused on theoretical problems. So, for example, computer science researchers are interested in general questions about distributed systems, where tech company engineers want to build working distributed systems to solve concrete problems.
**Matthew Housley**
A typical feedback loop between academia and silicon valley goes like this. Engineers build a distributed system and discover some kind of behavior. They write a paper about it. Academic researchers take the problem and develop a theoretical framework to explain it. Engineers then use this to improve their distributed systems.
**Matthew Housley**
There is definitely value in this feedback loop, but it’s a slow process.
**Matthew Housley**
This is not to say that _every_ academic research is strictly focused on theoretical problems, but this is the tendency. Professors who want to do concrete experimentation have to work very hard to stay up to date.
**Matthew Housley**
From my perspective, Martin Kleppmann has done a good job of staying up to date, partially based on his experience in the trenches.
**Matthew Housley**
Regarding incentives to publish: many technology employers don’t offer incentives for employees to publish. Worse still, employees may not be allowed to publish due to NDAs.
**Matthew Housley**
Google has done a good job of contributing back to the community through publications, but I’m sure they keep many things under wraps, and it’s possible that they’re less open now as a huge, mature company.
**Joseph Reis**
Re: q&a. We will try to answer all questions by 5pm MT every day of this book club. Ask away!
**Ricky McMaster**
Second question: I totally agree with your point about the importance of data engineers understanding business requirements from the stakeholders’ perspective. However, do you have any advice for junior engineers who do not have previous experience in more stakeholder-oriented roles such as data analysis, and whose main academic and career focus hitherto has been overwhelmingly technical? Would you consider it useful for such roles to be embedded in business unit teams?
**Joseph Reis**
Definitely. Embedding is a great way to build useful non-DE skills that actually make you a better DE. Learning the downstream user’s requirements builds empathy.
**Matthew Housley**
It’s a tricky problem to solve. To some extent, you’re at the mercy of the company that employs you, especially as a junior engineer. You’re stuck with their organizational dysfunction. Having said that, conversations go a long ways. Learn to have conversations with stakeholders you work with. Also, try to meet people in many roles across the company and talk to them about how they use data.
**Ricky McMaster**
Cool, good to know thanks. For me personally I come from a BI background so I’m definitely sold on the need for these conversations; it’s good to have it confirmed though that it’s important to prioritise the topic for data engineers, whether formally (e.g. embedding) or not.
I’ve often experienced quite a division between business and tech, but ultimately everyone loses if it persists.
**Grzegorz Sajko**
When it comes to breadth vs width when acquiring tech skills - when do you recommend to go deeper? What are your thoughts on specialization? As the tech landscape is changing very fast, is it better to be jack-of-all-trades?
**Matthew Housley**
I think you need to develop both general purpose and specific tech skills. If I’m looking to hire someone, I look for general purpose coding skills and what I’ll call “data intuition.”
**Matthew Housley**
I frequently find that software engineers who have mostly worked on services that handle single calls and events struggle when asked to handle bulk data. There’s a learning curve in transitioning from thinking of data as individual elements to viewing data as a large set.
**Matthew Housley**
This often leads to monstrosities at startups such as “ETL pipelines” that are a bunch of single event microservices stitched together.
**Matthew Housley**
I’ve seen many, many people make the transition from event level thinking to bulk data thinking, but there’s a learning, and ideally a potential hire has already made that transition.
**Matthew Housley**
If someone has general data skills, it’s relatively easy to retrain them on new data tools. If someone is good at Spark, they can probably learn other data frameworks with relative ease. If they know a realtime framework such as Flink, they can probably make the transition Beam.
**Matthew Housley**
So, my concrete advice is that you should experiment with data frameworks that you’re interested in. It’s critical that you move beyond frameworks that are traditionally single machine oriented (Pandas, R) to bulk data processing tools such as Spark and Beam. Also, get very good at SQL using database engines such as BigQuery, Snowflake or Redshift to solve analytics and data transformation problems. (SparkSQL is also great.) Even if you work primarily in Spark, SQL is an extremely useful tool, and it shows a potential employer that you have developed data intuition.
**Matthew Housley**
Also, as I mention in another reply, learn cloud infrastructure and orchestration skills. You can’t learn everything, but knowing one cloud and one orchestration tool is a good start for developing proficiency with other such tools.
**Grzegorz Sajko**
thanks! This is mine blind spot (single machine oriented / not cloud), because most personal projects don’t need scale.
**Alexey Grigorev**
Why so many data scientists are suddenly interested in data engineering? Is data science no longer the sexiest job?
**Erald David**
Curious to hear your perspective on this, Alexey Alexey Grigorev
Could this be because:
1. Many data scientists realized that a lot of company who hired them don’t have the sufficient data, so decide to learn data engineering also (sort of become fullstack), or
2. A lot of data scientist wanna be realized they fall in love with the engineering side of data science, so they flock to a more “engineering”-y position like data engineer or even analytics engineer (like [this great post](https://benn.substack.com/p/why-do-people-want-to-be-analytics)
from Benn Stancil)?
**Roman Zabolotin**
As for me, I think than I like work with data and I like programming too.
I think that working as a data scientist is more like research, math, science and so on.
But if your interest is programming, that data engineering will suit you more 😊
**serdar**
I am not even a data scientist but trying to make a change but in my last couple DS interviews I heard about interviewers complaning about the mediocre skills of the DS people. I guess those who really understand the needy greedy stuff find a new direction with DE. Just a thought..
**GerryK**
Maybe DS needs sector expertise related to data. DE has more software / tools/ infrastructure.
**Matthew Housley**
See the thread above on becoming “recovering data scientists.” [https://datatalks-club.slack.com/archives/C01H403LKG8/p1660777107448379?thread\_ts=1660703899.890469&cid=C01H403LKG8](https://datatalks-club.slack.com/archives/C01H403LKG8/p1660777107448379?thread_ts=1660703899.890469&cid=C01H403LKG8)
**Matthew Housley**
But I’ll add that increasingly, data teams are expected to put models into production.
**Matthew Housley**
At the height of the data science craze a few years back, data scientists would create a simple model on their laptop, hand some magical insight to a business stakeholder, and move on to the next project.
**Matthew Housley**
At least, that was the dream. In practice, businesses discovered that the value of their data initiatives was limited if they couldn’t get models into production.
**Matthew Housley**
Let’s led to a much greater emphasis on data engineering and ML engineering. As such, there seem to be more data engineering openings than data science openings, and data engineer demand far outstrips supply. (We’ll see if this trend holds with ongoing economics shifts.)
**Matthew Housley**
This blog post provides a great visualization of the foundations required for successful data science.
**Matthew Housley**
[https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007](https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007)
**Matthew Housley**
At any rate, I would argue that the prestige of data science and ML have actually increased in the last several years, but this has led to companies emphasizing better support structures for these initiatives, hence the demand for data engineers.
**Ning Wang**
My lab mates in college are mostly Data sciences, but I was an SWE before and a data engineer now (I am more interested in coding than building/tweaking data models). Personally I think with basic/standard tools, it could be very challenging to analyze most raw data. DEs build systems (standard or dedicated) to further process data so that DSs can get A LOT more values from the data and be A LOT more productive.
**Avinash M**
Joseph Reis would you suggest this book to a newbie in data engineering field like me?
**Joseph Reis**
yep!
**Avinash M**
Thank you!
**Aayush**
Thanks for this great book!
* How, according to you, should one approach Data Engineering at Reasonable scale, ie, at places that have data but not necessarily at terabyte scale? Do you think there is a need of implementing any modern data architecture at such organizations?
* At what point does one realize that they have reached a place where they now need specialised Data Engineers to look after their data needs?
* Currently, there is a heavy reliance on tool based learning for Data Engineers - learn SQL, learn spark, kafka and you are good to go. Do you think the heavy focus on tools rather than the Fundamentals of DE is a concern?
**Kevin Kho**
That third question is a very good question. Though SQL is universal so it’s not the same level as Spark and Kafka. Wanna hear their thoughts.
**Matthew Housley**
Regarding the first bullet point:
**Matthew Housley**
A really great thing about modern data engineering is that many of our tools now scale smoothly from gigabytes to petabytes.
**Matthew Housley**
Off the shelf tools such as Spark, BigQuery, Snowflake, etc. can quickly process a few megabytes or run a 1 PB query. It’s just a question of allocating resources at the correct scale for the problem at hand.
**Matthew Housley**
The relative simplicity of using these tools means that I generally advise organizations with small data problems to spin up one of these options rather than using self hosted Postgres (for example). The reason is that using a cloud based tool decreases operational overhead, and delivers better uptime and consistency, key considerations at any data scale.
**Matthew Housley**
Regarding the second bullet point: because scaling data is now relatively easy, the main considerations for dedicated data engineering talent have shifted somewhat.
**Matthew Housley**
I would ask these questions to assess the need for dedicated data engineers:
**Matthew Housley**
1. What are the quality expectations for the data? How difficult will it be to maintain quality? Are you ingesting data that is relatively dirty and complex, requiring complex pipelines for cleaning?
2. What is the expected service level agreement for the data?
3. How many different data sources are you handling? (Note that this is a separate question from the _size_ of the data.)
4. How quickly will new data sources be onboarded in the future?
5. How sensitive is the data? Are you handling data with proprietary company secrets or sensitive personal information?
**Matthew Housley**
Any such requirements can be a motivation for hiring dedicated data engineers even if the data is only gigabytes in size. One major problem with the modern data stack and the cloud is that it made data too easy, so that people with no security and engineering qualifications ended up causing data breeches or providing incorrect business data to business stakeholders.
**Matthew Housley**
Regarding the last bullet point, see my response to Grzegorz Sajko above.
**Srik**
Joseph Reis and Matthew Housley - Do the contents of the book vary by each region they are released? Example - [Amazon.com](http://amazon.com/)
indicates FoDE is 440 pages, [amazon.in](http://amazon.in/)
list it as 452 pages.
**Srik**
btw kindle edition has 740 pages
**Matthew Housley**
Not to my knowledge — I believe the editions are just formatted a bit differently. The Kindle version paginates into smaller pages, and the Indian version may have extra back or front matter pages. (I actually have a copy of this edition. I’ll have to check.)
**Srik**
Thank you
**Arianna Cooper**
Hi Joseph Reis and Matthew Housley! 😄 Given the years of experience and knowledge you both have in the industry, what have you both seen to be the best practices when it comes to senior level data engineers growing their early career level folks on their teams? What are the green or red flags that companies have demonstrated that show you that they really value (or don’t value) growing their junior and mid level data engineers? (I ask since I just graduated!)
**Joseph Reis**
Green flags - Showing you best practices and coaching you while you’re growing. Assigning you work and telling you what needs to be done, letting you figure out how. Stepping in to coach you when you’re astray or beyond stuck.
Red flags - No support or direction.
**Arianna Cooper**
Gotcha. Thank you so much for your input, really appreciate it!!
**Ning Wang**
Hi, Joseph Reis and Matthew Housley Thanks for writing the book! As a data engineer myself, I think the job is relatively new comparing to many other computer jobs, and many people may not have the right idea about what it is and what it can/should do. A bird-eye view could be very helpful for data engineers and other related people.
My question is: from some of your previous answers, it seems to me that you are supports of “data sharing”. So what are the main values of “data sharing” in your mind? In the high volume and high efficiency scenarios, what are the main points in your mind to justify “data sharing” vs efficiency? In the architecture in my company, we cares a lot about performance and efficiency, it makes data sharing to be trickier in my mind, so I am wondering what you think.
Finally, good luck to your book!
**Ning Wang**
And btw, the “Cut through marketing hype” part reminds me why I left my previous company. LOL.
**Matthew Housley**
Data sharing is generally used to share data in a database system that separates compute and storage. For example, BigQuery and Snowflake support flavors of data sharing, and Amazon S3 can be used for data sharing in a data lake environment.
**Matthew Housley**
The basic idea is that you grant another user, team or organization access to specific data, such as a dataset in BigQuery, a “share” in Snowflake or a specific set of objects in S3.
**Matthew Housley**
The users on the other end then spin up their own compute to consume the data as they wish. This saves you the trouble of having to grant access to account and clusters — they get read only access to the stored data itself.
**Matthew Housley**
Data sharing is a great way to publish datasets publicly. For example, a good deal of government data is available through shared BigQuery public dataset.
**Matthew Housley**
In addition, data sharing facilitates cross organizational collaboration. For example, you may be working with a partner ad tech company that needs access to certain data to create models. Data engineers build pipelines to appropriately scrub the data and load it into the shared datasets.
**Matthew Housley**
Finally, data sharing is interesting from the perspective of Zhamak Dehghani’s data mesh concept. Individual teams build pipelines to prepare data for external consumption, then share the data product across the company without granting any access to pipelines.
**Ning Wang**
Thanks for the details! Totally makes sense.
**Alexey Grigorev**
For data scientists who want to go into data engineering, what should they do apart from reading your book?
**Doink**
| | |
| --- | --- |
| enroll in <#C01FABYF2RG | course-data-engineering> 😉 ? |
**Joseph Reis**
What Doink said 😉
**Joseph Reis**
Also, build your network of people who can get you in front of great job and project opportunities.
**Joseph Reis**
Thanks everyone!
**Ricky McMaster**
Another one: something I have experienced more often than not is years-old, accumulating technical debt as a result of poorly designed and maintained operational/application databases. Sometimes, this is compounded by a switch to a cloud data warehouse, but with even less data integrity given how flexible they are in this respect.
I definitely acknowledge tools such as dbt filling the gap in data quality maintenance, but meanwhile do you detect a genuine rediscovery of solid relational database modelling principles, which are in large part decades old?
**Joseph Reis**
Data modeling is making a comeback for sure. Learn it, love it.
**Matthew Housley**
Unfortunately, technical debt is just a part of the job. Even for forward looking organizations that embrace new technologies, it takes time to retire old systems, and you don’t necessarily have the authority to accelerate this process. Interfacing with old systems is often a data engineering responsibility.
**Ricky McMaster**
\> Unfortunately, technical debt is just a part of the job.
Tell me about it… I actually don’t come from a technical academic background, and I would love to know if data modelling was given more of a priority in computer science degrees generally these days.
**Joseph Reis**
I’ve never seen CS degrees teach data modeling, at least as it pertains to analytics. A database class might teach relational algebra and the normal forms of relational data modeling.
**Ricky McMaster**
\> A database class might teach relational algebra and the normal forms of relational data modeling.
Yup - I feel like the correct procedures for even 3NF modelling are often overlooked though (at least from my perspective in Germany).
I don’t doubt that it’s part of my job to deal with the legacy issues, but I’d love to see modelling making a comeback for sure.
**Joseph Reis**
It’s hard. The amount of laziness to put in the hard work is high
**Ricky McMaster**
Haha well there is that
**Alexey Grigorev**
How important is it for data engineers to know how to build a dashboard? Usually it’s more a job for an analyst
**Joseph Reis**
While building a dashboard isn’t a direct requirement for a DE, if a DE knows how to build a dashboard, I think it helps the DE understand how to structure the data for delivery to the analyst. Zero downside for a DE to know dashboard basics
**Matthew Housley**
I second that. And learn to communicate with the people building dashboards so you can be better at your job.
**GerryK**
Hi Joseph Reis and Matthew Housley,
1. Are you touching on data validation during the ETL/ELT?
2. Do you use coding examples?
3. Are you touching on unit test /integration / functional tests for data pipelines?
**Joseph Reis**
1. Yes, this is covered throughout the book
2. No coding examples
3. We touch on testing throughout the book
**GerryK**
Sounds good!
**Arnthor S**
Hi Joseph Reis and Matthew Housley, thanks for doing this! I’m currently reading Designing Data-Intensive Applications, and wondering what the difference between the books are and if I should read FoDE when I’m finished with DDIA? Any important/interesting chapters/areas in this book that are missing in DDIA that you can highlight? I think I read somewhere here (can’t find it now) an answer from Joseph Reis where you recommended reading this book first and then DDIA, so maybe I should pause DDIA and start this one now?
**Joseph Reis**
DDIA shows you the ins and outs of building distributed systems. FoDE is oriented toward the big picture of data engineering. You can think of FoDE as the prequel to DDIA, and very much orthogonal to DDIA’s direction. If you’re a data engineer, I’d say read both, starting with FoDE.
A common thing I hear about DDIA is it throws you off the deep end quite quickly. FoDE will make your experience with DDIA much nicer.
Sidenote - DDIA authro Martin Kleppman was one of our tech reviewers
**Matthew Housley**
DDIA is important for two reasons in data engineering. First, if you work for a big tech company, you may be responsible for working on the guts of large scale distributed data systems. This book explains how the sausage is made. Second, if you work more with off the shelf technologies, you still need to understand how they work so you can debug problems.
**Arnthor S**
Thanks!
**Sergio Rozada**
Hi Joseph Reis and Matthew Housley, quick question here, what do you think about ML Engineers with core capabilities of doing Data Engineering? Unicorns? Better to have competences split into different roles?
**Joseph Reis**
Split the competencies into different roles. All in one is definitely a unicorn.
**Matthew Housley**
There’s also an ongoing debate about which responsibilities belong under ML engineering versus data engineering. The exact boundaries will be determined by the company you work for.
**Varun Nayyar**
Hey Joseph Reis and Matt.
Thanks for visiting here and taking the time out to answer our questions.
I just have a few teeny ones.
1. Would you recommend this book to someone with a little to no knowledge about data engineering practices?
2. Which cloud provider would you recommend as there is a lot of diverse advice in the market right now, keeping in mind future relevance?
3. What is a good starting point for data engg. Other than python and pandas? Or something entirely different?
4. How influential are data engineers in contributing to growth of projects? Is their role undermined by someone say a data scientist?
Thanks for your presence here.
Wish you great luck for your book.
**Joseph Reis**
1. Definitely. This book is geared toward people new to the field. That said, very senior DE’s have said they’ve also learned a ton of new stuff from FoDE.
2. I can’t suggest a particular cloud provider. Pick the one you’ll get most traction with (job prospects, joy, etc)
**Joseph Reis**
1. A good starting point for a DE? Read our book and find out 😉
2. The influence of DE’s largely depends on the team they’re operating in. DS’s shouldn’t undermine the role of a DE, and vice versa.
**Matthew Housley**
Regarding 2: GCP has great technology, but still somewhat limited mindshare. Azure appeals to companies that utilize Microsoft products; they are growing extremely fast and investing heavily in their data offerings. AWS still seems to be the mindshare leader for Silicon Valley engineers.
**Matthew Housley**
Regarding 4: many projects initially need data engineering more than they need data science. For example, customer facing analytics for a large number of customers can have a big impact on the growth of a SaaS platform, even if these analytics are relatively simple.
**Matthew Housley**
In the days of peak data science a few years back, companies hired data scientists like crazy, but they were stymied by a lack of data engineering. Now, there’s a recognition that data engineering acts as a catalyst for the success of data science and machine learning.
**Varun Nayyar**
Thanks for the responses.
Looking forward to reading your book.
**Joseph Reis**
Great questions so far. Keep ‘em coming.
**Eric Sims**
Not directly related to your book, but how did you each find yourselves working in data engineering? I imagine it wasn’t called DE then, and you probably didn’t start out expecting to write a book about it someday. Where did you start? Where did you think you were going? And why did you end up here?
**Matthew Housley**
I have a PhD in math, and my research was on the pure side of the discipline. That is, theoretical research problems generally not related to statistics, data, computer science, etc. When I started, “big data engineer” was a hot title, with an emphasis on managing on-premises Hadoop clusters and writing map reduce jobs. However, EMR (Amazon’s managed Hadoop service) and Redshift (Amazon’s managed columnar database) were starting to take off.
**Matthew Housley**
Both Joseph Reis and I refer to ourselves as recovering data scientists because we started out in data science but organically became data engineers because we needed to build data pipelines in order to deliver projects we were working on.
**Matthew Housley**
In my case, I worked on a number of cloud oriented data project and spotted that as a long term career growth opportunity.
**Joseph Reis**
I’ve only really worked with data in some capacity or another. My path has been very circuitous in the details, but the general direction has always been there for over 20 years.
**Philip Dießner**
Hello Matthew Housley and Joseph Reis Thanks for being here!
Following up on a previous question, what are good ways to learn about data modeling, especially when to use one or another modeling method? (Besides reading the section in your book 😉 )
**Joseph Reis**
A lot of trial and error. I suggest learning the big ideas, such as dimensional and relational modeling, and applying those to real world datasets. For example, you can use BigQuery’s numerous public datasets and experiment with various ways to model them.
**Alber Novo**
* Joseph Reis in a previous thread, you shared how you stay up to date by reading a lot. Could you share some of the sources you have found reliable?
* Matthew Housley, you mentioned about getting a cloud certification in another thread. In your opinion, which one would you recommend as a start point, AWS or GCP (Just to clarify, I don’t work in the area, but I’m considering work in this area eventually).
* And a question about the book, on chapter 11 you mentioned about emerging tools boosting spreadsheets with OLAP systems, could you mention some of the candidates for this new class of tools you’ve seen? Thank you for writing this book and taking time to answer questions here :)
**Matthew Housley**
Hmm… this is a tough question to answer. Personally, I prefer GCP’s data tools, but AWS certs will potentially give you access to more jobs given their massive mindshare.
**Matthew Housley**
Regarding our speculation on spreadsheets and new interactive data paradigms: there are various SaaS scalable spreadsheets, such as [https://www.gigasheet.com/](https://www.gigasheet.com/)
. We’ll see how they do in the marketplace.
**Matthew Housley**
Beyond that, I suspect that we’ll see a lot of interesting developments in terms of interactive analytics paradigms in the next few years.
**Alber Novo**
Thanks Matthew Housley. If it’s tough for you to give an opinion, you can imagine how hard it’s for me to decide 😅. I appreciate your input; I’m also inclined to start with GCP.
**Idil Ismiguzel**
Hi Matthew Housley and Joseph Reis thanks for being here and writing this wonderful book. You mention modern data engineering profession exists to serve downstream data consumers such as data scientists and ML engineers, and boundaries between these three roles are often blurry and depend on the organization’s data maturity. I was wondering your opinion on the main overlaps between ML Engineering and Data Engineering. In which areas an ML Engineer should be as expert as a Data Engineer even though the organization has the split between these two roles?
**Muhammad Awon**
I was trying to figure this out today but couldn’t find much help, glad you ask the question.
Thanks.
**Joseph Reis**
Look at the similarities - both DEs and MLEs need to move data between systems and storage. Additionally, they need to store and serve it.
The differences are the MLE needs to know ML pretty well, as that’s the use case the MLE is serving. The DE might also need to know ML, if that’s also the use case the DE is serving. And herein is where the lines get blurry…The big difference is the MLE focuses more on the ML lifecycle, which is quite distinct from the DE lifecycle
**Matthew Housley**
And MLEs build a lot of data pipelines. Each organization has to decide which pipelines are owned by MLEs and which by DEs.
**Joseph Reis**
is MLDE a job title yet?
**Grzegorz Sajko**
[https://www.getsphere.com/ml-engineering/data-engineering-for-machine-learning?source=ITP](https://www.getsphere.com/ml-engineering/data-engineering-for-machine-learning?source=ITP)
DE for ML
**Idil Ismiguzel**
Thank you for your answers!
**Joseph Reis**
Thanks for your questions and interest!
**Jk Jensen**
Thanks for putting in the effort to make a quality contribution to the space Matthew Housley and Joseph Reis! I’m curious what you see as the biggest problems to be solved in the Data Engineering space with regard to privacy? I am recently coming from the privacy infrastructure world and I would love to see an increased focus in this community on protecting the data we use.
**Joseph Reis**
There are no shortage of problems, that’s for sure. Privacy is becoming more top of mind, but it’s still a relatively young consideration for DE’s. If this is your specialty, you’ve got a bright future ahead of you!
**Matthew Housley**
In some respects, the modern data stack has increased issues with privacy by making data too easy. Basically, anyone handling sensitive data needs appropriate training, experience and best practices to do so. Cutting corners can lead to disaster.
**Matthew Housley**
Beyond that, I’m excited to see the continued emergence of automated sensitive data detection tools. One example is GCP DLP (data loss prevention).
**Matthew Housley**
Always follow best practices and be on the lookout for sensitive data, but automated tools can help to prevent human error and oversights.
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---
# Skills of a Successful Software Engineer – DataTalks.Club
AI Dev Tools Zoomcamp: Learn AI-powered coding assistants and agents [Register here!](https://airtable.com/appJRFiWKHBgmEt70/shrpw7rk55Ewr1jCG)
DataTalks.Club
--------------
Skills of a Successful Software Engineer
----------------------------------------
#### by [Fernando Doglio](https://datatalks.club/people/fernandodoglio.html)
##### The book of the week from 12 Sep 2022 to 16 Sep 2022
Skills to grow from a solo coder into a productive member of a software development team, with seasoned advice on everything from refactoring to acing an interview.
In Skills of a Successful Software Engineer you will learn:
* The skills you need to succeed on a software development team
* Best practices for writing maintainable code
* Testing and commenting code for others to read and use
* Refactoring code you didn’t write
* What to expect from a technical interview process
* How to be a tech leader
* Getting around gatekeeping in the tech community
Skills of a Successful Software Engineer is a best practices guide for succeeding on a software development team. The book reveals how to optimize both your code and your career, from achieving a good work-life balance to writing the kind of bug-free code delivered by pros. You’ll master essential skills that you might not have learned as a solo coder, including meaningful code commenting, unit testing, and using refactoring to speed up feature delivery. Timeless advice on acing interviews and setting yourself up for leadership will help you throughout your career. Crack open this one-of-a-kind guide, and you’ll soon be working in the professional manner that software managers expect
* [Book's page](https://www.manning.com/books/skills-of-a-successful-software-engineer)
* [Buy on Amazon](https://www.amazon.com/Skills-Successful-Software-Engineer-Fernando/dp/1617299707)
Questions and Answers
---------------------
**Ricky McMaster**
Hi Fernando Doglio, thanks a lot for doing this! Just wondering first of all how much you address effective communication within teams, and between tech and business?
**Fernando Doglio**
hey. Ricky, great question! I think I mention the importance of communication throughout the book, but I really make a point of it by the end, chapters 7 and 8. During chapter 7 I cover the importance of communication with your team and your manager in multiple places. As for the “business”, I do cover the subject of talking with clients from a consultant point of view. I do not cover the importance of effective communication with the business inside the company, if that’s what you were asking for.
**Ricky McMaster**
Thanks Fernando, that’s good to know.
**James Gough**
Hi Fernando, congrats on the book. It looks perfect for me and I think I will buy it so count me out of the free copy! 🙂 I have read Clean Code and am currently reading Code Complete. I chose these as I wanted to know wanted to approach software in the same way as others and given they’re foundational texts they seemed a good place to start. Does your book build on/update the best practices in these books or have best practices evolved a lot since they were written? Also, any criticism of these books and approaches I’d be interested to hear. I think older texts can be quite gate-keepy in their own way and project a very ‘there is only one right way’ kind of vibe.
Also, do you touch on how to get started on contributing to open source? That is something I think a lot of people (me included) feel intimidated about, especially if you’re not professionally part of a software engineering team.
**Fernando Doglio**
Hi James, thanks for the question! I honestly can critique those books since I haven’t read them. While I do have some chapters that cover coding and best practices, the focus of my book is not “just that”. So I do go into some details about best practices in general, and in regards to unit testing, but then I keep going into other topics.
**Fernando Doglio**
And as for your question of whether or not best practices evolve over time, I tend to agree there. Different architectural patterns either created, or re-discovered over time, and that brings light into different ways of tackling the same problems.
**Fernando Doglio**
when it comes to soft. development I would never say “there is one way to do this right” about anything really.
**James Gough**
Thank you!
**Di\_hash**
Hi Fernando Doglio , Thank you for giving us this opportunity to discuss this topic cuz i realize most people focus only on coding part and ignore other skills.
1 - Do you consider your book a technical book, non-technical or both?
2 - What are the few skills that you gathered much later tahn necessary?
3 - software engineering is more than memorizing algorithms, it’s about being able to work well with your team and be able to communicate your solutions well.
How do you see this saying, and is this what your book is about?
4 - If i hone skills/practices mentioned in your book, would i bridge the gap between being decent SWE to a good/great one?
Finally, wish you best of luck for your book, and thanks for your time.
**Fernando Doglio**
Hey Di\_hash! Great questions!
1. It’s actually both, I do cover some tech points, but most of them are not technical, the focus is the overall journey of a developer.
2. The main one would be communication. I neglected it thinking I should only focus on my tech skills. Eventually I realized that without being a good communicator, I would not be able to grow in my career.
3. Yeah, I’d say that this is one of the core principles at the center of my book.
4. Well, I certainly hope so! 🙂 Of course, it’s all up to you, reading the book alone will not make you better, you have to interiorize the teaching, and apply the lessons. It will definitely help you jump a few years ahead of everyone else at you same level, that’s for sure.
**Di\_hash**
Thank you, i appreciate your clarifications
**Fernando Doglio**
thank you for the great questions!
**Saba Azad**
Hi Fernando Doglio , thanks for the opportunity to ask you questions about your book.
My question is about ‘gatekeeping’ in tech. How to navigate through that? If a person who started late in tech and doesn’t have many years of experience, but they can do the work, how would they survive?
**Fernando Doglio**
Hey Saba, that’s a tough one. Mainly because the way I deal with “gatekeeping” is by ignoring it, but I understand not everyone can do it. My advice would be to keep reaching out to people until you find a positive support group. Internet is filled with developers looking to help others, some even specialize and create a lot of content and have a lot of advice for people switching careers into tech, if you run into gatekeepers, just turn around and keep looking.
**Fernando Doglio**
And if you don’t know where to look for these people, I would suggest reaching out to content creators that produce content around your areas of interest and asking for supportive communities, they’re probably exposed to multiple ones.
**Alber Novo**
Hi Fernando Doglio, thank you in advance for answer our questions 🙂
It seems to me that software development is getting further away from low-level programming to a high-level. Before, knowing how hardware and Operating Systems work was a must for a great programmer. Today it seems to be all about frameworks and how to connect them. All complexity is being hidden and forgotten (it may be good or bad:man-shrugging:). I know my statement is very opinionated, but it gives the context to my questions:
1. How do you see the future of software development?
2. What is your advice to someone pursuing a career in software development to be focused, and at the same time, be relevant in the market when a new and exciting framework is in demand every day!?
**Fernando Doglio**
1. This one is a tough one, because it depends on what you mean by “future”. I think on the long run, AI will definitely affect our jobs. Not replace us, mind you, but at least it will change the way we code. Allowing us to keep moving up on the abstraction layers, like you put it.
2. As for staying up-to-date with all the frameworks out there. that’s impossible, and I don’t think it should be anyone’s aim. Instead, you should aim to master the basics. No matter what your industry or area of expertise is, you should aim to understand the core of programming, and then you’ll see that picking up new frameworks becomes second nature. Focus on understanding how different paradigms work, learn about data structures and algorithms… learn about architectural patterns. The language is irrelevant and the framework even more. Once you know what MVC is (for example), you’ll start seeing it everywhere.
**Ashish Lalchandani**
Hi Luca Massaron Konrad Banachewicz, thanks for being here! My question is, how to get started with Kaggle? Is it something like Leetcode where there’s a problem statement and we start coding? As a beginner, do we just dive into ongoing competitions or practice a bit with older competitions? I am really wondering on how to get started with Kaggle, want to test my knowledge of basic concepts, and learn new things by reading others’s code, but don’t know how to get started.
**Doink**
Hi Fernando Doglio thank you for writing this book, I have one question, how to avoid getting burned out with the myriad of things to learn especially in ML where the rate of newer ideas are a lot. How to get optimal work-life balance?
**Fernando Doglio**
there is no magic formula for a balanced work-life scenario. Sadly. Everyone has different burnout limits, and I think it’s important to be aware of them.
The main thing is to remember that you’re need other things in your life other than your work to be happy, and that it’s OK to not know everything.
At the rate at which new technologies are developed, it’s impossible for a person to know everything and to have experience with everything, so you have to be able to draw that limit and say “here, I’m good here”, and learn to enjoy things other than coding, that way hobbies can take you out of your “work” mental space.
**Moritz Wilke**
Hi Fernando Doglio thanks for presenting your book. I am currently (re-)reading “The pragmatic programmer” which presents a similar “holistic” approach to writing software (albeit it seems to be a bit more focussed on technical patterns) and I enjoy it very much. What was your personal motivation for writing this book? Sections like “getting around gatekeeping in the tech community” sound as if you had to overcome very unproductive situations on your way? Do you “only” present personal strategies to navigate these problems or do you have ideas on what has to change in the whole community/profession?
**Fernando Doglio**
Hey Moritz, thanks for the question. I guess we could say 90% of the book is based on personal experience but there are other areas where I just present my ideas on how it could be improved. Problems like gatekeeping are very visible if you’re active on any social network where many developers interact with each other, you can see so many people trying to help others and some developers trying to prevent them from moving forward by imposing their own ideas of whats “right” or “wrong”.
**Fernando Doglio**
In the case of gatekeeping, I tried to show what I think are the best ways to move around it and most importantly, to be aware of it, because sadly internet is not filled with developers helping developers.
**Hyoson Yamanaka**
Hi Fernando Doglio, thanks for doing this session. I am currently in a team where the lead comes from a non-technical background and all members are DSs/SWEs. Would you recommend your book to a person less familiar with software development that wants to understand how to manage such processes? I am curious if you have any tips and experiences on how to bridge the gap between technical team members and non-technical leads/ decision makers.
**Fernando Doglio**
Hey Hyoson, thanks for the question. I wouldn’t recommend my book to this person. I didn’t write it with that kind of profile in mind, and if they’re already leading teams with no intention on becoming a developer, then the book really wouldn’t do them any good.
That said, bridging the gap between a non-technical lead and team members is hard, because sometimes it might seem like they speak completely different languages. The lead will usually have concerns about timeline, product delivery, reaching milestones, and if they’re not technical they will have to trust their team when it comes to tech issues (like generating/spending time fixing tech debt, time estimations, bug reports and many other areas). This person will have to actively look for advice from the team, or at least, from the senior members of the team.
From the PoV of the team members, they’ll also have to try and find a way to have an open dialog and accept that this person’s experience leading teams, even though they lack tech skills, might help them be an effective leader regardless. Sometimes tech folk tend to not show the same level of considerations or respect to non-tech leaders than what they would show to a senior technical person who becomes a leader. That has to change, if this person is leading the team, they deserve the benefit of the doubt.
**Fernando Doglio**
At least that’s how I would try to approach the situation from either end.
**Hyoson Yamanaka**
I appreciate your thoughts on the matter! There is indeed some tension in our group due to the issues you mentioned and it hasn’t been addressed directly yet, which I think it should.
**Fernando Doglio**
yeap definitely, all cards on the table , that’s the way I like to do it anyway.
**Tim Becker**
Hi Fernando Doglio, thank you for this very interesting book and for this opportunity to ask questions.
* Do you have any advice concerning how to handle configurational data and transactional data that several teams have to use? For example, a data science team and a software development team? How should the interface between the teams be designed?
* What are the most important skills for a tech leader?
* As a tech leader, how can you ensure high quality in your production code and at the same time provide meaningful tasks for junior developers in your team?
* How did you improve your communication skills?
**Fernando Doglio**
great questions!
Let me start in reverse order:
1. Writing did the trick for me. I started writing articles online, and that helped with my english and my ability to explain complicated concepts in simple ways.
2. I’m not sure I understand this one, if by “high quality in your production code” you mean MY actual code, I can’t, but that’s because as a tech lead I don’t code. It’s been my experience that as a tech lead I’m too busy with others managing tasks (estimations, client management, team management, etc) to be able to have enough time to code myself. Considering all the tasks I have at hand, it would be irresponsible to put coding tasks on my backlog as well, I wouldn’t be able to deliver them on time.
3. I would say they’re not technical. Communication and empathy are two big ones on my book. Tech people tend to be really bad at those, some of them like machines more than people, so they don’t focus on developing their communication skills, specially early on. Being able to interact with them in a way they feel heard, and then turn around and talk to the business in a way they understand what’s happening is a key skill any lead should have.
4. I’m not sure I understand this question either, can you elaborate a bit more? Maybe have a sample scenario?
**Tim Becker**
Fernando Doglio thank you for your answers. Seems like I wasn’t very precise in writing down my questions 😉
* Concerning question 3, yes, probably the lead does not code anymore. However, he/she will have to ensure that the products are running at all times, that there are no bugs and that the code stays maintainable. It seems to be that many junior developers are struggling with these objectives. How can you help junior developers grow and get better and avoid creating a mess in your products on the way?
* Concerning question 1, as an example, if your organisation has a data science team and a team with developers. The data science team creates models which produce predictions that should be displayed in a web application with a nice user interface. The web application is created by the developers. The data science team provides an API for the developers to use. For this purpose, a lot of data has to be shared between the teams. For example, how should the predictions be presented? Also the model might change over time, the inputs and outputs might have to be adjusted from time to time which needs to be communicated between the teams. How would you manage the interaction/communication between the teams. Or would you design the whole setup differently?
**Dustin Coates**
Hi Fernando Doglio thanks for the Q&A, and hello from a fellow Manning author.
My question for you: are the skills of a successful SWE different depending on the field or type of technology you work with? E.g. is someone working on maintaining old COBOL applications going to need different skills compared to a full-stack web developer or someone working on SFDC.
**Fernando Doglio**
Hey Dustin, great question! Interestingly enough, I cover this topic on chapter 1 of the book. The whole premise for the book is that the skills that make up a successful SWE are not technical. In fact, they’re mostly soft-skills. Technical skills differ from project to project, but you’ll always be dealing with people.
**Ajay Muppuri**
Hi Fernando Doglio thank you in advance 😊 What advice would you have for someone who is already a tech lead, so that they can keep up with the evolving technologies and still be good at their job of not just writing good code but also teaching others to do the same.
**Fernando Doglio**
Hey Ajay, great question. I would suggest starting to create technical content. My go-to would be written content, but if the aim is to stay close and up-to-date with code, you can do anything you like. Through this process you’ll be able to play around and get to know the latest trends in tech and you’ll have to become great at explaining concepts and transmitting that information to others.
This way, you get to see and experience new technologies without having to spend months working with them. If you have enough experience as a developer in general, that should give you all the tools you need to keep being a great leader and avoid falling behind trends your team is not using right now.
**Ajay Muppuri**
thank you 🙂
To take part in the book of the week event:
* [Register in our Slack](https://datatalks.club/slack.html)
* Join the `#book-of-the-week` channel
* Ask as many questions as you'd like
* The book authors answer questions from Monday till Thursday
* On Friday, the authors decide who wins free copies of their book
To see other books, check the [the book of the week](https://datatalks.club/books.html)
page.
Subscribe to our weekly newsletter and [join our Slack](https://datatalks.club/slack.html)
.
We'll keep you informed about our events, articles, courses, and everything else happening in the Club.
Email
Join
* * *
DataTalks.Club. Hosted on [GitHub Pages](https://github.com/DataTalksClub/datatalksclub.github.io)
. We use cookies.
---