- Dask DataFrame is not Pandas - Nov 22, 2021.
This article is the second article of an ongoing series on using Dask in practice. Each article in this series will be simple enough for beginners, but provide useful tips for real work. The next article in the series is about parallelizing for loops, and other embarrassingly parallel operations with dask.delayed.
Dask, Pandas, Python, Saturn Cloud
- Data Engineering Technologies 2021 - Sep 21, 2021.
Emerging technologies supporting the field of data engineering are growing at a rapid clip. This curated list includes the most important offerings available in 2021.
Abacus.ai, Dask, Data Engineering, Databricks, Dataiku, DataRobot, dbt, Fivetran, Pachyderm
- If You Can Write Functions, You Can Use Dask - Sep 21, 2021.
This article is the second article of an ongoing series on using Dask in practice. Each article in this series will be simple enough for beginners, but provide useful tips for real work. The first article in the series is about using LocalCluster.
Cloud, Dask, Python, Saturn Cloud
- Speeding up Neural Network Training With Multiple GPUs and Dask - Sep 14, 2021.
A common moment when training a neural network is when you realize the model isn’t training quickly enough on a CPU and you need to switch to using a GPU. It turns out multi-GPU model training across multiple machines is pretty easy with Dask. This blog post is about my first experiment in using multiple GPUs with Dask and the results.
Dask, GPU, Neural Networks, Training
- Building Machine Learning Pipelines using Snowflake and Dask - Jul 28, 2021.
In this post, I want to share some of the tools that I have been exploring recently and show you how I use them and how they helped improve the efficiency of my workflow. The two I will talk about in particular are Snowflake and Dask. Two very different tools but ones that complement each other well especially as part of the ML Lifecycle.
Dask, Machine Learning, Pipeline, Snowflake
Pandas not enough? Here are a few good alternatives to processing larger and faster data in Python - Jul 8, 2021.
While the Pandas library remains a crucial workhorse in data processing and management for data science, some limitations exist that can impact efficiencies, especially with very large data sets. Here, a few interesting alternatives to Pandas are introduced to improve your large data handling performance.
Dask, Modin, Pandas, Python, Scalability
- Dask and Pandas: No Such Thing as Too Much Data - Mar 4, 2021.
Do you love pandas, but don't love it when you reach the limits of your memory or compute resources? Dask provides you with the option to use the pandas API with distributed data and computing. Learn how it works, how to use it, and why it’s worth the switch when you need it most.
Dask, Distributed Computing, Pandas
Are You Still Using Pandas to Process Big Data in 2021? Here are two better options - Mar 1, 2021.
When its time to handle a lot of data -- so much that you are in the realm of Big Data -- what tools can you use to wrangle the data, especially in a notebook environment? Pandas doesn’t handle really Big Data very well, but two other libraries do. So, which one is better and faster?
Big Data, Dask, Data Preparation, Pandas, Python, Vaex
- Distributed and Scalable Machine Learning [Webinar] - Feb 17, 2021.
Mike McCarty and Gil Forsyth work at the Capital One Center for Machine Learning, where they are building internal PyData libraries that scale with Dask and RAPIDS. For this webinar, Feb 23 @ 2 pm PST, 5pm EST, they’ll join Hugo Bowne-Anderson and Matthew Rocklin to discuss their journey to scale data science and machine learning in Python.
Capital One, Dask, Distributed, Machine Learning, Python, scikit-learn, XGBoost
- Computer Vision at Scale With Dask And PyTorch - Nov 23, 2020.
A tutorial on conducting image classification inference using the Resnet50 deep learning model at scale with using GPU clusters on Saturn Cloud. The results were: 40x faster computer vision that made a 3+ hour PyTorch model run in just 5 minutes.
Computer Vision, Dask, Python, PyTorch, Scalability
Pandas on Steroids: End to End Data Science in Python with Dask - Nov 6, 2020.
End to end parallelized data science from reading big data to data manipulation to visualisation to machine learning.
Dask, Data Science, Pandas, Python
- Data Science in the Cloud with Dask - Oct 20, 2020.
Scaling large data analyses for data science and machine learning is growing in importance. Dask and Coiled are making it easy and fast for folks to do just that. Read on to find out how.
Cloud, Dask, Data Science, Python
- Machine Learning in Dask - Jun 22, 2020.
In this piece, we’ll see how we can use Dask to work with large datasets on our local machines.
Dask, Machine Learning, Pandas, Python
- Why and How to Use Dask with Big Data - Apr 15, 2020.
The Pandas library for Python is a game-changer for data preparation. But, when the data gets big, really big, then your computer needs more help to efficiency handle all that data. Learn more about how to use Dask and follow a demo to scale up your Pandas to work with Big Data.
Big Data, Dask, Data Engineering
- Five Interesting Data Engineering Projects - Mar 17, 2020.
As the role of the data engineer continues to grow in the field of data science, so are the many tools being developed to support wrangling all that data. Five of these tools are reviewed here (along with a few bonus tools) that you should pay attention to for your data pipeline work.
Dask, Data Engineering, dbt, DVC, Python
- K-means Clustering with Dask: Image Filters for Cat Pictures - Jun 18, 2019.
How to recreate an original cat image with least possible colors. An interesting use case of Unsupervised Machine Learning with K Means Clustering in Python.
Clustering, Dask, Image Classification, Image Recognition, K-means, Python, Unsupervised Learning
- Introducing Dask-SearchCV: Distributed hyperparameter optimization with Scikit-Learn - May 12, 2017.
We introduce a new library for doing distributed hyperparameter optimization with Scikit-Learn estimators. We compare it to the existing Scikit-Learn implementations, and discuss when it may be useful compared to other approaches.
Dask, Distributed Computing, Distributed Systems, Machine Learning, Optimization, scikit-learn
- Dask and Pandas and XGBoost: Playing nicely between distributed systems - Apr 27, 2017.
This blogpost gives a quick example using Dask.dataframe to do distributed Pandas data wrangling, then using a new dask-xgboost package to setup an XGBoost cluster inside the Dask cluster and perform the handoff.
Dask, Distributed Systems, Pandas, Python, XGBoost
- Introducing Dask for Parallel Programming: An Interview with Project Lead Developer - Sep 7, 2016.
Introducing Dask, a flexible parallel computing library for analytics. Learn more about this project built with interactive data science in mind in an interview with its lead developer.
Analytics, Continuum Analytics, Dask, Data Science, Distributed Computing, Parallelism, Python, Scientific Computing
