A data scientist working with medical claims data faced memory issues when merging large datasets. With 64GB of RAM, the datasets, which are extremely sparse, resulted in over 100 million rows after merging. The Jupyter notebook kernel crashed due to insufficient memory. Dask, a tool designed for handling data that doesn’t fit into memory, was employed. The data was read from CSV files, repartitioned for Dask, and intermediate results were saved as Parquet files to manage memory. Despite these efforts, the final merge still caused a crash. The process involved reading 11 CSV files, merging them step-by-step, and saving intermediate results to disk to free up memory. However, the last merge step overwhelmed the system, suggesting a need for alternative tools or strategies for handling such large-scale data operations.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
