Modin dataframes and IBM Cloud Object Storage

-
Modin is a  Python framework capable to efficiently scale Pandas dataframe.  To achieve this Modin uses a high performance distributed Ray framework. This short post explains how to use  Modin and read data objects from IBM Cloud Object Storage.

Requirements

IBM Cloud Object Storage account

If you doesn't have one already, navigate to IBM Cloud and choose IBM Cloud Object Storage. Using dashboard, create a new bucket and upload some CSV objects there. You will need to obtain HMAC credentials for the bucket, just follow simple steps as described here

Python and dependencies

I used Python 3.6 but i assume other versions will work as well.  Install the following packages:  IBM COS SDK for Python, smart_open (we will use smart_open to access IBM Cloud Object Storage) and modin

Example


import modin.pandas as pd
import ibm_boto3
import smart_open

if __name__ == '__main__':

      access_key = 'ACCESS KEY'
   secret_key = 'SECRET KEY'
   # service endpoint is the URL where your bucket was created. Please use without http prefix
   # for example:
   service_edndpoint = 's3-api.us-geo.objectstorage.softlayer.net'

   session = ibm_boto3.session.Session(aws_access_key_id=access_key,
             aws_secret_access_key=secret_key)
   # access bucket "mybucket" and read an object "dataset/data.csv"
   my_df = pd.read_csv(smart_open.smart_open('s3://mybucket/dataset/data.csv',
                            host = service_endpoint, s3_session = session))
   print (my_df.columns)

Comments

Post a Comment

Popular posts from this blog

Brew your beer at home - cooling the wort

-
Cooling the wort is a very important phase in the beer making process. Once you finished boiling the wort, it need to be cooled  to reach about 24℃ as fast as possible. The faster you cool the wort, so less it will became contaminated with unwanted bacterias.  I tried all kind of approaches to cool the wort.  I initially bought a cooling spiral. While on the paper this was a perfect approach, in reality it was a waste of money for me. The main problem was, how to flow the cold water throw the cooling spiral. Connecting spiral to water tap is not good  enough as the water from the tap is not so cold.  I then tried to put a water tank filled with ice above the beer vessel, connect one pipe to the water tank and siphon water from the second pipe.  This allowed to water flow from the tank throw cooling spiral, while i collect water on the other side. This was still complicated and from time to time pipes were disconnecting causing flooding. Investing in buying ...
Read more

What is Big Data and is it really so Big?

-
"Big Data" is a hot term this days. We all hear  "Big Data here",  "Big Data there", "Big Data everywhere".  All sounds fancy and yet mysterious a bit. But what is Big Data? Was there Big Data in the past or it was born recently?  What does "Big Data around us"  means?  Can we touch it? Obviously there is no a single definition for the Big Data and everyone may define it differently while also confusing the true meaning of this term. The funny part is that few decades ago a sentence " I have a challenge how to process our Big Data" - would probably imply that employee is not qualified enough. But saying the same this days sounds good and perhaps implies that employee is well qualified and working on the real challenges.  So what have changed in the last few decades? Imagine a first grade kid who just got his first homework assignment and he is about to share the shocking news with his parents "Mom, Dad! I got 10 pages ...
Read more