Handling and Converting Data Types in Python Pandas

Handling and Converting Data Types in Python Pandas

Data types are one of those things that you don’t tend to care about until you get an error or some unexpected results. It is also one of the first things you should check once you load a new data into pandas for further analysis.

Chris Moffit

In this short tutorial, Chris shows how to the pandas dtypes map to the numpy and base Python data types.

A screenshot of the data type mapping.

Moreover, Chris demonstrates how to handle and convert data types so you can speed up your data analysis. Both using custom functions and anonymous lambda functions.

A snapshot from the original blog.

A very handy guide indeed, after which you will be able to read in your datasets into Python in the right format from the get-go!

Using data type casting, lambda functions, and functional programming to read in data in Python. Via pbpython.com/pandas_dtypes.html

10 Guidelines to Better Table Design

10 Guidelines to Better Table Design

Jon Schwabisch recently proposed ten guidelines for better table design.

Next to the academic paper, Jon shared his recommendations in a Twitter thread.

Let me summarize them for you:

  • Right-align your numbers
  • Left-align your texts
  • Use decimals appropriately (one or two is often enough)
  • Display units (e.g., $, %) sparsely (e.g., only on first row)
  • Highlight outliers
  • Highlight column headers
  • Use subtle highlights and dividers
  • Use white space between rows and columns
  • Use white space (or dividers) to highlight groups
  • Use visualizations for large tables
Afbeelding
Highlights in a table. Via twitter.com/jschwabish/status/1290324966190338049/photo/2
Afbeelding
Visualizations in a table. Via twitter.com/jschwabish/status/1290325409570197509/photo/3
Afbeelding
Example of a well-organized table. Via twitter.com/jschwabish/status/1290325663543627784/photo/2
How most statistical tests are linear models

How most statistical tests are linear models

Jonas Kristoffer Lindeløv wrote a great visual explanation of how the most common statistical tests (t-test, ANOVA, ANCOVA, etc) are all linear models in the back-end.

Jonas’ original blog uses R programming to visually show how the tests work, what the linear models look like, and how different approaches result in the same statistics.

George Ho later remade a Python programming version of the same visual explanation.

If I was thought statistics and methodology this way, I sure would have struggled less! Have a look yourself: https://lindeloev.github.io/tests-as-linear/

The 10 Fundamental Concepts of JavaScript

The 10 Fundamental Concepts of JavaScript

Another pearl of a resource on Twitter is this thread by Madison on 10 of fundamentalal concepts of Javascript — and programming in general for that matter.

For your convience, I copied the links below. Just click them to browse to the resource and learn more about the concept

Click to learn more about each concept

  1. Variables & Scoping
  2. Data types
  3. Objects, Funtions & Arrays
  4. Document Object Model (DOM)
  5. Prototypes & this.
  6. Events
  7. Flow Control (specifically, for-loops)
  8. Security & (web) Accesibility
  9. Good coding practices (to which I’ve linked before)
  10. Async

This 10-step list was compiled as apart of this interesting podcast, which I recommend you listen to as well.

Want to learn more?

According to many, this is the best book to continue learning more about JavaScript.

There’s a (now classic) conference talk that comes with this book, which I can also recommend you watch:

Data Science vs. Data Alchemy – by Lucas Vermeer

Data Science vs. Data Alchemy – by Lucas Vermeer

How do scurvy, astronomy, alchemy and data science relate to each other?

In this goto conference presentation, Lucas Vermeer — Director of Experimentation at Booking.com — uses some amazing storytelling to demonstrate how the value of data (science) is largely by organizations capability to gather the right data — the data they actually need.

It’s a definite recommendation to watch for data scientists and data science leaders out there.

Here are the slides, and they contain some great oneliners:

@lucasvermeer
@lucasvermeer
Book tip: On the Clock

Book tip: On the Clock

Suppose you operate a warehouse where workers work 11-hour shifts. In order to meet your productivity KPIs, a significant number of them need to take painkillers multiple times per shift. Do you…

  1. Decrease or change the KPI (goals)
  2. Make shifts shorter
  3. Increase the number or duration of breaks
  4. Increase the medical staff
  5. Install vending machines to dispense painkillers more efficiently

Nobody in their right mind would take option 5… Right?

Yet, this is precisely what Amazon did according to Emily Guendelsberger in her insanely interesting and relevant book “On the clock(note the paradoxal link to Amazon’s webshop here).

Emily went undercover as employee at several organizations to experience blue collar jobs first-hand. In her book, she discusses how tech and data have changed low-wage jobs in ways that are simply dehumanizing.

These days, with sensors, timers, and smart nudging, employees are constantly being monitored and continue working (hard), sometimes at the cost of their own health and well-being.

I really enjoyed the book, despite the harsh picture it sketches of low wage jobs and malicious working conditions these days. The book poses several dilemma’s and asks multiple reflective questions that made me re-evaluate and re-appreciate my own job. Truly an interesting read!

Some quotes from the book to get you excited:

“As more and more skill is stripped out of a job, the cost of turnover falls; eventually, training an ever-churning influx of new unskilled workers becomes less expensive than incentivizing people to stay by improving the experience of work or paying more.”

Emily Guendelsberger, On the Clock

“Q: Your customer-service representatives handle roughly sixty calls in an eighty-hour shift, with a half-hour lunch and two fifteen-minute breaks. By the end of the day, a problematic number of them are so exhausted by these interactions that their ability to focus, read basic conversational cues, and maintain a peppy demeanor is negatively affected. Do you:

A. Increase staffing so you can scale back the number of calls each rep takes per shift — clearly, workers are at their cognitive limits

B. Allow workers to take a few minutes to decompress after difficult calls

C. Increase the number or duration of breaks

D. Decrease the number of objectives workers have for each call so they aren’t as mentally and emotionally taxing

E. Install a program that badgers workers with corrective pop-ups telling them that they sound tired.

Seriously—what kind of fucking sociopath goes with E?”

Emily Guendelsberger, On the Clock
My copy of the book
(click picture to order your own via affiliate link)

Cover via Freepik