My Areas of Expertise

My Areas of Expertise

Controlling NVIDIA GPU Fans on a headless Ubuntu system

When performing data science on an Ubuntu Linux machine remotely, the fans on NVIDIA GPUs may not spin up in response to increased load. This is due to fact that the NVIDIA controlling software generally requires logging into the GUI Desktop. Simply forwarding the xwindow with SSH has little effect,

Google's 'Coral' Edge TPU Dev Boards

Google's 'Coral' Edge TPU Dev Boards

I received a pair of Google's 'Coral' Edge TPU Dev Boards today. They have the exact same dimensions as Raspberry Pi devices, including the location of the mounting post holes. This is convenient if you have project boxes or mounting plates that you normally use for

Google's 'Coral' Edge TPU Accelerator

Google's 'Coral' Edge TPU Accelerator

Feeling like a kid on Christmas, today I finally received a couple of the Google 'Coral' Edge TPU Accelerators in the mail. First impressions upon unboxing is that they're not as physically robust as the Intel Neural Compute Stick 2. The entire Intel NCS2 case is

Pandas - Opening and Selecting Data

Pandas - Opening and Selecting Data

Data assets used in these examples * Titanic [https://www.kaggle.com/c/titanic] * City population [http://ezlocal.com/blog/post/Top-5000-US-Cities-by-Population.aspx] * USGS ShakeMap Atlas [https://earthquake.usgs.gov/data/shakemap/] -------------------------------------------------------------------------------- Reading data from CSV files Reading CSV files and using one of the columns

Connect Jupyter to Remote Spark Clusters With Apache Toree

Scala [https://www.scala-lang.org/] is a fun language which gives you all the power of Java [https://www.java.com], with the simplicity of Python [https://www.python.org/], and the power of functional programming [https://en.wikipedia.org/wiki/Functional_programming]. Though there are a variety of

How to run TensorFlow with GPU on Windows 10 in a Jupyter Notebook

How to run TensorFlow with GPU on Windows 10 in a Jupyter Notebook

Install CUDA ToolKit The first step in our process is to install the CUDA ToolKit, which is what gives us the ability to run against the the GPU CUDA cores. Because TensorFlow [https://www.tensorflow.org/] is very version specific, you'll have to go to the CUDA ToolKit

Transpose data with Spark

Transpose data with Spark

A short user defined function written in Scala which allows you to transpose a dataframe without performing aggregation functions.

Convert Spark Vectors to DataFrame Columns

Convert Spark Vectors to DataFrame Columns

Vectors are typically required for Machine Learning tasks, but are otherwise not commonly used. Sometimes you end up with an assembled Vector [https://spark.apache.org/docs/latest/ml-features.html#vectorassembler] that you just want to disassemble into its individual component columns so you can do some Spark SQL

Pivoting data with Spark

Pivoting data with Spark

One of the common data engineering tasks is taking a deep dataset and turning into a wide dataset with some sort of aggregation function. Let's take a quick look at an example dataset to see why we would want to perform this action. Our goal: To determine if

Renaming All Columns In A Spark DataFrame

Renaming All Columns In A Spark DataFrame

Here's an easy example of how to rename all columns in an Apache Spark DataFrame. Tehcnically, we're really creating a second DataFrame with the correct names. // IMPORT DEPENDENCIES import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.{SQLContext,

Using Spark, Scala and XGBoost On The Titanic Dataset from Kaggle

Using Spark, Scala and XGBoost On The Titanic Dataset from Kaggle

The Titanic: Machine Learning from Disaster [https://www.kaggle.com/c/titanic] competition on Kaggle [https://www.kaggle.com/] is an excellent resource for anyone wanting to dive into Machine Learning [https://en.wikipedia.org/wiki/Machine_learning]. There are forums [https://www.kaggle.com/c/titanic/discussion] where you

Family of Elephants in the Mara

Family of Elephants in the Mara

Copyright: James Conner Camera: Canon EOS 5D Mark III Lens: Canon 100-400L F4 with 1.4x Internal Extender Stats: 560mm/ƒ/5.6/1/1600s/ISO 640 Taken: March 5 2015

List All Additional Jars Loaded in Spark

List All Additional Jars Loaded in Spark

Once in a while, you need to verify the versions of your jars which have been loaded into your Spark session. Fortunately, there's a relatively easy way to do this: the listJars method. As you can see from the example below, the listJars method shows all jars loaded

Spark Vector of Vectors

Spark Vector of Vectors

I recently ran into a problem with creating a features vector for a machine learning project. If the number of features in your dataframe is too large, the JVM will crash during the Catalyst optimizer process because the number of constant variables generated exceeds the JVM limit of 65,536.

Joining Spark DataFrames Without Duplicate or Ambiguous Column Names

Joining Spark DataFrames Without Duplicate or Ambiguous Column Names

When performing joins in Spark, one question keeps coming up: When joining multiple dataframes, how do you prevent ambiguous column name errors? 1) Let's start off by preparing a couple of simple example dataframes // Create first example dataframe val firstDF = spark.createDataFrame(Seq( (1, 1, 2, 3, 8,

Selecting Dynamic Columns In Spark DataFrames (aka Excluding Columns)

Selecting Dynamic Columns In Spark DataFrames (aka Excluding Columns)

I often need to perform an inverse selection of columns in a dataframe, or exclude some columns from a query. This is a very easy method, and I use it frequently when arranging features into vectors for machine learning tasks. import org.apache.spark.sql.Column // Create an example dataframe

Photography laser trigger trap

Photography laser trigger trap

I've decided to try my hand at droplet and splash photography. Rather than using a manual method (setting up a constant drip method and trying to snap a picture the moment a droplet lands), I decided to go the engineered route (surprise!). I did a lot of searching

Cheetah in tall grass in the Maasai Mara

Cheetah in tall grass in the Maasai Mara

Copyright: James Conner Camera: Canon EOS 5D Mark III Lens: Canon 100-400L F4 with 1.4x Internal Extender Stats: 560mm/ƒ/5.6/1/1600s/ISO 640 Taken: March 5 2015

Intent lion cub in the Maasai Mara

Intent lion cub in the Maasai Mara

Copyright: James Conner Camera: Canon EOS 5D Mark III Lens: Canon 100-400L F4 with 1.4x Internal Extender Stats: 366mm/ƒ/5.6/1/400s/ISO 400 Taken: February 28 2015

Joining Data Frames in Spark SQL

Joining Data Frames in Spark SQL

Data The data that I'm using for this test comes from Kaggle's [https://www.kaggle.com/] Titanic Project [https://www.kaggle.com/c/titanic]. The purpose of the Titanic project is to create a machine learning model to predict the survivability of the Titanic passengers. In

Spark with Hadoop Crash Course Workshop

Spark with Hadoop Crash Course Workshop

Create an Apache Spark cluster using Raspberry Pi 2 Nodes

Create an Apache Spark cluster using Raspberry Pi 2 Nodes

Everyone talks about Spark in the context of "Big Data", but I've been having fun with it on a tiny platform: Raspberry Pi 2 (RP2) micro computers! Though the specs of the RP2 are modest, you can easily run Spark on a single node ... but where&

Spark Word Count

Spark Word Count

In the world of Big Data, your first program isn't "Hello, World!", it's Word Count, because it's easy to understand and conceptualize. In Java MapReduce, this would take about 4 pages of code. In Scala/Spark, it can be done in a