The term “Big Data” will fade away, the title “Data Scientist” will lose value, but the importance of “Data Science” will continue to grow..

This post contains my predictions for “Big Data”, “Data Scientist” and “Data Science”. Rather than start each sentence with “in my opinion” or “my best guess is”, I will state everything that follows as if it were fact. Please argue with me if you disagree with any of it.

“Big Data” has been a buzzword recently and is actually a useful term right now to use when talking about advances made in technologies and methods for handling and utilizing large data sets. Many business and others are leveraging statistical analysis of large data sets for the first time. As data analytics advocates across the world test their influence while navigating budget committees and corner offices, they benefit from the memory-locking powers of talking points and buzzwords. Since I am a fan of the growth of data analytics I support any tool that helps spread its prevalence, including buzzwords. Over time, however, as “Big Data Science” approaches the asymptote of ubiquity, the term “Big Data” will be less useful as a buzzword. Another problem for the term “Big Data” is that the size that a data set must be to qualify as “Big Data” is temporally bound, always growing as a function of time. Data storage hardware is analogous in this respect. We generally do not refer to a piece of data storage hardware as “big”. Instead, we specify its size when expressing its bigness. For these reasons the use of the term “Big Data” will decline.

The title “Data Scientist” is destining to befall a similar fate as the title “<fill-in-the-blank> Architect”. With no universally accepted definition the continuum between an Analyst and a Data Scientist will blur.  The term will succumb to market pressures from job seekers who will prefer a title that is perceived to have status and improve career prospects. There are already positions posted for entry level “Jr. Data Scientists”.

“Data Science” itself is not going anywhere. As more and more data is available, understanding that data is increasingly necessary for organizations to succeed. Statisticians and data architects will be in increasing demand. People that can bridge those skills will be more valuable still.

Logic and Probability Puzzle

I heard a good puzzle a few weeks ago on The Skeptics Guide to the Universe, a podcast about scientific skepticism. I strongly recommend this weekly podcast to anyone interested in learning the pitfalls of flawed reasoning while keeping up with the latest science news. Here is the puzzle:

 “I have two children. One of them is a boy born on a Tuesday. What is the probability that I have two boys?”

Spoiler Alert! Stop now if you want to figure this out on your own.

The answer is thirteen twenty-sevenths. My gut reaction to the solution was to question how the day of the week that one child is born can influence the probability of the other child being a boy. That was incorrect reasoning because the information, “One of them is a boy born on a Tuesday”, is not information about one of the children. It is information about the pair.

In other words, it is not equivalent to say:

 “I have two children. Child #1 is a boy born on a Tuesday. What is the probability that child #2 is a boy?”

The answer to this altered version is clearly 50%. The difference being that the original puzzle does not give certainty that either child in particular is a boy born on a Tuesday. Try thinking through the puzzle starting with the boy born on a Tuesday as a given. Then cycle through all the possible combinations of genders and days of week for the other child. When you get to “boy born on a Tuesday” for the other child you can no longer take “boy born on a Tuesday” as a given for the first child. The logic in the previous sentence is easy to miss when thinking through the puzzle this way. If you do miss this fact, then you are left with the non-equivalent altered puzzle and incorrectly conclude that the answer is 50%.

A better way to think through this is to start by splitting the universe into equally probably parts; the universe being the set of people with two children. For each child there are 14 possibilities (2 genders times 7 days of the week). For each of those possibilities there are 14 possibilities for the other child. That means there are 14 times 14 total possibilities which equals 196 equally probable combinations. If we limit these 196 combinations to just those with at least one of the children being a boy born on a Tuesday, we are left with 27 equally probable combinations. Of those, 13 have two boys; therefore, the probability is 13/27.

If you change the puzzle to, "I have two children. One of them is a boy. What is the probability that I have two boys?", then answer becomes 1/3. The universe of two-child families is [(boy,boy), (boy,girl), (girl,boy), (girl,girl)]. We learn that at least one child is a boy which excludes (girl,girl). We are left with three equally probable combinations; one of which is two boys. 1/3.

If you want to read a robust debate and discussion of this puzzle, check out this Skeptics Guide to this Universe forums thread.

I'll finish with my T-SQL solution:

WITH child AS
   (
   SELECT g.gender, w.weekday_born
   FROM
       (VALUES('boy'),('girl')) AS g(gender)
   CROSS APPLY
       (VALUES('Sun'),('Mon'),('Tue'),('Wed'),('Thu'),('Fri'),('Sat'))
       AS w(weekday_born)
   ) 
SELECT
   CAST(SUM(CASE
       WHEN c1.gender = 'boy' AND c2.gender = 'boy'
       THEN 1 ELSE 0 END) AS FLOAT)
   / COUNT(*) AS answer 
FROM
   child AS c1 
CROSS JOIN
   child AS c2 
WHERE
   (c1.gender = 'boy' AND c1.weekday_born = 'Tue')
   OR
   (c2.gender = 'boy' AND c2.weekday_born = 'Tue')

/*
answer
----------------------
0.481481481481481
*/

Using a Recursive CTE for Logic in VAE Protocol

Until this week, all of the recursive CTE posts I had read used an employee hierarchy example. Then my colleague pointed me to the best technical blog post I have ever read where Brad Schulz uses a sales running total example in his blog post entitled,  “This Article On Recursion Is Entitled “This Article On Recursion Is Entitled “This Article… ∞” ” Even if you are not interested in SQL you should check out this article for the great story, writing style, examples or recursion, and links to Wikipedia pages about things like fractals and mobius strips.

My blog post is about using a recursive CTE in SQL Server for part of the logic in the Center for Disease Control’s (CDC) Ventilator Associated Event (VAE) Protocol. 

Imagine you have a temp table called #vent_days where each row represents a patient and a day that they were on a mechanical ventilator. You have already identified which vent days meet the criteria for a VAE day of event except you haven’t yet taken into account the rule: when a VAE day of event occurs, another VAE day of event cannot occur for 14 days. You have filtered the temp table down to just those patient visits that have a VAE. Your temp table has the columns represented by the following SELECT statement.

 
SELECT
   patient_visit_id
   , vent_day_date      /*date of mechanical ventilation*/
   , vent_day           /*the number of days a patient is on a mechanical ventilator as of the vent_day_date*/
   , pos_VAE_doe_flg    /*possible VAE day of event flag. 1 indicates that this day meets the
                       criteria for a VAE day of event except the rule that a VAE day of event may not occur until 14 days after a previous VAE day of event*/
   , pos_VAE_doe_order /*only populated where pos_VAE_doe_flg is 1. The ordered number of possible VAE days of event per patient visit.*/ 
FROM
   #vent_days  

Below is how you can use a recursive CTE to complete the logic by fulfilling the criteria that a VAE day of event must not occur until 14 days have passed since a previous VAE day of event.

; 
WITH VAE_doe (patient_visit_id, vent_day_date, vent_day, pos_VAE_doe_order, prev_doe, doe_flg) 
AS (
   /*  the anchor query is the first possible VAE day of event for each patient visit. doe_flag is set to 1 because it will always be an actual day of event.*/
   SELECT
       v.patient_visit_id
       , v.vent_day_date
       , v.vent_day
       , v.pos_VAE_doe_order
       , 0 AS prev_doe
       , 1 AS doe_flag
   FROM
       #vent_days AS v
   WHERE
       v.pos_VAE_doe_order = 1
       AND v.pos_VAE_doe_flg = 1
   UNION ALL
   /*  second query recurses across the anchor query*/
   SELECT
       v.patient_visit_id
       , v.vent_day_date
       , v.vent_day
       , v.pos_VAE_doe_order
       /*  when the previous VAE date of event occurs less than 14 days early
           then do not change prev_doe, otherwise set prev_doe equal to vent_day
           and assign a 1 to doe_flag. */
       , CASE   WHEN v.vent_day - VAE_doe.doe_vent_day >= 14
               THEN VAE_doe.vent_day ELSE VAE_doe.prev_doe END AS prev_doe
       , CASE   WHEN v.vent_day - VAE_doe.doe_vent_day >= 14
               THEN 1 ELSE 0 END AS doe_flag
   FROM
       #vent_days AS v
   INNER JOIN
       VAE_doe
       ON v.patient_visit_id = VAE_doe.patient_visit_id
       AND v.pos_VAE_doe_order = VAE_doe.pos_VAE_doe_order + 1 /*for each visit recurse by order of possible VAE days of envet*/
   WHERE
       v.pos_VAE_doe_flg = 1
   ) 


SELECT
   v.patient_visit_id
   , v.vent_day_date
   , v.vent_day
   , COALESCE(VAE_doe.doe_flg,0) AS doe_flg 
FROM
   #vent_days AS v 
LEFT OUTER JOIN
   VAE_doe
   ON v.patient_visit_id = VAE_doe.patient_visit_id
   AND v.vent_day = VAE_doe.vent_day 
ORDER BY
   v.patient_visit_id
   , v.vent_day 
OPTION(MAXRECURSION 25);

If I had read Brad Schulz’s post before coming up with this I would have used a WHILE loop. Recursive CTEs do not use set-based processing but process everything row-by-row using a Stack Spool. I don’t know if I will ever use a recursive CTE again. 

Besides a possible argument for convenience, does a legitimate reason for using a recursive CTE in SQL Server exist?

Using Regular Expressions to Clean Data

Regular expressions are a very useful tool for any data professional. It is often the case in a data analytics project that the vast majority of the work is preparing the data. Among other things, Regular Expressions allow for advanced logic in filter, find, or find and replace criteria.

For example, the following regular expression is intended to only match a valid Medicare HIC number.

I found this on Regular Expression Library (regexlib.com).

 (?![A-z](\d)\1{5,})(^[A-z]{1,3}(\d{6}|\d{9})$)|(^\d{9}[A-z][0-9|A-z]?$)  

This expression says, "Except in the case when it is one letter followed by the same digit 6 or more times in a row, match on one to three letters followed by six or nine digits. Alternatively, match on 9 digits followed by a letter then a digit or a letter."

More robust validation rules could be applied. For example, there is a more specific subset of valid suffixes than a letter followed by a digit or a letter. Also, I used this in a RegexMatch function in SQL Server and found it slow. Getting more precise and using set based logic in SQL would be more effective and possibly more efficient; however, that would take time away from other work. By doing a quick Google search to find a regular expression a task like validating a HIC number can be completed in minutes.

Architects and Scientists

Is the “Architect” in “Analytics Architect”, “Infrastructure Architect” or “Data Architect” analogues to the “Scientist” in “Data Scientist”? Clearly the answer is no. 

“Architect” in a technical job title generally implies that responsibilities include structural design of the information system, like a traditional Architect’s responsibilities include structural design of a building or other physical structure. In these technical job titles the word “Architect” is not intended to be interpreted as a traditional Architect. Traditional Architects’ opinions on this seem to range from insult to flattery. See this article titled I'm an Architect by Amanda Kolson Hurley at architectmagazine.com.

 The “Scientist” in “Data Scientist” is intended to say that the person is an actual scientist. As I discussed in my first post to this blog, a data scientist generally does not publish their work, an important part of the scientific method; therefore, I am not convinced that the work they do is actually science and so I do not believe that they are actual scientists.

In conclusion, an Analytics Architect is not an actual architect and a Data Scientist is not an actual scientist. The title “Analytics Architect” is not intended to mean that the person is an actual architect. The title “Data Scientist” is intended to mean that the person is an actual scientist. Stopping there is not entirely fair because while an Analytics Architect’s work barely resembles the work of an actual architect, a Data Scientist’s work comes pretty close to being actual science. 

For my next post I will cover a technical topic.