Hurricane Alex

It is only January, but we have had the first Atlantic hurricane of the 2016 season.

Hurricane Alex was a category one storm over the Azores. Wikipedia has an excellent article on the storm.

The storm had an interesting history, forming near the Bahamas, before bringing gales to Bermuda. It then tracked southerly, south of the Azores and began to intensify.

It is generally stated that hurricanes require sea surface temperature of 26C to form, but the sea surface was just 20C.

Upper atmosphere temperatures, however, were unusually low. It is the difference in temperature that drives the hurricane heat engine.

Some have argued that global warming, with rising sea temperatures might not result in more intense storms if there is also an equivalent warming of upper atmosphere.

I am guessing that the warming of the atmosphere and oceans will not be in sync, so there will likely be a period of strange storms such as hurricane Alex in January.

From wikipedia:

An eye feature soon appeared, marking intensification, within a complex of several banding features. The 20 mi (25 km) wide feature cleared out early on January 14 and was surrounded by a ring of −76°F (−60°C) cloud tops. The storm remained vertically stacked with a cold-core low, though the development of upper-level outflow indicated the system was becoming increasingly tropical.

Despite moving over 68°F (20°C) waters, Alex continued to deepen and transitioned into a full-fledged tropical cyclone by 09:00 UTC. The transition was enabled by colder-than-average upper-tropospheric temperatures which created greater instability than would otherwise be expected.

Wikipedia also states that there was less damage than feared in the Azores. I am wondering what the preciptation was like for the storm. Rainfall has been a factor in damage for some of the recent storms, together with flooding.

Hurricane Joaquin pressure

In [1]:
# Tell matplotlib to plot in line
%matplotlib inline

# import pandas
import pandas

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()
In [8]:
infile = "../files/pijessie_weather.csv"
In [2]:
!scp 192.168.0.127:Adafruit_Python_BMP/weather.csv $infile
weather.csv                                   100% 9735KB   4.8MB/s   00:02    
In [9]:
""" assume it is csv and let pandas do magic

  index_col tells it to use the 'date' column in the data
  as the row index, plotting picks up on this and uses the
  date on the x-axis

  The *parse_dates* bit just tells it to try and figure out
  the date/time in the columne labeled 'date'.
"""
data = pandas.read_csv(infile, index_col='date', parse_dates=['date'])

See the pressure plummet as Joaquin approaches Bermuda

Pressure is still dropping as of 17.50pm BDA time.

It looks like pressure was starting to bottom out around 10pm BDA time.

No surprise that is when my power went out :(

I will have to look into more reliable power for the raspberry pi for the next storm.

In [4]:
data.pressure[-1*24*24:].plot()
Out[4]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fe585742860>
In [12]:
# See how this compares to "normal" pressure
# Plot the last 10 days

data.pressure[-10*24*24:].plot()
Out[12]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fe58565db00>
In [10]:
data.tail()
Out[10]:
temp pressure altitude sealevel_pressure humidity temp_dht
date
2015-10-05 00:58:00.057913 26.2 98395 245.990163 98410 96.199997 25.799999
2015-10-05 00:59:00.673876 26.2 98383 247.865819 98389 96.099998 25.700001
2015-10-05 01:00:01.316217 26.2 98383 247.013206 98383 95.900002 25.700001
2015-10-05 01:01:04.485819 26.2 98408 247.098464 98395 95.900002 25.700001
2015-10-05 01:02:05.128270 26.1 98414 244.455788 98420 95.900002 25.700001
In [6]:
!pwd
/home/jng/devel/peakrisk/posts

Bermuda Weather Radar

I found the Bermuda weather radar invaluable for watchin Joaquin

Below is a screenshot from 19.39 BDA time.

http://weather.bm/tools/animateimages.asp?name=RADAR_250KM_SRI

In [7]:
from IPython import display
display.Image('../galleries/Joaquin/joaquin.png')
Out[7]:

Walking in the Peak District

This week I am over in the UK walking in the Peak District National Park.

I spent a lot of time walking and cycling here when growing up. It is good to be back.

So far the weather has been kind, with some great views of the peaks in the Peak.

../galleries/PeakDistrict/20150923_101002.jpg

Here are some more pictures.

Comparing pressure data from two sensors

I have had two simple raspberry pi weather stations running for a while now.

Both have pressure, temperature and humidity sensors.

One I have in the carefully controlled environment of my study, the other is hanging out of the window.

The study one is known as pijessie as it started life as a Raspberry Pi running the Jessie version of Raspbian.

The outside station is known as kittycam as I intend at some point to attach a camera so I can watch our cat come and go.

For a while I have been noticing that the pressure values have been quite a way apart. The software I am includes a conversion to altitude and I find these numbers more natural for me to think about.

The altitude conversion assumes the pressure at sea level is 1023.25 hPa, which is the mean pressure at sea level.

When the pressure is higher than this the altitude comes out below sea level, when pressure is lower than this above sea level.

As always, Wikipedia has good information on this: https://en.wikipedia.org/wiki/Atmospheric_pressure

For a while I had been noticing the two sensors giving values differing by about 10 metres altitude.

I had put this down to the sensors not being calibrated accurately, but also noticed that kittycam was more prone to weird glitches.

Now the glitches I put down to the fact I have one process collecting data every minute and another process creating a display on my laptop so I can glance over and see what the weather is doing. The latter was just polling the sensor every 10 minutes.

The code does not do anything smart like get a lock and my guess was that the two processes were occasionally trampling on each other's feet.

Long story short, I decided to take a closer look.

In [34]:
# Tell matplotlib to plot in line
%matplotlib inline

import datetime

# import pandas
import pandas

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()
In [35]:
# input files: the data from the two sensors
infiles = ["../files/kittycam_weather.csv", "../files/pijessie_weather.csv"]
In [36]:
# Read the data

data = []

for infile in infiles:
    data.append(pandas.read_csv(infile, index_col='date', parse_dates=['date']))
In [37]:
# take a look at what we got
data[0].describe()
Out[37]:
temp pressure altitude sealevel_pressure humidity temp_dht
count 42768.000000 42768.000000 42768.000000 42768.000000 42764.000000 42764.000000
mean 28.114357 101385.375842 -5.068642 101387.094487 76.806599 27.730374
std 2.205370 430.276252 36.334073 409.738250 8.512122 2.068583
min 22.800000 56117.000000 -1314.018026 67136.000000 29.500000 22.299999
25% 26.500000 101141.750000 -26.939647 101142.000000 71.599998 26.200001
50% 27.700000 101411.000000 -7.157439 101412.000000 77.099998 27.400000
75% 29.900000 101648.000000 15.246750 101649.000000 83.500000 29.299999
max 43.200000 102242.000000 3397.334521 118353.000000 94.300003 47.099998
In [38]:
# plots are always good 

data[0].plot(subplots=True)
Out[38]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f1a6b1e4128>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f1a6a7e40f0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f1a69ef79b0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f1a6afef0f0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f1a6ae8d7b8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f1a6aeb2908>], dtype=object)

Now the two sets of data have different indices since the processes collecting the data are not in sync.

So we need to align the data and then fill in missing values

In [39]:
# align returns two new dataframes, now aligned
d1, d2 = data[0].align(data[1])
In [40]:
# have a look, note the count is just the valid data.
# Things have been aligned, but missing values are set ton NaN
d1.describe()
Out[40]:
temp pressure altitude sealevel_pressure humidity temp_dht
count 42768.000000 42768.000000 42768.000000 42768.000000 42764.000000 42764.000000
mean 28.114357 101385.375842 -5.068642 101387.094487 76.806599 27.730374
std 2.205370 430.276252 36.334073 409.738250 8.512122 2.068583
min 22.800000 56117.000000 -1314.018026 67136.000000 29.500000 22.299999
25% 26.500000 101141.750000 -26.939647 101142.000000 71.599998 26.200001
50% 27.700000 101411.000000 -7.157439 101412.000000 77.099998 27.400000
75% 29.900000 101648.000000 15.246750 101649.000000 83.500000 29.299999
max 43.200000 102242.000000 3397.334521 118353.000000 94.300003 47.099998
In [41]:
# Use interpolation to fill in the missing values
d1 = d1.interpolate(method='time')
d2 = d2.interpolate(method='time')
In [42]:
# Now plot
d1.altitude.plot()
print(len(d1))
80476
In [43]:
# For convenience, add a new series to d1 with the altitude data from d2
d1['altitude2'] = d2.altitude
In [44]:
# Now plot the two
d1[['altitude', 'altitude2']][10000:30000].clip(-60,60).plot()
Out[44]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f1a6aa1fe48>
In [45]:
(d1.altitude - d1.altitude2)[10000:30000].clip(-20,15).plot()
Out[45]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f1a6a463630>

So we do have a difference around 5m. More interestingly, there seems to be some sort of daily pattern to the data.

Latest Weather

Simple quick update latest weather

In [1]:
# Tell matplotlib to plot in line
%matplotlib inline

# import pandas
import pandas

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()

def smooth(data, thresh=None):
    
    means = data.mean()

    if thresh is None:
        sds = data.std()
    else:
        sds = thresh
    
    delta = data - data.shift()
    
    good = delta[abs(delta) < sds]

    #print(good.describe())
    
    return delta.where(good, 0.0)
In [2]:
infile = "../files/pijessie_weather.csv"

!scp 192.168.0.127:Adafruit_Python_BMP/weather.csv $infile
weather.csv                                   100%   13MB   6.4MB/s   00:02    
In [3]:
""" assume it is csv and let pandas do magic

  index_col tells it to use the 'date' column in the data
  as the row index, plotting picks up on this and uses the
  date on the x-axis

  The *parse_dates* bit just tells it to try and figure out
  the date/time in the columne labeled 'date'.
"""
data = pandas.read_csv(infile, index_col='date', parse_dates=['date'])
#data = smooth(data)

# smooth the data to filter out bad temps and pressures
#data.altitude = (smooth(data.altitude, 5.0).cumsum() + data.altitude[0])
#data.temp = (smooth(data.temp, 5.0).cumsum() + data.temp[0])
In [4]:
data.altitude.plot()
Out[4]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc4afcb70>

Last 24 hours:

In [5]:
# reading is once a minute, so take last 24 * 60 readings
def plotem(data, n=-60):
    
    
    if n < 0:
        start = n
        end = len(data)
    else:
        start = 0
        end = n
        
    data[['temp', 'altitude', 'humidity']][n:].plot(subplots=True)
        
plotem(data, -24*60)
In [6]:
data.altitude[-8*60:].plot()
Out[6]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc2992e10>

Last week

In [7]:
# reading is once a minute, so take last 7 * 24 * 60 readings
plotem(data, -7*24*60)
In [8]:
plotem(data)

Look at all the data

In [9]:
data.describe()
Out[9]:
temp pressure altitude sealevel_pressure humidity temp_dht
count 134436.000000 134436.000000 134436.000000 134436.000000 134430.000000 134430.000000
mean 27.861387 101168.977112 12.937777 101170.173919 68.160873 27.322403
std 1.599058 429.541753 33.097760 446.960346 9.849979 1.711857
min 22.300000 50042.000000 -73.255830 48765.000000 36.599998 1.200000
25% 27.000000 100935.000000 -10.068943 100937.000000 62.900002 26.400000
50% 28.000000 101151.000000 14.413017 101153.000000 69.199997 27.600000
75% 28.900000 101445.000000 32.352991 101447.000000 74.300003 28.500000
max 121.400000 117173.000000 2740.700691 102210.000000 98.000000 30.700001
In [10]:
data.tail()
Out[10]:
temp pressure altitude sealevel_pressure humidity temp_dht
date
2015-10-30 00:49:37.821883 28.3 100873 37.699238 100872 80.000000 26.400000
2015-10-30 00:50:38.421900 28.3 100864 38.200590 100873 80.000000 26.400000
2015-10-30 00:51:39.022023 28.4 100869 38.284151 100865 80.099998 26.500000
2015-10-30 00:52:39.622038 27.7 100860 38.534838 100868 80.000000 26.500000
2015-10-30 00:53:40.206103 27.4 100864 38.534838 100869 81.599998 26.299999

I currently have two temperature sensors:

  • DHT22 sensor which gives temperature and humidity.
  • BMP180 sensor which gives pressure and temperature.

The plot below shows the two temperature plots.

Both these sensors are currently in my study. For temperature and humidity I would like to have some readings from outside. If I can solder them to a phone jack then I can just run phone cable to where they need to be.

Below plots the current values from these sensors. This is handy for calibration.

In [11]:
data[['temp', 'temp_dht']].plot()
Out[11]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc30cae48>

Dew Point

The warmer air is, the more moisture it can hold. The dew point is the temperature at which air would be totally saturated if it had as much moisture as it currently does.

Given the temperature and humidity the dew point can be calculated, the actual formula is pretty complex.

It is explained in more detail here: http://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html

If you are interested in a simpler calculation that gives an approximation of dew point temperature if you know >the observed temperature and relative humidity, the following formula was proposed in a 2005 article by Mark G. >Lawrence in the Bulletin of the American Meteorological Society:

$$Td = T - ((100 - RH)/5.)$$
In [12]:
data['dewpoint'] = data.temp - ((100. - data.humidity)/5.)
In [13]:
data[['temp', 'dewpoint', 'humidity']].plot()
Out[13]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc30aa3c8>
In [14]:
data[['temp', 'dewpoint', 'humidity']].plot(subplots=True)
Out[14]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f5dc3064518>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f5dc32126d8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f5dc3330160>], dtype=object)
In [15]:
data[['temp', 'dewpoint']].plot()
Out[15]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc0b35048>
In [16]:
data.altitude.plot()
Out[16]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f5dc1b8a668>

Latest Weather

Simple quick update latest weather

  • switched to new sensor 8am BDA 2015-08-21 Old sensor was giving systematic bias and other weirdness.
In [145]:
# Tell matplotlib to plot in line
%matplotlib inline

import datetime

# import pandas
import pandas

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()

def smooth(data, thresh=None):
    """ This smoothing function is meant to spot and eliminate bad readings.
    
    """ 
    
    means = data.mean()

    if thresh is None:
        sds = data.std()
    else:
        sds = thresh
    
    delta = data - data.shift()
    
    good = delta[abs(delta) < sds]

    #print(good.describe())
    
    return delta.where(good, 0.0)
In [182]:
infile = "../files/kittycam_weather.csv"

!scp 192.168.0.121:Adafruit_Python_BMP/weather.csv $infile
weather.csv                                   100% 5852KB   5.7MB/s   00:01    
In [147]:
!wc ../files/weather.csv
  564  1127 56016 ../files/weather.csv
In [148]:
""" assume it is csv and let pandas do magic

  index_col tells it to use the 'date' column in the data
  as the row index, plotting picks up on this and uses the
  date on the x-axis

  The *parse_dates* bit just tells it to try and figure out
  the date/time in the columne labeled 'date'.
"""
data = pandas.read_csv(infile, index_col='date', parse_dates=['date'])
In [149]:
data.plot()
Out[149]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f54f9f60>
In [150]:
# smooth the data to filter out bad temps and pressures
#data.altitude = (smooth(data.altitude, 5.0).cumsum() + data.altitude[0])
#data.temp = (smooth(data.temp, 5.0).cumsum() + data.temp[0])
#data.altitude = data.altitude.clip(-100, 100)
In [151]:
data.altitude.plot()
Out[151]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f4aaa748>
In [152]:
data.tail()
Out[152]:
temp pressure altitude sealevel_pressure humidity temp_dht
date
2015-10-04 14:59:04.029159 26.4 99788 128.924064 99791 82.500000 25.9
2015-10-04 15:00:04.671297 26.4 99808 127.069788 99808 82.500000 25.9
2015-10-04 15:01:05.262094 26.5 99791 129.429833 99788 82.500000 25.9
2015-10-04 15:02:05.904264 26.5 99766 129.935627 99767 82.599998 25.9
2015-10-04 15:03:06.546427 26.5 99758 130.525751 99773 82.400002 25.9
In [153]:
data.describe()
Out[153]:
temp pressure altitude sealevel_pressure humidity temp_dht
count 60281.000000 60281.000000 60281.000000 60281.000000 60281.000000 60281.000000
mean 27.892394 101246.221778 6.548877 101247.144722 63.323885 27.275608
std 1.717909 315.289819 26.273290 315.299968 12.424934 1.718179
min 22.500000 99758.000000 -55.052172 99767.000000 28.700001 21.799999
25% 26.900000 101027.000000 -8.738071 101028.000000 58.099998 26.299999
50% 28.000000 101251.000000 6.079362 101252.000000 66.500000 27.500000
75% 29.000000 101430.000000 24.756020 101431.000000 71.599998 28.400000
max 31.400000 101985.000000 130.525751 101985.000000 88.500000 30.600000

Dew Point

The warmer air is, the more moisture it can hold. The dew point is the temperature at which air would be totally saturated if it had as much moisture as it currently does.

Given the temperature and humidity the dew point can be calculated, the actual formula is pretty complex.

It is explained in more detail here: http://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html

If you are interested in a simpler calculation that gives an approximation of dew point temperature if you know >the observed temperature and relative humidity, the following formula was proposed in a 2005 article by Mark G. >Lawrence in the Bulletin of the American Meteorological Society:

$$Td = T - ((100 - RH)/5.)$$
In [154]:
data['dewpoint'] = data.temp - ((100. - data.humidity)/5.)

Last 24 hours:

In [155]:
print(datetime.datetime.now())
print(data.ix[-1:])
2015-10-04 12:04:12.700504
                            temp  pressure    altitude  sealevel_pressure  \
date                                                                        
2015-10-04 15:03:06.546427  26.5     99758  130.525751              99773   

                             humidity  temp_dht  dewpoint  
date                                                       
2015-10-04 15:03:06.546427  82.400002      25.9     22.98  
In [156]:
# reading is once a minute, so take last 24 * 60 readings
def plotem(data, n=-60, start=None):
        
    
    if n < 0:
        start = n
        end = len(data)
    else:
        start = start or 0
        end = start + n
        
    data[['temp', 'altitude', 'humidity', 'dewpoint']][start:end].plot(subplots=True)

# last few hours
plotem(data, -24*60)

Last week

In [157]:
# reading is once a minute, so take last 7 * 24 * 60 readings
plotem(data, -7*24*60)
In [158]:
plotem(data)
In [159]:
plotem(data, -5)

Look at all the data

In [160]:
data.describe()
Out[160]:
temp pressure altitude sealevel_pressure humidity temp_dht dewpoint
count 60281.000000 60281.000000 60281.000000 60281.000000 60281.000000 60281.000000 60281.000000
mean 27.892394 101246.221778 6.548877 101247.144722 63.323885 27.275608 20.557171
std 1.717909 315.289819 26.273290 315.299968 12.424934 1.718179 3.586928
min 22.500000 99758.000000 -55.052172 99767.000000 28.700001 21.799999 8.660000
25% 26.900000 101027.000000 -8.738071 101028.000000 58.099998 26.299999 19.820000
50% 28.000000 101251.000000 6.079362 101252.000000 66.500000 27.500000 21.620000
75% 29.000000 101430.000000 24.756020 101431.000000 71.599998 28.400000 22.939999
max 31.400000 101985.000000 130.525751 101985.000000 88.500000 30.600000 26.020000

I currently have two temperature sensors:

  • DHT22 sensor which gives temperature and humidity.
  • BMP180 sensor which gives pressure and temperature.

The plot below shows the two temperature plots.

Both these sensors are currently in my study. For temperature and humidity I would like to have some readings from outside. If I can solder them to a phone jack then I can just run phone cable to where they need to be.

Below plots the current values from these sensors. This is handy for calibration.

In [161]:
data[['temp', 'temp_dht']].plot()
Out[161]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f42c5e80>
In [162]:
data[['temp', 'dewpoint', 'humidity']].plot()
Out[162]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f431d908>
In [163]:
data[['temp', 'dewpoint', 'humidity']].plot(subplots=True)
Out[163]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f06f432a9e8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f06f4c719e8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f06f3ea7f60>], dtype=object)
In [164]:
data[['temp', 'dewpoint']].plot()
Out[164]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f41a6630>
In [165]:
from numpy import fft
In [166]:
data['fft_altitude'] = fft.fft(data.altitude)
data['fft_alt_real'] = data.fft_altitude.real
data['fft_alt_imag'] = data.fft_altitude.imag
In [167]:
data['alt_power'] = ((data.fft_alt_real ** 2.0) + (data.fft_alt_imag ** 2.0)) ** 0.5
In [168]:
import numpy
N = len(data)
TWELVE = N//(12*59)

def hours(pos):
    return N/(pos*60.) 


eleven_elevator = N/(12*59.0)   # Scottish elevator
print(eleven_elevator, TWELVE)
data['pos'] = numpy.arange(N)
data[['fft_alt_imag', 'fft_alt_real', 'alt_power']][TWELVE//3:2*TWELVE].plot()
85.14265536723164 85
Out[168]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f4034978>
In [169]:
data[TWELVE//3:2*TWELVE].plot(x='pos', y='alt_power')
Out[169]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f38c70b8>
In [170]:
data[TWELVE-4: TWELVE+10].plot(x='pos', y='alt_power')
print(TWELVE, eleven_elevator)
85 85.14265536723164
In [171]:
#FIXME - the data points are probably a fraction over a minute apart.
for x in range(TWELVE-5, TWELVE+5):
    print(x, "%6.4f" % hours(x))
80 12.5585
81 12.4035
82 12.2522
83 12.1046
84 11.9605
85 11.8198
86 11.6824
87 11.5481
88 11.4169
89 11.2886
In [172]:
data.tail(1)
Out[172]:
temp pressure altitude sealevel_pressure humidity temp_dht dewpoint fft_altitude fft_alt_real fft_alt_imag alt_power pos
date
2015-10-04 15:03:06.546427 26.5 99758 130.525751 99773 82.400002 25.9 22.98 (250838.166745-103601.889966j) 250838.166745 -103601.889966 271391.115368 60280
In [173]:
data.tail(10*24*60).head(1)
Out[173]:
temp pressure altitude sealevel_pressure humidity temp_dht dewpoint fft_altitude fft_alt_real fft_alt_imag alt_power pos
date
2015-09-24 10:03:41.189865 24.2 101185 11.91222 101187 35.799999 23.6 11.36 (-46.1226318013-240.511319049j) -46.122632 -240.511319 244.893838 45881
In [174]:
hours(54)
Out[174]:
18.605246913580245
In [175]:
data[TWELVE//3:3 + (2*TWELVE)].clip(-60000, 60000).plot(x='pos', y='alt_power')
Out[175]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f3f4e710>
In [176]:
xx = data.fft_altitude.copy()
xx[TWELVE:-TWELVE] = 0.0
data['smooth_alt'] = [x.real for x in fft.ifft(xx)]
In [177]:
data[['smooth_alt', 'altitude']].plot()
Out[177]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f3f2dcf8>
In [178]:
data.smooth_alt.plot()
Out[178]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f06f3f42470>
In [179]:
def power(x, y):
    
    return ((x ** 2.0) + (y ** 2.0)) ** 0.5
In [180]:
P = TWELVE
xx = data.fft_altitude.copy()
xx[:] = 0.0
xx[P:-P] = data.fft_altitude[P:-P]
data['twelve'] = [x.real for x in fft.ifft(xx)]
data['itwelve'] = [x.imag for x in fft.ifft(xx)]
data['twelve_pow'] = power(data.twelve, data.itwelve)
data['smoothed'] = data.altitude - data.twelve
data[['twelve', 'altitude', 'smoothed', 'smooth_alt']].plot(subplots=True)
Out[180]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f06f391c6d8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f06f31a07f0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f06f3354eb8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f06f386e3c8>], dtype=object)

Machine Learning in the atmosphere

Fourier analysis

So, the story so far:

Looking to see if there is a relationship between the tidal pattern in pressure data and the moon.

The pressure data gives a waveform. The Fourier transform of this data we can converts the data from a time series to give us the underlying frequencies driving that data.

In [111]:
# Tell matplotlib to plot in line
%matplotlib inline
import pandas as pd

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()

from matplotlib import pyplot

import mpld3
In [112]:
# interactive plots -- except in ein

#mpld3.enable_notebook()
In [113]:
infile="../files/moon_weather.csv"
In [114]:
import numpy
from numpy import fft
In [115]:
data = pd.read_csv(infile, index_col='date', parse_dates=['date'])
In [116]:
data.altitude[:24480].plot()
Out[116]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e56dbe0>
In [117]:
data.describe()
Out[117]:
altitude pressure sealevel_pressure temp moon_phase moon_glat moon_glon tide
count 30397.000000 0 0 30397.000000 30397.000000 30397.000000 30397.000000 30397.000000
mean -31.879161 NaN NaN 29.392555 58.143320 0.009210 3.272469 12.141895
std 20.589351 NaN NaN 0.894486 36.001448 0.063526 1.943755 6.996605
min -69.792586 NaN NaN 27.200000 0.172035 -0.087831 0.000126 -0.087386
25% -50.325302 NaN NaN 28.700000 22.124449 -0.062341 1.274654 5.998249
50% -34.975717 NaN NaN 29.400000 69.537071 0.026644 3.743553 12.296909
75% -10.941520 NaN NaN 30.000000 91.296631 0.070193 4.961365 18.230285
max 8.379000 NaN NaN 31.500000 99.811104 0.087505 6.283139 24.070675
In [118]:
24480/720.
#(60. * 12)
Out[118]:
34.0
In [119]:
N = 24480
fft_altitude = fft.fft(data[:N].altitude)
In [120]:
# Plot the transform.  Most of the energy in the signal is at low frequencies
pd.Series(x.real for x in fft_altitude).clip(-1000, 1000)[:].plot()
Out[120]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e6e4908>
In [121]:
# spike of energy at location 34
pd.Series(x.real for x in fft_altitude)[20:40].plot()
Out[121]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e70bdd8>
In [150]:
# Convert location into period
point = 34
for p in (point-1, point, point+1):
    print(p)
    print("%8.3f %s" % (N/p, "minutes"))
    print("%8.3f %s" % (N/(p * 60), "hours"))
    print("%8.3f %s" % (N/(p * 60 * 24), "days"))
33
 741.818 minutes
  12.364 hours
   0.515 days
34
 720.000 minutes
  12.000 hours
   0.500 days
35
 699.429 minutes
  11.657 hours
   0.486 days

The spike of energy at location 34 corresponds to a signal with a period of 12 hours.

Note that the amount of data available limits the resolution of transform.

A peak at position 33 would give a period roughly 22 minutes longer, at 35 it would be 21 minutes shorter.

It looks like the tidal pattern in the pressure data has a period of 24 hours.

Hence, it looks like it is driven by the sun, not the moon.

In [123]:
# transform is complex, look at the imaginary component
pd.Series(x.imag for x in fft_altitude)[0:60].plot()
Out[123]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e620f98>
In [124]:
# remove 12 hour and higher freq components:
xx = fft_altitude.copy()
xx[point:-point] = 0.0
xx[int(point/2)] = 0.0
#xx[point+1:] = 0.0
In [125]:
smooth_altitude = fft.ifft(xx)
In [126]:
pd.Series(y.real for y in smooth_altitude).plot()
Out[126]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e4ccbe0>
In [133]:
df = pd.DataFrame()
df['altitude'] = data.altitude[:N]
df['smooth_altitude'] = [y.real for y in smooth_altitude]

df.plot(subplots=False)
Out[133]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535e1801d0>
In [147]:
smdf = pd.DataFrame(dict(smooth_altitude=smooth_altitude, fft_alt=fft.fft(smooth_altitude)))
smdf.fft_alt[6:50].plot()
/home/jng/.virtualenvs/peakrisk2/lib/python3.4/site-packages/numpy/core/numeric.py:462: ComplexWarning: Casting complex values to real discards the imaginary part
  return array(a, dtype, copy=False, order=order)
Out[147]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f535dd28908>
In [157]:
# moon takes 29.53 days to be in same position relative to earth/sun

N/(29.53 * 24 * 60)
Out[157]:
0.5756857433118862
In [135]:
pd.Series(fft.fft(df.smooth_altitude)
Out[135]:
array([-931957.98424361    +0.j        ,  255374.61463530+17993.92648614j,
         92024.57013790+88700.72926733j, ...,
        -31904.27748675-39364.56851422j,   92024.57013790-88700.72926733j,
        255374.61463530-17993.92648614j])
In [129]:
import math
sine = [numpy.sin(x/math.pi) for x in range(1000)]
In [130]:
xfft = fft.fft(sine)

df = pd.DataFrame()
df['sine'] =sine
df['ffti'] = [x.imag for x in xfft]
df['fftr'] = [x.real for x in xfft]
df[:].plot(subplots=True)
Out[130]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x7f535e1fd908>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f535e0be6a0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7f535e306f60>], dtype=object)
In [131]:
df.describe()
Out[131]:
sine ffti fftr
count 1000.000000 1.000000e+03 1.000000e+03
mean 0.005197 -5.684342e-17 -9.947598e-17
std 0.706704 1.081976e+01 1.955480e+01
min -0.999981 -1.990662e+02 -3.577702e+02
25% -0.699680 -4.233174e-01 -5.654399e-02
50% 0.012879 -5.190293e-15 2.330771e-01
75% 0.709885 4.233174e-01 2.929166e-01
max 0.999983 1.990662e+02 1.862269e+02
In [132]:
df.head(20)
Out[132]:
sine ffti fftr
0 0.000000 0.000000 5.197500
1 0.312962 0.052948 5.199377
2 0.594481 0.106019 5.205016
3 0.816273 0.159337 5.214444
4 0.956056 0.213027 5.227705
5 0.999785 0.267219 5.244862
6 0.943067 0.322045 5.265998
7 0.791600 0.377643 5.291215
8 0.560603 0.434158 5.320637
9 0.273282 0.491741 5.354412
10 -0.041494 0.550553 5.392711
11 -0.352102 0.610767 5.435733
12 -0.627335 0.672567 5.483709
13 -0.839540 0.736151 5.536901
14 -0.967398 0.801737 5.595609
15 -0.998063 0.869561 5.660173
16 -0.928453 0.939881 5.730982
17 -0.765564 1.012983 5.808478
18 -0.525759 1.089186 5.893164
19 -0.233132 1.168842 5.985611
$$A_k = \sum_{m=0}^{n-1} a_m \exp\left\{-2\pi i{mk \over n}\right\} \qquad k = 0,\ldots,n-1.$$
$$a_m = \exp\{2\pi i\,f m\Delta t\}$$

where $\Delta t$ is the sampling interval

Sixty-seven Minutes

Nelson Mandela Day

July 18th is Nelson Mandela's birthday, he would have been 97.

The United Nations calls it Mandela Day, or international day for freedom, justice and democracy. They say:

Every year on 18 July — the day Nelson Mandela was born — the UN joins a call by the Nelson Mandela Foundation to devote 67 minutes of time to helping others, as a way to mark Nelson Mandela International Day.

For 67 years Nelson Mandela devoted his life to the service of humanity — as a human rights lawyer, a prisoner of conscience, an international peacemaker and the first democratically elected president of a free South Africa.

There is more here: http://www.un.org/en/events/mandeladay/

There are a number of events being organised in Bermuda.

It looks like things might kick off early with this event at Bermuda College:

On the Eve of Nelson Mandela Day

Collaboration by:

ADHT, Bermuda College, Chewstick, HRC, Imagine Bermuda

Presents

Another Kind of ‘Happy Hour’

Based @ the Bermuda College Library

FRIDAY, JULY 17TH 5.30 – 7.30 pm

MUSIC……….POETRY………STORY-SHARING

NETWORKING…….NETPLAYING

Meet and Greet,

Even stump your feet,

Although Challenging,

Remember, Life is Sweet!

There are others planning events, I will try to post details here.

I am thinking of going to different events and doing sixty-seven minutes of pi.

Trivia

In [7]:
import math
from ipython_doctester import test

# FIXME 
# ipython_doctester broken, might be this prob: https://github.com/catherinedevlin/ipython_doctester/pull/5
#@test
def prime(n):
    """ Simple primality tester

    Examples:
    
    >>> prime(15)
    False

    >>> prime(67)
    True
    """

    # Deal with some small numbers
    if n <= 2: return True

    factor = 2

    while factor < math.sqrt(n):
        if 0 == (n % factor): return False
        
        factor += 1

    return True
In [9]:
# Is 67 prime?
print("Is 67 prime?")
print(prime(67))

# Mandela would have been 97
print("Is 97 prime?")
print(prime(97))
Is 67 prime?
True
Is 97 prime?
True

Sixty-seven minutes of $\pi$

Angles are measured in different units. Mostly, people seem to use degrees.

A right angle is 90 degrees. The angles in a triangle add up to 180 degrees.

In mathematics it is often useful to use different units, radians.

$\pi$ radians is equal to 180 degrees. So the angles in a triangle add up to 180 degrees.

Now just as there are 60 minutes in an hour, there are also 60 minutes in a degree.

So, 67 minutes of $\pi$ is just 67 minutes from 180 degrees.

$$ x = \pi * 67 / (180 * 60)$$
In [10]:
# In python

math.pi * 67 / (180 * 60)
Out[10]:
0.019489509980603347
In [11]:
# Now 67 minutes of pi number begins 0.01948.

1948 + 67 
Out[11]:
2015

So if Mandela had started his campaigning on his birthday in 1948 he would still be at it today.

The UN are just asking you to spend 67 minutes helping others on his birthday.