The LAMP stack has been around for years, Linux, Apache, MySQL and PHP/Perl/Python/Parrot?

It is a stack built on free software that drove a significant phase in the growth of the internet around 2000.

Linux, a free operating system.

Apache, a free webserver

MySQL, a free database, a friend described it to me as, "faster than a very fast thing*.

P, free languages, many beginning with P.

At the time this stack became popular I was working in the corporate finance world.

The company was looking at future tech choices and looking for a database.

There were basically two candidates, MySQL and Microsoft SQL Server.

The LIMP Stack¶

The company already had a heavy investment in Microsoft and was starting to use SQL Server more and more.

Many key products they depended on also uses SQL Server.

In the end this swung the decision. Oh and a belief that MySQL was not ready for prime time.

The company was already running some of its internal modelling on Linux. It was a natural migration from proprietary Unix that had been used previously. It worked.

But they were short of free software expertise and understanding. And the main advocate in the company was sick with an (as yet) undiagnosed neuro-muscular disease as well as mildly bi-polar.

So the company went with Linux, IIS, Micrsoft and Python.

I call it the LIMP stack, but you have to walk before you can run.

But working cross platform is harder for everyone. I was delighted to be able to use python and linux. I knew I was way more productive in that environment.

I knew that it would give us flexibility. It would allow us to use cheaper hardware and get more out of it. It would future proof us through the freedom that the free software licenses were granting.

It allowed us to understand at every level, how the software was actually working.

Most of all, it allowed others to do the same. And they did. And just by using this stuff they helped make it better.

Now just this week I read that NSA built is key infrastructure on top of free software. The GPL license gives them the freedom to use the software how they want.

It trusts that, people will in general use it for good purposes. Defining good is just too hard, so it does not try.

I just blogged about Mandela Day. Mandela used to be described as a terrorist. Now he is recognised by the United Nations as a relentless campaigner for freedom, justice and democracy.

Perspectives change.

But the free software community is full of people with good motives. If they feel a project is going in a bad direction, it may be forked into a new project.

Forks cause some harm, as their is division of effort. Eventually, one project wins out. More often than not projects will rejoin.

There is currently a fork in the MySQL project. Oracle own the MySQL code (they acquired it from Sun), but it is published under the GPL.

The original authors of MySQL have started a fork project, MariaDB.

I expect these projects will one day re-unite.

Then there is the postgres database. It has a special Geographic Information System sister project called postGIS.

The neat thing is that switching databases in many projects is not too hard, particularly if you are going from one free database to another.

For example, the django web framework supports multiple backend databases.

It is the same story wherever you use free software.

LIMP was a very bold choice at the time and it served the company very well. It was a great place to work, a very open organisation.

I am really enjoying having this weather station. I say weather station, but it is just a raspberry pi with a pressure and temperature sensor attached to it. Computers are versatile, you can run any software on them, so they can do a lot of different things.

But the pi takes this to a whole different level. They are $40 or so, depending on taxes and stuff. Now you need a whole bunch of other stuff: leads, keyboards, sensors, cameras, touch screens and lots more.

I have been using Adafruit. They have been very helpful with orders.

I found browsing for parts and finding what I needed to get started with the weather stuff a bit confusing for a while.

I had other stuff to do anyway, I needed to get used to just installing software on them and setting up environments where I can figure out how things are working.

I found a great posting about building a weather station, with enough detail I thought I would be able to work through it.

I order a cheap humidity sensor ($5) that also should do temperature. I haven't got it working yet, not sure if it is hardware or software. Meanwhile, I now have a better sensor. The humidity here is often outside the range the cheaper sensor is supposed to work.

The good thing is that I should be able to find a use for it at some point.

I also have a camera for this thing and a touchscreen. I am thinking of trying to combine them to make a camera.

The one on my phone can do some neat stuff, but the interface keeps changing and not in ways that are making things easier.

The night skies have been spectacular of late, with Venus and Jupiter close together in the early evening sky to the west. If you see two bright stars after sunset, that is probably them.

Then the moon is just past full. Here in Bermuda there is often cloud, not heavy, but patchy clouds. The humidity is often very high, so the atmosphere is interesting.

The best sunsets usually have some clouds for the setting sun to reflect off. The same with sunrises.

And the full moon too. Tonight it rose behind cloud. Last night it was clearer and it rose orange/red.

Back to weather¶

Since I got this thing working it has been hot and settled weather. There is a strong Bermuda high in place. Someone described it to me as like a pit-bull, once it gets hold it does not let go. So, we may be in for a long hot spell.

With the pressure changing quite smoothly, the plots I created still showed quite an interesting pattern.

Skip the next bit, or go back to the earlier post, it is just setting things up to do some plotting.

# Tell matplotlib to plot in line
%matplotlib inline

# import pandas
import pandas

# seaborn magically adds a layer of goodness on top of Matplotlib
# mostly this is just changing matplotlib defaults, but it does also
# provide some higher level plotting methods.
import seaborn

# Tell seaborn to set things up
seaborn.set()



def smooth(data, thresh=None):
    
    means = data.mean()

    if thresh is None:
        sds = data.std()
    else:
        sds = thresh
    
    delta = data - data.shift()
    
    good = delta[abs(delta) < sds]

    print(good.describe())
    
    return delta.where(good, 0.0)

# set the path to the file we are going to analyse
infile = "../files/light.csv"

!scp 192.168.0.127:Adafruit_Python_BMP/light.csv ../files

light.csv                                     100% 2025KB   2.0MB/s   00:01

""" assume it is csv and let pandas do magic

  index_col tells it to use the 'date' column in the data
  as the row index, plotting picks up on this and uses the
  date on the x-axis

  The *parse_dates* bit just tells it to try and figure out
  the date/time in the columne labeled 'date'.
"""
data = pandas.read_csv(infile, index_col='date', parse_dates=['date'])

# incantation to extract the first record in the data
start = data[['temp', 'altitude']].irow(0)

# smooth the data to filter out bad temps and pressures
sdata = (smooth(data, 5.0).cumsum() + start)

# now use smooth to throw away dodgy data, and plot the temp and altitude fieldsa
sdata[['temp', 'altitude']].plot(subplots=True)

               temp      pressure      altitude  sealevel_pressure
count  30304.000000  16155.000000  30077.000000       16182.000000
mean       0.000043      0.017580      0.000820          -0.000309
std        0.034749      2.460094      0.516706           2.460453
min       -0.400000     -4.000000     -2.330888          -4.000000
25%        0.000000     -2.000000     -0.331984          -2.000000
50%        0.000000      0.000000      0.000000           0.000000
75%        0.000000      2.000000      0.332114           2.000000
max        1.300000      4.000000      2.748358           4.000000

array([<matplotlib.axes._subplots.AxesSubplot object at 0x7fed54cfdfd0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed54a521d0>], dtype=object)

Temperature¶

The temperature plot shows a steady up and down, warming during the days, cooling a little, but only 2C at night.

There is one day, where I think we had some thunder storms where it dropped during the day.

The last week or so the temperature has been steadily climbing.

Pressure¶

The Pressure also shows slow drifting up and down. But there is this other strange ripple up and down.

I mentioned this to a meteorologist and immediately got the reply that it was because the atmosphere is tidal.

So the pattern we see in the pressure should be driven largely by the moon.

import astropy

from astropy import units
from astropy import find_api_page

# find_api_page is handy, even opens a browser windo.

#find_api_page(units)

# astropy is cool, but I need data for the moon.  Let's try pyephem

# uncommented the line below and run this cell if you need to install using
# pip.  This will install into the environment that is being used to run your
# ipython notebook server.

#!pip install pyephem

import ephem

# Create a Moon

moon = ephem.Moon()

# Tell it to figure out where it is
moon.compute()

# pring out the phase
moon.phase

12.865459442138672

def moon_orbitals(index):
    """ Given an index of times create a DataFrame of moon orbitals
    
    For now, just Phase, geocentric latitude and geocentric longitude
    """
    
    # Create dataframe with index as the index
    df = pandas.DataFrame(index=index)
    
    # Add three series
    df['phase'] = pandas.Series()
    df['glat'] = pandas.Series()
    df['glon'] = pandas.Series()
    
    # Now generate the data
    # NB this is slow, solpy might work out faster
    moon = ephem.Moon()
    for ix, timestamp in enumerate(index):
        
        # Compute the moon posigion
        moon.compute(timestamp.strftime("%Y/%m/%d %H:%M:%S"))
        
        df.phase[ix] = moon.phase
        df.glat[ix] = moon.hlat
        df.glon[ix] = moon.hlon
        
    return df

# See what we got

moon = moon_orbitals(data.index)
moon.describe()

moon.plot(subplots=True)

array([<matplotlib.axes._subplots.AxesSubplot object at 0x7fed54facdd8>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed547378d0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed547ad630>], dtype=object)

# Try feeding in a longer time series
days = pandas.date_range('7/7/2015', periods=560, freq='D')

moon_orbitals(days).plot(subplots=True)

array([<matplotlib.axes._subplots.AxesSubplot object at 0x7fed544abd30>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed5451c2b0>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed54841b70>], dtype=object)

sdata['moon_phase'] = moon.phase
sdata['moon_glat'] = moon.glat
sdata['moon_glon'] = moon.glon

# FIXME -- must be a pandas one liner eg data += moon ?

sdata[['temp', 'altitude', 'moon_phase', 'moon_glon', 'moon_glat']].plot(subplots=True)

array([<matplotlib.axes._subplots.AxesSubplot object at 0x7fed54641978>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed545cb470>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed54596518>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed54550550>,
       <matplotlib.axes._subplots.AxesSubplot object at 0x7fed544d6d30>], dtype=object)

The latitude is almost in phase with the phase of the moon, at least at the moment.

Next job is to add these series to our data frame and then take a look at scikit-learn

print(sdata.index[0])
sdata.index[0].hour + (sdata.index[0].minute / 60.)

2015-06-25 15:07:37.174400

15.116666666666667

sdata.describe()

def tide_proxy(df):
    # Create dataframe with index as the index
    series = pandas.Series(index=df.index)
    
    for ix, timestamp in enumerate(df.index):
        hour_min = timestamp.hour + (timestamp.minute / 60.)
	
        hour_min += df.moon_glat[ix]

        series[ix] = hour_min
        
    return series

xx = tide_proxy(sdata)

xx.plot()

xx.describe()

count    30397.000000
mean        12.141895
std          6.996605
min         -0.087386
25%          5.998249
50%         12.296909
75%         18.230285
max         24.070675
dtype: float64

# See what we got
moon = moon_orbitals(data.index)
moon.describe()

sdata['tide'] = tide_proxy(sdata)

fields = ['altitude', 'temp', 'tide']

sdata[fields].plot()

<matplotlib.axes._subplots.AxesSubplot at 0x7fed54167be0>

sdata.temp.plot()

<matplotlib.axes._subplots.AxesSubplot at 0x7fed5425d550>

with open("../files/moon_weather.csv", 'w') as outfile:
     sdata.to_csv(outfile)

	phase	glat	glon
count	30397.000000	30397.000000	30397.000000
mean	58.143320	0.009210	3.272469
std	36.001448	0.063526	1.943755
min	0.172035	-0.087831	0.000126
25%	22.124449	-0.062341	1.274654
50%	69.537071	0.026644	3.743553
75%	91.296631	0.070193	4.961365
max	99.811104	0.087505	6.283139

	altitude	pressure	sealevel_pressure	temp	moon_phase	moon_glat	moon_glon
count	30397.000000	0	0	30397.000000	30397.000000	30397.000000	30397.000000
mean	-31.879161	NaN	NaN	29.392555	58.143320	0.009210	3.272469
std	20.589351	NaN	NaN	0.894486	36.001448	0.063526	1.943755
min	-69.792586	NaN	NaN	27.200000	0.172035	-0.087831	0.000126
25%	-50.325302	NaN	NaN	28.700000	22.124449	-0.062341	1.274654
50%	-34.975717	NaN	NaN	29.400000	69.537071	0.026644	3.743553
75%	-10.941520	NaN	NaN	30.000000	91.296631	0.070193	4.961365
max	8.379000	NaN	NaN	31.500000	99.811104	0.087505	6.283139

	phase	glat	glon
count	30397.000000	30397.000000	30397.000000
mean	58.143320	0.009210	3.272469
std	36.001448	0.063526	1.943755
min	0.172035	-0.087831	0.000126
25%	22.124449	-0.062341	1.274654
50%	69.537071	0.026644	3.743553
75%	91.296631	0.070193	4.961365
max	99.811104	0.087505	6.283139

LAMP and LIMP stacks

The LIMP Stack¶

Air pressure and the tides

Back to weather¶

Temperature¶

Pressure¶