It is also a slide deck for the talk given at the ScalaSyd meetup May 2018.
To switch between modes press a number as follows :

Arrow keys navigate to next or previous section or slide

Refactor

Ugly Scala Code

9th May 2018

 

Code Weaver

 

By

Karl Roberts

@MrK4rl

To present this document press 2. Press Esc to get back to document view. Left and Right arrow keys to navigate. See suited.js

In this blog I’ll be talking about working with Scala to build applications.

I have a bunch of justifications for this behaviour they may not be good excuses but here they are.

The code written in the style, that you are going to see, runs ok (I think) but I am always wary of it, because I know how hard I had to work to get it running. Coding it was a trial and error affair, with contant compiling and running and attaching debuggers to test what was happening to my mutable placeholders.

The trick is to refactor it to a more imutable functional style as soon as it is “working”.

It is always best to refactor runnable code because we can see that the refactor caused no disernable difference to the outcome by simple running it.

Tests help of course, but the problem with testing mutable sections of code is that you have to test from the outside, as alluded to above it is hard to test an isolated fragment to see if it works.

This code is extracted from a Crypto currency tool I wrote to help me keep track of my position on coinspot.

Unfortunatly at the time coinspot api did not allow me to get all the data from them so I needed to extract my trade history as csv, parse it and build up a data structure to allow me to query my position and other useful info.

Using my Position data structure I can see what currencies I’m making money in and ask questions ike “how much bitcoin did I buy below this price?”

The big problem solved above is that the trade history showed prices for cross trades in terms of each other eg bought 1.2 ETH for .05 BTC but I want to know my position in AUD for each currency, so I was forced to look to the internet for historical prices of the coins in AUD at the price of the time of the trade to work backwards.

Here you can see I have a list of previously known prices from which I build up a PriceTable which is a Map of PriceTableForCurrency.

This is really a big map of maps. The caller of this function (in this case the main function) saves the newly discovered prices to a file that I read and parse on start-up to contruct my known prices List.

In short what I need to do is:-

For context you should know that to help me work around the Map of Maps in the imperative style I created a few type aliases and helper classes to make it easier to reason about, the main structures are as follows:-

// a type alias
type PriceTable = Map[Currency, PriceTableForCurrency]

Here PriceTableForCurrency is just a case class that wraps a Map but has a helpful API that allows me to augment it and build it up from HistoricPrice objects which I get from parsing the CSV file of previously discovered historic prices.

case class PriceTableForCurrency(currency: Currency, prices: Map[LocalDateTime,Double]) {
  /**
    * Will overwrite if we alredy have a price at this time
    * @param historicPrice
    * @return
    */
  def +=(historicPrice: HistoricPrice) = {
    val hp = (historicPrice.localDateTime, historicPrice.quotePrice)
    this.copy(prices = prices + hp)
  }
  def ++=(other:PriceTableForCurrency) = if(other.currency == currency) this.copy(prices = prices ++ other.prices) else this

  def asHistoricPrices: List[HistoricPrice] = {
    import scala.collection.mutable.ListBuffer
    val lb = ListBuffer[HistoricPrice]()
    prices.foreach(tup => {
      lb += HistoricPrice(tup._1,currency,tup._2)
    })
    lb.toList
  }
}

The first part I want to look at is building up the initial PriceTable (effectivly a Map of Maps) from the known prices list, which I read in from a CSV file.

Building a Map of Maps is something that I do a lot while coding, and it is a pain every time.

Walking a list and examining the Map to see if I have the Key in the map already and adding it and a value if I don’t have it or modifying the value if I do have it is tedious for a single Map. It is exponentially more tedious and error prone for each level of nested Map.

If only there was a better way.

This makes sense. A Map can be thought of as a list of Key and Value pairs.

So if I can convert my List into a list of Key/Value pairs, ie a Tuple2 or (Key,Value), I should be able to turn it into a Map.

Awesome. I can do it using the standard library.

Mmm. This is giving me a clue. Maybe I can refactor this.

This is good I’m getting a raging clue now.

That was great. Now I have a functional tool to construct maps and maps of maps.

List.groupBy

Now fix the ugly foreach

Ok now that is done lets look at the ugle walk of the xtrades (cross trades) list using foreach.

While its better than a forloop (which could adds another possible iteration bug) the body of the foreach is convoluted. The problem is that it is doing too many things at once.

I need to break it down into smaller problems which will not only make it easier to test and debug but will structure the code around clearer thinking.

That was simple. As a side benefit, now I am not iterativley Building the lists and Maps, I can get rid of all the var placeholder Maps and Buffers

A close look at the the code shows that I was able to stuff the new prices into the imutable PriceTable by folding down the new prices.

Ok

So now the code is tidy and manageble, is it time for me to speed up that slow price fetch from the internet?

$ git checkout start_5
$ # clear out the existing prefetched prices
$ cat /dev/null >  $HOME/uglycode/coinspotpos/historicPrices.csv
$ sbt clean run

So lets have a look at how much tidier the code is now.

Nice.

It is also a slide deck for a talk given at the ScalaSyd meetup Sept 2017.
To switch between modes press a number as follows :

Arrow keys navigate to next or previous section or slide

A Markdown combinator parser in

Scala

13th September 2017

To present this document press 2. Press Esc to get back to document view. Left and Right arrow keys to navigate. See suited.js

A Markdown combinator parser in

Scala.js

13th September 2017

To present this document press 2. Press Esc to get back to document view. See suited.js

This Blog is my talk for ScalaSyd meetup on September 13th 2017. It is about Scala combinator parsers and Scala.js

Before diving into nitty-gritty details it’s helpful to explain my motivation for this.

So apart from getting down with all the transpiling kool kids I actually have a need to do some JavaScript jiggery-pokery.

Suited is cool and I’ve given many talks using it and the same document can be a blog or article too. But we always want it to do more for instance we wanted to write out talks in markdown.

So we needed a markdown plugin, but we also wanted a magic syntax to add slides and figures in markdown, Suited.js uses <section data-figure> and <section data-slide> to mark out sections of the contents to appear as navigable sections. Slides are just visible in the slide show whereas figures are visible in doc mode and also in the slide show.

~~* **your bold stuff** *~~

~~: **your bold and _italic_ stuff** :~~

So we implemented a markdown plugin, but delegated to markdown-it.

But suited has a fatal flaw. It has no parser to implement fragments

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<section data-slide>
### Why?
* &nbsp;
  - &nbsp;
</section>

<section data-slide>
### Why?
* because
  - &nbsp;
</section>

<section data-figure>
### Why?
* because
  - it's like that!
</section>

Because I want to rely on a few libraries as possible while experimenting with Scala.js I decided to use Mark Hibberd’s Combinator Parser demo code from his previous Scalasyd talk “Dont fear the parser” as a starting point.

case class Parser[A](run: String => ParseState[A])

1
2
3
4

sealed trait ParseState[A]

case class ParseOk[A](input: String, value: A) extends ParseState[A]
case class ParseKo[A](message: String) extends ParseState[A]

Parsing is OK but I also want to render the parse tree into HTML, this is what Transformers are for.

1
2
3
4
5
6

trait Transformer[T] {
  type IN
  type OUT

  def run(t: T, in: IN): OUT
}

1
2
3
4
5
6
7
8
9
10
11

// syntax pimps
implicit class TransformerOps[T0](foo: T0) {

  /**
   * magic wand. pimp alias of Transformer.run eg a transform function
   */
  def ---*[A,B](bar: A)(implicit aux: Transformer.Aux[T0,A,B]) : B = {
    aux.run(foo, bar)
  }
  ...
}

See my previous talk on the Aux Pattern

Now lets look at a specific member of the Transformer typeclass. The MarkdownToHtml transformer.

I wanted to split the transform function into two parts, one to parse, and of course I want to use my markdown parser’s run function in this place, and one to render the ParseResult into HTML.

Not only does this make it easier to reason about but also (I Hope) easier to build a plugin interface where you can supply a function to parse modified markdown as long as it still produces a ParseResult[Markdown] and one to enrich the HTML or even transform it completely, say into rich text or PDF

1
2
3

case class MarkdownToHtml(
    p: String => ParseState[MarkdownDoc],
    r: ParseState[MarkdownDoc] => Html)

1
2
3
4
5
6
7
8
9
10

object MarkdownToHtml {
  import Transformer._
  import ast._

  // Use `instance` typeclass constructor to add MarkdownToHtml to the Transformer typeclass
  implicit val m2hTransformer:
     Transformer.Aux[MarkdownToHtml,String, parser.Html] =
       instance( (t, in) => t.r(t.p(in)) )
  ...
}

1

simple ---* """## this is h2 i presume"""

So let’s demonstrate the parser and renderer in the console.

me@host $ sbt
sbt:Markdownem> console
scala> :paste consoleimports.txt
Pasting file consoleimports.txt...
import parser._
import parser.markdownParser._
import transformers._
import transformers.Transformer._
import transformers.MarkdownToHtml._

Notice that

1

[warn] Scala REPL doesn't work with Scala.js. You are running a JVM REPL.

The Scala REPL is only available to the Scala code. So only code built without depending on JavaScript libraries will work in this REPL demo.

But that’s OK, this demo has no JavaScript dependencies, it just generate JavaScript output.

1

scala> headerParser.run("""## this is h2 i presume""")

1
2

res0: parser.ParseState[ast.Markdown] =
  ParseOk(,H2(List(this is h2 i presume)))

1
2

scala> headerParser.run("""## this is h2 i presume
     | more input not in header""")

1
2
3

res3: parser.ParseState[ast.Markdown] =
  ParseOk(more input not in header,
    H2(List(this is h2 i presume)))

Notice that this is a successful parse, and that the ParseOK case class also contains the remaining input for further processing

Also notice that the H2 contains a List of more Markdown, in this case rawHtml. This is because the link text could contain more inline markdown, like so…

1

scala> headerParser.run("""## this _is_ h2 i **presume**""")

1
2
3
4

res1: parser.ParseState[ast.Markdown] =
  ParseOk(,H2(List(this ,
                Italic(List(is)),
                h2 i , Bold(List(presume)))))

1

scala> headerParser.run("""this _is_ NOT h2 i presume""")

1
2
3
4

scala> headerParser.run("""this _is_ NOT h2 i presume""")
headerParser.run("""this _is_ NOT h2 i presume""")
res0: parser.ParseState[ast.Markdown] =
  ParseKo(Input failed to match predicate.: instead saw char 't')

Admittedly, this is not the best error message (for better errors use the built in Scala combinator parser library rather than this demo). It failed to see a # as the first char in a header and so produced a ParseKo instead of ParseOk.

The important thing to see is that a parser stops as soon as it can go no further.

In order to parse a whole document we can use the combinator methods, such as ||| i.e. or on Parser and some of the parsers such a list to compose more powerful parsers that continue parsing input by trying one parser after another.

An example is the markdownParser which will try all blockParsers followed by the inlineParsers continuously until it can parse no more.

However that will only occur when there is no more input because the last parser it tries is always the rawHtml parser which will always succeed because any input that is not parsed as markdown must be is just treated as RawHtml.

1
2

scala> markdownParser.run("""this _is_ NOT h2 i presume
     | ## but this _is **h2**_""")

1
2
3
4
5
6
7

res4: parser.ParseState[List[ast.Markdown]] =
  ParseOk(,List(this ,
             Italic(List(is)),
              NOT h2 i presume,
             Hardwrap,
             H2(List(but this ,
               Italic(List(is , Bold(List(h2))))))))

Now we can see that the input parses into a List of Markdown. This is all very well but it’d be nice to see the rendered HTML.

This is where simple MarkdownToHtml Transformer comes in.

I can now use the ---* magic wand function

1
2

scala> simple ---* """this _is_ NOT h2 i presume
     | ## but this _is **h2**_"""

1
2
3
4

res1: transformers.MarkdownToHtml.m2hTransformer.OUT =
"this <em>is</em> NOT h2 i presume
<h2>but this <em>is <strong>h2</strong></em></h2>
"

1
2
3
4
5
6
7
8

scala> simple ---* """* unordered list item
     |   1. _nested_ list item
     |   2. **another _nested_** list item
     | * unorderd [link](http://foo.bar) list item
     | 
     | [a ref link][1] whose url detail is at the end
     | 
     | [1] http://the.reflink.com"""

1
2
3
4
5
6
7
8
9
10
11
12
13
14

res0: transformers.MarkdownToHtml.m2hTransformer.OUT =
"<ul>
    <li>unordered list item</li>
    <ol>
        <li><em>nested</em> list item</li>
        <li><strong>another <em>nested</em></strong> list item</li>
    </ol>
    <li>unorderd <a href="http://foo.bar">link</a> list item</li>
</ul>

<a href="http://the.reflink.com">a ref link</a> whose url detail is at the end
<p>
<!-- [1] http://the.reflink.com --></p>
"

What about Performance?

I haven’t run proper benchmarks yet, but as a quick and dirty test I have added timing output to the main function.

1

scalaJSUseMainModuleInitializer := true

1
2
3
4
5
6
7
8
9

sbt
[info] Loading settings from idea.sbt ...
[info] Loading global plugins from /home/robertk/.sbt/1.0/plugins
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /home/robertk/projects/skunk/markdownem_js/project
[info] Loading settings from build.sbt ...
[info] Set current project to Markdownem (in build file:/home/robertk/projects/skunk/markdownem_js/)
[info] sbt server started at 127.0.0.1:5300
sbt:Markdownem>

1
2
3
4

sbt:Markdownem> run
[info] Running tutorial.webapp.TestApp
[error] java.io.IOException: Cannot run program "node": error=2, No such file or directory
...

What happened?

It is an error from sbt as it is trying to run the default JavaScript runtime Node.js to execute the generated JavaScript.

sbt can also run using other JavaScript runners such as PhantomJS, Selenium or Rhino see the docs

I had better load NodeJS onto my PATH, i use nvm for this.

1
2
3
4
5
6
7
8
9

sbt:Markdownem> exit
$ nvm use v7.9.0
Now using node v7.9.0 (npm v4.2.0)
$ sbt
[info] Loading settings from idea.sbt ...
...
[info] Set current project to Markdownem (in build file:/home/robertk/projects/skunk/markdownem_js/)
[info] sbt server started at 127.0.0.1:5300
sbt:Markdownem>

1
2
3
4
5
6
7
8
9
10
11

sbt:Markdownem> run
[info] Running tutorial.webapp.TestApp
>>>>>>>>>>> ms: -> 690
>>>>>>>>>>> ms: -> 443
>>>>>>>>>>> ms: -> 434
>>>>>>>>>>> ms: -> 447
>>>>>>>>>>> ms: -> 423
>>>>>>>>>>> ms: -> 430
[success] Total time: 4 s, completed 21/09/2017 9:53:01 AM
sbt:Markdownem>
...

The TestApp just renders 112 lines of markdown into HTML a few times.

We can see that after the initial run where it starts the node environment it ends up taking about 430ms.

That seems slow!

If we comment the JS lines in build.sbt we can build and run the main app in standard Scala on the JVM

1
2
3
4
5
6
7
8
9

// enablePlugins(ScalaJSPlugin)

// This is an application with a main method
// change this to true if you want the The TestApp main class to be a JS "Application"
// scalaJSUseMainModuleInitializer := true

name := "Markdownem"
scalaVersion := "2.12.2"
...

1
2
3
4
5
6
7
8
9
10
11
12
13

sbt:Markdownem> reload
sbt:Markdownem> run
[info] Compiling 8 Scala sources to /home/markdownem_js/target/scala-2.12/classes ...
[info] Done compiling.
[info] Packaging /home/markdownem_js/target/scala-2.12/markdownem_2.12-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[info] Running tutorial.webapp.TestApp
>>>>>>>>>>> ms: -> 592
>>>>>>>>>>> ms: -> 263
>>>>>>>>>>> ms: -> 234
>>>>>>>>>>> ms: -> 229
>>>>>>>>>>> ms: -> 225
>>>>>>>>>>> ms: -> 235

1
2
3
4
5
6
7
8
9

enablePlugins(ScalaJSPlugin)

// This is an application with a main method
// change this to true if you want the The TestApp main class to be a JS "Application"
scalaJSUseMainModuleInitializer := true

name := "Markdownem"
scalaVersion := "2.12.2"
...

1
2

sbt:Markdownem> reload
sbt:Markdownem> set scalaJSStage in Global := FullOptStage

1

sbt:Markdownem> set scalaJSStage in Global := FastOptStage

1
2
3
4
5
6
7
8
9
10

sbt:Markdownem> run
[info] Full optimizing ./target/scala-2.12/markdownem-opt.js
[info] Closure: 0 error(s), 0 warning(s)
[info] Running tutorial.webapp.TestApp
>>>>>>>>>>> ms: -> 590
>>>>>>>>>>> ms: -> 298
>>>>>>>>>>> ms: -> 281
>>>>>>>>>>> ms: -> 296
>>>>>>>>>>> ms: -> 270
>>>>>>>>>>> ms: -> 259

Runing in the sbt prompt is all very well.

But how can I run this code in my browser?

1

import scala.scalajs.js.annotation.JSExportTopLevel

1
2

@JSExportTopLevel("mdmagic")
def transform(md: String): String = simple ---* md

1
2
3
4
5
6

sbt:Markdownem> fullOptJS
[info] Compiling 8 Scala sources to ./target/scala-2.12/classes ...
[info] Done compiling.
[info] Full optimizing /home/robertk/projects/skunk/markdownem_js/target/scala-2.12/markdownem-opt.js
[info] Closure: 0 error(s), 0 warning(s)
[success] Total time: 16 s, completed 21/09/2017 11:26:04 AM

1

<script type="text/javascript" src="./target/scala-2.12/markdownem-opt.js"></script>

1
2
3

element = document.getElementById(elementId)
theContent = element.innerHTML
element.innerHTML = mdmagic(theContent)

It is also a slide deck for a talk givn at Scalasyd April 2017.
To switch between modes press a number as follows :

Arrow keys navigate to next or previous section. or slide

The Rise (and hopefully fall) of

The Aux Pattern

12th April 2017

To present this document press 2. Press Esc to get back to document view. Left and Right arrow keys to navigate. See suited.js

This Blog is my talk for Scalasyd April 12th 2017. It is about the Aux Pattern as seen all over the Shapeless library.

We’ve all seen code like this:-

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

object IsHCons {
  def apply[L <: HList](implicit isHCons: IsHCons[L]): Aux[L, isHCons.H, isHCons.T] = isHCons

  type Aux[L <: HList, H0, T0 <: HList] = IsHCons[L] { type H = H0; type T = T0 }

  implicit def hlistIsHCons[H0, T0 <: HList]: Aux[H0 :: T0, H0, T0] =
    new IsHCons[H0 :: T0] {
      type H = H0
      type T = T0

      def head(l : H0 :: T0) : H = l.head
      def tail(l : H0 :: T0) : T = l.tail
      def cons(h : H0, t : T0) : H0 :: T0 = h :: t
    }
}

Alphabet soup if ever there was some. Hopefully at the end of this, we’ll all be able to read this. But before I skip on, notice the Aux all over that code. This is the source of the name of the Aux pattern.

The first thing to realise is that it is not a Pattern that can apply to a programming paradidm It is really a hack. An elegant hack, but never-the-less a hack. Definatly a hack.

To be kinds to it, it has become quite pervasive as a way to work around a problem so I guess I can call it an idiom specific to Scala and its type system.

Being an idom it has become a common way to get stuff done in scala, but what is it solving? To answer that we need a little ramble around the scala type system.

Type inference

First let’s talk about type inference

A pretty useful feature to have, as I’m sure you’ll agree, as it helps cut down on “Type boiler-plate”, while letting the compiler asset correctness for us.

So let’s look an simple example

object InferenceTest1 extends Application {
  val x = 1 + 2 * 3         // the type of x is Int
  val y = x.toString()      // the type of y is String
  def succ(x: Int) = x + 1  // method succ returns Int values
}

As we can see the compiler is able to infer some types from the context. The next example shows that even with generic types we can infer types at the call site. I.e. when the specific type is first used.

case class MyPair[A, B](x: A, y: B);
object InferenceTest3 extends Application {
  def id[T](x: T) = x            // return type: T
  val p = new MyPair(1, "scala") // type: MyPair[Int, String]
  val q = id(1)                  // type: Int
}

with no hints the compiler knows that p is a MyPair[Int, String] and q is an int

as an a2tion" sometimes mislabled “type annotation”, for example here is the some code with more type ascription than is necessary in scala

def id[T](x: T): T = x
val p: MyPair[Int, String] = new MyPair[Int, String](1, "scala")
val q: Int = id[Int](1)

val b = 2 : Byte

The only necessary ascription here was on on val b which we want to be a Byte rather than let the default inference of Int.

Path Dependent Types

OK no lets switch to another Scala feature, path dependant types.

Why do I care?

let’s see an example

trait Food {
  override def toString(): String
}

object grass extends Food {
  override def toString: String = "grass"
}

object dogfood extends Food {
  override def toString: String = "dogfood"
}

In good OOP style I’ve used inheritance to create a type hierarchy.

I can then use these types to define some object instances.

abstract class BadAnimal  {
  def eat(fodder: Food): Unit = {
    println(s" ${this.getClass} Eating my food: "+ fodder)
  }
}

object badfeeder {
  def feed(animal: BadAnimal, food: Food): Unit = {
    animal.eat(food)
  }
}

class Cow extends BadAnimal with Food {
  override def toString = "Cow"
}

Now I can use the badfeeder object to feed Daisy the cow.

val Daisy = new Cow

//Acceptable
badfeeder.feed(Daisy, grass)

val Daisy = new Cow

//naughty
badfeeder.feed(Daisy, dogfood)

val Daisy = new Cow

//totally unacceptable. Canibalism
badfeeder.feed(Daisy, Daisy)

As you can see badfeeder was allowed by the compiler to turn Daisy into a masochistic canible.

The problem is that the feeder was allowed to give Food to an BadAnimal, and Daisy was Food as well as an BadAnimal . What we need is that an Animal can only eat food suitable for that animal.

But at the time we declare the Animal supertype we don’t know what is suitable food. The solution is to create an abstract type in Animal that will be specialised by each animal kind when we extend Animal. So as before…

trait Food {
  override def toString(): String
}

object grass extends Food {
  override def toString: String = "grass"
}

object dogfood extends Food {
  override def toString: String = "dogfood"
}

abstract class GoodAnimal {
  // path dependent type
  type SuitableFood <: Food

  def eat(food: SuitableFood): Unit = {
      println(s" ${this.getClass} Eating my food: "+ food)
  }
}

class Cow extends GoodAnimal with Food {
  type SuitableFood = grass.type
}

Now we are saying that an Animal can only eat food that is suitable, we don’t know what that is when we create the GoodAnimal class so we make it abstract and give it an Abstract type.

Subclasses can declare what the type is. Notice that we can refer to the abstract type in the method eat.

Now let’s try to define the feed function. We might be tempted to try using the # operator (type projectionn) to access the type defined inside the GoodAnimal …

def feed(animal: GoodAnimal, food: GoodAnimal#SuitableFood): Unit = {



  animal.eat(food)
}

def feed(animal: GoodAnimal, food: GoodAnimal#SuitableFood): Unit = {
//    OOPS cant do this it says
//     expected animal.Suitablefood, actual:GoodAnimal#SuitableFood
//
  animal.eat(food)
}

It won’t compile. Carefully reading the error message we see that we need animal.Suitablefood, notice that this is specific to this animal object instance!

//
//
//
def feed1(animal: GoodAnimal, food: animal.SuitableFood): Unit = {
      animal.eat(food)
}

// OOPs cant resolve symbol animal
// can't re-refer to a param in same parameter list
// type system works left to right
def feed1(animal: GoodAnimal, food: animal.SuitableFood): Unit = {
      animal.eat(food)
}

The problem here is that the Scala compiler’s type inferencer expects to be able to resolve all types statically in a parameter list, but when declaring food we don’t know yet what the type of actual type of animal is, we only know its abstract supertype. The compiler has a rule to prevent this problem, as it says in the error, you can't re-refer to a param in same parameter list

This is getting tricky. How can I use this fancy good animal?

The solution is to give the Compiler and type inference system a chance to work it out statically using another scala feature

Multiple parameter lists

// 
// 
def feed2(animal: GoodAnimal)( food: animal.SuitableFood): Unit = {
  animal.eat(food)
}

// solution is add a second parameter list now type is known at call site
// type resolution works left to right
def feed2(animal: GoodAnimal)( food: animal.SuitableFood): Unit = {
  animal.eat(food)
}

As shown in the previous section, by using multiple parameter lists we can use the type animal.SuitableFood in the second list because the compiler will have been able to reason about the type of GoodAnimal in the first list from its use at the call site.

Now the bad example are impossible.

val Daisy = new Cow
// Acceptable
goodfeeder.feed2(Daisy)(grass)

// naughty
// 
//

// totally unacceptable. Canabalism
//
//

val Daisy = new Cow
//Acceptable
goodfeeder.feed2(Daisy)(grass)

// naughty
// wont compile says: expected Cow:SuitableFood, actual dogfood.type
//goodfeeder.feed2(Daisy)(dogfood)

//
// 
//

val Daisy = new Cow
//Acceptable
goodfeeder.feed2(Daisy)(grass)

//naughty
// wont compile says: expected Cow:SuitableFood, actual dogfood.type
//goodfeeder.feed2(Daisy)(dogfood)

//totally unacceptable. Canabalism
// wont compile says: expected Cow:SuitableFood, actual GoodAnimal.Daisy.type
// goodfeeder.feed2(Daisy)(Daisy)

So far so good.

You will often see this kind of code in scala libraries.

def join[A: Monoid](a: A, b: A): A = {
  a |+| b
}

the form [S : T] in the type parameter list is actually syntactic sugar, called context bound, for this:-

def join[A](a: A, b: A)(implicit ev: Monoid[A]): A = {
  a |+| b
}

There is implicit evidence available that a Monaid[A] exists, and so we can use the |+| function from Monoid, nowing that an implicit converion to Monoid is available.

trait MyTypeClass[T] {
  type In
  type Out

  def doMagic(foo: T, param1: In): Out
}

not only are there too many implicit param lists but we also have no way to tie the In0 type to ev.In

object MyTypeClass {
//aux pattern
type Aux[T0, In0, Out0] = MyTypeClass[T0] {type In = In0; type Out = Out0}

// convenience Summoner so I don't need to use instead of `implicitly` 
def apply[T](implicit evidence: MyTypeClass[T]): Aux[T, evidence.In, evidence.Out] = evidence
}

So we create a type alias to my type, that allows all the path dependent types to be seen on the outside.

We also create a handy typeclass summoner that an be used instead of implitly when we need it.

case class Greeter(name: String) {

  def greeting(greeting: String): String = {
    s"$greeting, $name"
  }
}

case object ImAnAlien {

  @tailrec
  private def sayThis(thing: String, accumlator: String, numTimes: Int): String = numTimes match {
    //base case
    case 0 => accumlator
    //recurse
    case _ => sayThis(thing, accumlator + " " + thing, (numTimes - 1))

  }

  def say(num: Int): String = {
    sayThis("I'm an Alien", "", num)
  }
}

def foo[A,B](a:A):B 

  // known type class members
  implicit val greeterToTypeclass: Aux[Greeter, String, String] = new MyTypeClass[Greeter] {
    type In = String
    type Out = String

    override def doMagic(foo: Greeter, param1: In): Out = foo.greeting(param1)
  }

  implicit val imAnAlienToTypeclass: Aux[ImAnAlien.type, Int, String] = new MyTypeClass[ImAnAlien.type] {
    type In = Int
    type Out = String

    override def doMagic(foo: ImAnAlien.type, param1: In): Out = foo.say(param1)
  }   
}

object SomeApi {
  def voodoo[T, In0, Out0](foo: T, sayThis: In0)(implicit ev: MyTypeClass.Aux[T, In0, Out0]): Out0 = {
    val tc = MyTypeClass[T]
    tc.doMagic(foo, sayThis)
    ev.doMagic(foo, sayThis)
  }
}

Aliens:- 
 I'm an Alien I'm an Alien I'm an Alien

1
2
3
4
5
6
7
8
9
10
11

import shapeless.{HList, ::, HNil}
import shapeless.ops.hlist.Last

//use the implicitly
implicitly[Last[String :: Int :: HNil]]
// res6: shapeless.ops.hlist.Last[shapeless.::[String,shapeless.::[Int,shapeless.HNil]]] =shapeless.ops.hlist$Last$$anon$34@20bd5df0

// use the summoner
Last[String :: Int :: HNil]
// res7: shapeless.ops.hlist.Last\
// [shapeless.::[String,shapeless.::[Int,shapeless.HNil]]]{type Out = Int} =shapeless.ops.hlist$Last$$anon$34@4ac2f6f

Notice that with implicitly we loose information that Out is Int

object IsHCons {
  def apply[L <: HList](implicit isHCons: IsHCons[L]): Aux[L, isHCons.H, isHCons.T] = isHCons

  type Aux[L <: HList, H0, T0 <: HList] = IsHCons[L] { type H = H0; type T = T0 }

  implicit def hlistIsHCons[H0, T0 <: HList]: Aux[H0 :: T0, H0, T0] =
    new IsHCons[H0 :: T0] {
      type H = H0
      type T = T0

      def head(l : H0 :: T0) : H = l.head
      def tail(l : H0 :: T0) : T = l.tail
      def cons(h : H0, t : T0) : H0 :: T0 = h :: t
    }
}

We can now see that we have an Aux type that promotes the HCons’s abstrat types for head and tail of the HList, H and T into type parameters so they can be captured or bound.

We provide an implict function to produce an Aux tpe when needed that uses the IsHCons typeclass and fills in or binds the H and T to the type parameters in the Aux. and we also provide a summoner method to fetch the Aux from the implicit scope if it is available, ie if there is an IsHCons implicity available that matches the desired type parameters.

It seems that they compiler restriction to prevent multiple implicit parameter lists is simply a syntactic sugar restriction. Miles Sabin, the initiator of the Shapeless library has proposed a change to remove the restriction, which looks like it will get accepted into a future version of Scala.

So while many use cases of the Aux pattern will be able to be expressed at the call site in terms of multiple implicit parameter lists, the Aux pattern is still a useful object lesson in how to use the type system to gain access to the Abstract types defined inside other types which could still be useful to keep the API of a library simple.

I was thinking about how to run a very fast native function, e.g. compiled with Rust with no dynamically linked libs (all static) and found an article showing how to launch the executable even though it is not yet directly supported by AWS by wrapping it in a node.js lambda.

This got me thinking, is node the fastest way to do this, after all if I’m a speed freak and writing Rust then launch times are important to me…

So I looked at at launching a process from node and python and timing it, I did not investigate C# or Java as they will always take longer to start because their runtime is doing more stuff.

This is not a statistically viable experiment (I only ran it once :-) but I built a shell tool called launch.sh, see the gist, to spit out a date including nanoseconds (my laptop supports this) then kick of a python app that uses a subprocess to run the same date command twice from within the python app. It then does the same using node.js.

Running it I get this data :-

me@storm:~/bin$ ./launch.sh 
start launch python data subprocess...
09:57:25 1489532245 519378382
09:57:25 1489532245 530775400
09:57:25 1489532245 531650629

start launch python compiled data subprocess...
09:57:25 1489532245 533984993
09:57:25 1489532245 541690675
09:57:25 1489532245 542552837

Now using node v7.6.0 (npm v4.1.2)
start launch node subprocess
09:57:25 1489532245 712281203
09:57:25 1489532245 750742945
09:57:25 1489532245 751821095

The first date in each triplet is the date at the shell before launching python or node followed by the subprocess date commands inside the python or node script.

The last set of numbers on each line is nanoseconds.

I also pre-compiled a python script to see if it was faster than executing the script, but forgot that python would do a search for pre-compiled scripts version of commands so the results of the first two are probably actually running the compiled version, the first is slower to launch presumably searching to see if there is a pre-compiled version in the directory (N.B. this is a guess as I don’t have enough data here to prove it). Regardless I removed the compiled python and ran again to get un-compiled time data:

start launch python data subprocess...
10:34:30 1489534470 041312697
10:34:30 1489534470 054794071
10:34:30 1489534470 055933101

The results are :-

Uncompiled python takes 13481374 ns to launch and run the subprocess
Uncompiled python takes  1139030 ns to launch and run a subprocess once it is already running

  Compiled python takes  7705682 ns to launch and run the subprocess
  Compiled python takes   862162 ns to launch and run a subprocess once it is already running 

node.js           takes 38461742 ns to launch and run the subprocess
node.js           takes  1078150 ns to launch and run a subprocess once it is already running 

So the results show it is currently best to use compiled python to kick off a sub process e.g. a Rust or Go procedure in AWS lambda

When Go or Rust are natively supported by AWS Lambda containers no doubt the story will change.

For those who don’t know, an AWS Lambda enables so-called “Serverless” event processing. On an “event” or “request” AWS will run your Lambda code function, and charge you just for the time it takes to run rather than paying for the time of a full EC2 instance running for hours. So if your application only need to sporadically answer requests it may be a good cost trade-off.

The down side is that your application becomes more fragmented, which can become an Operations and maintenance issue, and you are now definitely tied to your cloud provider. If you need super responsive queries you may be better to run a traditional application on EC2 that can be ready and listening and pay for its uptime.

Anyway back to my topic of AWS Lambda response times…

According to this link they are run in “containers” I imagine that is similar to a docker container but a custom to AWS “chroot” jail?

If the container is still running from a previous invocation then there is no start up time for the container just the start-up time for the AWS lambda function.

Even if there is a container start-up time, if the underlying host is running (likely) then Docker containers are quick to start, like starting a process and a custom chroot jail may be quicker still. However some developer forums have stated that they can get 50-100ms response times from their Lambda’s but occasionally see significant 12-15 second delays for infrequent invocations.

One can only guess at the reason but some kind of provisioning or resource sharing bottleneck seems likely. No doubt AWS teams are working on this as “Serverless” compute becomes the new Cloud buzzword and battleground.

But, as I mentioned earlier, the true benefit of Serverless is cheaper running for request or event processing not blazingly fast performance and response times

Premature optimization is the root of all evil ... in programming

It is far better to have clean code and optimise where necessary after measuring everything in a production like environment.

In fact it may not be necessary to mess up the code, perhaps the solution is to run on better hardware?

Check out the video below.

I recently upgraded my laptop after 4 years of usage, and as well as the usual smug appreciation of my new hardware, I am blown away by the performance hike. Double your speed with a hardware upgrade

Like watching grass grow we don’t usually notice the ongoing benefit of the advances in computer hardware until you get to see it in time-lapse.

My old laptop was no donkey. A Samsung New Series 9 and yet next to my new Dell XPS 9560 it looks positively pedestrian.

I have the same version of Ubuntu 16.04 with the latest patches running on both machines, and I downloaded the same project I’m currently working on. I then started a Maven build side-by-side to compile and package the whole application I’m currently working on.

My new laptop is almost twice as fast as my previous fast laptop

New Dell XPS 9560 => 12.329 seconds vs old Samsung “New” series 9 => 21.365 seconds

A 12 second pause is no way as distracting as a 20 second pause, allowing me to stay in context and keep coding.

Nice.

sudo apt-get update
sudo apt-get upgrade

But recently I got a new laptop with the latest chipsets and peripherals and decided to upgrade to the new Ubuntu Xenial 16.04 LTS while I’m at it.

I’m sad to say that the combination of new distro with bleeding edge hardware has twice given me a problem.

I’m sure it’ll be stable soon but in the mean time I’ve had to revert to a previous kernel twice now, so this blog will help me remember what to do if it happens again.

The symptoms are dramatic. After a re-boot my laptop had no network drivers and no graphics drivers, the display reverts to an emergency emulated mode which has very slow mouse response times and it looks huge, like a “my first computer” children’s toy!

Not what I want from my brand new hardware.

I don’t have time or want to fix all the drivers, so the quickest fix is to revert to a previous kernel that was working, remove the broken one, which has the effect of blacklisting it then try the next kernel when it comes out.

Howto

Boot from previous kernel

1. Hold the shift key when you see the Grub screen, to get to the grub options.
2. you may have better luck holding the shift key all the time through the boot if you have a fast system.
3. Choose Advanced options for Ubuntu
4. Make a note of the current kernel numbers (top of the list) you’ll need them later to remove it.
5. Use the arrow key to pick a previous kernel and boot into it, hit enter.

Remove the dodgy kernel

This will remove the broken kernel and drivers, and lets the package manager know that you don’t want it again if you do an update. You should remove the specific broken kernel and it’s headers, don’t remove the super package linux-generic this is the package that Ubuntu uses to upgrade the kernel and headers when they become available. If you remove it you wont get kernel updates automatically and will have to specifically run apt-get to get them.

# use the kernel numbers from previous step to confirm that the broken kernel has been installed
# eg if the currently broken kernel was linux-image-4.4.0-64-generic it should show up in the following command.
dpkg -l | grep linux-image

# remove the broken kernel
sudo apt-get purge linux-image-4.4.0-64-generic

# remove its headers too
sudo apt-get purge linux-headers-4.4.0-64-generic

Reboot

On reboot you should boot into the previous safe version, hold shift down on boot and confirm that the broken kernel is not a choice in the grub advanced settings.

Because we haven’t removed the linux-generic package itself Ubuntu will still attempt to get a new kernel when one is available, just not the one you specifically purged.

That’s it.

After using docker for a while you may find that you want more control over the images you want to base your containers on.

If you have a Dockerfile, you can always use it to build an image locally using

docker build .

Alternatively if you want the image to have a tag name to make it easier to recognise

docker build -t myimage .

After the base image has downloaded and all the commands in the Docker file have run you’ll have an image locally.

This is great as it gives you the ability to spin up a container based on the image and run a command there, such as to kick off your micro-service.

# find the correct image id

$ docker image ls
REPOSITORY                                     TAG                 IMAGE ID            CREATED             SIZE
<none>                                         myimage             5f157f802a51        2 minutes ago       533 MB
ubuntu                                         xenial-20160914     45bc58500fa3        4 months ago        127 MB

# start the container and ls the home dir as an example rather than start
# my super-cool microservice
$ docker run --name mycont1 5f157f802a51 ls

A potential issue with the above process is that if I want a co-worker to kick off his micro service, or we want to run it in multiple “environments” then he will have to run the docker build first and then run it.

This takes some time

Imagine how long it would take if your container was based on ubuntu and in the Dockerfile you ran apt-get update and apt-get upgrade -y, not to mention the time it takes to install necessary software such as Java, Mono, Python or Ruby. The whole process could easily take 10 minutes or more to run the build.

But that isn’t the biggest problem. What if a Python dependency or Ruby Gem had changes since you ran your build? Not only could the install fail because, in this case, other different dependencies would need to be pre-installed but also your co-worker’s build will now be different to yours.

All your version controlled guaranties are now moot.

Clearly we want him to have the same exact image as us.

Create an image from your container

The way to do this is to create an image from your container. In Docker-speak we will commit the image. Think of it like committing all the changes to the layered file system that you have made in the container after starting it from an image.

If you run up your container as described above, then you need to find out its container ID.

If it is still running use:

$ docker ps

If it has stopped use:

$ docker ps -a

You will see info about your container, this is where naming the container using the -name flag, see above, will be useful. For example this is what I see:

$ docker ps -a
CONTAINER ID    IMAGE          COMMAND        CREATED         STATUS                       PORTS   NAMES
2fdc41c14fc6    5f157f802a51   "/bin/sh ls"   7 minutes ago   Exited (137) 13 seconds ago          mycont1

So the container Id for mycont1 is 2fdc41c14fc6. I can now use this to commit an image. Note that you can even cut an image if the container is running because by default it will pause the container before committing, see man docker commit

Now you can create the image

$ docker commit -m "my microservice added to ubuntu xenial" -a "Karl" 2fdc41c14fc6 ubuntu-xenial-mymicro

You can see the newly minted image as before

$ docker image ls
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
ubuntu-xenial-mymicro   latest              d2c713fdb832        10 seconds ago      795 MB
ubuntu                  xenial-20160914     45bc58500fa3        4 months ago        127 M

Push to a repository

Now we have an image we want to push it to our repository so our co-workers can pull and run up a container from it.

The trick to all of this is that Docker uses the image tag name in a special way. When you pull or run an image the name you give it is actually a location (URI) that also refers to the repository host. By default if no host is specified then the Docker hub repo is assumed.

Therefore if you wanted to push your image to Docker Hub, you would first create an account to get a Docker ID that also maps to a “repo” in Docker Hub, e.g. if my Docker ID was karlcode I would create an image tag that referenced my repo in Docker Hub. Remember that “Docker Hub” is assumed so I could do this:

docker tag ubuntu-xenial-mymicro karlcode/ubuntu-xenial-mymicro:latest
docker push karlcode/ubuntu-xenial-mymicro:latest

This creates a tag of my image, you can use docker image ls to confirm this, and then uses the tag to push the image to the repository host/repo of the same name as the tag.

Of course you would need to let docker know how to login to your docker hub account in order to run the push.

This is done with the docker login command.

It asks for your Docker Hub Docker ID and password and stores a token in your HOME directory at ~/.docker/config.json if you have not run docker login before, so you don’t need to login every time you push or pull.

Push to your private repository

The same principle applies to push to a private repository such as your company’s own Docker repository. The only difference is that you need to specify the repository host in the tag name.

Lets say that my repository was running at docker.owtelse.com running on port 443 and that my project is called magicmicro I may choose to do this:-

$ docker login docker.owtelse.com:443
$ docker tag ubuntu-xenial-mymicro docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:latest
$ docker tag ubuntu-xenial-mymicro docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:20170125

I would then have local images which we can see like:-

$ docker image ls
REPOSITORY                                                TAG              IMAGE ID      CREATED          SIZE
docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro   20170125         d2c713fdb832  10 minutes ago   795 MB
docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro   latest           d2c713fdb832  10 minutes ago   795 MB
ubuntu-xenial-mymicro                                     latest           d2c713fdb832  10 minutes ago   795 MB
ubuntu                                                    xenial-20160914  45bc58500fa3  4 months ago     127 M

You may notice that I have two alias for my image with two tags one with the date and the other with the latest tag.

Images with the latest tag are pulled by default if the tag is not specified.

This allows a user to easily get the latest version by not specifying a preference or by picking a specific release, e.g. for debugging a particular version or for a production environment.

Now to push it:-

$ docker push docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:latest
$ docker push docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:20170125

And that is it!

Test it

To prove this to yourself, delete the local images, remember to use your own image ID in the command below :-

$ docker rmi -f docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro
$ docker rmi -f d2c713fdb832

Then you can pull the image from your private repo and use docker image ls to confirm you have it

$ docker pull docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:latest

Alternatively you and you co-worker can simply try to run it. If you have already run docker login docker.owtelse.com:443 the image will download and run, when you type:

$ docker run --name mymicro docker.owtelse.com:443/magicmicro/ubuntu-xenial-mymicro:latest

For more info see the man page man docker push.

I hope you find this information useful and you are now able take more advantage of Docker in both the development and production environment.

The end.

You may just want to make sure that you webcam, usually connected to the internal USB bus, is turned off.

In my case my laptop webcam has become intermittently faulty. This has caused my laptop to hang on shutdown while the kernel tries to power down the device.

Clearly I want to disable the camera’s port without disabling all the other USB ports on the laptop…

Identify the port

The first step was to identify the camera’s port.

You can trawl through dmesg logs looking at all USB entries until you find the webcam, e.g.

$ dmesg
...
[    2.996705] usb 1-1.6: Product: Webcam SC-13HDL11624N
...

Here you can see that my webcam was on bus 1 - port 1 on the 6th device on that port e.g. 1-1.6

However Linux has some commands and a pseudo “sys” file system that make it easier, for instance you could use lsusb

$ lsusb -t
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 2: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
        |__ Port 2: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 5: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
        |__ Port 5: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
        |__ Port 6: Dev 5, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 6: Dev 5, If 1, Class=Video, Driver=uvcvideo, 480M

If we walk this tree we can see bus 01 Port 1 is a USB hub and port 6 on the hub is a video device, which is my webcam… but this is a bit obtuse and confusing, and it doesn’t yield the literal string 1-1.6 that I’ll need to disable the webcam port.

A better method is to walk the sys pseudo file system that gives info on all devices attached to the kernel system.

$ for device in $(ls /sys/bus/usb/devices/*/product); do echo $device;cat $device;done
/sys/bus/usb/devices/1-1.2/product
Dell USB Entry Keyboard
/sys/bus/usb/devices/1-1.6/product
Webcam SC-13HDL11624N
/sys/bus/usb/devices/2-1/product
Amazon Kindle
/sys/bus/usb/devices/2-2/product
USB Receiver
/sys/bus/usb/devices/usb1/product
EHCI Host Controller
/sys/bus/usb/devices/usb2/product
xHCI Host Controller
/sys/bus/usb/devices/usb3/product
xHCI Host Controller

Here we can see that the file /sys/bus/usb/devices/1-1.6/product contains the Webcam SC-13HDL11624N

So in this case the USB device (or port) we need is 1-1.6.

Because the Webcam is attached to the internal USB hub, it will always be listed at the same address, this is helpful as it means I can hard-code 1-1.6 where I need it rather than parsing the output of the command.

Turn off power to the Webcam

Now we know the USB device number, it is simple to turn off the camera using the sys file system.

By writing values to the “files” in the sys file system you can effect the devices that the file represents. Obviously you need to be root to do this, or be a user that has sudo permissions.

On Ubuntu

We can send a command to the USB driver to unbind a port,

$ echo '1-1.6' | sudo tee /sys/bus/usb/drivers/usb/unbind

Obviously replace “1-1.6” with whichever usb port your webcam is on (see above).

To turn it back on,

$ echo '1-1.6' | sudo tee /sys/bus/usb/drivers/usb/bind

This command pushed the USB device name, “1-1.6”, to a special socket that acts like a command API to the USB driver, in other words rather than directly control the power of the device we ask the USB driver to do it for us.

Run it at start-up

Now I have control over my USB ports I want to disable this port at start-up. Ubuntu uses the anacron cron daemon which allows a special syntax, @reboot, to hook a command to the reboot sequence.

Simple edit root’s crontab

$ sudo crontab -e

And then append the following

@reboot echo '1-1.6' > /sys/bus/usb/drivers/usb/unbind

Again edit ‘1-1.6’ to your usb device’s number.

And that’s it. My laptop can now shutdown without hanging.

Cheers.

The published: false statement is meant to prevent the blog from being prematurely published, eg while it is in progress.

The internet told me that running rake generate before rake deploy was supposed to remove blogs marked as published: false from the files to be published. However it was not working for me.

Octopress does allow blogs marked as published: false to be used in the preview task, so you can see your work in progress locally.

Looking through the Rakefile I discovered that a file called .preview-mode is used to handle this exemption.

It turned out that I had accidentally committed the .preview-mode file to git while running the preview. It was now always there! This messed up the deploy and enabled all my in flight blogs to be published.

The fix was simple.

1
2

git rm -f ./.preview-mode
git commit -m "removed .preview-mode, which was accidentally added"

SOLVED. rake generate and rake deploy now work properly again.

:-)

This blog intends to catalogue how I got my JS examples into the blog.

What do I need? …

I need to add a script tag in the blog post to include the JavaScript.

Luckily the Octopress markdown syntax that blogs are written in allows pure html, so I can simply add the script tags e.g. Here is how to use D3 to generate an SVG

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<div id="rect1" class="chart"></div>

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.5.0/d3.min.js"></script>

<script>
var square = d3.select("#rect1");

var svgContainer = square.append("svg")
    .attr("width", 100).attr("height", 100);

//Draw the Rectangle
var rectangle = svgContainer.append("rect")
    .attr("x", 10).attr("y", 10).attr("width", 50)
    .attr("height", 60).attr("fill", "red")
    .attr("stroke", "blue").attr("stroke-width", 5);
</script>

Notice that the <div> tag to hold the generated SVG came before the JavaScript?

If it had come after the JS the Square would not have rendered because the script would run as soon as it was encountered in the normal HTML way but the DOM element it was looking to populate (something with the id=“rect1”) was not yet on the page.

I could have simply looked for the body and appended it to that in JS or had the JavaScript run in an onload callback e.g.

1

window.onload = function drawSquare() {...}

This is all OK but there are a couple of problems I can see.

1. I may end up importing popular libraries more than once on a page
   • especially in the blog index that shows many blog pages
   • multiple large downloads may impact render times
2. Because the index page shows many posts on one page..
   • I may get JavaScript variable name conflicts.

The solution to point 1 is found in the octopress documentation

If you want to add scripts or tags to the `` take a look at `/source/_includes/custom/head.html` ...

If you'd rather inject scripts at the bottom of the page, you can add that to `/source/_includes/custom/after_footer.html`.

So, I can simply include my common libraries by adding the script tag in /source/_includes/custom/after_footer.html

Problem 2 can be solved by adopting a convention. All JavaScript I write will be closed inside an immediate function named after the blog page and the div’s the code interacts with shall have an id that is prefixed by the blog page.

For example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<div id="blogpost1-rect1" class="chart"></div>

<script>
(function blogpost1() {
var square = d3.select("#blogpost1-rect1");

var svgContainer = square.append("svg")
    .attr("width", 100).attr("height", 100);

//Draw the Rectangle
var rectangle = svgContainer.append("rect")
    .attr("x", 10).attr("y", 10).attr("width", 50)
    .attr("height", 60).attr("fill", "red")
    .attr("stroke", "blue").attr("stroke-width", 5);
})();
</script>

There are gotcha’s when working with remote state in terraform that this blog attempts to explain and workaround.

Terraform is a pretty cool infrastructure provisioner. It lets me set up infrastructure that can span cloud providers, eg AWS and/or Azure.

working with terraform

By writing terraform config files you declaritivly describe the infrastructure that you want, for example which AWS IAM users, groups, roles and policies on which S3 buckets and EC2 instances.

by running terraform plan terraform create a terraform.tfstate file that describes the full state as described in your config and compares it to the previous tfstate file to show you what changes will be made.

This is cool, you can see what will happen before you run terraform apply.

What’s the catch?

It is important to realise that terraform is a state machine.

if your previous state added a user, and your new state does not mention that user (ie you removed him from one of the config files) then next time you run terraform plan or terraform apply you will see that the user will be removed so that the terraformed environment matches the new desired state.

That doesn’t seem so bad. Your config (which I presume you are version controlling in git?) grows and consistently matches the desired end stateof your environment, effectivly documenting the infrastructure. Cool.

So what’s the problem?

The problem

What happens when your mate runs apply from his dev machine?

Obviously terraform will apply the state to the environment as described in his config. But what if he deletes something that you are still depending on? You can quite quickly destroy each others infrastructure.

This can be avoided using normal Version control practice, eg rebasing from git before you run terraform plan and terraform apply but almost enevitably you will always be merging conflicts by hand in the terraform.tfstate JSON file. This is not too bad if you always have a conflict as you’ll know that you need to merge, but as with all structured text files it’s possible that a clause will be inserted at a different line to your changes that doesn’t conflict in a diff sense but does change the meaning of the file in an inconsitent way, ege a group gould be removed from one place but depended on in another. terraform will fail in this case but then you’ll need to manually fix it up, which is a pain, but I can imagine changes that could happen that would not be conflicts and dont screw up the file, but do dramatically impact the generated infrastucture, such as adding a policy or group that together work to lower the expected security contraints of the infrastructure.

How do we deal with this?

Terraform handles this scenario by allowing a Remote statefile, that can live in Consul, AWS S3 or Atlas.

Running the terraform remote command can set up your local terraform.tfstate file to match the remote one before seeing what the diffs are.

Gotcha 1 NB even with a remote statefile terraform does not support concurrent terraform apply commands as it doesn’t manage locking of the stanza’s in the statefile, so talk to your mate before running it!

This is all OK now…. or is it?

There are still a couple of anoying problems that prevent this feature working as you’d expect.

The way I’d like to work with this, is prevent git from commiting the .terraform/terraform.tfstate file in the gitignore file, so that the remote one is auto downloaded to compare with my generated new state before calulateing the combined next state.

Gotcha 2 If you don’t have a local terraform.tfstate file then terraform apply fails beacuse it assumes it needs to create resources that already may already exist but it doesn’t know because it doesn’t have the current state.

The workaround

You could fetch the latest remote state file and copy it to .terraform/terraform.tfstate the first time, but this means you be forever explaining to people what to do.

my current recommendation is to use a Makefile to run terraform, which by default sets up the copy of remote state if it doesn’t exist and then runs terraform plan with a seperate target for terraform apply.

In this way you don’t need to check in the statefile so you cna be sure that you are in sync and you don’t need to remember to get it first (Chicken and Egg scenario)

here is the Makefile using and AWS bucket for this terraform remote state:-

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# Makefile to kick of the terraform for this project
#
# You should set the following environment variable to authenticate 
# with AWS so you can store and retrieve the remote state befor you run this Makefile.
#
# export AWS_ACCESS_KEY_ID= <your key>
# export AWS_SECRET_ACCESS_KEY= <your secret>
# export AWS_DEFAULT_REGION= <your bucket region eg ap-southeast-2>
# export TF_VAR_access_key=$AWS_ACCESS_KEY # exposed as access_key in the terraform scripts
# export TF_VAR_secret_key=$AWS_SECRET_ACCESS_KEY
#
# ####################################################
#
STATEBUCKET = my-statefile-bucket-name
PREFIX = myterraformprojectname

# # Before we start test that we have the manditory executables avilable
 EXECUTABLES = git terraform
 K := $(foreach exec,$(EXECUTABLES),\
  $(if $(shell which $(exec)),some string,$(error "No $(exec) in PATH, consider apt-get install $(exec)")))
#
#     .PHONY: all s3bucket plan


.PHONY: all plan apply

all: init.txt plan
  echo "All"

plan: 
  @echo "running terraform plan"
  terraform plan

apply:
  @echo running terraform apply
  terraform apply

# little hack target to prevent it running again without need
# for second nested Makefile
init.txt:
  @echo "initialise remote statefile"
  terraform remote config -backend=s3 -backend-config="bucket=terrastate" -backend-config="key=$(PREFIX)/terraform.tfstate"
  echo "ran terraform remote config -backend=s3 -backend-config=\"bucket=$(STATEBUCKET)\" -backend-config=\"key=$(PREFIX)/terraform.tfstate\"" > ./init.txt

here is the .gitignore file

1
2
3

*.swp
.terraform/terraform.tfstate*
init.txt

here is the terraform config to set up remote state, remote.tf :-

# found from env var TF_VAR_acces_key
variable "access_key" {}

variable "secret_key" {}

provider "aws" {
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
    region = "ap-southeast-2"
}

resource "terraform_remote_state" "remote_state" {
    backend = "s3"
    config {
      bucket = "my-statefile-bucket-name"
      key    = "myterraformprojectname/terraform.tfstate"
     # region = "ap-southeast-2"
    }
}

Usage

1
2

# run 'terraform plan'
make

1
2

# run 'terraform apply'
make apply

This is cool because it is very reliable and extremly cheap. You don’t pay to have an EC2 instance constantly running

• you just get charged s3 costs
• $0.03 per GB and $0.004 per 10000 requests.

But what if I want to host my company private web pages? Can I use S3? Will other people be able to see my stuff?

The answer is yes you can use S3.

The trick is you need to modify the bucket policy to have constraints.

For example here is a public s3 buckets policy:-

1
2
3
4
5
6
7
8
9
10
11
12
13

{
  "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "PublicReadGetObject",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::testfoo/*"
        }
      }
    ]
}

And here is a policy that allows access only from my ip address or CIDR range and denies a some other specified ip-addreses that would otherise be included.

NB any other ip is automatically excluded too

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

{
  "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "PublicReadGetObject",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::testfoo/*",
        "Condition": {
          "IpAddress": {
            "aws:SourceIp": "150.101.204.0/24"
          },
          "NotIpAddress": {
            "aws:SourceIp": "150.101.204.188/32"
          }
        }
      }
    ]
}

The full list of AWS Condition keys is available here

Cheap private web hosting here I come!

First I create the ssh key

1

ssh-keygen -t rsa

When asked for the password for the key simply hit enter key, and again. This will create a private key and public key in

1
2

~/.ssh/id_rsa
~/.ssh/id_rsa.pub

Over on the Synology box (I assume you have ssh’d there as root)

1

ssh root@synology

Some hoops need to be jumped.

By default you can’t get to the home directory of a user, it is mapped to a fake place Get around that by:-

• go to Users admin page
  • click advanced
  • turn on “home services”

Now you need to modify the home dir permissions

1
2
3

cd /var/services/homes

chmod 755 user user # by default synology setts 777 and sshd is picky

Now you need to actually give your user a shell. as root edit /etc/passwd

You need an entry like

1

rsync:x:1031:100:linux backup user:/var/services/homes/rsync:/bin/ash

Notice that the last section is a real shell /bin/ash if it is not a shell you can’t log in.

Now you need to modify /etc/ssh/sshd_config make sure it has the following lines:-

1
2
3

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile  .ssh/authorized_keys

Finally you need to create the authorized_keys file in the users account an add a public key to it. Beware that file permissions are crucial here or ssh will refuse you.

1
2
3
4
5
6
7
8

su -s /bin/ash rsync
cd $HOME
pwd # make sure you are in /var/services/homes/rsync

mkdir ./.ssh
chmod 700 ./.ssh
touch ./.ssh/authorized_keys
chmod 600 ./.ssh/authorized_keys

Then add a public key corresponding to a private key that you own on one line in the file either use vi an paste it in or from another PC you can use ssh and cat (you’ll need to use password for the user until you are done)

Assume my pub key is at ~/.ssh/id_rsa.pub and I want the rsync users authorized_keys file to hold contain that key..

1

ssh rsync@synology "/bin/cat >> ./.ssh/authorized_keys" < ~/.ssh/id_rsa.pub

After being prompted for the password the key will be in place.. now restart synology or quicker just get the ssh daemon to re-read its config

1
2
3
4

synology> ps | grep sshd
16783 root     16772 S    /usr/bin/sshd

synology> kill -HUP 16783

You should now be able to ssh to synology as rsync with no password

1

ssh rsync@synology

and now you want to test a Service or Controller quickly.

sure you can (and should) write a unit test but you just need to sanity check the service is doing the right thing.

Well the browser has a javascript console (f12) which has all your code loaded in it, so we should be able just run the service.

But the service is wrapped in a clousure, how can I get it to run it’s methods?

Luckily Angular uses a dependency Injection (DI) mechanism, and we can get access to the injector. Once we have the injector we can ask for the service “by Name” :-)

Assume I have a service called RouteService that has a method called routes() on it that returns an array of route Objects. I can run the method like this from the console:-

var myinjector = angular.element(document.body).injector();   
var myRouteService = myinjector.get('RouteService');
var routes = myRouteService.routes();

voila! we now have the routes in a variable… we can see them by typing the name of the varible in the javascript console:-

routes
[Object, Object, Object]

In Chrome the object is compressed onto one line with a little “expandme” triangle next to it, click the triangle to see the values in the route Objects.

As usual, I was thinking about algorithms while doing my household chores. This morning I was emptying the cutlery from my dishwasher. My normal way to do this is to select all the knives, hold them all between two fingers in one hand, then select all the forks, hold them all between another couple of fingers and repeat for spoons and teaspoons. Then with a big bunch of cutlery I walk to the drawer and quickly and efficiently grab each finger full of cutlery and dump that whole lot in it’s tray. All the knives go at once, then forks etc. etc.

This pre-sorting algorithm works fine for me and minimises my trips from the dishwasher to the cutlery drawer. Also no-one is waiting on my output. But what if my house was a busy restaurant and someone was waiting to lay the tables and there was no cutlery in the tray? In this case my dishwasher emptying algorithm is sub-optimal because the person waiting would have to wait for me to fully empty the dishwasher before they could get any work done. It would be better in that case if I simply grab a handful of any items and sort them directly into the tray. In this way the other person can start some work almost immediately.

Little’s Law shows that by shrinking the Work In Progress (WIP) we can reduce flow time through the system while holding throughput constant. We can make use of this in our development cycles.

Developers have, IMHO, a natural tendency to hoard their work, in the same way an artist may refuse to let you see a painting before it is finished, a developer will not release his code until it is beautiful. It’s as if we believe that “showing the workings” exposes the magic trick. The beautiful unique snowflake, that is our software, was not magicked into existence in one fell swoop but was instead wrought out of the aether by hard work, determination and skill. We seem to think that doing this would in some way cheapen the final product.

This is a shame. We see examples everywhere that if people see the work in progress then the final product is improved, either directly or by providing critical feedback. For example how many times have you explained some software design to BA’s or project stakeholders and they just looked at you funny? But when you show them a demo GUI they can instantly grasp the concept and help refine it.

Most people are quite good at incrementally improving a design, but not all are good at initialising a design. Getting “eyes on early” is a great strategy for producing great products. Engineers have done this since time immemorial, by producing prototypes to allow for early and cheaper testing and re-design.

Gathering critical feedback and design improvements is not the only benefit from transparently showing your workings. More eyes mean less bugs! This is true whether it is a restaurant with an openly visible kitchen, allowing customers to check hygiene standards for themselves or the open source software community where transparency allows Techos to find and fix bugs and security flaws preventing mal-ware and viruses from quietly taking hold.

So clearly releasing work early and often is going to be a big win. How big? I urge you to watch Don Reinersten’s talk at YOW 2012. IMHO it should be recommended or required? viewing for all developers and managers in general.

But how do we do this? The key is to understand what a batch, and hence batch size, is in the Software Development Life cycle (SDLC). A Batch for a developer is a deployable artefact it just needs to compile and run. Once an artefact is released the next people or robots in the chain can work with it, be they QA Testers or Load testers or client representatives who can provide feedback quickly. Quick turnaround of feedback can massively improve the development time, one simple example is that the context of the work will probably still be in the developer’s head allowing him to immediately fix or modify the code without a lengthy analysis session.

Shrinking the batch time of each stage in the SDLC pipeline is critical to building quality software and reducing development time.

Think Dev-Ops.

This makes it trivial to add methods to previously closed classes. However there are some gotcha’s to beware of. In particular make sure you read the rules

The reason I’m writing this blog is that I fell foul of restriction 3. It took me ages to work it out.

Restriction 3 says:-

“There may not be any method, member or object in scope with the same name as the implicit class.

Note: This means an implicit class cannot be a case class."

The mistake I made was to have a companion Object for my Implicit class. The compiler spits out an error message like this :

[error] /home/projects/myproj/src/test/scala/com/owtelse/FooTest.scala:27: value myPimpedMethod is not a member of com.owtelse.Foo
[error]     val f2: Foo = foo.myPimpedMethod(false)
[error]                       ^
[error] one error found
[error] (scazelcast-api/test:compile) Compilation failed
[error] Total time: 2 s, completed 15/12/2014 8:24:05 AM

This had me pulling my hair out! I could see my implicit class was in scope and so the method should have been available to Foo. The problem was of course that I had an object named the same as my implicit class as a companion object. Rather than tell me that the compiler simply discounted my implicit class and simply reported that the myPimpedMethod was not available on type Foo.

As soon as I deleted the companion object every thing worked.

While I was a bit annoyed with this behaviour, it makes sense. The only purpose of the implicit class is as a convenience to wrap the underlying class you wish to pimp and is only meant to be used as an implicit conversion. As such you should have no need to instantiate the class in any other way and so should have no need for companion object, plus I bet it makes it easier for the compiler writers to apply the conversion where needed if they don’t have to disambiguate symbols :-)

Pro

• don’t need your admin to assign special “impersonate” permissions to the service account (once assigned it can impersonate anyone in the domain)
• don’t need a Google Domain (see link ???)
• app can use the refresh token to continue to fetch from Google API’s until the user is revoked.

Con

• A user needs to accept the Scope of the app, not too bad as once accepted it can continue to use the refresh token to keep accessing the API
• con you can’t choose the OAuth2 redurect_URI just get magig top of page or http://localhost (any port you want) so no good for webapps… but then they have a Web-AppClient if you want that.

Create a project in Google Dev Console

The Google page explaining how to use it is here but here is a concreate example.

1. Click “Create a client ID” -> “Installed application”

2. This give you a client with these details:-

   Client ID for native application

   CLIENT ID 758647508586-0vnrjg9dv7gr9h1qvqle58rd4kq0lu45.apps.googleusercontent.com CLIENT SECRET _Br_xUcZSfgmsuPrRmxkavV9

   REDIRECT URIS
   urn:ietf:wg:oauth:2.0:oob http://localhost

3. Request the authentication code I’m requesting auth for the contacts API see the scope (here

4. The request is made up of :-
5. The Google oath code request URI = https://accounts.google.com/o/oauth2/auth
6. The Google ContactAPI read/write scope = https://www.google.com/m8/feeds
7. The redirect uri to put the code in the page title and code div = urn:ietf:wg:oauth:2.0:oob see googledocs
8. response_type = code

9. client_id = the Client_ID found in the dev console, see above

   https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.google.com%2Fm8%2Ffeeds&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&client_id=758647508586-0vnrjg9dv7gr9h1qvqle58rd4kq0lu45.apps.googleusercontent.com

10. look at the response there will be a code in the page title and also in a div with id=‘code’ in the page like below

    4/7x7B5YMMVMQ1S7iWQ1PnOcFRGqyj.cuXlXIEu_icYdJfo-QBMszswfSocjgI

11. now we have the authorisation code we need to exchange it for an access token, this is done with a POST see here

    POST https://accounts.google.com/o/oauth2/token

    code=4/7x7B5YMMVMQ1S7iWQ1PnOcFRGqyj.cuXlXIEu_icYdJfo-QBMszswfSocjgI&client_id=758647508586-0vnrjg9dv7gr9h1qvqle58rd4kq0lu45.apps.googleusercontent.com&client_secret=_Br_xUcZSfgmsuPrRmxkavV9&redirect_uri=urn:ietf:wg:oauth:2.0:oob&grant_type=authorization_code

12. parse the result (see below) we’ll need to store the access_token and the refresh_token.

    { “access_token” : “ya29.OwAPW_yxF27ZMB4AAACPwV13K2FYvOqzrCppl-9wp4poGBiYfvl6ibeqHPwYgg”, “token_type” : “Bearer”, “expires_in” : 3600, “refresh_token” : “1/m8wiWvc63swre62YXwrECE-SEqlQBf1Vb62Zb28-3lE” }

13. Use the access token to finally make a Google API call

14. here I’m calling the Contacts API asking for all contacts (for the user who authenticated in step 1)

15. you can add the token in a header or as a parameter (header is better as it won’t be stored in the URL)

    https://www.google.com/m8/feeds/contacts/default/full

    Authorization: Bearer ya29.OwAPW_yxF27ZMB4AAACPwV13K2FYvOqzrCppl-9wp4poGBiYfvl6ibeqHPwYgg

    eg

    curl https://www.google.com/m8/feeds/contacts/default/full?access_token=ya29.OwAPW_yxF27ZMB4AAACPwV13K2FYvOqzrCppl-9wp4poGBiYfvl6ibeqHPwYgg

    or curl -H “Authorization: Bearer ya29.OwAPW_yxF27ZMB4AAACPwV13K2FYvOqzrCppl-9wp4poGBiYfvl6ibeqHPwYgg” https://www.google.com/m8/feeds/contacts/default/full

16. Read the results :-) Enjoy.

I want to set up a server side App that can read list and filter the Google contacts in our company domain “mycompany.com”. This server side app can then be queried by some of our internal webapps to display helpful info.

Why not simply do it all in javascript?

Dunno. maybe that would be better. We’ll See.

Problems

There are a number of gotchas involved, and misleading documentation to contend with, so I’m going to go through setting up a service account to access your users contact info.

Google Project Authentication Authorisation

This is likely to be the most confusing part.

For any app to use Googles API’s it’ll need to ble to authenticate with Google and be authorised to use the API’s you want to use, or a subset of the API, eg the ability to read contact info but not edit it, it may also need to be granted permission from a user to see their data.

To allow any of this you need to set up a “Google Project” for your app in the Google Developer Console. The project manages the app’s authentication as well as which API’s it can use. In the “API’s & Auth” -> API section, add the Contacts api. Don’t add any others yet, it is best to keep things as simple as possible while setting up your app and to test that your app can connect to the requested API before other authorisations are layered on it.

Gotchas

• You NEED to make the scope exact including not trailing slashes!
  • eg use https://www.google.com/m8/feeds not https://www.google.com/m8/feeds/
• The Sevice Account ID is the Service account EMAIL NOT it’s Client ID
  • eg blahblah@@developer.gserviceaccount.com not blahblah.apps.googleusercontent.com
• The service account must be delegated access to the the SCOPE you want it to use
  • see The Admin SDK Google Docs
• The sevice account has no contacts of it’s own so you must supply a User who’s contactas you want to snarf, eg GoogleCredential.Builder().setServiceAccountUser("me@mydomain.com")

The Octopress Doco does a pretty good job but it misses out one crucial step.

• You need to commit the master branch and push it up to github.

But before you do please read on to see how to do that the octopress way.

Where are all my blogs and pages?

If you’ve followed octpresses Doco, then under your octopress directory you’ll have a source folder.

• Web pages are in directories directly under source.
• Blog posts are under the _posts directory.

running rake generate will copy the content to the _deploy directory and check it into git. running rake deploy commits it and pushes the deploy directory to github

Job done right?

Unfortunatly it’s not always that easy. All the code under source will be on the git branch source. If you follow the instructions in the doco you’ll commit this code and push it to git hub with this command

git add .
git commit -m 'your message'
git push origin source

This pushes your “source” to github into the source directory.

Why can’t I see my stuff in Github

The problem is the pages that get rendered as the blog in github is the stuff in the master branch. That was supposed to be pushed by running rake deploy

However sometimes that git simply failes to do that for you. Why? Well perhaps you have accidentally modified the code under _deploy and git is actually complaining but the rake task doesn’t show you that?

To check simply

cd _deploy
git status # notice that the _deploy dir is in the master branch :-)
git push origin master # this may fail and tell you why

For me the last step did fail. I fixed it by doing

cd _deploy
git fetch origin master
git merge origin/master  # you may need to fix conflicts here
git push origin master

And that sorted it :-)

From then on my rake deploy worked as expected. lesson learned, leave the _deploy dir alone!