This Blog is about Scala Combinator Parsers but excitingly about compiling the Scala to JavaScript using Scala.js.
It is also a slide deck for a talk given at the ScalaSyd meetup Sept 2017.
To switch between modes press a number as follows :
- ‘1’ -> Doc mode:
- shows the document as intended.
- ‘2’ -> Deck mode, see the slides
- see the slides
- ‘4’ -> Lecture Mode
- enter zooms current navigated to section
- click zooms div or block clicked
Arrow keys navigate to next or previous section or slide
A Markdown combinator parser in
Scala
2
. Press Esc
to get back to document view. Left and Right arrow keys to navigate.
See suited.js
A Markdown combinator parser in
Scala.js
This Blog is my talk for ScalaSyd meetup on September 13th 2017. It is about Scala combinator parsers and Scala.js
Intro
- This talk is about Scala
Intro
- This talk is about Scala.js
Intro
- This talk is about Scala.js
- Combinator Parsing
Intro
- This talk is about Scala.js
- Combinator Parsing
- Markdown
Intro
- This talk is about Scala.js
- Combinator Parsing
- Markdown
- suited.js
Before diving into nitty-gritty details it’s helpful to explain my motivation for this.
Why?
Why?
- LambdaJam 2017
-
-
Why?
- LambdaJam 2017
- compile all the things to other languages
-
-
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
-
-
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
- Wanted to explore Scala.js
-
-
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
- Wanted to explore Scala.js
- I needed yet another partial project
-
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
- Wanted to explore Scala.js
- I needed yet another partial project
- devs seem to like starting projects
-
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
- Wanted to explore Scala.js
- I needed yet another partial project
- devs seem to like starting projects
- I had too much time on my hands one day.
Why?
- LambdaJam 2017
- compile all the things to other languages
- treat javascript like assembly lang for the web
- Wanted to explore Scala.js
- I needed yet another partial project
- devs seem to like starting projects
- I had too much time on my hands one day.
- solved!
So apart from getting down with all the transpiling kool kids I actually have a need to do some JavaScript jiggery-pokery.
What?
- This talk is presented using suited.js
- a JavaScript library
- Allows a single document to render as page or slide deck
What?
- This talk is presented using suited.js
- a JavaScript library
- Allows a single document to render as page or slide deck
- this slide deck is actually my in blog
What?
- This talk is presented using suited.js
- a JavaScript library
- Allows a single document to render as page or slide deck
- this slide deck is actually my in blog
- hint: hit key 1,2 or 4 for fun
- mode 4 will zoom on
enter
or click
- mode 4 will zoom on
suited.js
- was written by Dirk and myself
- a small library
- uses no other js lib
- event driven
- plugin architecture
suited.js
- but then we wanted markdown
Suited is cool and I’ve given many talks using it and the same document can be a blog or article too. But we always want it to do more for instance we wanted to write out talks in markdown.
So we needed a markdown plugin, but we also wanted a magic syntax to add slides and figures in markdown,
Suited.js uses <section data-figure>
and <section data-slide>
to mark out sections of the contents to appear as navigable
sections. Slides are just visible in the slide show whereas figures are visible in doc mode and also in the slide show.
markdown slides
~~* **your bold stuff** *~~
markdown figures
~~: **your bold and _italic_ stuff** :~~
So we implemented a markdown plugin, but delegated to markdown-it.
But suited has a fatal flaw. It has no parser to implement fragments
suited.js
- Fragments are no fun
fragments
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
markdown-it plugin?
- I dont want to write another markdownit plugin
- especially a complex one
- it will probably add javascript to the markup
so I wrote my own markdown parser for Javascript?
- Using Scala.js
- how hard can it be?
Because I want to rely on a few libraries as possible while experimenting with Scala.js I decided to use Mark Hibberd’s Combinator Parser demo code from his previous Scalasyd talk “Dont fear the parser” as a starting point.
Parser recap
case class Parser[A](run: String => ParseState[A])
Parser recap
1 2 3 4 |
|
Parsing is OK but I also want to render the parse tree into HTML, this is what Transformers are for.
Transformer typeclass
1 2 3 4 5 6 |
|
Syntax pimp and Aux Pattern
1 2 3 4 5 6 7 8 9 10 11 |
|
See my previous talk on the Aux Pattern
Now lets look at a specific member of the Transformer typeclass. The MarkdownToHtml
transformer.
I wanted to split the transform function into two parts, one to parse, and of course I want to use my markdown parser’s run function in this place, and one to render the ParseResult
into HTML.
Not only does this make it easier to reason about but also (I Hope) easier to build a plugin interface where you can supply a function to parse modified markdown as long as it still produces a ParseResult[Markdown]
and one to enrich the HTML or even transform it completely, say into rich text or PDF
MarkdownToHtml
1 2 3 |
|
- notice I’ve split it into 2 functions
- one to parse, p
- and one to render, r
MarkdownToHtml
use instance
to join the Transformer typeclass
1 2 3 4 5 6 7 8 9 10 |
|
see MarkdownToHtml
MarkdownToHtml ---*
function
- Now MarkdownToHtml is in the Transformer typeclass
- It has access to the transformer Magic Wand function
- so my
simple
implementation can use it like so
1
|
|
Demolition time
- Scala parser and render in the sbt console
So let’s demonstrate the parser and renderer in the console.
Demo parser: start console
me@host $ sbt
sbt:Markdownem> console
scala> :paste consoleimports.txt
Pasting file consoleimports.txt...
import parser._
import parser.markdownParser._
import transformers._
import transformers.Transformer._
import transformers.MarkdownToHtml._
Notice that
1
|
|
The Scala REPL is only available to the Scala code. So only code built without depending on JavaScript libraries will work in this REPL demo.
But that’s OK, this demo has no JavaScript dependencies, it just generate JavaScript output.
Demo headerParser
1
|
|
1 2 |
|
Demo headerParser
1 2 |
|
The header parser just parses a single header.
1 2 3 |
|
Notice that this is a successful parse, and that the ParseOK
case class also contains the remaining input for further processing
Also notice that the H2 contains a List of more Markdown, in this case rawHtml
.
This is because the link text could contain more inline markdown, like so…
Demo headerParser
1
|
|
1 2 3 4 |
|
Demo headerParser failing
1
|
|
1 2 3 4 |
|
Admittedly, this is not the best error message (for better errors use the built in Scala combinator parser library rather than this demo).
It failed to see a #
as the first char in a header and so produced a ParseKo
instead of ParseOk
.
The important thing to see is that a parser stops as soon as it can go no further.
In order to parse a whole document we can use the combinator methods, such as ||| i.e. or
on Parser and some of the parsers such a list to compose more powerful
parsers that continue parsing input by trying one parser after another.
An example is the markdownParser which will try all blockParsers followed by the inlineParsers continuously until it can parse no more.
However that will only occur when there is no more input because the last parser it tries is always the rawHtml parser which will always succeed because any input that is not parsed as markdown must be is just treated as RawHtml
.
Demo markdownParser
1 2 |
|
1 2 3 4 5 6 7 |
|
Now we can see that the input parses into a List of Markdown
. This is all very well but it’d be nice to see the rendered HTML.
This is where simple MarkdownToHtml Transformer comes in.
I can now use the ---*
magic wand function
Demo simple
render
1 2 |
|
1 2 3 4 |
|
Demo simple
render a complex list
1 2 3 4 5 6 7 8 |
|
Demo simple
render a complex list result
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
|
So… Performance?
What about Performance?
I haven’t run proper benchmarks yet, but as a quick and dirty test
I have added timing output to the main
function.
remember this in build.sbt
1
|
|
- After compiling to JavaScript
- adds a call to the
main
function in theMainClass
- at the end of the JavaScript.
- adds a call to the
- This makes the main function run when the JS is loaded
We can get sbt
run task to do the same for us
1 2 3 4 5 6 7 8 9 |
|
We can get sbt
run task to do the same for us
1 2 3 4 |
|
What happened?
It is an error from sbt as it is trying to run the default JavaScript runtime Node.js
to execute the generated JavaScript.
sbt can also run using other JavaScript runners such as PhantomJS
, Selenium
or Rhino
see the docs
I had better load NodeJS onto my PATH, i use nvm
for this.
WTF!
WTF!
- It was trying to use
node
to run the JavaScript.
WTF!
- It was trying to use
node
to run the JavaScript. - I better load it onto my PATH
1 2 3 4 5 6 7 8 9 |
|
We can get sbt
running NodeJs
1 2 3 4 5 6 7 8 9 10 11 |
|
The TestApp just renders 112 lines of markdown into HTML a few times.
We can see that after the initial run where it starts the node environment it ends up taking about 430ms.
That seems slow!
seems a bit slow!
112 lines of markdown in 430ms
seems a bit slow!
112 lines of markdown in 430ms
lets compare the JS to a scala/JVM run
If we comment the JS lines in build.sbt we can build and run the main app in standard Scala on the JVM
comment Scala.js from build.sbt
1 2 3 4 5 6 7 8 9 |
|
now run as a ScalaJVM app
1 2 3 4 5 6 7 8 9 10 11 12 13 |
|
On the JVM same code runs 200ms faster
112 lines of markdown in ~ 230ms
On the JVM same code runs 200ms faster
112 lines of markdown in ~ 230ms
But this is not really a fair comparison with NodeJS
On the JVM same code runs 200ms faster
112 lines of markdown in ~ 230ms
But this is not really a fair comparison with NodeJS
The Javascript was not fully optomised because it would take longer to compile
On the JVM same code runs 200ms faster
112 lines of markdown in ~ 230ms
But this is not really a fair comparison with NodeJS
The Javascript was not fully optomised because it would take longer to compile
So lets optomise it.
use the optimising compiler
- The Javascript was compiled with the
FastOptStage
compiler- meaning it was fast to compile
- We can tell sbt to run with the
FullOptStage
compiler- which will optimise much more
un-comment Scala.js, put it back into build.sbt
1 2 3 4 5 6 7 8 9 |
|
tell sbt to use Full optimisation
1 2 |
|
- N.B. we can set it back with
1
|
|
now run optimised Javascript
1 2 3 4 5 6 7 8 9 10 |
|
Javascript in NodeJS almost as quick as JVM
- 112 lines of markdown in 260ms
But still it seems slow
- Combinator parsers are recursive decent parsers
- every
or
|||
branch backtracks on the input- and re-parses of the first parser fails
But still it seems slow
- Combinator parsers are recursive decent parsers
- every
or
|||
branch backtracks on the input- and re-parses of the first parser fails
- Packrat parsing’s memoization can massivly improve it.
But still it seems slow
- Combinator parsers are recursive decent parsers
- every
or
|||
branch backtracks on the input- and re-parses of the first parser fails
- Packrat parsing’s memoization can massivly improve it.
caveat: also I’ve done no code optimisation yet
Runing in the sbt prompt is all very well.
But how can I run this code in my browser?
So… how do I run it in my browser?
export a javascript function to the “toplevel”
- this allows other javascript ion a page to call it.
- add
@JSExportTopLevel
annotation to the method to export- so in MarkdownToHtml.scala line 3 uncomment the
import
- so in MarkdownToHtml.scala line 3 uncomment the
1
|
|
export a javascript function to the “toplevel”
- add
@JSExportTopLevel
annotation to the method to export- so in MarkdownToHtml.scala line 83 uncomment the annotation and expose a top level function called
mdmagic
- so in MarkdownToHtml.scala line 83 uncomment the annotation and expose a top level function called
1 2 |
|
compile the optimised JavaScript
1 2 3 4 5 6 |
|
import the markdownem-opt.js in a html page
- see test.html
1
|
|
NB you probably want to set scalaJSUseMainModuleInitializer := false
in build.sbt
to prevent the slow main app test from running when the page loads
simply use the mdmagic function in some javascript
1 2 3 |
|
For Remaining topics eg
Javascript interop
TODO….
- just use scala lib PackratParsers
or sbt-rats
- better error messages
- faster
- add the plugin interface
- write the fragments plugin
- re-write suited.js in scala.js
- look at a recursion scheme for render
- Free Monad/Applicative for renderer/interpreter
The End
Thanks ….
me @MrK4rl
code used in demo https://bitbucket.org/suited/markdownem
- deck and talk (http://karlcode.owtelse.com/blog/2017/07/09/scala-js-markdown-combinator-parser/?mode=deck#slide-0)
- ref Scala.js doco
- ref “Dont fear the parser” - Mark Hibberd - scalasyd talk
- ref “Dont fear the parser” - Mark Hibberd - demo code
- ref The Type Astronaut’s Guide to Shapeless http://underscore.io/books/shapeless-guide/
- ref “The rise and fall of the Aux pattern” - Karl Roberts - scalasyd talk