Santiago Saavedra

• Free Software + functional programming
• 1st time speaker @ FOSDEM
• Maintains displaylink-rpm¹, GPUL² and Trackula³

Who is OpenShine?

• Small consultancy firm
• Based in Madrid, Spain
• Remote friendly

Answer questions about code without running it.

• Terminates?
• Max memory needed?
• Output for a given input? (Restrictions)
• Can this pointer be null?
• Is this variable always initialized?
• Is this dead code?
• Does this expression have the correct type?

Not only answering questions, but also modifications.

E.g., -O3 in the GNU Compiler Collection.

• Pipeline assets reuse
• Constant extraction
• Type inference
• Query optimization
• Code rearrangement
• Cost analysis
• Cost optimization

• Curry-Howard-Lambek correspondence
• Proofs are types are proofs.
• Richer type systems
• Better compile-time error reporting
• Model checking

E.g.

E. Albert, P. Arenas, S. Genaim, G. Puebla, and D. Zanardini. 2007. Cost analysis of java bytecode. In Proceedings of the 16th European Symposium on Programming (ESOP'07), Rocco De Nicola (Ed.). Springer-Verlag, Berlin, Heidelberg, 157-172.

ElasticSearch queries are not exactly compiled. Lucene AST Query.

• Elastic does not provide documentation or easy API access to internals before execution.

1. Parse program text
2. Optimize AST (if required)
3. Traverse AST
4. (Rearrange/optimize/error)

Our cost analysis is generated from the parse tree directly.

It's a recursive tree. Costs are calculated from children in a single pass, which leads to \(O(n)\).

More sophisticated analysis possible.

The cost grammar.

Make nodes only depend on themselves and their children.

Our grammar:

GET _searchv?size=0
{"aggs": { /* root aggregation */
  "sessions": { /* sub-aggregation */
    "terms": { "field": "sessionId", "size": 10 },
    "aggs": {
      "byDay": {/* Leaf aggregation */
	"date_histogram": { "field": "@date", "interval": "day" }
      }
    }
  }
}}

• Better model: include synthetic nodes
  • Caching
  • Network transmission of shard data
  • Coordination
  • Reduction steps
  • …

Not available. Elastic does not expose enough API surface.

Search engine would need to expose "compiled ASTs" a state where steps can be traced to the execution model effectively. À-la Spark's Catalyst Optimizer.

• Scala 2.12
• json4s-native (to avoid conflicts with Jackson versions)
• typesafe-config
• pureconfig
• Akka, optionally

• More expressive Type System than Java.
• Faster development in functional programming
• Pureconfig for configuration (relies on typesafe-config but is actually typesafe)

Elastic does not even provide basic access to parsed information

/**
  * Hackity hack. For some reason, the Elastic team has not provided the
  * only useful method in all the AST: recursion.
  */
def getSubAggregations(ag: AggregationBuilder)
    : Seq[AggregationBuilder] = {
  import implicits._
  val factoriesBuilder = ag.getPrivateFieldValue[
      AggregatorFactories.Builder]("factoriesBuilder")

  factoriesBuilder.getAggregatorFactories.asScala
}

We reimplement our visibility needs. With dragons.

implicit class PrivateMethodAccessor[A](val instance: A)(
  implicit c: ClassTag[A]) {
  def getPrivateFieldValue[B](fieldName: String): B = {
    AccessController.doPrivileged(new PrivilegedAction[B] {
      override def run(): B = {
	val f: Field = c.runtimeClass.getDeclaredField(fieldName)
	f.setAccessible(true)
	f.get(instance).asInstanceOf[B]
      }
  })
}

When used as ES Plugin, quite hard security conditions:

grant {
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};

All deployments can be configured in the same way: config.yaml

# config.yaml
default.maxTreeHeight: 3
default.whitelistChildren: ["terms", "date_histogram"]
custom.terms.defaultNodeCost: 10
custom.date_histogram.whitelistChildren: ["terms"]

ESCOVA is a library.

Sub-projects to deploy as:

• Elasticsearch plugin (hosted in the ES cluster)
• Web frontend to Elasticsearch
• Stand-alone web server without ES fallback

• Created SBT structure for building ES plugins
• Created from the Gradle code by ES
• Cannot guarantee stability if Elastic doesn't
• Sample plugin structure, take us as a working example

• Intercepts all requests to /_searchv
• Other requests go to $ELASTICSEARCH_BACKEND, if available
• Built with Akka-http
• Sample project, adapt to your own needs

Chart available in the repo¹. Configurable at values.yaml.

service:
  name: escova
  type: ClusterIP
  externalPort: 9200
backend: # Elasticsearch backend
  enabled: false # Redirect non-escova calls to backend ES
  host: my-internal-elasticsearch
  port: 9200
# Example configuration limiting the tree size
config: |
  default.defaultNodeCost: 1
  default.maxTreeSize: 4

• We need a logo!
• Future-proof for ES 7+
• Analyze statistics about dynamic usage e.g., cross data against Profiler
• Take measurements from data
• Improve static analysis