saas: Stream as a service

• you post an edn config onto a kafka topic
  • it gets converted to a kafka stream
  • spins up the workers according to sane defaults (e.g. # of partitions of input topic) / what you specify
• Snazzy def UI

Config

• EDN
• onyx-like config
  • basically, from Onyx Concepts
    • workflow
    • Catalog
      • simpler since only kafka :input/:output are supported, no plugins
        • no need to know about zk
        • no need to know about kafka bootstrap servers
          • defaults to current cluster
        • defaults to hiccup SerDe (overridable in global conf, fine.)
    • Flow conditions, as is
    • Function, mostly as is, but:
      • functions need to be fully specified:
        • :group.id:artifact-id:version:package.name/functionName
        • e.g. :apache.org:apache-commons:1.8.0:org.apache.commons.strings/parseBoolean
        • shortcuts for :group.id:artifact-id:version can be defined in global config
        • a number of trusted, stable, non breaking jars are required, they default to latest, and don't need full qualification
          • clojure.core -> :clojure.core/inc
          • medley.core -> :medley.core/map-vals
        • can require :all in global config?
        • jars available on MVN Central are automatically fetched n cached
        • a private nexus can configured in the global config
    • Plugin no.
      • Kafka all the things
      • But support kafka connect?
    • Virtual peer
      • no need, just input topic with one partition
    • Sentinels: sure

Schemas / SerDe

• schemaless:
  • edn
  • hiccup
• shchemaful:
  • Avro:
    • read -> no need
    • write -> attach a malli spec and a schema will be genned and posted to the schema registry for you.

Updating Jobs

• Just push another config with the same Kafka Key?

UI

• interface lets you preview in real time output of your choices by only streaming a few dozens of data points
• see Preview of Pyroclast