Roadmap

This document is a living road map of the Adama Platform. As such, it contains the investment details for the entire vision and future products.

Developer relations & adoption

The current story for developers is "meh", so these items help improve and modernize the developer experience.

projectIPmilestones/description
vs code extensionX(1.) Syntax highlighting, (2.) Language server protocol (LSP) - local
sublime extensionSince sublime is so fast, try to get core Adama language support for syntax highlighting
Android client(1) Write a simplified web-socket interface, (2) implement interface with OkHttp, (3) update apikit code generator to produce a SDK talking to the web socket interface.
iOS client(1) Write a simplified web-socket interface, (2) implement interface with ?, (3) update apikit code generator to produce a SDK talking to the web socket interface.
capacitor.js templateA template to turn an RxHTML project into a mobile app with deployment pipeline
integrate-linterintegrate the linter to detect issues prior launch; address #62
lib-reactlibrary to use Adama with React
lib-vuelibrary to use Adama with Vue
lib-sveltelibrary to use Adama with svelte

Documentation

projectIPdescription
kickoff demosXSee https://asciinema.org/ for more information
client-walkA detailed walkthough about how to use the client library, the expectations, and core APIs
improve overviewMake the overview easier to understand, more details, etc
cheat-sheetdocument the vital aspects of the language and provide copy-pasta examples
micro-examplesa mini stack-overflow of sorts
tutorial-appwalk through the basics of building an Adama with just Adama and RxHTML
tutorial-twiliobuild a twilio bot as an example with details on calling services
tutorial-weba HOWTO host a static website with Adama
tutorial-domaina HOWTO use Adama's domain hosting support
zero-heroa breakdown of using the bootstrap tooling to build a complete app
feature-complexWrite about complex number support
feature-maybe(1) write about how maybes work, (2) write about maybe field deref, (3) write about math and maybe
feature-listswrite more lists
feature-mapwrite about map transforms
feature-dynamicwrite about dynamic types
feature-viewerwrite about @viewer
feature-contextwrite about @context
feature-webwrite about @headers / @parameters
map/reduce-loveXreduce love along with maps
functionsprocedure, aborts, functions, methods
enumeration/dispatchtalk about dispatch
feature-servicestalk about services and linkage to first party along with third party http (and encryption of secrets)
feature-asynctalk about async await,decide,fetch, choose, and result
result typetalk about the result type
feature-smtalk about the state machine, invoke, transition, transition-in
web-puttalk about the web processing
stdlib code-gengenerate documentation from the stdlib (i.e. embed docs in java as annotations) (embed documentation in type)
document or kill rpcThe rpc helper is an interesting way of making channels, but it isn't used right now... kill or document
assets
rxhtml auth stuff
auto ids in RxHTML

Standard Library

projectIPdescription
statsbuild out a statistics package that is decent and correct (median isn't good right now) (sum, average, median, product, count_non_zero)
to/from Base64convert a string to and from Base64 with maybe<string>
substituteFirstfind the needle in a haystack and replace the first instance
substituteLastfind the needle in a haystack and replace the last instance
substituteAllfind the needle in a haystack and replace the every instance
substituteNthfind the needle in a haystack and replace the n'th instance
formatneed var_args (just use a string[]) support, but the idea is to allow efficient string construction from many known finite pieces
split /w limitonly split a certain degree
findfind a string within a string
findAllfind a string within a string and produce a list if indicies
propertake a string, split on spaces, normalize white space, turn every string into a camel case word, join together
initialstake a string, split on spaces, normalize white space, turn every string into a concat of just first letters capitalized
convex hulla 2D version at first
matrix mathinverse, multiply, etc...
exact-mathmath that doesn't overflow, and if it does, empty maybe
random+gaussian random + other statisitical random functions
even/oddsimple and fun functions
factorial
gamma factorialhttps://en.wikipedia.org/wiki/Gamma_function
colorhsv/cmyk and a color typing library

Web management

The online web portal needs a lot of work to be useful. The web portal should be upgraded to provide a manager view along with tools to get started along with steps to make progress.

projectIPmilestones/description
render-planRender and explain the deployment plan
render-routesRender and explain the routing including both rxhtml and web instructions
better-debuggerXThe debugger sucks
support fbauth
metricsA metrics explorer
service callsA way to explore which documents are making which service calls and the results

Contributer Experience

If your name isn't Jeff, then the current environment is not great.

projectIPmilestones/description
shell script loveImprove the build experience outside of Ubuntu
test MacOSXWork through issues with unit tests on MacOS and any productivity issues with the python build script
test WindowsXWork through issues with unit tests on Windows and any productivity issues with the python build script
local modeXAdama should be able to run locally with a special version of Adama just for applications and local development
faster unit testsImprove the testing to not leverage shared resources (stdout, cough) such that testing can be made parallel
write documentation about structureWrite a document to outlining the high level mono-repo structure

Performance/Capacity Issues

projectIPmilestones/description
bubble view filteringBubbles should recompute only with related viewer changes happen rather than the viewer changed
compressed tablesfor tables that don't have deep subscriptions, let's compact records
table-paging-{disk/db}for tables that don't have deep subscription and stale data, let's introduce an offload database for caching
java-compiler-serviceinstead of every adama instance compiling java, let's cache some byte code and maybe offload to an isolated service

Language

The language is going to big with many features!

projectIPmilestones/description
index-tensorTables should be indexable based on integer tuples. Beyond efficiency, language extensions can help work with tables in a more natural array style (think 2D grids)
index-graphTables should be able to become hyper graphs for efficient navigation between records using a graph where edges can be annotated (this maps)
full-text-naiveXstring =? string is a full text operator that is naive
full-text-indexintroduce full indexing where records describe a rich query language
dynamic-orderintroduce a special command language for runtime ordering of lists
dynamic-queryintroduce a special language for queries to be dynamic
math-matrixThe type system and math library should come with vectors and matrices out of the box
2d-primitivescircle, rectangle
xml supportConvert messages to XML
metrics emit $num;XThe language should have first class support for metrics (counters, inflight, distributions)
metrics emit $string;The language should support emitting tags with metrics to help shred and classifiy metrics
auto-convert-msgthe binding of messages can be imprecise, need to simplify and automate @convert primarily for services
bubble + privacyXAdd a way to annotate a bubble with a privacy policy to simplify privacy logic
privacy-policy cachingXinstead of making privacy policies executable every single time, cache them by person and invalidate on data changes
table-protocolintroduce a way to expose a table protocol for reading and writing tables via a data-grid component
sum/bag typesa sum type is going to be a special type of message
auto-convertauto convert messages to dynamic
normalize messagesmessages of the same structure should auto convert
proper-lambdasXI should be able to accept functions as argruments to other functions
extension methodsExtend every message/record based on a structural pattern
message filteringApply rules for parsing messages like "trim/lowercase" to help ensure only valid data enters system
message abortsApply simple rules to reject messages that are invalid
decimal typefor arbitray rationals

Notes on bag type


enum E {Aa, Ab, B, C}
bag<E> B {
  A* {
    int x;
  }
  B {
    int x;
    int y;
  }
  C { }
}

The bag creates a record with the union of all fields along with various along with a virtual type for each instance which maps to the appropriate subset. We can derefence it via a match

B my_bag;
#sm {
  match B as o {
    A* {
      o.x = 123;
    }
    B {
      o.x = 42;
      o.y = 7;
    }
    C {
    }
  }
}

Implicitly a bag has a type of the enumeration, and ingestion would be the way to set the type along with parameters.

my_bag <- {type: E::Aa, x: 42};

This will typecheck that the bag has everything needed and will clear other fields out. This is all sketch work, but the key design goal is to provide some degree of compact representation of data based on the kind of a thng.

Infrastructure - Protocols

For integration across different ecosystems, there are more protocols to bridge gaps.

projectIPmilestones/description
mqttX(1) Write design document how to adapt Adama to MQTT, (2) Build it with TLS, (3) Built it with plain-text
sse(1) Write design document how to adapt Adama to Server-Sent Events, (2) Build it with TLS

Infrastructure - Enterprise data

projectIPmilestones/description
smaller deltas(1) Define a log format that leverages binary serialization and compacts field definitions, (2) convert JSON deltas to binary in the logger to measure impact (throughput and latency), (3) leverage format upstream to minimize future network
healing logImplement a log data structure that can heal (anti entropy) across machines using Adama's network stack
raftImplement raft leader election and log append using Adama's network stack
control plane(1) Manual definition of raft shards definitions, (2) automatic machine management

Infrastructure - Multi-region & massive scale

At some point, Adama is going to be at the edge with hundreds of nodes across the world.

projectIPmilestones
diagramXdiagram the usage of the database in the adama service
billingXhave billing route partial metering records to billing document ( and globalize )
proxy-modeXproxy the WS API from region A to region B (or global important services )
remote-finderXextend WS API to implement a Finder for region A to do core tasks (or globalize)
finder in adamaXTurn core service into a finder cache for web tier
region-isolateAllow regions to have storage for local documents
capacity-globalglobalize capacity management
test-heat-capvalidate when an adama host heats up that traffic sheds
test-cold-capvalidate when an adama host cools off that traffic returns for density
cap-configmake high/low vectors dynamic configurable
ro-replica(1) introduce new observe command which is a read-only version of connect, (2) have web routes go to an in-region replica
reconcileXevery adama host should be lazy with unknown spaces and also reconcile capacity if it should redeploy (due to missed deployment message)
space-delete-botmake run only in the global region
dead-detectoronly run in global region
leader-election-overlordinstead of a single host, get leader election in place and have one adama assign work to other adama instances
all-make-targetinstead of overlord making targets, have every adama instance expose a targets file
monitoringinstead of using a local prometheus, let's use a service
gc-2.0garbage collection should probably run on adama
finish global CPXAdama is going to have a primary region with a control plane proxy to DB
write clients for global CPXwrap SelfClient to talk to Adama control region
e2e region testsrethink the testing strategy to create a global multi-region foot print in unit tests (maybe with a customized runner)

Infrastructure - Core Service

Adama is a service.

projectIPmilestones/description
env-bootstrapautomatic the memory and other JVM args
third-party replicationXthe language should allow specification of a endpoint to replace a piece of data to on data changes. This requires maintaining a local copy in the document along with a state machine about status. The tricky bit requires a delete notification. There is also the need to load every document on a deployment.
replication-searchprovide a "between document" search using replication tech
replication-graphsimilar to search, replicate part of the document into a graph database
metricsXdocuments should able to emit metrics
fix-keysXdocument keys are stored with both private and public keys, and this is really bad; we should only store the public key and version the keys along with an expiry (core work done, need to remove old code)
op-query-engineXa tool to debug the status of a document live
portletsmaybe part of replication, but a subdocument that can emit messages that are independent subscriptions (for SSE/MQTT) and for Adama to consume
adama-actorXimplement Adama as a special first class service
twilio-serviceimplement twilio as a first party service
stripe-serviceXimplement stripe as a first party service
discord-serviceimplement discord as a first party service
slack-serviceimplement slack as a first party service
http-servicestudy postman and design a generic HTTP service for third party integration
results-streamfigure out how to ensure deliveries can overwrite prior entries
portlets - passive netif we express the desigre for a result to hold the stream results for a foreign document's portlet then this creates a graph such that documents need to be reloaded. that is, we need a persistent subscription
firebase - notificationswe should be able to send notifications
document filterXinstead of relying on privacy and wasting compute, a viewer should be able to opt into which fields to view.

Infrastructure - Web

Adama is a web host provider of sorts!

projectIPmilestones/description
web-async deleteXallow DELETEs to contain async calls
web-async getXallow GETs to contain async calls
request cachingXrespect the cache_ttl_ms
doc auth - expiryinfer cache_ttl_ms as an expiry for doc auth
asset transformsimplement some basic asset transforms
web-abort put/deleteXweb calls that write should support abort
@contextXensure web operations can access context
web-metricsadd an API for the client to emit metrics
add auth for webthe principal for web* is currently @no_one; it should be a valid user
build delta accumulatorslow clients may be get overwhelmed, the edge should buffer deltas and merge them together

Infrastructure - Overlord

Overlord is how the fleet is managed and aggregator.

projectIPmilestones/description
canaryfor testing the service health and correctness; overlord should maintain a constant state of various high-value API calls
operationalize-superthe "super" service needs a secure environment
ui for querydynamic queries
billing-sendXSimplify the billing engine and remove the overlord need
ntplook into time sync

RxHTML

RxHTML is a small set of extensions to allow HTML to be dynamic on the fly. The spiritual question is "what minimal number of things does HTML need to build apps?"

projectIPmilestones/description
headwindcssPort tailwindcss to Java for vertical control
componentsXBring clarity for single javascript extentions for new controls
timeXCustom component for selecting a time of day (Blocked on components model)
dateXCustom component for selecting a date or a date range (Blocked on components model)
colorXCustom component for selecting a color (Blocked on components model)
graphCustom component with rich config to visualize graphs
server-sideCreate a customized shell for each page such that server side rendering allows faster presence
convert-reactConvert the RxHTML forest into a complete React app
gcfigure out if there is still a bug with "rxhtml fire delete" isn't cleaning up pubsub
remove-colremove the rxhtml column from the spaces column and move into document; #127
auto-idsA big pain point in HTML is the pair bonding of label to input via id.
typecheckerA tool to validate an RxHTML forest doesn't have issues
ai-testerA browser extension to walk and validate a site is working and collecting tracking tokens

RxHTML - Mobile

projectIPmilestones/description
capacitorXshould be able to convert an RxHTML SPA to a mobile app
notification support

Roslin (RxImage)

The vision is for a runtime just for games. See this post for more information

projectIPmilestones/description
design documentWrite a design document from Jeff's notes
runtime-androidImplement a starting runtime for web using android
gameboard demo(s)Write a game board demo of various games
runtime-webImplement runtime for web using rust
rxhtml-integIntegrate runtime-web into RxHTML as a new component

Saul (RxApp)

Similar to RxHTML, the question is how to build a minimal runtime for iOS and Android applications. Tactically speaking, we can use RxHTML with capacitor.

projectIPmilestones
design documentdesign a simple XML only way to build Android applications using reactive data binding similar to RxHTML