This document is a living road map of the Adama Platform. As such, it contains the investment details for the entire vision and future products.
The current story for developers is "meh", so these items help improve and modernize the developer experience.
| project | IP | milestones/description |
|---|
| vs code extension | X | (1.) Syntax highlighting, (2.) Language server protocol (LSP) - local |
| sublime extension | | Since sublime is so fast, try to get core Adama language support for syntax highlighting |
| Android client | | (1) Write a simplified web-socket interface, (2) implement interface with OkHttp, (3) update apikit code generator to produce a SDK talking to the web socket interface. |
| iOS client | | (1) Write a simplified web-socket interface, (2) implement interface with ?, (3) update apikit code generator to produce a SDK talking to the web socket interface. |
| capacitor.js template | | A template to turn an RxHTML project into a mobile app with deployment pipeline |
| integrate-linter | | integrate the linter to detect issues prior launch; address #62 |
| lib-react | | library to use Adama with React |
| lib-vue | | library to use Adama with Vue |
| lib-svelte | | library to use Adama with svelte |
| project | IP | description |
|---|
| kickoff demos | X | See https://asciinema.org/ for more information |
| client-walk | | A detailed walkthough about how to use the client library, the expectations, and core APIs |
| improve overview | | Make the overview easier to understand, more details, etc |
| cheat-sheet | | document the vital aspects of the language and provide copy-pasta examples |
| micro-examples | | a mini stack-overflow of sorts |
| tutorial-app | | walk through the basics of building an Adama with just Adama and RxHTML |
| tutorial-twilio | | build a twilio bot as an example with details on calling services |
| tutorial-web | | a HOWTO host a static website with Adama |
| tutorial-domain | | a HOWTO use Adama's domain hosting support |
| zero-hero | | a breakdown of using the bootstrap tooling to build a complete app |
| feature-complex | | Write about complex number support |
| feature-maybe | | (1) write about how maybes work, (2) write about maybe field deref, (3) write about math and maybe |
| feature-lists | | write more lists |
| feature-map | | write about map transforms |
| feature-dynamic | | write about dynamic types |
| feature-viewer | | write about @viewer |
| feature-context | | write about @context |
| feature-web | | write about @headers / @parameters |
| map/reduce-love | X | reduce love along with maps |
| functions | | procedure, aborts, functions, methods |
| enumeration/dispatch | | talk about dispatch |
| feature-services | | talk about services and linkage to first party along with third party http (and encryption of secrets) |
| feature-async | | talk about async await,decide,fetch, choose, and result |
| result type | | talk about the result type |
| feature-sm | | talk about the state machine, invoke, transition, transition-in |
| web-put | | talk about the web processing |
| stdlib code-gen | | generate documentation from the stdlib (i.e. embed docs in java as annotations) (embed documentation in type) |
| document or kill rpc | | The rpc helper is an interesting way of making channels, but it isn't used right now... kill or document |
| assets | | |
| rxhtml auth stuff | | |
| auto ids in RxHTML | | |
Standard Library
| project | IP | description |
|---|
| stats | | build out a statistics package that is decent and correct (median isn't good right now) (sum, average, median, product, count_non_zero) |
| to/from Base64 | | convert a string to and from Base64 with maybe<string> |
| substituteFirst | | find the needle in a haystack and replace the first instance |
| substituteLast | | find the needle in a haystack and replace the last instance |
| substituteAll | | find the needle in a haystack and replace the every instance |
| substituteNth | | find the needle in a haystack and replace the n'th instance |
| format | | need var_args (just use a string[]) support, but the idea is to allow efficient string construction from many known finite pieces |
| split /w limit | | only split a certain degree |
| find | | find a string within a string |
| findAll | | find a string within a string and produce a list if indicies |
| proper | | take a string, split on spaces, normalize white space, turn every string into a camel case word, join together |
| initials | | take a string, split on spaces, normalize white space, turn every string into a concat of just first letters capitalized |
| convex hull | | a 2D version at first | |
| matrix math | | inverse, multiply, etc... |
| exact-math | | math that doesn't overflow, and if it does, empty maybe |
| random+ | | gaussian random + other statisitical random functions |
| even/odd | | simple and fun functions |
| factorial | | |
| gamma factorial | | https://en.wikipedia.org/wiki/Gamma_function |
| color | | hsv/cmyk and a color typing library |
The online web portal needs a lot of work to be useful.
The web portal should be upgraded to provide a manager view along with tools to get started along with steps to make progress.
| project | IP | milestones/description |
|---|
| render-plan | | Render and explain the deployment plan |
| render-routes | | Render and explain the routing including both rxhtml and web instructions |
| better-debugger | X | The debugger sucks |
| support fbauth | | |
| metrics | | A metrics explorer |
| service calls | | A way to explore which documents are making which service calls and the results |
If your name isn't Jeff, then the current environment is not great.
| project | IP | milestones/description |
|---|
| shell script love | | Improve the build experience outside of Ubuntu |
| test MacOS | X | Work through issues with unit tests on MacOS and any productivity issues with the python build script |
| test Windows | X | Work through issues with unit tests on Windows and any productivity issues with the python build script |
| local mode | X | Adama should be able to run locally with a special version of Adama just for applications and local development |
| faster unit tests | | Improve the testing to not leverage shared resources (stdout, cough) such that testing can be made parallel |
| write documentation about structure | | Write a document to outlining the high level mono-repo structure |
| project | IP | milestones/description |
|---|
| bubble view filtering | | Bubbles should recompute only with related viewer changes happen rather than the viewer changed |
| compressed tables | | for tables that don't have deep subscriptions, let's compact records |
| table-paging-{disk/db} | | for tables that don't have deep subscription and stale data, let's introduce an offload database for caching |
| java-compiler-service | | instead of every adama instance compiling java, let's cache some byte code and maybe offload to an isolated service |
The language is going to big with many features!
| project | IP | milestones/description |
|---|
| index-tensor | | Tables should be indexable based on integer tuples. Beyond efficiency, language extensions can help work with tables in a more natural array style (think 2D grids) |
| index-graph | | Tables should be able to become hyper graphs for efficient navigation between records using a graph where edges can be annotated (this maps) |
| full-text-naive | X | string =? string is a full text operator that is naive |
| full-text-index | | introduce full indexing where records describe a rich query language |
| dynamic-order | | introduce a special command language for runtime ordering of lists |
| dynamic-query | | introduce a special language for queries to be dynamic |
| math-matrix | | The type system and math library should come with vectors and matrices out of the box |
| 2d-primitives | | circle, rectangle |
| xml support | | Convert messages to XML |
| metrics emit $num; | X | The language should have first class support for metrics (counters, inflight, distributions) |
| metrics emit $string; | | The language should support emitting tags with metrics to help shred and classifiy metrics |
| auto-convert-msg | | the binding of messages can be imprecise, need to simplify and automate @convert primarily for services |
| bubble + privacy | X | Add a way to annotate a bubble with a privacy policy to simplify privacy logic |
| privacy-policy caching | X | instead of making privacy policies executable every single time, cache them by person and invalidate on data changes |
| table-protocol | | introduce a way to expose a table protocol for reading and writing tables via a data-grid component |
| sum/bag types | | a sum type is going to be a special type of message |
| auto-convert | | auto convert messages to dynamic |
| normalize messages | | messages of the same structure should auto convert |
| proper-lambdas | X | I should be able to accept functions as argruments to other functions |
| extension methods | | Extend every message/record based on a structural pattern |
| message filtering | | Apply rules for parsing messages like "trim/lowercase" to help ensure only valid data enters system |
| message aborts | | Apply simple rules to reject messages that are invalid |
| decimal type | | for arbitray rationals |
enum E {Aa, Ab, B, C}
bag<E> B {
A* {
int x;
}
B {
int x;
int y;
}
C { }
}
The bag creates a record with the union of all fields along with various along with a virtual type for each instance which maps to the appropriate subset.
We can derefence it via a match
B my_bag;
#sm {
match B as o {
A* {
o.x = 123;
}
B {
o.x = 42;
o.y = 7;
}
C {
}
}
}
Implicitly a bag has a type of the enumeration, and ingestion would be the way to set the type along with parameters.
my_bag <- {type: E::Aa, x: 42};
This will typecheck that the bag has everything needed and will clear other fields out. This is all sketch work, but the key design goal is to provide some degree of compact representation of data based on the kind of a thng.
For integration across different ecosystems, there are more protocols to bridge gaps.
| project | IP | milestones/description |
|---|
| mqtt | X | (1) Write design document how to adapt Adama to MQTT, (2) Build it with TLS, (3) Built it with plain-text |
| sse | | (1) Write design document how to adapt Adama to Server-Sent Events, (2) Build it with TLS |
| project | IP | milestones/description |
|---|
| smaller deltas | | (1) Define a log format that leverages binary serialization and compacts field definitions, (2) convert JSON deltas to binary in the logger to measure impact (throughput and latency), (3) leverage format upstream to minimize future network |
| healing log | | Implement a log data structure that can heal (anti entropy) across machines using Adama's network stack |
| raft | | Implement raft leader election and log append using Adama's network stack |
| control plane | | (1) Manual definition of raft shards definitions, (2) automatic machine management |
At some point, Adama is going to be at the edge with hundreds of nodes across the world.
| project | IP | milestones |
|---|
| diagram | X | diagram the usage of the database in the adama service |
| billing | X | have billing route partial metering records to billing document ( and globalize ) |
| proxy-mode | X | proxy the WS API from region A to region B (or global important services ) |
| remote-finder | X | extend WS API to implement a Finder for region A to do core tasks (or globalize) |
| finder in adama | X | Turn core service into a finder cache for web tier |
| region-isolate | | Allow regions to have storage for local documents |
| capacity-global | | globalize capacity management |
| test-heat-cap | | validate when an adama host heats up that traffic sheds |
| test-cold-cap | | validate when an adama host cools off that traffic returns for density |
| cap-config | | make high/low vectors dynamic configurable |
| ro-replica | | (1) introduce new observe command which is a read-only version of connect, (2) have web routes go to an in-region replica |
| reconcile | X | every adama host should be lazy with unknown spaces and also reconcile capacity if it should redeploy (due to missed deployment message) |
| space-delete-bot | | make run only in the global region |
| dead-detector | | only run in global region |
| leader-election-overlord | | instead of a single host, get leader election in place and have one adama assign work to other adama instances |
| all-make-target | | instead of overlord making targets, have every adama instance expose a targets file |
| monitoring | | instead of using a local prometheus, let's use a service |
| gc-2.0 | | garbage collection should probably run on adama |
| finish global CP | X | Adama is going to have a primary region with a control plane proxy to DB |
| write clients for global CP | X | wrap SelfClient to talk to Adama control region |
| e2e region tests | | rethink the testing strategy to create a global multi-region foot print in unit tests (maybe with a customized runner) |
Adama is a service.
| project | IP | milestones/description |
|---|
| env-bootstrap | | automatic the memory and other JVM args |
| third-party replication | X | the language should allow specification of a endpoint to replace a piece of data to on data changes. This requires maintaining a local copy in the document along with a state machine about status. The tricky bit requires a delete notification. There is also the need to load every document on a deployment. |
| replication-search | | provide a "between document" search using replication tech |
| replication-graph | | similar to search, replicate part of the document into a graph database |
| metrics | X | documents should able to emit metrics |
| fix-keys | X | document keys are stored with both private and public keys, and this is really bad; we should only store the public key and version the keys along with an expiry (core work done, need to remove old code) |
| op-query-engine | X | a tool to debug the status of a document live |
| portlets | | maybe part of replication, but a subdocument that can emit messages that are independent subscriptions (for SSE/MQTT) and for Adama to consume |
| adama-actor | X | implement Adama as a special first class service |
| twilio-service | | implement twilio as a first party service |
| stripe-service | X | implement stripe as a first party service |
| discord-service | | implement discord as a first party service |
| slack-service | | implement slack as a first party service |
| http-service | | study postman and design a generic HTTP service for third party integration |
| results-stream | | figure out how to ensure deliveries can overwrite prior entries |
| portlets - passive net | | if we express the desigre for a result to hold the stream results for a foreign document's portlet then this creates a graph such that documents need to be reloaded. that is, we need a persistent subscription |
| firebase - notifications | | we should be able to send notifications |
| document filter | X | instead of relying on privacy and wasting compute, a viewer should be able to opt into which fields to view. |
Adama is a web host provider of sorts!
| project | IP | milestones/description |
|---|
| web-async delete | X | allow DELETEs to contain async calls |
| web-async get | X | allow GETs to contain async calls |
| request caching | X | respect the cache_ttl_ms |
| doc auth - expiry | | infer cache_ttl_ms as an expiry for doc auth |
| asset transforms | | implement some basic asset transforms |
| web-abort put/delete | X | web calls that write should support abort |
| @context | X | ensure web operations can access context |
| web-metrics | | add an API for the client to emit metrics |
| add auth for web | | the principal for web* is currently @no_one; it should be a valid user |
| build delta accumulator | | slow clients may be get overwhelmed, the edge should buffer deltas and merge them together |
Overlord is how the fleet is managed and aggregator.
| project | IP | milestones/description |
|---|
| canary | | for testing the service health and correctness; overlord should maintain a constant state of various high-value API calls |
| operationalize-super | | the "super" service needs a secure environment |
| ui for query | | dynamic queries |
| billing-send | X | Simplify the billing engine and remove the overlord need |
| ntp | | look into time sync |
RxHTML is a small set of extensions to allow HTML to be dynamic on the fly. The spiritual question is "what minimal number of things does HTML need to build apps?"
| project | IP | milestones/description |
|---|
| headwindcss | | Port tailwindcss to Java for vertical control |
| components | X | Bring clarity for single javascript extentions for new controls |
| time | X | Custom component for selecting a time of day (Blocked on components model) |
| date | X | Custom component for selecting a date or a date range (Blocked on components model) |
| color | X | Custom component for selecting a color (Blocked on components model) |
| graph | | Custom component with rich config to visualize graphs |
| server-side | | Create a customized shell for each page such that server side rendering allows faster presence |
| convert-react | | Convert the RxHTML forest into a complete React app |
| gc | | figure out if there is still a bug with "rxhtml fire delete" isn't cleaning up pubsub |
| remove-col | | remove the rxhtml column from the spaces column and move into document; #127 |
| auto-ids | | A big pain point in HTML is the pair bonding of label to input via id. |
| typechecker | | A tool to validate an RxHTML forest doesn't have issues |
| ai-tester | | A browser extension to walk and validate a site is working and collecting tracking tokens |
| project | IP | milestones/description |
|---|
| capacitor | X | should be able to convert an RxHTML SPA to a mobile app |
| notification support | |
The vision is for a runtime just for games. See this post for more information
| project | IP | milestones/description |
|---|
| design document | | Write a design document from Jeff's notes |
| runtime-android | | Implement a starting runtime for web using android |
| gameboard demo(s) | | Write a game board demo of various games |
| runtime-web | | Implement runtime for web using rust |
| rxhtml-integ | | Integrate runtime-web into RxHTML as a new component |
Similar to RxHTML, the question is how to build a minimal runtime for iOS and Android applications. Tactically speaking, we can use RxHTML with capacitor.
| project | IP | milestones |
|---|
| design document | | design a simple XML only way to build Android applications using reactive data binding similar to RxHTML |