Best rapid development single AJAX page with table fill

14ned

As I've mentioned elsewhere in the forum, I've been busy at work recently making a very high performance REST API server which queries a bespoke Petabyte scale database, and returns results in whichever format the Accept header asked for, including JSON.

My first question is this: is 30,000 queries answered per sec per kernel thread good? That's on the loopback device, I'm very sure it would be much faster on a real high end NIC. But as not a web developer, I actually don't know if it's any good or not, hence asking for feedback.

My second question is this: I need to quickly throw together a pretty looking web page which lets people input queries and get back records matching the queries. The records returned almost certainly would fill a table with rows. The last time I wrote an AJAX web page was in the mid-2000s, and I used jQuery back then.

I assume that technology has marched considerably further than jQuery, but I have no idea what. I'm a C++ dev, not a web dev, I've not kept up to date with web technologies. Can anyone better informed than I suggest a framework or solution which can query a REST API server using AJAX and update a page? Is there some sort of drag and drop UI designer nowadays which makes pretty looking UIs very quickly?

I'm basically looking for "Visual Basic for webpages", if there is such a thing. As in, with two hours of work, the whole thing is working perfectly, and looks professional.

Thanks in advance.

Niall

S.M.B.

Not aware of any drag and drop web builder that would do it all for you.

I'd imagine the quickest approach would be do something similar to your previous effort but use react instead of jquery.

Or if the UI is extremely basic you could knock up a solution in vanilla JS very quickly.

Shai

I find some of the things mentioned in your post a bit confusing. I apologise in advance for some of my comments below being a bit basic. I know neither you nor your development capabilities, so I'm casting a wide net here. No disrespect is intended.

- the 30k queries/thread is not really relevant in itself as I would expect the database to become a bottleneck here and not your REST api server. Do you know how many queries you'll actually need to deal with and what kind of load they will introduce on the database?

- when you say that you want to let people input queries through a webpage, does this mean that you expect people to write Query Language statements directly in an input form and just send that over to the database to be executed? Or is there more to this than that? Is there a validation layer somewhere that will prevent dangerous queries and injection attacks from being executed?

- jQuery is just fine for getting some data back and filling out a table. Using a JS framework for this would be some serious over-engineering. If you need a library to make your page look a bit nicer, then I would suggest using Bootstrap for now.

antimatterx

Use the vanilla Javascript `Fetch` method to hit an endpoint and return you JSON data.

14ned

S.M.B. wrote: »

Not aware of any drag and drop web builder that would do it all for you.

I'd imagine the quickest approach would be do something similar to your previous effort but use react instead of jquery.

I did a bit of research based on your reply here, looking into React and Vue. I suspect neither would be faster to deliver for what I need than hand coding the thing in jquery or even plain Javascript. However, I also don't think they'd be slower, and I can see an advantage to maintenance plus a chance to upskill. So thanks for the suggestion!

- the 30k queries/thread is not really relevant in itself as I would expect the database to become a bottleneck here and not your REST api server. Do you know how many queries you'll actually need to deal with and what kind of load they will introduce on the database?

The database is of my own design and implementation, and has an average single item lookup latency of just under one microsecond if the query dataset is currently in RAM. The DB scales linearly with CPU core count as it performs no synchronisation, and can work with a few petabytes of data per node. This is my particular field of deep specialisation, I was specifically hired to design this DB after two previous well funded attempts failed.

For your purposes you can assume that the DB has zero overhead relative to the REST API call (in fact the whole query + JSON serialisation routine is benching at about 3 microseconds per API call, it's almost statistical noise compared to the rest of the REST and TCP ceremony).

- when you say that you want to let people input queries through a webpage, does this mean that you expect people to write Query Language statements directly in an input form and just send that over to the database to be executed? Or is there more to this than that? Is there a validation layer somewhere that will prevent dangerous queries and injection attacks from being executed?

As a fixed feature database, there is no query language. It's all hardcoded around the dataset. Custom DB for custom data. So no attacks of that kind are possible. The queries are actually specified using plain old HTTP GET e.g. get all records for symbol X between timestamps Y and Z where field value A is between B and C.

The DB is a streaming results implementation, so if you ask for a billion records (several billion rows are added per day), that's absolutely fine. It only fetches the beginning of the results, and it'll go incrementally get more as you pull the results. Each node has several 40 GBit NICs, so the network is by far the limiting factor in any case.

Besides, none of these nodes will ever see the public internet. Clients get a private dedicated fabric connection to each node, for which they pay handsomely. That said, I'll be deploying a raft of fuzz and permutation testing on the HTTP GET input parsers, because I don't trust anyone.

Niall

antimatterx

Ignore my comment though

Colonel Panic

So you can build some insane, hardcoded, petabytes scale database, but can't figure out what's good performance when exposing it via HTTP?

Is this just some weird humblebrag?

14ned

Colonel Panic wrote: »

So you can build some insane, hardcoded, petabytes scale database, but can't figure out what's good performance when exposing it via HTTP?

Is this just some weird humblebrag?

Well, it's not my domain of expertise. I've no idea what's considered fast outside what I know.

(And if it was humblebrag, it was unintentional. More likely just me replying late at night after a long day of coding, and I chose the wrong balance of detail vs response)

In terms of research, the best I've found for comparison is https://bitbucket.org/sobjectizerteam/restinio-benchmark/src/default/ where:

- Raw Linux epoll() does 445k ops/sec/thread
- Boost.Beast does > 200k ops/sec/thread

On that basis, my ~30k ops/sec/thread looks awful. It's why I was asking, because I really don't know.

Niall

Colonel Panic

Apologies for my tone, a late one for me too last night and here I am up at 6:30

To give a more measured response, I don’t think anyone in general will know, since most people’s concerns with http based APIs is concurrent connections and first few bytes response time, and payloads tend to be pretty small.

You mentioned epoll there, is your http server implementation using using it or other asynchronous io solution to server up requests? Where are the bottlenecks in turning your raw data into whatever is specified in the request header? And has the customer actually requested super high perf via a REST api? Is those the only interface to your DB?

I used to have to worry about these things a bit more, but I moved away from back end high performance stuff into a more wishy washy domain!

14ned

Colonel Panic wrote: »

Apologies for my tone, a late one for me too last night and here I am up at 6:30

Christmas is very tiring when you have smaller children. They wake at dawn no matter what, with no regard to what the parents were doing the night before.

To give a more measured response, I don’t think anyone in general will know, since most people’s concerns with http based APIs is concurrent connections and first few bytes response time, and payloads tend to be pretty small.

For the benchmark results I mentioned, response payload is a single record, just a few hundred bytes. It's a single kernel thread for the server, but 15 client threads doing the querying to simulate real life.

You mentioned epoll there, is your http server implementation using using it or other asynchronous io solution to server up requests?

It uses the industry standard in C++ https://think-async.com/Asio/.

That's epoll on Linux, kqueues on BSD and Mac OS, IOCP on Windows.

Where are the bottlenecks in turning your raw data into whatever is specified in the request header?

I reckon the total time per request spent in our DB and formatting the JSON response is under 3 microseconds. So the other ~30 microseconds is code outside my control. I reckon a good 18 microseconds is latency in the loopback device.

You've made me wonder able Nagle's actually. I'll try turning that off when I return to work in January.

And has the customer actually requested super high perf via a REST api? Is those the only interface to your DB?

Actually, the customer wants the data returned in a space delimited text record format, one line per record. Like CSV, but more legacy.

The JSON api is rather faster, unsurprisingly, and we're building that for what we think the client's engineers will actually use in practice . We'll also be throwing together a demo query webpage, and for that JSON makes far more sense.

The REST server serves both formats incidentally, based on the Accept MIME header.

Niall