1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. Note: prior to version 11, this would be embedded_httpd_processes on Linux and embedded_httpd_threads everywhere else. It now means embedded_httpd_hybrid everywhere supported and embedded_httpd_threads everywhere else. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. Please note: on nginx make sure you add `scgi_param PATH_INFO $document_uri;` to the config! 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 490 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 491 492 /++ 493 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 494 +/ 495 version(Demo) 496 unittest { 497 import arsd.cgi; 498 499 mixin DispatcherMain!( 500 "/".serveStaticFileDirectory(null, true) 501 ); 502 } 503 504 /++ 505 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 506 +/ 507 version(Demo) 508 unittest { 509 import arsd.cgi; 510 511 void requestHandler(Cgi cgi) { 512 cgi.dispatcher!( 513 "/".serveStaticFileDirectory(null, true) 514 ); 515 } 516 517 // mixin GenericMain!requestHandler would add this function: 518 void main(string[] args) { 519 // this is all the content of [cgiMainImpl] which you can also call 520 521 // cgi.d embeds a few add on functions like real time event forwarders 522 // and session servers it can run in other processes. this spawns them, if needed. 523 if(tryAddonServers(args)) 524 return; 525 526 // cgi.d allows you to easily simulate http requests from the command line, 527 // without actually starting a server. this function will do that. 528 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 529 return; 530 531 RequestServer server; 532 // you can change the default port here if you like 533 // server.listeningPort = 9000; 534 535 // then call this to let the command line args override your default 536 server.configureFromCommandLine(args); 537 538 // here is where you could print out the listeningPort to the user if you wanted 539 540 // and serve the request(s) according to the compile configuration 541 server.serve!(requestHandler)(); 542 543 // or you could explicitly choose a serve mode like this: 544 // server.serveEmbeddedHttp!requestHandler(); 545 } 546 } 547 548 /++ 549 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 550 otherwise run through the rest of the internal mechanisms to call your functions without actually 551 spinning up a server. 552 +/ 553 version(Demo) 554 unittest { 555 import arsd.cgi; 556 557 void requestHandler(Cgi cgi) { 558 559 } 560 561 // D doesn't let me embed a unittest inside an example unittest 562 // so this is a function, but you can do it however in your real program 563 /* unittest */ void runTests() { 564 auto tester = new CgiTester(&requestHandler); 565 566 auto response = tester.GET("/"); 567 assert(response.code == 200); 568 } 569 } 570 571 /++ 572 The session system works via a built-in spawnable server. 573 574 Bugs: 575 Requires addon servers, which are not implemented yet on Windows. 576 +/ 577 version(Posix) 578 version(Demo) 579 unittest { 580 import arsd.cgi; 581 582 struct SessionData { 583 string userId; 584 } 585 586 void handler(Cgi cgi) { 587 auto session = cgi.getSessionObject!SessionData; 588 589 if(cgi.pathInfo == "/login") { 590 session.userId = cgi.queryString; 591 cgi.setResponseLocation("view"); 592 } else { 593 cgi.write(session.userId); 594 } 595 } 596 597 mixin GenericMain!handler; 598 } 599 600 static import std.file; 601 602 static import arsd.core; 603 version(Posix) 604 import arsd.core : makeNonBlocking; 605 606 607 // for a single thread, linear request thing, use: 608 // -version=embedded_httpd_threads -version=cgi_no_threads 609 610 version(Posix) { 611 version(CRuntime_Musl) { 612 613 } else version(minimal) { 614 615 } else { 616 version(FreeBSD) { 617 // I never implemented the fancy stuff there either 618 } else { 619 version=with_breaking_cgi_features; 620 version=with_sendfd; 621 version=with_addon_servers; 622 } 623 } 624 } 625 626 version(Windows) { 627 version(minimal) { 628 629 } else { 630 // not too concerned about gdc here since the mingw version is fairly new as well 631 version=with_breaking_cgi_features; 632 } 633 } 634 635 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 636 void cloexec(int fd) { 637 version(Posix) { 638 import core.sys.posix.fcntl; 639 fcntl(fd, F_SETFD, FD_CLOEXEC); 640 } 641 } 642 643 void cloexec(Socket s) { 644 version(Posix) { 645 import core.sys.posix.fcntl; 646 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 647 } 648 } 649 650 // the servers must know about the connections to talk to them; the interfaces are vital 651 version(with_addon_servers) 652 version=with_addon_servers_connections; 653 654 version(embedded_httpd) { 655 version(OSX) 656 version = embedded_httpd_threads; 657 else 658 version=embedded_httpd_hybrid; 659 /* 660 version(with_openssl) { 661 pragma(lib, "crypto"); 662 pragma(lib, "ssl"); 663 } 664 */ 665 } 666 667 version(embedded_httpd_hybrid) { 668 version=embedded_httpd_threads; 669 version(cgi_no_fork) {} else version(Posix) 670 version=cgi_use_fork; 671 version=cgi_use_fiber; 672 } 673 674 version(cgi_use_fork) 675 enum cgi_use_fork_default = true; 676 else 677 enum cgi_use_fork_default = false; 678 679 version(embedded_httpd_processes) 680 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 681 682 version(embedded_httpd_threads) { 683 // unless the user overrides the default.. 684 version(cgi_session_server_process) 685 {} 686 else 687 version=cgi_embedded_sessions; 688 } 689 version(scgi) { 690 // unless the user overrides the default.. 691 version(cgi_session_server_process) 692 {} 693 else 694 version=cgi_embedded_sessions; 695 } 696 697 // fall back if the other is not defined so we can cleanly version it below 698 version(cgi_embedded_sessions) {} 699 else version=cgi_session_server_process; 700 701 702 version=cgi_with_websocket; 703 704 enum long defaultMaxContentLength = 5_000_000; 705 706 /* 707 708 To do a file download offer in the browser: 709 710 cgi.setResponseContentType("text/csv"); 711 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 712 */ 713 714 // FIXME: the location header is supposed to be an absolute url I guess. 715 716 // FIXME: would be cool to flush part of a dom document before complete 717 // somehow in here and dom.d. 718 719 720 // these are public so you can mixin GenericMain. 721 // FIXME: use a function level import instead! 722 public import std.string; 723 public import std.stdio; 724 public import std.conv; 725 import std.uri; 726 import std.uni; 727 import std.algorithm.comparison; 728 import std.algorithm.searching; 729 import std.exception; 730 import std.base64; 731 static import std.algorithm; 732 import std.datetime; 733 import std.range; 734 735 import std.process; 736 737 import std.zlib; 738 739 740 T[] consume(T)(T[] range, int count) { 741 if(count > range.length) 742 count = range.length; 743 return range[count..$]; 744 } 745 746 int locationOf(T)(T[] data, string item) { 747 const(ubyte[]) d = cast(const(ubyte[])) data; 748 const(ubyte[]) i = cast(const(ubyte[])) item; 749 750 // this is a vague sanity check to ensure we aren't getting insanely 751 // sized input that will infinite loop below. it should never happen; 752 // even huge file uploads ought to come in smaller individual pieces. 753 if(d.length > (int.max/2)) 754 throw new Exception("excessive block of input"); 755 756 for(int a = 0; a < d.length; a++) { 757 if(a + i.length > d.length) 758 return -1; 759 if(d[a..a+i.length] == i) 760 return a; 761 } 762 763 return -1; 764 } 765 766 /// If you are doing a custom cgi class, mixing this in can take care of 767 /// the required constructors for you 768 mixin template ForwardCgiConstructors() { 769 this(long maxContentLength = defaultMaxContentLength, 770 string[string] env = null, 771 const(ubyte)[] delegate() readdata = null, 772 void delegate(const(ubyte)[]) _rawDataOutput = null, 773 void delegate() _flush = null 774 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 775 776 this(string[] args) { super(args); } 777 778 this( 779 BufferedInputRange inputData, 780 string address, ushort _port, 781 int pathInfoStarts = 0, 782 bool _https = false, 783 void delegate(const(ubyte)[]) _rawDataOutput = null, 784 void delegate() _flush = null, 785 // this pointer tells if the connection is supposed to be closed after we handle this 786 bool* closeConnection = null) 787 { 788 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 789 } 790 791 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 792 } 793 794 /// thrown when a connection is closed remotely while we waiting on data from it 795 class ConnectionClosedException : Exception { 796 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 797 super(message, file, line, next); 798 } 799 } 800 801 802 version(Windows) { 803 // FIXME: ugly hack to solve stdin exception problems on Windows: 804 // reading stdin results in StdioException (Bad file descriptor) 805 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 806 private struct stdin { 807 struct ByChunk { // Replicates std.stdio.ByChunk 808 private: 809 ubyte[] chunk_; 810 public: 811 this(size_t size) 812 in { 813 assert(size, "size must be larger than 0"); 814 } 815 do { 816 chunk_ = new ubyte[](size); 817 popFront(); 818 } 819 820 @property bool empty() const { 821 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 822 } 823 @property nothrow ubyte[] front() { return chunk_; } 824 void popFront() { 825 enforce(!empty, "Cannot call popFront on empty range"); 826 chunk_ = stdin.rawRead(chunk_); 827 } 828 } 829 830 import core.sys.windows.windows; 831 static: 832 833 T[] rawRead(T)(T[] buf) { 834 uint bytesRead; 835 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 836 837 if (!result) { 838 auto err = GetLastError(); 839 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 840 return buf[0..0]; 841 // Some other error, throw it 842 843 char* buffer; 844 scope(exit) LocalFree(buffer); 845 846 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 847 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 848 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 849 throw new Exception(to!string(buffer)); 850 } 851 enforce(!(bytesRead % T.sizeof), "I/O error"); 852 return buf[0..bytesRead / T.sizeof]; 853 } 854 855 auto byChunk(size_t sz) { return ByChunk(sz); } 856 857 void close() { 858 std.stdio.stdin.close; 859 } 860 } 861 } 862 863 /// The main interface with the web request 864 class Cgi { 865 public: 866 /// the methods a request can be 867 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 868 // these are defined in the standard, but idk if they are useful for anything 869 OPTIONS, TRACE, CONNECT, 870 // These seem new, I have only recently seen them 871 PATCH, MERGE, 872 // this is an extension for when the method is not specified and you want to assume 873 CommandLine } 874 875 876 /+ 877 878 ubyte[] perRequestMemoryPool; 879 void[] perRequestMemoryPoolWithPointers; 880 // might want to just slice the buffer itself too when we happened to have gotten a full request inside it and don't need to decode 881 // then the buffer also can be recycled if it is set. 882 883 // we might also be able to set memory recyclable true by default, but then the property getters set it to false. but not all the things are property getters. but realistically anything except benchmarks are gonna get something lol so meh. 884 885 /+ 886 struct VariableCollection { 887 string[] opIndex(string name) { 888 889 } 890 } 891 892 /++ 893 Call this to indicate that you've not retained any reference to the request-local memory (including all strings returned from the Cgi object) outside the request (you can .idup anything you need to store) and it is thus free to be freed or reused by another request. 894 895 Most handlers should be able to call this; retaining memory is the exception in any cgi program, but since I can't prove it from inside the library, it plays it safe and lets the GC manage it unless you opt into this behavior. All cgi.d functions will duplicate strings if needed (e.g. session ids from cookies) so unless you're doing something yourself, this should be ok. 896 897 History: 898 Added 899 +/ 900 public void recycleMemory() { 901 902 } 903 +/ 904 905 906 /++ 907 Cgi provides a per-request memory pool 908 909 +/ 910 void[] allocateMemory(size_t nBytes) { 911 912 } 913 914 /// ditto 915 void[] reallocateMemory(void[] old, size_t nBytes) { 916 917 } 918 919 /// ditto 920 void freeMemory(void[] memory) { 921 922 } 923 +/ 924 925 926 /* 927 import core.runtime; 928 auto args = Runtime.args(); 929 930 we can call the app a few ways: 931 932 1) set up the environment variables and call the app (manually simulating CGI) 933 2) simulate a call automatically: 934 ./app method 'uri' 935 936 for example: 937 ./app get /path?arg arg2=something 938 939 Anything on the uri is treated as query string etc 940 941 on get method, further args are appended to the query string (encoded automatically) 942 on post method, further args are done as post 943 944 945 @name means import from file "name". if name == -, it uses stdin 946 (so info=@- means set info to the value of stdin) 947 948 949 Other arguments include: 950 --cookie name=value (these are all concated together) 951 --header 'X-Something: cool' 952 --referrer 'something' 953 --port 80 954 --remote-address some.ip.address.here 955 --https yes 956 --user-agent 'something' 957 --userpass 'user:pass' 958 --authorization 'Basic base64encoded_user:pass' 959 --accept 'content' // FIXME: better example 960 --last-event-id 'something' 961 --host 'something.com' 962 --session name=value (these are added to a mock session, changes to the session are printed out as dummy response headers) 963 964 Non-simulation arguments: 965 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 966 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 967 968 */ 969 970 /** Initializes it with command line arguments (for easy testing) */ 971 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 972 rawDataOutput = _rawDataOutput; 973 // these are all set locally so the loop works 974 // without triggering errors in dmd 2.064 975 // we go ahead and set them at the end of it to the this version 976 int port; 977 string referrer; 978 string remoteAddress; 979 string userAgent; 980 string authorization; 981 string origin; 982 string accept; 983 string lastEventId; 984 bool https; 985 string host; 986 RequestMethod requestMethod; 987 string requestUri; 988 string pathInfo; 989 string queryString; 990 991 bool lookingForMethod; 992 bool lookingForUri; 993 string nextArgIs; 994 995 string _cookie; 996 string _queryString; 997 string[][string] _post; 998 string[string] _headers; 999 1000 string[] breakUp(string s) { 1001 string k, v; 1002 auto idx = s.indexOf("="); 1003 if(idx == -1) { 1004 k = s; 1005 } else { 1006 k = s[0 .. idx]; 1007 v = s[idx + 1 .. $]; 1008 } 1009 1010 return [k, v]; 1011 } 1012 1013 lookingForMethod = true; 1014 1015 scriptName = args[0]; 1016 scriptFileName = args[0]; 1017 1018 environmentVariables = cast(const) environment.toAA; 1019 1020 foreach(arg; args[1 .. $]) { 1021 if(arg.startsWith("--")) { 1022 nextArgIs = arg[2 .. $]; 1023 } else if(nextArgIs.length) { 1024 if (nextArgIs == "cookie") { 1025 auto info = breakUp(arg); 1026 if(_cookie.length) 1027 _cookie ~= "; "; 1028 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 1029 } 1030 if (nextArgIs == "session") { 1031 auto info = breakUp(arg); 1032 _commandLineSession[info[0]] = info[1]; 1033 } 1034 1035 else if (nextArgIs == "port") { 1036 port = to!int(arg); 1037 } 1038 else if (nextArgIs == "referrer") { 1039 referrer = arg; 1040 } 1041 else if (nextArgIs == "remote-address") { 1042 remoteAddress = arg; 1043 } 1044 else if (nextArgIs == "user-agent") { 1045 userAgent = arg; 1046 } 1047 else if (nextArgIs == "authorization") { 1048 authorization = arg; 1049 } 1050 else if (nextArgIs == "userpass") { 1051 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 1052 } 1053 else if (nextArgIs == "origin") { 1054 origin = arg; 1055 } 1056 else if (nextArgIs == "accept") { 1057 accept = arg; 1058 } 1059 else if (nextArgIs == "last-event-id") { 1060 lastEventId = arg; 1061 } 1062 else if (nextArgIs == "https") { 1063 if(arg == "yes") 1064 https = true; 1065 } 1066 else if (nextArgIs == "header") { 1067 string thing, other; 1068 auto idx = arg.indexOf(":"); 1069 if(idx == -1) 1070 throw new Exception("need a colon in a http header"); 1071 thing = arg[0 .. idx]; 1072 other = arg[idx + 1.. $]; 1073 _headers[thing.strip.toLower()] = other.strip; 1074 } 1075 else if (nextArgIs == "host") { 1076 host = arg; 1077 } 1078 // else 1079 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1080 1081 nextArgIs = null; 1082 } else if(lookingForMethod) { 1083 lookingForMethod = false; 1084 lookingForUri = true; 1085 1086 if(arg.asLowerCase().equal("commandline")) 1087 requestMethod = RequestMethod.CommandLine; 1088 else 1089 requestMethod = to!RequestMethod(arg.toUpper()); 1090 } else if(lookingForUri) { 1091 lookingForUri = false; 1092 1093 requestUri = arg; 1094 1095 auto idx = arg.indexOf("?"); 1096 if(idx == -1) 1097 pathInfo = arg; 1098 else { 1099 pathInfo = arg[0 .. idx]; 1100 _queryString = arg[idx + 1 .. $]; 1101 } 1102 } else { 1103 // it is an argument of some sort 1104 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1105 auto parts = breakUp(arg); 1106 _post[parts[0]] ~= parts[1]; 1107 allPostNamesInOrder ~= parts[0]; 1108 allPostValuesInOrder ~= parts[1]; 1109 } else { 1110 if(_queryString.length) 1111 _queryString ~= "&"; 1112 auto parts = breakUp(arg); 1113 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1114 } 1115 } 1116 } 1117 1118 acceptsGzip = false; 1119 keepAliveRequested = false; 1120 requestHeaders = cast(immutable) _headers; 1121 1122 cookie = _cookie; 1123 cookiesArray = getCookieArray(); 1124 cookies = keepLastOf(cookiesArray); 1125 1126 queryString = _queryString; 1127 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1128 get = keepLastOf(getArray); 1129 1130 postArray = cast(immutable) _post; 1131 post = keepLastOf(_post); 1132 1133 // FIXME 1134 filesArray = null; 1135 files = null; 1136 1137 isCalledWithCommandLineArguments = true; 1138 1139 this.port = port; 1140 this.referrer = referrer; 1141 this.remoteAddress = remoteAddress; 1142 this.userAgent = userAgent; 1143 this.authorization = authorization; 1144 this.origin = origin; 1145 this.accept = accept; 1146 this.lastEventId = lastEventId; 1147 this.https = https; 1148 this.host = host; 1149 this.requestMethod = requestMethod; 1150 this.requestUri = requestUri; 1151 this.pathInfo = pathInfo; 1152 this.queryString = queryString; 1153 this.postBody = null; 1154 this.requestContentType = null; 1155 } 1156 1157 private { 1158 string[] allPostNamesInOrder; 1159 string[] allPostValuesInOrder; 1160 string[] allGetNamesInOrder; 1161 string[] allGetValuesInOrder; 1162 } 1163 1164 CgiConnectionHandle getOutputFileHandle() { 1165 return _outputFileHandle; 1166 } 1167 1168 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1169 1170 /** Initializes it using a CGI or CGI-like interface */ 1171 this(long maxContentLength = defaultMaxContentLength, 1172 // use this to override the environment variable listing 1173 in string[string] env = null, 1174 // and this should return a chunk of data. return empty when done 1175 const(ubyte)[] delegate() readdata = null, 1176 // finally, use this to do custom output if needed 1177 void delegate(const(ubyte)[]) _rawDataOutput = null, 1178 // to flush teh custom output 1179 void delegate() _flush = null 1180 ) 1181 { 1182 1183 // these are all set locally so the loop works 1184 // without triggering errors in dmd 2.064 1185 // we go ahead and set them at the end of it to the this version 1186 int port; 1187 string referrer; 1188 string remoteAddress; 1189 string userAgent; 1190 string authorization; 1191 string origin; 1192 string accept; 1193 string lastEventId; 1194 bool https; 1195 string host; 1196 RequestMethod requestMethod; 1197 string requestUri; 1198 string pathInfo; 1199 string queryString; 1200 1201 1202 1203 isCalledWithCommandLineArguments = false; 1204 rawDataOutput = _rawDataOutput; 1205 flushDelegate = _flush; 1206 auto getenv = delegate string(string var) { 1207 if(env is null) 1208 return std.process.environment.get(var); 1209 auto e = var in env; 1210 if(e is null) 1211 return null; 1212 return *e; 1213 }; 1214 1215 environmentVariables = env is null ? 1216 cast(const) environment.toAA : 1217 env; 1218 1219 // fetching all the request headers 1220 string[string] requestHeadersHere; 1221 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1222 if(k.startsWith("HTTP_")) { 1223 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1224 } 1225 } 1226 1227 this.requestHeaders = assumeUnique(requestHeadersHere); 1228 1229 requestUri = getenv("REQUEST_URI"); 1230 1231 cookie = getenv("HTTP_COOKIE"); 1232 cookiesArray = getCookieArray(); 1233 cookies = keepLastOf(cookiesArray); 1234 1235 referrer = getenv("HTTP_REFERER"); 1236 userAgent = getenv("HTTP_USER_AGENT"); 1237 remoteAddress = getenv("REMOTE_ADDR"); 1238 host = getenv("HTTP_HOST"); 1239 pathInfo = getenv("PATH_INFO"); 1240 1241 queryString = getenv("QUERY_STRING"); 1242 scriptName = getenv("SCRIPT_NAME"); 1243 { 1244 import core.runtime; 1245 auto sfn = getenv("SCRIPT_FILENAME"); 1246 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1247 } 1248 1249 bool iis = false; 1250 1251 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1252 if(requestUri.length == 0) { 1253 // IIS sometimes includes the script name as part of the path info - we don't want that 1254 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1255 pathInfo = pathInfo[scriptName.length .. $]; 1256 1257 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1258 1259 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1260 1261 // FIXME: this works for apache and iis... but what about others? 1262 } 1263 1264 1265 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1266 getArray = assumeUnique(ugh); 1267 get = keepLastOf(getArray); 1268 1269 1270 // NOTE: on shitpache, you need to specifically forward this 1271 authorization = getenv("HTTP_AUTHORIZATION"); 1272 // this is a hack because Apache is a shitload of fuck and 1273 // refuses to send the real header to us. Compatible 1274 // programs should send both the standard and X- versions 1275 1276 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1277 // unnecessary with mod_rewrite, so it is commented 1278 1279 //if(authorization.length == 0) // if the std is there, use it 1280 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1281 1282 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1283 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1284 port = to!int(getenv("SERVER_PORT")); 1285 else 1286 port = 0; // this was probably called from the command line 1287 1288 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1289 if(ae.length && ae.indexOf("gzip") != -1) 1290 acceptsGzip = true; 1291 1292 accept = getenv("HTTP_ACCEPT"); 1293 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1294 1295 auto ka = getenv("HTTP_CONNECTION"); 1296 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1297 keepAliveRequested = true; 1298 1299 auto or = getenv("HTTP_ORIGIN"); 1300 origin = or; 1301 1302 auto rm = getenv("REQUEST_METHOD"); 1303 if(rm.length) 1304 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1305 else 1306 requestMethod = RequestMethod.CommandLine; 1307 1308 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1309 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1310 1311 // FIXME: DOCUMENT_ROOT? 1312 1313 // FIXME: what about PUT? 1314 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1315 version(preserveData) // a hack to make forwarding simpler 1316 immutable(ubyte)[] data; 1317 size_t amountReceived = 0; 1318 auto contentType = getenv("CONTENT_TYPE"); 1319 1320 // FIXME: is this ever not going to be set? I guess it depends 1321 // on if the server de-chunks and buffers... seems like it has potential 1322 // to be slow if they did that. The spec says it is always there though. 1323 // And it has worked reliably for me all year in the live environment, 1324 // but some servers might be different. 1325 auto cls = getenv("CONTENT_LENGTH"); 1326 auto contentLength = to!size_t(cls.length ? cls : "0"); 1327 1328 immutable originalContentLength = contentLength; 1329 if(contentLength) { 1330 if(maxContentLength > 0 && contentLength > maxContentLength) { 1331 setResponseStatus("413 Request entity too large"); 1332 write("You tried to upload a file that is too large."); 1333 close(); 1334 throw new Exception("POST too large"); 1335 } 1336 prepareForIncomingDataChunks(contentType, contentLength); 1337 1338 1339 int processChunk(in ubyte[] chunk) { 1340 if(chunk.length > contentLength) { 1341 handleIncomingDataChunk(chunk[0..contentLength]); 1342 amountReceived += contentLength; 1343 contentLength = 0; 1344 return 1; 1345 } else { 1346 handleIncomingDataChunk(chunk); 1347 contentLength -= chunk.length; 1348 amountReceived += chunk.length; 1349 } 1350 if(contentLength == 0) 1351 return 1; 1352 1353 onRequestBodyDataReceived(amountReceived, originalContentLength); 1354 return 0; 1355 } 1356 1357 1358 if(readdata is null) { 1359 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1360 if(processChunk(chunk)) 1361 break; 1362 } else { 1363 // we have a custom data source.. 1364 auto chunk = readdata(); 1365 while(chunk.length) { 1366 if(processChunk(chunk)) 1367 break; 1368 chunk = readdata(); 1369 } 1370 } 1371 1372 onRequestBodyDataReceived(amountReceived, originalContentLength); 1373 postArray = assumeUnique(pps._post); 1374 filesArray = assumeUnique(pps._files); 1375 files = keepLastOf(filesArray); 1376 post = keepLastOf(postArray); 1377 this.postBody = pps.postBody; 1378 this.requestContentType = contentType; 1379 cleanUpPostDataState(); 1380 } 1381 1382 version(preserveData) 1383 originalPostData = data; 1384 } 1385 // fixme: remote_user script name 1386 1387 1388 this.port = port; 1389 this.referrer = referrer; 1390 this.remoteAddress = remoteAddress; 1391 this.userAgent = userAgent; 1392 this.authorization = authorization; 1393 this.origin = origin; 1394 this.accept = accept; 1395 this.lastEventId = lastEventId; 1396 this.https = https; 1397 this.host = host; 1398 this.requestMethod = requestMethod; 1399 this.requestUri = requestUri; 1400 this.pathInfo = pathInfo; 1401 this.queryString = queryString; 1402 } 1403 1404 /// Cleans up any temporary files. Do not use the object 1405 /// after calling this. 1406 /// 1407 /// NOTE: it is called automatically by GenericMain 1408 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1409 void dispose() { 1410 foreach(file; files) { 1411 if(!file.contentInMemory) 1412 if(std.file.exists(file.contentFilename)) 1413 std.file.remove(file.contentFilename); 1414 } 1415 } 1416 1417 private { 1418 struct PostParserState { 1419 string contentType; 1420 string boundary; 1421 string localBoundary; // the ones used at the end or something lol 1422 bool isMultipart; 1423 bool needsSavedBody; 1424 1425 ulong expectedLength; 1426 ulong contentConsumed; 1427 immutable(ubyte)[] buffer; 1428 1429 // multipart parsing state 1430 int whatDoWeWant; 1431 bool weHaveAPart; 1432 string[] thisOnesHeaders; 1433 immutable(ubyte)[] thisOnesData; 1434 1435 string postBody; 1436 1437 UploadedFile piece; 1438 bool isFile = false; 1439 1440 size_t memoryCommitted; 1441 1442 // do NOT keep mutable references to these anywhere! 1443 // I assume they are unique in the constructor once we're all done getting data. 1444 string[][string] _post; 1445 UploadedFile[][string] _files; 1446 } 1447 1448 PostParserState pps; 1449 } 1450 1451 /// This represents a file the user uploaded via a POST request. 1452 static struct UploadedFile { 1453 /// If you want to create one of these structs for yourself from some data, 1454 /// use this function. 1455 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1456 Cgi.UploadedFile f; 1457 f.filename = name; 1458 f.content = cast(immutable(ubyte)[]) data; 1459 f.contentInMemory = true; 1460 return f; 1461 } 1462 1463 string name; /// The name of the form element. 1464 string filename; /// The filename the user set. 1465 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1466 1467 /** 1468 For small files, cgi.d will buffer the uploaded file in memory, and make it 1469 directly accessible to you through the content member. I find this very convenient 1470 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1471 often want to inspect and modify the file anyway!) 1472 1473 I find the file is very large, it is undesirable to eat that much memory just 1474 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1475 to the constructor so they are accepted, cgi.d will write the content to a temporary 1476 file that you can re-read later. 1477 1478 You can override this behavior by subclassing Cgi and overriding the protected 1479 handlePostChunk method. Note that the object is not initialized when you 1480 write that method - the http headers are available, but the cgi.post method 1481 is not. You may parse the file as it streams in using this method. 1482 1483 1484 Anyway, if the file is small enough to be in memory, contentInMemory will be 1485 set to true, and the content is available in the content member. 1486 1487 If not, contentInMemory will be set to false, and the content saved in a file, 1488 whose name will be available in the contentFilename member. 1489 1490 1491 Tip: if you know you are always dealing with small files, and want the convenience 1492 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1493 a large file comes in, it simply throws an exception (and HTTP error response) 1494 instead of trying to handle it. 1495 1496 The default value of maxContentLength in the constructor is for small files. 1497 */ 1498 bool contentInMemory = true; // the default ought to always be true 1499 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1500 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1501 1502 /// 1503 ulong fileSize() const { 1504 if(contentInMemory) 1505 return content.length; 1506 import std.file; 1507 return std.file.getSize(contentFilename); 1508 1509 } 1510 1511 /// 1512 void writeToFile(string filenameToSaveTo) const { 1513 import std.file; 1514 if(contentInMemory) 1515 std.file.write(filenameToSaveTo, content); 1516 else 1517 std.file.rename(contentFilename, filenameToSaveTo); 1518 } 1519 } 1520 1521 // given a content type and length, decide what we're going to do with the data.. 1522 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1523 pps.expectedLength = contentLength; 1524 1525 auto terminator = contentType.indexOf(";"); 1526 if(terminator == -1) 1527 terminator = contentType.length; 1528 1529 pps.contentType = contentType[0 .. terminator]; 1530 auto b = contentType[terminator .. $]; 1531 if(b.length) { 1532 auto idx = b.indexOf("boundary="); 1533 if(idx != -1) { 1534 pps.boundary = b[idx + "boundary=".length .. $]; 1535 pps.localBoundary = "\r\n--" ~ pps.boundary; 1536 } 1537 } 1538 1539 // while a content type SHOULD be sent according to the RFC, it is 1540 // not required. We're told we SHOULD guess by looking at the content 1541 // but it seems to me that this only happens when it is urlencoded. 1542 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1543 pps.isMultipart = false; 1544 pps.needsSavedBody = false; 1545 } else if(pps.contentType == "multipart/form-data") { 1546 pps.isMultipart = true; 1547 enforce(pps.boundary.length, "no boundary"); 1548 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1549 // save the body so the application can handle it 1550 pps.isMultipart = false; 1551 pps.needsSavedBody = true; 1552 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1553 // save the body so the application can handle it 1554 pps.needsSavedBody = true; 1555 pps.isMultipart = false; 1556 } else { 1557 // the rest is 100% handled by the application. just save the body and send it to them 1558 pps.needsSavedBody = true; 1559 pps.isMultipart = false; 1560 } 1561 } 1562 1563 // handles streaming POST data. If you handle some other content type, you should 1564 // override this. If the data isn't the content type you want, you ought to call 1565 // super.handleIncomingDataChunk so regular forms and files still work. 1566 1567 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1568 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1569 // input anyway, so I'm not going to get too worked up about it right now. 1570 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1571 if(chunk.length == 0) 1572 return; 1573 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1574 // if we're passed big chunks, it might throw unnecessarily. 1575 // just pass it smaller chunks at a time. 1576 if(pps.isMultipart) { 1577 // multipart/form-data 1578 1579 1580 // FIXME: this might want to be factored out and factorized 1581 // need to make sure the stream hooks actually work. 1582 void pieceHasNewContent() { 1583 // we just grew the piece's buffer. Do we have to switch to file backing? 1584 if(pps.piece.contentInMemory) { 1585 if(pps.piece.content.length <= 10 * 1024 * 1024) 1586 // meh, I'm ok with it. 1587 return; 1588 else { 1589 // this is too big. 1590 if(!pps.isFile) 1591 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1592 else { 1593 // a file this large is probably acceptable though... let's use a backing file. 1594 pps.piece.contentInMemory = false; 1595 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1596 1597 int count = 0; 1598 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1599 // odds are this loop will never be entered, but we want it just in case. 1600 while(std.file.exists(pps.piece.contentFilename)) { 1601 count++; 1602 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1603 } 1604 // I hope this creates the file pretty quickly, or the loop might be useless... 1605 // FIXME: maybe I should write some kind of custom transaction here. 1606 std.file.write(pps.piece.contentFilename, pps.piece.content); 1607 1608 pps.piece.content = null; 1609 } 1610 } 1611 } else { 1612 // it's already in a file, so just append it to what we have 1613 if(pps.piece.content.length) { 1614 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1615 std.file.append(pps.piece.contentFilename, pps.piece.content); 1616 pps.piece.content = null; 1617 } 1618 } 1619 } 1620 1621 1622 void commitPart() { 1623 if(!pps.weHaveAPart) 1624 return; 1625 1626 pieceHasNewContent(); // be sure the new content is handled every time 1627 1628 if(pps.isFile) { 1629 // I'm not sure if other environments put files in post or not... 1630 // I used to not do it, but I think I should, since it is there... 1631 pps._post[pps.piece.name] ~= pps.piece.filename; 1632 pps._files[pps.piece.name] ~= pps.piece; 1633 1634 allPostNamesInOrder ~= pps.piece.name; 1635 allPostValuesInOrder ~= pps.piece.filename; 1636 } else { 1637 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1638 1639 allPostNamesInOrder ~= pps.piece.name; 1640 allPostValuesInOrder ~= cast(string) pps.piece.content; 1641 } 1642 1643 /* 1644 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1645 pps.piece.content.length < 1000 1646 ? 1647 to!string(pps.piece.content) 1648 : 1649 "too long"); 1650 */ 1651 1652 // FIXME: the limit here 1653 pps.memoryCommitted += pps.piece.content.length; 1654 1655 pps.weHaveAPart = false; 1656 pps.whatDoWeWant = 1; 1657 pps.thisOnesHeaders = null; 1658 pps.thisOnesData = null; 1659 1660 pps.piece = UploadedFile.init; 1661 pps.isFile = false; 1662 } 1663 1664 void acceptChunk() { 1665 pps.buffer ~= chunk; 1666 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1667 } 1668 1669 immutable(ubyte)[] consume(size_t howMuch) { 1670 pps.contentConsumed += howMuch; 1671 auto ret = pps.buffer[0 .. howMuch]; 1672 pps.buffer = pps.buffer[howMuch .. $]; 1673 return ret; 1674 } 1675 1676 dataConsumptionLoop: do { 1677 switch(pps.whatDoWeWant) { 1678 default: assert(0); 1679 case 0: 1680 acceptChunk(); 1681 // the format begins with two extra leading dashes, then we should be at the boundary 1682 if(pps.buffer.length < 2) 1683 return; 1684 assert(pps.buffer[0] == '-', "no leading dash"); 1685 consume(1); 1686 assert(pps.buffer[0] == '-', "no second leading dash"); 1687 consume(1); 1688 1689 pps.whatDoWeWant = 1; 1690 goto case 1; 1691 /* fallthrough */ 1692 case 1: // looking for headers 1693 // here, we should be lined up right at the boundary, which is followed by a \r\n 1694 1695 // want to keep the buffer under control in case we're under attack 1696 //stderr.writeln("here once"); 1697 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1698 // throw new Exception("wtf is up with the huge mime part headers"); 1699 1700 acceptChunk(); 1701 1702 if(pps.buffer.length < pps.boundary.length) 1703 return; // not enough data, since there should always be a boundary here at least 1704 1705 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1706 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1707 // we *should* be at the end here! 1708 assert(pps.buffer[0] == '-'); 1709 consume(1); 1710 assert(pps.buffer[0] == '-'); 1711 consume(1); 1712 1713 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1714 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1715 "not lined up on boundary " ~ pps.boundary); 1716 consume(pps.boundary.length); 1717 1718 assert(pps.buffer[0] == '-'); 1719 consume(1); 1720 assert(pps.buffer[0] == '-'); 1721 consume(1); 1722 1723 assert(pps.buffer[0] == '\r'); 1724 consume(1); 1725 assert(pps.buffer[0] == '\n'); 1726 consume(1); 1727 1728 assert(pps.buffer.length == 0); 1729 assert(pps.contentConsumed == pps.expectedLength); 1730 break dataConsumptionLoop; // we're done! 1731 } else { 1732 // we're not done yet. We should be lined up on a boundary. 1733 1734 // But, we want to ensure the headers are here before we consume anything! 1735 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1736 if(headerEndLocation == -1) 1737 return; // they *should* all be here, so we can handle them all at once. 1738 1739 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1740 "not lined up on boundary " ~ pps.boundary); 1741 1742 consume(pps.boundary.length); 1743 // the boundary is always followed by a \r\n 1744 assert(pps.buffer[0] == '\r'); 1745 consume(1); 1746 assert(pps.buffer[0] == '\n'); 1747 consume(1); 1748 } 1749 1750 // re-running since by consuming the boundary, we invalidate the old index. 1751 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1752 assert(headerEndLocation >= 0, "no header"); 1753 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1754 1755 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1756 1757 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1758 1759 // now we'll parse the headers 1760 foreach(h; pps.thisOnesHeaders) { 1761 auto p = h.indexOf(":"); 1762 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1763 string hn = h[0..p]; 1764 string hv = h[p+2..$]; 1765 1766 switch(hn.toLower) { 1767 default: assert(0); 1768 case "content-disposition": 1769 auto info = hv.split("; "); 1770 foreach(i; info[1..$]) { // skipping the form-data 1771 auto o = i.split("="); // FIXME 1772 string pn = o[0]; 1773 string pv = o[1][1..$-1]; 1774 1775 if(pn == "name") { 1776 pps.piece.name = pv; 1777 } else if (pn == "filename") { 1778 pps.piece.filename = pv; 1779 pps.isFile = true; 1780 } 1781 } 1782 break; 1783 case "content-type": 1784 pps.piece.contentType = hv; 1785 break; 1786 } 1787 } 1788 1789 pps.whatDoWeWant++; // move to the next step - the data 1790 break; 1791 case 2: 1792 // when we get here, pps.buffer should contain our first chunk of data 1793 1794 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1795 throw new Exception("wtf is up with the huge mime part buffer"); 1796 1797 acceptChunk(); 1798 1799 // so the trick is, we want to process all the data up to the boundary, 1800 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1801 // want to wait for the next chunk. We start by looking for the whole boundary 1802 // in the buffer somewhere. 1803 1804 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1805 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1806 if(boundaryLocation != -1) { 1807 // this is easy - we can see it in it's entirety! 1808 1809 pps.piece.content ~= consume(boundaryLocation); 1810 1811 assert(pps.buffer[0] == '\r'); 1812 consume(1); 1813 assert(pps.buffer[0] == '\n'); 1814 consume(1); 1815 assert(pps.buffer[0] == '-'); 1816 consume(1); 1817 assert(pps.buffer[0] == '-'); 1818 consume(1); 1819 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1820 pps.weHaveAPart = true; 1821 pps.whatDoWeWant = 1; // back to getting headers for the next part 1822 1823 commitPart(); // we're done here 1824 } else { 1825 // we can't see the whole thing, but what if there's a partial boundary? 1826 1827 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1828 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1829 bool potentialBoundaryFound = false; 1830 1831 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1832 // we grow the boundary a bit each time. If we think it looks the 1833 // same, better pull another chunk to be sure it's not the end. 1834 // Starting small because exiting the loop early is desirable, since 1835 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1836 // the best we can do. 1837 if(a > pps.buffer.length) 1838 break; // FIXME: is this right? 1839 assert(a <= pps.buffer.length); 1840 assert(a > 0); 1841 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1842 // ok, there *might* be a boundary here, so let's 1843 // not treat the end as data yet. The rest is good to 1844 // use though, since if there was a boundary there, we'd 1845 // have handled it up above after locationOf. 1846 1847 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1848 consume(pps.buffer.length - a); 1849 pieceHasNewContent(); 1850 potentialBoundaryFound = true; 1851 break boundaryCheck; 1852 } 1853 } 1854 1855 if(!potentialBoundaryFound) { 1856 // we can consume the whole thing 1857 pps.piece.content ~= pps.buffer; 1858 pieceHasNewContent(); 1859 consume(pps.buffer.length); 1860 } else { 1861 // we found a possible boundary, but there was 1862 // insufficient data to be sure. 1863 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1864 1865 return; // wait for the next chunk. 1866 } 1867 } 1868 } 1869 } while(pps.buffer.length); 1870 1871 // btw all boundaries except the first should have a \r\n before them 1872 } else { 1873 // application/x-www-form-urlencoded and application/json 1874 1875 // not using maxContentLength because that might be cranked up to allow 1876 // large file uploads. We can handle them, but a huge post[] isn't any good. 1877 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1878 throw new Exception("wtf is up with such a gigantic form submission????"); 1879 1880 pps.buffer ~= chunk; 1881 1882 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1883 if(pps.buffer.length == pps.expectedLength) { 1884 if(pps.needsSavedBody) 1885 pps.postBody = cast(string) pps.buffer; 1886 else 1887 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1888 version(preserveData) 1889 originalPostData = pps.buffer; 1890 } else { 1891 // just for debugging 1892 } 1893 } 1894 } 1895 1896 protected void cleanUpPostDataState() { 1897 pps = PostParserState.init; 1898 } 1899 1900 /// you can override this function to somehow react 1901 /// to an upload in progress. 1902 /// 1903 /// Take note that parts of the CGI object is not yet 1904 /// initialized! Stuff from HTTP headers, including get[], is usable. 1905 /// But, none of post[] is usable, and you cannot write here. That's 1906 /// why this method is const - mutating the object won't do much anyway. 1907 /// 1908 /// My idea here was so you can output a progress bar or 1909 /// something to a cooperative client (see arsd.rtud for a potential helper) 1910 /// 1911 /// The default is to do nothing. Subclass cgi and use the 1912 /// CustomCgiMain mixin to do something here. 1913 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1914 // This space intentionally left blank. 1915 } 1916 1917 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1918 /// *closeConnection will be set to true if you should close the connection after handling this request 1919 this(BufferedInputRange ir, bool* closeConnection) { 1920 isCalledWithCommandLineArguments = false; 1921 import al = std.algorithm; 1922 1923 immutable(ubyte)[] data; 1924 1925 void rdo(const(ubyte)[] d) { 1926 //import std.stdio; writeln(d); 1927 sendAll(ir.source, d); 1928 } 1929 1930 auto ira = ir.source.remoteAddress(); 1931 auto irLocalAddress = ir.source.localAddress(); 1932 1933 ushort port = 80; 1934 if(auto ia = cast(InternetAddress) irLocalAddress) { 1935 port = ia.port; 1936 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1937 port = ia.port; 1938 } 1939 1940 // that check for UnixAddress is to work around a Phobos bug 1941 // see: https://github.com/dlang/phobos/pull/7383 1942 // but this might be more useful anyway tbh for this case 1943 version(Posix) 1944 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1945 else 1946 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1947 } 1948 1949 /** 1950 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1951 1952 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1953 1954 Params: 1955 inputData = the incoming data, including headers and other raw http data. 1956 When the constructor exits, it will leave this range exactly at the start of 1957 the next request on the connection (if there is one). 1958 1959 address = the IP address of the remote user 1960 _port = the port number of the connection 1961 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1962 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1963 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1964 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1965 closeConnection = if the request asks to close the connection, *closeConnection == true. 1966 */ 1967 this( 1968 BufferedInputRange inputData, 1969 // string[] headers, immutable(ubyte)[] data, 1970 string address, ushort _port, 1971 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1972 bool _https = false, 1973 void delegate(const(ubyte)[]) _rawDataOutput = null, 1974 void delegate() _flush = null, 1975 // this pointer tells if the connection is supposed to be closed after we handle this 1976 bool* closeConnection = null) 1977 { 1978 // these are all set locally so the loop works 1979 // without triggering errors in dmd 2.064 1980 // we go ahead and set them at the end of it to the this version 1981 int port; 1982 string referrer; 1983 string remoteAddress; 1984 string userAgent; 1985 string authorization; 1986 string origin; 1987 string accept; 1988 string lastEventId; 1989 bool https; 1990 string host; 1991 RequestMethod requestMethod; 1992 string requestUri; 1993 string pathInfo; 1994 string queryString; 1995 string scriptName; 1996 string[string] get; 1997 string[][string] getArray; 1998 bool keepAliveRequested; 1999 bool acceptsGzip; 2000 string cookie; 2001 2002 2003 2004 environmentVariables = cast(const) environment.toAA; 2005 2006 idlol = inputData; 2007 2008 isCalledWithCommandLineArguments = false; 2009 2010 https = _https; 2011 port = _port; 2012 2013 rawDataOutput = _rawDataOutput; 2014 flushDelegate = _flush; 2015 nph = true; 2016 2017 remoteAddress = address; 2018 2019 // streaming parser 2020 import al = std.algorithm; 2021 2022 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 2023 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 2024 while(idx == -1) { 2025 inputData.popFront(0); 2026 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 2027 } 2028 2029 assert(idx != -1); 2030 2031 2032 string contentType = ""; 2033 string[string] requestHeadersHere; 2034 2035 size_t contentLength; 2036 2037 bool isChunked; 2038 2039 { 2040 import core.runtime; 2041 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 2042 } 2043 2044 2045 int headerNumber = 0; 2046 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 2047 if(line.length) { 2048 headerNumber++; 2049 auto header = cast(string) line.idup; 2050 if(headerNumber == 1) { 2051 // request line 2052 auto parts = al.splitter(header, " "); 2053 if(parts.front == "PRI") { 2054 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 2055 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 2056 // to bring me benefit) 2057 throw new HttpVersionNotSupportedException(); 2058 } 2059 requestMethod = to!RequestMethod(parts.front); 2060 parts.popFront(); 2061 requestUri = parts.front; 2062 2063 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 2064 scriptName = requestUri[0 .. pathInfoStarts]; 2065 2066 auto question = requestUri.indexOf("?"); 2067 if(question == -1) { 2068 queryString = ""; 2069 // FIXME: double check, this might be wrong since it could be url encoded 2070 pathInfo = requestUri[pathInfoStarts..$]; 2071 } else { 2072 queryString = requestUri[question+1..$]; 2073 pathInfo = requestUri[pathInfoStarts..question]; 2074 } 2075 2076 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2077 getArray = cast(string[][string]) assumeUnique(ugh); 2078 2079 if(header.indexOf("HTTP/1.0") != -1) { 2080 http10 = true; 2081 autoBuffer = true; 2082 if(closeConnection) { 2083 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2084 *closeConnection = true; 2085 } 2086 } 2087 } else { 2088 // other header 2089 auto colon = header.indexOf(":"); 2090 if(colon == -1) 2091 throw new Exception("HTTP headers should have a colon!"); 2092 string name = header[0..colon].toLower; 2093 string value = header[colon+2..$]; // skip the colon and the space 2094 2095 requestHeadersHere[name] = value; 2096 2097 if (name == "accept") { 2098 accept = value; 2099 } 2100 else if (name == "origin") { 2101 origin = value; 2102 } 2103 else if (name == "connection") { 2104 if(value == "close" && closeConnection) 2105 *closeConnection = true; 2106 if(value.asLowerCase().canFind("keep-alive")) { 2107 keepAliveRequested = true; 2108 2109 // on http 1.0, the connection is closed by default, 2110 // but not if they request keep-alive. then we don't close 2111 // anymore - undoing the set above 2112 if(http10 && closeConnection) { 2113 *closeConnection = false; 2114 } 2115 } 2116 } 2117 else if (name == "transfer-encoding") { 2118 if(value == "chunked") 2119 isChunked = true; 2120 } 2121 else if (name == "last-event-id") { 2122 lastEventId = value; 2123 } 2124 else if (name == "authorization") { 2125 authorization = value; 2126 } 2127 else if (name == "content-type") { 2128 contentType = value; 2129 } 2130 else if (name == "content-length") { 2131 contentLength = to!size_t(value); 2132 } 2133 else if (name == "x-forwarded-for") { 2134 remoteAddress = value; 2135 } 2136 else if (name == "x-forwarded-host" || name == "host") { 2137 if(name != "host" || host is null) 2138 host = value; 2139 } 2140 // FIXME: https://tools.ietf.org/html/rfc7239 2141 else if (name == "accept-encoding") { 2142 if(value.indexOf("gzip") != -1) 2143 acceptsGzip = true; 2144 } 2145 else if (name == "user-agent") { 2146 userAgent = value; 2147 } 2148 else if (name == "referer") { 2149 referrer = value; 2150 } 2151 else if (name == "cookie") { 2152 cookie ~= value; 2153 } else if(name == "expect") { 2154 if(value == "100-continue") { 2155 // FIXME we should probably give user code a chance 2156 // to process and reject but that needs to be virtual, 2157 // perhaps part of the CGI redesign. 2158 2159 // FIXME: if size is > max content length it should 2160 // also fail at this point. 2161 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2162 2163 // FIXME: let the user write out 103 early hints too 2164 } 2165 } 2166 // else 2167 // ignore it 2168 2169 } 2170 } 2171 2172 inputData.consume(idx + 4); 2173 // done 2174 2175 requestHeaders = assumeUnique(requestHeadersHere); 2176 2177 ByChunkRange dataByChunk; 2178 2179 // reading Content-Length type data 2180 // We need to read up the data we have, and write it out as a chunk. 2181 if(!isChunked) { 2182 dataByChunk = byChunk(inputData, contentLength); 2183 } else { 2184 // chunked requests happen, but not every day. Since we need to know 2185 // the content length (for now, maybe that should change), we'll buffer 2186 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2187 auto data = dechunk(inputData); 2188 2189 // set the range here 2190 dataByChunk = byChunk(data); 2191 contentLength = data.length; 2192 } 2193 2194 assert(dataByChunk !is null); 2195 2196 if(contentLength) { 2197 prepareForIncomingDataChunks(contentType, contentLength); 2198 foreach(dataChunk; dataByChunk) { 2199 handleIncomingDataChunk(dataChunk); 2200 } 2201 postArray = assumeUnique(pps._post); 2202 filesArray = assumeUnique(pps._files); 2203 files = keepLastOf(filesArray); 2204 post = keepLastOf(postArray); 2205 postBody = pps.postBody; 2206 this.requestContentType = contentType; 2207 2208 cleanUpPostDataState(); 2209 } 2210 2211 this.port = port; 2212 this.referrer = referrer; 2213 this.remoteAddress = remoteAddress; 2214 this.userAgent = userAgent; 2215 this.authorization = authorization; 2216 this.origin = origin; 2217 this.accept = accept; 2218 this.lastEventId = lastEventId; 2219 this.https = https; 2220 this.host = host; 2221 this.requestMethod = requestMethod; 2222 this.requestUri = requestUri; 2223 this.pathInfo = pathInfo; 2224 this.queryString = queryString; 2225 2226 this.scriptName = scriptName; 2227 this.get = keepLastOf(getArray); 2228 this.getArray = cast(immutable) getArray; 2229 this.keepAliveRequested = keepAliveRequested; 2230 this.acceptsGzip = acceptsGzip; 2231 this.cookie = cookie; 2232 2233 cookiesArray = getCookieArray(); 2234 cookies = keepLastOf(cookiesArray); 2235 2236 } 2237 BufferedInputRange idlol; 2238 2239 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2240 string[string] ca; 2241 foreach(k, v; arr) 2242 ca[k] = v[$-1]; 2243 2244 return assumeUnique(ca); 2245 } 2246 2247 // FIXME duplication 2248 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2249 UploadedFile[string] ca; 2250 foreach(k, v; arr) 2251 ca[k] = v[$-1]; 2252 2253 return assumeUnique(ca); 2254 } 2255 2256 2257 private immutable(string[][string]) getCookieArray() { 2258 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2259 return assumeUnique(forTheLoveOfGod); 2260 } 2261 2262 /++ 2263 Very simple method to require a basic auth username and password. 2264 If the http request doesn't include the required credentials, it throws a 2265 HTTP 401 error, and an exception to cancel your handler. Do NOT catch the 2266 `AuthorizationRequiredException` exception thrown by this if you want the 2267 http basic auth prompt to work for the user! 2268 2269 Note: basic auth does not provide great security, especially over unencrypted HTTP; 2270 the user's credentials are sent in plain text on every request. 2271 2272 If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2273 application. Either use Apache's built in methods for basic authentication, or add 2274 something along these lines to your server configuration: 2275 2276 ``` 2277 RewriteEngine On 2278 RewriteCond %{HTTP:Authorization} ^(.*) 2279 RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2280 ``` 2281 2282 To ensure the necessary data is available to cgi.d. 2283 +/ 2284 void requireBasicAuth(string user, string pass, string message = null, string file = __FILE__, size_t line = __LINE__) { 2285 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2286 throw new AuthorizationRequiredException("Basic", message, file, line); 2287 } 2288 } 2289 2290 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2291 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2292 /// Use setResponseExpires and updateResponseExpires for more control 2293 void setCache(bool allowCaching) { 2294 noCache = !allowCaching; 2295 } 2296 2297 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2298 /// who can accept it 2299 bool gzipResponse; 2300 2301 immutable bool acceptsGzip; 2302 immutable bool keepAliveRequested; 2303 2304 /// Set to true if and only if this was initialized with command line arguments 2305 immutable bool isCalledWithCommandLineArguments; 2306 2307 /// This gets a full url for the current request, including port, protocol, host, path, and query 2308 string getCurrentCompleteUri() const { 2309 ushort defaultPort = https ? 443 : 80; 2310 2311 string uri = "http"; 2312 if(https) 2313 uri ~= "s"; 2314 uri ~= "://"; 2315 uri ~= host; 2316 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2317 version(none) 2318 if(!(!port || port == defaultPort)) { 2319 uri ~= ":"; 2320 uri ~= to!string(port); 2321 } 2322 +/ 2323 uri ~= requestUri; 2324 return uri; 2325 } 2326 2327 /// You can override this if your site base url isn't the same as the script name 2328 string logicalScriptName() const { 2329 return scriptName; 2330 } 2331 2332 /++ 2333 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2334 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2335 Note setResponseStatus() must be called *before* you write() any data to the output. 2336 2337 History: 2338 The `int` overload was added on January 11, 2021. 2339 +/ 2340 void setResponseStatus(string status) { 2341 assert(!outputtedResponseData); 2342 responseStatus = status; 2343 } 2344 /// ditto 2345 void setResponseStatus(int statusCode) { 2346 setResponseStatus(getHttpCodeText(statusCode)); 2347 } 2348 private string responseStatus = null; 2349 2350 /// Returns true if it is still possible to output headers 2351 bool canOutputHeaders() { 2352 return !isClosed && !outputtedResponseData; 2353 } 2354 2355 /// Sets the location header, which the browser will redirect the user to automatically. 2356 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2357 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2358 void setResponseLocation(string uri, bool important = true, string status = null) { 2359 if(!important && isCurrentResponseLocationImportant) 2360 return; // important redirects always override unimportant ones 2361 2362 if(uri is null) { 2363 responseStatus = "200 OK"; 2364 responseLocation = null; 2365 isCurrentResponseLocationImportant = important; 2366 return; // this just cancels the redirect 2367 } 2368 2369 assert(!outputtedResponseData); 2370 if(status is null) 2371 responseStatus = "302 Found"; 2372 else 2373 responseStatus = status; 2374 2375 responseLocation = uri.strip; 2376 isCurrentResponseLocationImportant = important; 2377 } 2378 protected string responseLocation = null; 2379 private bool isCurrentResponseLocationImportant = false; 2380 2381 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2382 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2383 /// Note: the when parameter is different than setCookie's expire parameter. 2384 void setResponseExpires(long when, bool isPublic = false) { 2385 responseExpires = when; 2386 setCache(true); // need to enable caching so the date has meaning 2387 2388 responseIsPublic = isPublic; 2389 responseExpiresRelative = false; 2390 } 2391 2392 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2393 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2394 responseExpires = whenFromNow; 2395 setCache(true); // need to enable caching so the date has meaning 2396 2397 responseIsPublic = isPublic; 2398 responseExpiresRelative = true; 2399 } 2400 private long responseExpires = long.min; 2401 private bool responseIsPublic = false; 2402 private bool responseExpiresRelative = false; 2403 2404 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2405 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2406 /// output as a whole is as cacheable as the least cachable part in the chain. 2407 2408 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2409 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2410 void updateResponseExpires(long when, bool isPublic) { 2411 if(responseExpires == long.min) 2412 setResponseExpires(when, isPublic); 2413 else if(when < responseExpires) 2414 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2415 } 2416 2417 /* 2418 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2419 /// Should generally be false if the user is logged in. It assumes private cache only. 2420 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2421 void setPublicCaching(bool allowPublicCaches) { 2422 publicCaching = allowPublicCaches; 2423 } 2424 private bool publicCaching = false; 2425 */ 2426 2427 /++ 2428 History: 2429 Added January 11, 2021 2430 +/ 2431 enum SameSitePolicy { 2432 Lax, 2433 Strict, 2434 None 2435 } 2436 2437 /++ 2438 Sets an HTTP cookie, automatically encoding the data to the correct string. 2439 expiresIn is how many milliseconds in the future the cookie will expire. 2440 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2441 Note setCookie() must be called *before* you write() any data to the output. 2442 2443 History: 2444 Parameter `sameSitePolicy` was added on January 11, 2021. 2445 +/ 2446 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2447 assert(!outputtedResponseData); 2448 string cookie = std.uri.encodeComponent(name) ~ "="; 2449 cookie ~= std.uri.encodeComponent(data); 2450 if(path !is null) 2451 cookie ~= "; path=" ~ path; 2452 // FIXME: should I just be using max-age here? (also in cache below) 2453 if(expiresIn != 0) 2454 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2455 if(domain !is null) 2456 cookie ~= "; domain=" ~ domain; 2457 if(secure == true) 2458 cookie ~= "; Secure"; 2459 if(httpOnly == true ) 2460 cookie ~= "; HttpOnly"; 2461 final switch(sameSitePolicy) { 2462 case SameSitePolicy.Lax: 2463 cookie ~= "; SameSite=Lax"; 2464 break; 2465 case SameSitePolicy.Strict: 2466 cookie ~= "; SameSite=Strict"; 2467 break; 2468 case SameSitePolicy.None: 2469 cookie ~= "; SameSite=None"; 2470 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2471 break; 2472 } 2473 2474 if(auto idx = name in cookieIndexes) { 2475 responseCookies[*idx] = cookie; 2476 } else { 2477 cookieIndexes[name] = responseCookies.length; 2478 responseCookies ~= cookie; 2479 } 2480 } 2481 private string[] responseCookies; 2482 private size_t[string] cookieIndexes; 2483 2484 /// Clears a previously set cookie with the given name, path, and domain. 2485 void clearCookie(string name, string path = null, string domain = null) { 2486 assert(!outputtedResponseData); 2487 setCookie(name, "", 1, path, domain); 2488 } 2489 2490 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2491 void setResponseContentType(string ct) { 2492 assert(!outputtedResponseData); 2493 responseContentType = ct; 2494 } 2495 private string responseContentType = null; 2496 2497 /// Adds a custom header. It should be the name: value, but without any line terminator. 2498 /// For example: header("X-My-Header: Some value"); 2499 /// Note you should use the specialized functions in this object if possible to avoid 2500 /// duplicates in the output. 2501 void header(string h) { 2502 customHeaders ~= h; 2503 } 2504 2505 /++ 2506 I named the original function `header` after PHP, but this pattern more fits 2507 the rest of the Cgi object. 2508 2509 Either name are allowed. 2510 2511 History: 2512 Alias added June 17, 2022. 2513 +/ 2514 alias setResponseHeader = header; 2515 2516 private string[] customHeaders; 2517 private bool websocketMode; 2518 2519 void flushHeaders(const(void)[] t, bool isAll = false) { 2520 StackBuffer buffer = StackBuffer(0); 2521 2522 prepHeaders(t, isAll, &buffer); 2523 2524 if(rawDataOutput !is null) 2525 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2526 else { 2527 stdout.rawWrite(buffer.get()); 2528 } 2529 } 2530 2531 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2532 string terminator = "\n"; 2533 if(rawDataOutput !is null) 2534 terminator = "\r\n"; 2535 2536 if(responseStatus !is null) { 2537 if(nph) { 2538 if(http10) 2539 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2540 else 2541 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2542 } else 2543 buffer.add("Status: ", responseStatus, terminator); 2544 } else if (nph) { 2545 if(http10) 2546 buffer.add("HTTP/1.0 200 OK", terminator); 2547 else 2548 buffer.add("HTTP/1.1 200 OK", terminator); 2549 } 2550 2551 if(websocketMode) 2552 goto websocket; 2553 2554 if(nph) { // we're responsible for setting the date too according to http 1.1 2555 char[29] db = void; 2556 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2557 buffer.add("Date: ", db[], terminator); 2558 } 2559 2560 // FIXME: what if the user wants to set his own content-length? 2561 // The custom header function can do it, so maybe that's best. 2562 // Or we could reuse the isAll param. 2563 if(responseLocation !is null) { 2564 buffer.add("Location: ", responseLocation, terminator); 2565 } 2566 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2567 if(responseExpiresRelative) { 2568 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2569 buffer.add(responseExpires); 2570 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2571 } else { 2572 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2573 char[29] db = void; 2574 printDateToBuffer(cast(DateTime) expires, db[]); 2575 buffer.add("Expires: ", db[], terminator); 2576 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2577 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2578 buffer.add(terminator); 2579 } 2580 } 2581 if(responseCookies !is null && responseCookies.length > 0) { 2582 foreach(c; responseCookies) 2583 buffer.add("Set-Cookie: ", c, terminator); 2584 } 2585 if(noCache) { // we specifically do not want caching (this is actually the default) 2586 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2587 buffer.add("Expires: 0", terminator); 2588 buffer.add("Pragma: no-cache", terminator); 2589 } else { 2590 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2591 buffer.add("Cache-Control: public", terminator); 2592 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2593 } 2594 } 2595 if(responseContentType !is null) { 2596 buffer.add("Content-Type: ", responseContentType, terminator); 2597 } else 2598 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2599 2600 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2601 buffer.add("Content-Encoding: gzip", terminator); 2602 } 2603 2604 2605 if(!isAll) { 2606 if(nph && !http10) { 2607 buffer.add("Transfer-Encoding: chunked", terminator); 2608 responseChunked = true; 2609 } 2610 } else { 2611 buffer.add("Content-Length: "); 2612 buffer.add(t.length); 2613 buffer.add(terminator); 2614 if(nph && keepAliveRequested) { 2615 buffer.add("Connection: Keep-Alive", terminator); 2616 } 2617 } 2618 2619 websocket: 2620 2621 foreach(hd; customHeaders) 2622 buffer.add(hd, terminator); 2623 2624 // FIXME: what about duplicated headers? 2625 2626 // end of header indicator 2627 buffer.add(terminator); 2628 2629 outputtedResponseData = true; 2630 } 2631 2632 /// Writes the data to the output, flushing headers if they have not yet been sent. 2633 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2634 assert(!closed, "Output has already been closed"); 2635 2636 StackBuffer buffer = StackBuffer(0); 2637 2638 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2639 // actually gzip the data here 2640 2641 auto c = new Compress(HeaderFormat.gzip); // want gzip 2642 2643 auto data = c.compress(t); 2644 data ~= c.flush(); 2645 2646 // std.file.write("/tmp/last-item", data); 2647 2648 t = data; 2649 } 2650 2651 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2652 prepHeaders(t, isAll, &buffer); 2653 } 2654 2655 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2656 if (autoBuffer && !isAll) { 2657 outputBuffer ~= cast(ubyte[]) t; 2658 } 2659 if(!autoBuffer || isAll) { 2660 if(rawDataOutput !is null) 2661 if(nph && responseChunked) { 2662 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2663 // we're making the chunk here instead of in a function 2664 // to avoid unneeded gc pressure 2665 buffer.add(toHex(t.length)); 2666 buffer.add("\r\n"); 2667 buffer.add(cast(char[]) t, "\r\n"); 2668 } else { 2669 buffer.add(cast(char[]) t); 2670 } 2671 else 2672 buffer.add(cast(char[]) t); 2673 } 2674 } 2675 2676 if(rawDataOutput !is null) 2677 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2678 else 2679 stdout.rawWrite(buffer.get()); 2680 2681 if(maybeAutoClose && isAll) 2682 close(); // if you say it is all, that means we're definitely done 2683 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2684 } 2685 2686 /++ 2687 Convenience method to set content type to json and write the string as the complete response. 2688 2689 History: 2690 Added January 16, 2020 2691 +/ 2692 void writeJson(string json) { 2693 this.setResponseContentType("application/json"); 2694 this.write(json, true); 2695 } 2696 2697 /// Flushes the pending buffer, leaving the connection open so you can send more. 2698 void flush() { 2699 if(rawDataOutput is null) 2700 stdout.flush(); 2701 else if(flushDelegate !is null) 2702 flushDelegate(); 2703 } 2704 2705 version(autoBuffer) 2706 bool autoBuffer = true; 2707 else 2708 bool autoBuffer = false; 2709 ubyte[] outputBuffer; 2710 2711 /// Flushes the buffers to the network, signifying that you are done. 2712 /// You should always call this explicitly when you are done outputting data. 2713 void close() { 2714 if(closed) 2715 return; // don't double close 2716 2717 if(!outputtedResponseData) 2718 write("", true, false); 2719 2720 // writing auto buffered data 2721 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2722 if(!nph) 2723 stdout.rawWrite(outputBuffer); 2724 else 2725 write(outputBuffer, true, false); // tell it this is everything 2726 } 2727 2728 // closing the last chunk... 2729 if(nph && rawDataOutput !is null && responseChunked) 2730 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2731 2732 if(flushDelegate) 2733 flushDelegate(); 2734 2735 closed = true; 2736 } 2737 2738 // Closes without doing anything, shouldn't be used often 2739 void rawClose() { 2740 closed = true; 2741 } 2742 2743 /++ 2744 Gets a request variable as a specific type, or the default value of it isn't there 2745 or isn't convertible to the request type. 2746 2747 Checks both GET and POST variables, preferring the POST variable, if available. 2748 2749 A nice trick is using the default value to choose the type: 2750 2751 --- 2752 /* 2753 The return value will match the type of the default. 2754 Here, I gave 10 as a default, so the return value will 2755 be an int. 2756 2757 If the user-supplied value cannot be converted to the 2758 requested type, you will get the default value back. 2759 */ 2760 int a = cgi.request("number", 10); 2761 2762 if(cgi.get["number"] == "11") 2763 assert(a == 11); // conversion succeeds 2764 2765 if("number" !in cgi.get) 2766 assert(a == 10); // no value means you can't convert - give the default 2767 2768 if(cgi.get["number"] == "twelve") 2769 assert(a == 10); // conversion from string to int would fail, so we get the default 2770 --- 2771 2772 You can use an enum as an easy whitelist, too: 2773 2774 --- 2775 enum Operations { 2776 add, remove, query 2777 } 2778 2779 auto op = cgi.request("op", Operations.query); 2780 2781 if(cgi.get["op"] == "add") 2782 assert(op == Operations.add); 2783 if(cgi.get["op"] == "remove") 2784 assert(op == Operations.remove); 2785 if(cgi.get["op"] == "query") 2786 assert(op == Operations.query); 2787 2788 if(cgi.get["op"] == "random string") 2789 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2790 --- 2791 +/ 2792 T request(T = string)(in string name, in T def = T.init) const nothrow { 2793 try { 2794 return 2795 (name in post) ? to!T(post[name]) : 2796 (name in get) ? to!T(get[name]) : 2797 def; 2798 } catch(Exception e) { return def; } 2799 } 2800 2801 /// Is the output already closed? 2802 bool isClosed() const { 2803 return closed; 2804 } 2805 2806 private SessionObject commandLineSessionObject; 2807 2808 /++ 2809 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2810 +/ 2811 Session!Data getSessionObject(Data)() { 2812 if(testInProcess !is null) { 2813 // test mode 2814 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2815 if(obj !is null) 2816 return cast(typeof(return)) obj; 2817 else { 2818 auto o = new MockSession!Data(); 2819 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2820 return o; 2821 } 2822 } else { 2823 // FIXME: the changes are not printed out at the end! 2824 if(_commandLineSession !is null) { 2825 if(commandLineSessionObject is null) { 2826 auto clso = new MockSession!Data(); 2827 commandLineSessionObject = clso; 2828 2829 2830 foreach(memberName; __traits(allMembers, Data)) { 2831 if(auto str = memberName in _commandLineSession) 2832 __traits(getMember, clso.store_, memberName) = to!(typeof(__traits(getMember, Data, memberName)))(*str); 2833 } 2834 } 2835 2836 return cast(typeof(return)) commandLineSessionObject; 2837 } 2838 2839 // normal operation 2840 return new BasicDataServerSession!Data(this); 2841 } 2842 } 2843 2844 // if it is in test mode; triggers mock sessions. Used by CgiTester 2845 version(with_breaking_cgi_features) 2846 private CgiTester testInProcess; 2847 2848 /* Hooks for redirecting input and output */ 2849 private void delegate(const(ubyte)[]) rawDataOutput = null; 2850 private void delegate() flushDelegate = null; 2851 2852 /* This info is used when handling a more raw HTTP protocol */ 2853 private bool nph; 2854 private bool http10; 2855 private bool closed; 2856 private bool responseChunked = false; 2857 2858 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2859 immutable(ubyte)[] originalPostData; 2860 2861 /++ 2862 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2863 2864 It is intended to be used for JSON and XML request content types, but also may be used 2865 for other content types your application can handle. But it will NOT be populated 2866 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2867 parsed into the post and postArray members. 2868 2869 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2870 will be discarded to the client with an error. This helps keep this array from being exploded in size 2871 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2872 client in certain build modes.) 2873 2874 History: 2875 Added January 5, 2021 2876 Documented February 21, 2023 (dub v11.0) 2877 +/ 2878 public immutable string postBody; 2879 alias postJson = postBody; // old name 2880 2881 /++ 2882 The content type header of the request. The [postBody] member may hold the actual data (see [postBody] for details). 2883 2884 History: 2885 Added January 26, 2024 (dub v11.4) 2886 +/ 2887 public immutable string requestContentType; 2888 2889 /* Internal state flags */ 2890 private bool outputtedResponseData; 2891 private bool noCache = true; 2892 2893 const(string[string]) environmentVariables; 2894 2895 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2896 partially because it logically is (your code doesn't change what the user requested...) 2897 and partially because I hate how bad programs in PHP change those superglobals to do 2898 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2899 2900 For some of these, you'll want to refer to the http or cgi specs for more details. 2901 */ 2902 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2903 2904 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2905 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2906 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2907 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2908 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2909 immutable(char[]) scriptFileName; /// The physical filename of your script 2910 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2911 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2912 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2913 2914 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2915 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2916 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2917 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2918 2919 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2920 */ 2921 immutable(char[]) referrer; 2922 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2923 2924 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2925 2926 immutable bool https; /// Was the request encrypted via https? 2927 immutable int port; /// On what TCP port number did the server receive the request? 2928 2929 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2930 2931 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2932 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2933 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2934 2935 /// added later 2936 alias query = get; 2937 2938 /** 2939 Represents user uploaded files. 2940 2941 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2942 */ 2943 immutable(UploadedFile[][string]) filesArray; 2944 immutable(UploadedFile[string]) files; 2945 2946 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2947 /// the order of the arrays is the order the data arrives 2948 immutable(string[][string]) getArray; /// like get, but an array of values per name 2949 immutable(string[][string]) postArray; /// ditto for post 2950 immutable(string[][string]) cookiesArray; /// ditto for cookies 2951 2952 private string[string] _commandLineSession; 2953 2954 // convenience function for appending to a uri without extra ? 2955 // matches the name and effect of javascript's location.search property 2956 string search() const { 2957 if(queryString.length) 2958 return "?" ~ queryString; 2959 return ""; 2960 } 2961 2962 // FIXME: what about multiple files with the same name? 2963 private: 2964 //RequestMethod _requestMethod; 2965 } 2966 2967 /// use this for testing or other isolated things when you want it to be no-ops 2968 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2969 // we want to ignore, not use stdout 2970 if(outputSink is null) 2971 outputSink = delegate void(const(ubyte)[]) { }; 2972 2973 string[string] env; 2974 env["REQUEST_METHOD"] = to!string(method); 2975 env["CONTENT_LENGTH"] = to!string(data.length); 2976 2977 auto cgi = new Cgi( 2978 0, 2979 env, 2980 { return data; }, 2981 outputSink, 2982 null); 2983 2984 return cgi; 2985 } 2986 2987 /++ 2988 A helper test class for request handler unittests. 2989 +/ 2990 version(with_breaking_cgi_features) 2991 class CgiTester { 2992 private { 2993 SessionObject[TypeInfo] mockSessions; 2994 SessionObject getSessionOverride(TypeInfo ti) { 2995 if(auto o = ti in mockSessions) 2996 return *o; 2997 else 2998 return null; 2999 } 3000 void setSessionOverride(TypeInfo ti, SessionObject so) { 3001 mockSessions[ti] = so; 3002 } 3003 } 3004 3005 /++ 3006 Gets (and creates if necessary) a mock session object for this test. Note 3007 it will be the same one used for any test operations through this CgiTester instance. 3008 +/ 3009 Session!Data getSessionObject(Data)() { 3010 auto obj = getSessionOverride(typeid(typeof(return))); 3011 if(obj !is null) 3012 return cast(typeof(return)) obj; 3013 else { 3014 auto o = new MockSession!Data(); 3015 setSessionOverride(typeid(typeof(return)), o); 3016 return o; 3017 } 3018 } 3019 3020 /++ 3021 Pass a reference to your request handler when creating the tester. 3022 +/ 3023 this(void function(Cgi) requestHandler) { 3024 this.requestHandler = requestHandler; 3025 } 3026 3027 /++ 3028 You can check response information with these methods after you call the request handler. 3029 +/ 3030 struct Response { 3031 int code; 3032 string[string] headers; 3033 string responseText; 3034 ubyte[] responseBody; 3035 } 3036 3037 /++ 3038 Executes a test request on your request handler, and returns the response. 3039 3040 Params: 3041 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 3042 args = additional arguments. Same format as cgi's command line handler. 3043 +/ 3044 Response GET(string url, string[] args = null) { 3045 return executeTest("GET", url, args); 3046 } 3047 /// ditto 3048 Response POST(string url, string[] args = null) { 3049 return executeTest("POST", url, args); 3050 } 3051 3052 /// ditto 3053 Response executeTest(string method, string url, string[] args) { 3054 ubyte[] outputtedRawData; 3055 void outputSink(const(ubyte)[] data) { 3056 outputtedRawData ~= data; 3057 } 3058 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 3059 cgi.testInProcess = this; 3060 scope(exit) cgi.dispose(); 3061 3062 requestHandler(cgi); 3063 3064 cgi.close(); 3065 3066 Response response; 3067 3068 if(outputtedRawData.length) { 3069 enum LINE = "\r\n"; 3070 3071 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 3072 assert(idx != -1, to!string(outputtedRawData)); 3073 auto headers = cast(string) outputtedRawData[0 .. idx]; 3074 response.code = 200; 3075 while(headers.length) { 3076 auto i = headers.locationOf(LINE); 3077 if(i == -1) i = cast(int) headers.length; 3078 3079 auto header = headers[0 .. i]; 3080 3081 auto c = header.locationOf(":"); 3082 if(c != -1) { 3083 auto name = header[0 .. c]; 3084 auto value = header[c + 2 ..$]; 3085 3086 if(name == "Status") 3087 response.code = value[0 .. value.locationOf(" ")].to!int; 3088 3089 response.headers[name] = value; 3090 } else { 3091 assert(0); 3092 } 3093 3094 if(i != headers.length) 3095 i += 2; 3096 headers = headers[i .. $]; 3097 } 3098 response.responseBody = outputtedRawData[idx + 4 .. $]; 3099 response.responseText = cast(string) response.responseBody; 3100 } 3101 3102 return response; 3103 } 3104 3105 private void function(Cgi) requestHandler; 3106 } 3107 3108 3109 // should this be a separate module? Probably, but that's a hassle. 3110 3111 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3112 string makeDataUrl(string mimeType, in void[] data) { 3113 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3114 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3115 } 3116 3117 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3118 /// Represents a url that can be broken down or built up through properties 3119 struct Uri { 3120 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3121 3122 // scheme//userinfo@host:port/path?query#fragment 3123 3124 string scheme; /// e.g. "http" in "http://example.com/" 3125 string userinfo; /// the username (and possibly a password) in the uri 3126 string host; /// the domain name 3127 int port; /// port number, if given. Will be zero if a port was not explicitly given 3128 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3129 string query; /// the stuff after the ? in a uri 3130 string fragment; /// the stuff after the # in a uri. 3131 3132 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3133 // the decode ones need to keep different names anyway because we can't overload on return values... 3134 static string encode(string s) { return std.uri.encodeComponent(s); } 3135 static string encode(string[string] s) { return encodeVariables(s); } 3136 static string encode(string[][string] s) { return encodeVariables(s); } 3137 3138 /// Breaks down a uri string to its components 3139 this(string uri) { 3140 reparse(uri); 3141 } 3142 3143 private void reparse(string uri) { 3144 // from RFC 3986 3145 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3146 // it was a nice experiment but just not worth it. 3147 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3148 /* 3149 Captures: 3150 0 = whole url 3151 1 = scheme, with : 3152 2 = scheme, no : 3153 3 = authority, with // 3154 4 = authority, no // 3155 5 = path 3156 6 = query string, with ? 3157 7 = query string, no ? 3158 8 = anchor, with # 3159 9 = anchor, no # 3160 */ 3161 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3162 // instead, I will DIY and cut that down to 0.6s on the same computer. 3163 /* 3164 3165 Note that authority is 3166 user:password@domain:port 3167 where the user:password@ part is optional, and the :port is optional. 3168 3169 Regex translation: 3170 3171 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3172 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3173 Path cannot have any ? or # in it. It is optional. 3174 Query must start with ? and must not have # in it. It is optional. 3175 Anchor must start with # and can have anything else in it to end of string. It is optional. 3176 */ 3177 3178 this = Uri.init; // reset all state 3179 3180 // empty uri = nothing special 3181 if(uri.length == 0) { 3182 return; 3183 } 3184 3185 size_t idx; 3186 3187 scheme_loop: foreach(char c; uri[idx .. $]) { 3188 switch(c) { 3189 case ':': 3190 case '/': 3191 case '?': 3192 case '#': 3193 break scheme_loop; 3194 default: 3195 } 3196 idx++; 3197 } 3198 3199 if(idx == 0 && uri[idx] == ':') { 3200 // this is actually a path! we skip way ahead 3201 goto path_loop; 3202 } 3203 3204 if(idx == uri.length) { 3205 // the whole thing is a path, apparently 3206 path = uri; 3207 return; 3208 } 3209 3210 if(idx > 0 && uri[idx] == ':') { 3211 scheme = uri[0 .. idx]; 3212 idx++; 3213 } else { 3214 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3215 idx = 0; 3216 } 3217 3218 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3219 // we have an authority.... 3220 idx += 2; 3221 3222 auto authority_start = idx; 3223 authority_loop: foreach(char c; uri[idx .. $]) { 3224 switch(c) { 3225 case '/': 3226 case '?': 3227 case '#': 3228 break authority_loop; 3229 default: 3230 } 3231 idx++; 3232 } 3233 3234 auto authority = uri[authority_start .. idx]; 3235 3236 auto idx2 = authority.indexOf("@"); 3237 if(idx2 != -1) { 3238 userinfo = authority[0 .. idx2]; 3239 authority = authority[idx2 + 1 .. $]; 3240 } 3241 3242 if(authority.length && authority[0] == '[') { 3243 // ipv6 address special casing 3244 idx2 = authority.indexOf(']'); 3245 if(idx2 != -1) { 3246 auto end = authority[idx2 + 1 .. $]; 3247 if(end.length && end[0] == ':') 3248 idx2 = idx2 + 1; 3249 else 3250 idx2 = -1; 3251 } 3252 } else { 3253 idx2 = authority.indexOf(":"); 3254 } 3255 3256 if(idx2 == -1) { 3257 port = 0; // 0 means not specified; we should use the default for the scheme 3258 host = authority; 3259 } else { 3260 host = authority[0 .. idx2]; 3261 if(idx2 + 1 < authority.length) 3262 port = to!int(authority[idx2 + 1 .. $]); 3263 else 3264 port = 0; 3265 } 3266 } 3267 3268 path_loop: 3269 auto path_start = idx; 3270 3271 foreach(char c; uri[idx .. $]) { 3272 if(c == '?' || c == '#') 3273 break; 3274 idx++; 3275 } 3276 3277 path = uri[path_start .. idx]; 3278 3279 if(idx == uri.length) 3280 return; // nothing more to examine... 3281 3282 if(uri[idx] == '?') { 3283 idx++; 3284 auto query_start = idx; 3285 foreach(char c; uri[idx .. $]) { 3286 if(c == '#') 3287 break; 3288 idx++; 3289 } 3290 query = uri[query_start .. idx]; 3291 } 3292 3293 if(idx < uri.length && uri[idx] == '#') { 3294 idx++; 3295 fragment = uri[idx .. $]; 3296 } 3297 3298 // uriInvalidated = false; 3299 } 3300 3301 private string rebuildUri() const { 3302 string ret; 3303 if(scheme.length) 3304 ret ~= scheme ~ ":"; 3305 if(userinfo.length || host.length) 3306 ret ~= "//"; 3307 if(userinfo.length) 3308 ret ~= userinfo ~ "@"; 3309 if(host.length) 3310 ret ~= host; 3311 if(port) 3312 ret ~= ":" ~ to!string(port); 3313 3314 ret ~= path; 3315 3316 if(query.length) 3317 ret ~= "?" ~ query; 3318 3319 if(fragment.length) 3320 ret ~= "#" ~ fragment; 3321 3322 // uri = ret; 3323 // uriInvalidated = false; 3324 return ret; 3325 } 3326 3327 /// Converts the broken down parts back into a complete string 3328 string toString() const { 3329 // if(uriInvalidated) 3330 return rebuildUri(); 3331 } 3332 3333 /// Returns a new absolute Uri given a base. It treats this one as 3334 /// relative where possible, but absolute if not. (If protocol, domain, or 3335 /// other info is not set, the new one inherits it from the base.) 3336 /// 3337 /// Browsers use a function like this to figure out links in html. 3338 Uri basedOn(in Uri baseUrl) const { 3339 Uri n = this; // copies 3340 if(n.scheme == "data") 3341 return n; 3342 // n.uriInvalidated = true; // make sure we regenerate... 3343 3344 // userinfo is not inherited... is this wrong? 3345 3346 // if anything is given in the existing url, we don't use the base anymore. 3347 if(n.scheme.empty) { 3348 n.scheme = baseUrl.scheme; 3349 if(n.host.empty) { 3350 n.host = baseUrl.host; 3351 if(n.port == 0) { 3352 n.port = baseUrl.port; 3353 if(n.path.length > 0 && n.path[0] != '/') { 3354 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3355 if(b.length == 0) 3356 b = "/"; 3357 n.path = b ~ n.path; 3358 } else if(n.path.length == 0) { 3359 n.path = baseUrl.path; 3360 } 3361 } 3362 } 3363 } 3364 3365 n.removeDots(); 3366 3367 return n; 3368 } 3369 3370 void removeDots() { 3371 auto parts = this.path.split("/"); 3372 string[] toKeep; 3373 foreach(part; parts) { 3374 if(part == ".") { 3375 continue; 3376 } else if(part == "..") { 3377 //if(toKeep.length > 1) 3378 toKeep = toKeep[0 .. $-1]; 3379 //else 3380 //toKeep = [""]; 3381 continue; 3382 } else { 3383 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3384 //continue; // skip a `//` situation 3385 toKeep ~= part; 3386 } 3387 } 3388 3389 auto path = toKeep.join("/"); 3390 if(path.length && path[0] != '/') 3391 path = "/" ~ path; 3392 3393 this.path = path; 3394 } 3395 3396 unittest { 3397 auto uri = Uri("test.html"); 3398 assert(uri.path == "test.html"); 3399 uri = Uri("path/1/lol"); 3400 assert(uri.path == "path/1/lol"); 3401 uri = Uri("http://me@example.com"); 3402 assert(uri.scheme == "http"); 3403 assert(uri.userinfo == "me"); 3404 assert(uri.host == "example.com"); 3405 uri = Uri("http://example.com/#a"); 3406 assert(uri.scheme == "http"); 3407 assert(uri.host == "example.com"); 3408 assert(uri.fragment == "a"); 3409 uri = Uri("#foo"); 3410 assert(uri.fragment == "foo"); 3411 uri = Uri("?lol"); 3412 assert(uri.query == "lol"); 3413 uri = Uri("#foo?lol"); 3414 assert(uri.fragment == "foo?lol"); 3415 uri = Uri("?lol#foo"); 3416 assert(uri.fragment == "foo"); 3417 assert(uri.query == "lol"); 3418 3419 uri = Uri("http://127.0.0.1/"); 3420 assert(uri.host == "127.0.0.1"); 3421 assert(uri.port == 0); 3422 3423 uri = Uri("http://127.0.0.1:123/"); 3424 assert(uri.host == "127.0.0.1"); 3425 assert(uri.port == 123); 3426 3427 uri = Uri("http://[ff:ff::0]/"); 3428 assert(uri.host == "[ff:ff::0]"); 3429 3430 uri = Uri("http://[ff:ff::0]:123/"); 3431 assert(uri.host == "[ff:ff::0]"); 3432 assert(uri.port == 123); 3433 } 3434 3435 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3436 // the possibilities. 3437 unittest { 3438 auto url = Uri("cool.html"); // checking relative links 3439 3440 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3441 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3442 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3443 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3444 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3445 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3446 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3447 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3448 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3449 3450 url = Uri("/something/cool.html"); // same server, different path 3451 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3452 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3453 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3454 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3455 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3456 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3457 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3458 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3459 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3460 3461 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3462 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3463 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3464 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3465 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3466 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3467 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3468 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3469 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3470 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3471 3472 url = Uri("/test/bar"); 3473 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3474 assert(Uri("../").basedOn(url) == "/"); 3475 3476 url = Uri("http://example.com/"); 3477 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3478 3479 //auto uriBefore = url; 3480 url = Uri("#anchor"); // everything should remain the same except the anchor 3481 //uriBefore.anchor = "anchor"); 3482 //assert(url == uriBefore); 3483 3484 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3485 3486 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3487 3488 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3489 3490 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3491 3492 // FIXME: add something for port too 3493 } 3494 3495 // these are like javascript's location.search and location.hash 3496 string search() const { 3497 return query.length ? ("?" ~ query) : ""; 3498 } 3499 string hash() const { 3500 return fragment.length ? ("#" ~ fragment) : ""; 3501 } 3502 } 3503 3504 3505 /* 3506 for session, see web.d 3507 */ 3508 3509 /// breaks down a url encoded string 3510 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3511 auto vars = data.split(separator); 3512 string[][string] _get; 3513 foreach(var; vars) { 3514 auto equal = var.indexOf("="); 3515 string name; 3516 string value; 3517 if(equal == -1) { 3518 name = decodeComponent(var); 3519 value = ""; 3520 } else { 3521 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3522 // stupid + -> space conversion. 3523 name = decodeComponent(var[0..equal].replace("+", " ")); 3524 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3525 } 3526 3527 _get[name] ~= value; 3528 if(namesInOrder) 3529 (*namesInOrder) ~= name; 3530 if(valuesInOrder) 3531 (*valuesInOrder) ~= value; 3532 } 3533 return _get; 3534 } 3535 3536 /// breaks down a url encoded string, but only returns the last value of any array 3537 string[string] decodeVariablesSingle(string data) { 3538 string[string] va; 3539 auto varArray = decodeVariables(data); 3540 foreach(k, v; varArray) 3541 va[k] = v[$-1]; 3542 3543 return va; 3544 } 3545 3546 /// url encodes the whole string 3547 string encodeVariables(in string[string] data) { 3548 string ret; 3549 3550 bool outputted = false; 3551 foreach(k, v; data) { 3552 if(outputted) 3553 ret ~= "&"; 3554 else 3555 outputted = true; 3556 3557 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3558 } 3559 3560 return ret; 3561 } 3562 3563 /// url encodes a whole string 3564 string encodeVariables(in string[][string] data) { 3565 string ret; 3566 3567 bool outputted = false; 3568 foreach(k, arr; data) { 3569 foreach(v; arr) { 3570 if(outputted) 3571 ret ~= "&"; 3572 else 3573 outputted = true; 3574 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3575 } 3576 } 3577 3578 return ret; 3579 } 3580 3581 /// Encodes all but the explicitly unreserved characters per rfc 3986 3582 /// Alphanumeric and -_.~ are the only ones left unencoded 3583 /// name is borrowed from php 3584 string rawurlencode(in char[] data) { 3585 string ret; 3586 ret.reserve(data.length * 2); 3587 foreach(char c; data) { 3588 if( 3589 (c >= 'a' && c <= 'z') || 3590 (c >= 'A' && c <= 'Z') || 3591 (c >= '0' && c <= '9') || 3592 c == '-' || c == '_' || c == '.' || c == '~') 3593 { 3594 ret ~= c; 3595 } else { 3596 ret ~= '%'; 3597 // since we iterate on char, this should give us the octets of the full utf8 string 3598 ret ~= toHexUpper(c); 3599 } 3600 } 3601 3602 return ret; 3603 } 3604 3605 3606 // http helper functions 3607 3608 // for chunked responses (which embedded http does whenever possible) 3609 version(none) // this is moved up above to avoid making a copy of the data 3610 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3611 const(ubyte)[] ret; 3612 3613 ret = cast(const(ubyte)[]) toHex(data.length); 3614 ret ~= cast(const(ubyte)[]) "\r\n"; 3615 ret ~= data; 3616 ret ~= cast(const(ubyte)[]) "\r\n"; 3617 3618 return ret; 3619 } 3620 3621 string toHex(long num) { 3622 string ret; 3623 while(num) { 3624 int v = num % 16; 3625 num /= 16; 3626 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3627 ret ~= d; 3628 } 3629 3630 return to!string(array(ret.retro)); 3631 } 3632 3633 string toHexUpper(long num) { 3634 string ret; 3635 while(num) { 3636 int v = num % 16; 3637 num /= 16; 3638 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3639 ret ~= d; 3640 } 3641 3642 if(ret.length == 1) 3643 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3644 3645 return to!string(array(ret.retro)); 3646 } 3647 3648 3649 // the generic mixins 3650 3651 /++ 3652 Use this instead of writing your own main 3653 3654 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3655 +/ 3656 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3657 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3658 } 3659 3660 /++ 3661 Boilerplate mixin for a main function that uses the [dispatcher] function. 3662 3663 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3664 3665 History: 3666 Added July 9, 2021 3667 +/ 3668 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3669 /// forwards to [CustomCgiDispatcherMain] with default args 3670 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3671 } 3672 3673 /// ditto 3674 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3675 class GenericPresenter : WebPresenter!GenericPresenter {} 3676 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3677 } 3678 3679 /++ 3680 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3681 3682 History: 3683 Added May 13, 2023 (dub v11.0) 3684 +/ 3685 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3686 /++ 3687 Handler to the generated presenter you can use from your objects, etc. 3688 +/ 3689 Presenter activePresenter; 3690 3691 /++ 3692 Request handler that creates the presenter then forwards to the [dispatcher] function. 3693 Renders 404 if the dispatcher did not handle the request. 3694 3695 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3696 +/ 3697 void handler(Cgi cgi) { 3698 auto presenter = new Presenter; 3699 activePresenter = presenter; 3700 scope(exit) activePresenter = null; 3701 3702 if(cgi.pathInfo.length == 0) { 3703 cgi.setResponseLocation(cgi.scriptName ~ "/"); 3704 return; 3705 } 3706 3707 if(cgi.dispatcher!DispatcherArgs(presenter)) 3708 return; 3709 3710 switch(cgi.pathInfo) { 3711 case "/style.css": 3712 cgi.setCache(true); 3713 cgi.setResponseContentType("text/css"); 3714 cgi.write(presenter.style(), true); 3715 break; 3716 case "/script.js": 3717 cgi.setCache(true); 3718 cgi.setResponseContentType("application/javascript"); 3719 cgi.write(presenter.script(), true); 3720 break; 3721 default: 3722 presenter.renderBasicError(cgi, 404); 3723 } 3724 } 3725 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3726 } 3727 3728 /// ditto 3729 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3730 class GenericPresenter : WebPresenter!GenericPresenter {} 3731 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3732 3733 } 3734 3735 private string simpleHtmlEncode(string s) { 3736 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3737 } 3738 3739 string messageFromException(Throwable t) { 3740 string message; 3741 if(t !is null) { 3742 debug message = t.toString(); 3743 else message = "An unexpected error has occurred."; 3744 } else { 3745 message = "Unknown error"; 3746 } 3747 return message; 3748 } 3749 3750 string plainHttpError(bool isCgi, string type, Throwable t) { 3751 auto message = messageFromException(t); 3752 message = simpleHtmlEncode(message); 3753 3754 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3755 isCgi ? "Status:" : "HTTP/1.1", 3756 type, message.length, message); 3757 } 3758 3759 // returns true if we were able to recover reasonably 3760 bool handleException(Cgi cgi, Throwable t) { 3761 if(cgi.isClosed) { 3762 // if the channel has been explicitly closed, we can't handle it here 3763 return true; 3764 } 3765 3766 if(cgi.outputtedResponseData) { 3767 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3768 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3769 } else { 3770 // no headers are sent, we can send a full blown error and recover 3771 cgi.setCache(false); 3772 cgi.setResponseContentType("text/html"); 3773 cgi.setResponseLocation(null); // cancel the redirect 3774 cgi.setResponseStatus("500 Internal Server Error"); 3775 cgi.write(simpleHtmlEncode(messageFromException(t))); 3776 cgi.close(); 3777 return true; 3778 } 3779 } 3780 3781 bool isCgiRequestMethod(string s) { 3782 s = s.toUpper(); 3783 if(s == "COMMANDLINE") 3784 return true; 3785 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3786 if(s == member) 3787 return true; 3788 return false; 3789 } 3790 3791 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3792 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3793 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3794 void main(string[] args) { 3795 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3796 } 3797 } 3798 3799 version(embedded_httpd_processes) 3800 __gshared int processPoolSize = 8; 3801 3802 // Returns true if run. You should exit the program after that. 3803 bool tryAddonServers(string[] args) { 3804 if(args.length > 1) { 3805 // run the special separate processes if needed 3806 switch(args[1]) { 3807 case "--websocket-server": 3808 version(with_addon_servers) 3809 websocketServers[args[2]](args[3 .. $]); 3810 else 3811 printf("Add-on servers not compiled in.\n"); 3812 return true; 3813 case "--websocket-servers": 3814 import core.demangle; 3815 version(with_addon_servers_connections) 3816 foreach(k, v; websocketServers) 3817 writeln(k, "\t", demangle(k)); 3818 return true; 3819 case "--session-server": 3820 version(with_addon_servers) 3821 runSessionServer(); 3822 else 3823 printf("Add-on servers not compiled in.\n"); 3824 return true; 3825 case "--event-server": 3826 version(with_addon_servers) 3827 runEventServer(); 3828 else 3829 printf("Add-on servers not compiled in.\n"); 3830 return true; 3831 case "--timer-server": 3832 version(with_addon_servers) 3833 runTimerServer(); 3834 else 3835 printf("Add-on servers not compiled in.\n"); 3836 return true; 3837 case "--timed-jobs": 3838 import core.demangle; 3839 version(with_addon_servers_connections) 3840 foreach(k, v; scheduledJobHandlers) 3841 writeln(k, "\t", demangle(k)); 3842 return true; 3843 case "--timed-job": 3844 scheduledJobHandlers[args[2]](args[3 .. $]); 3845 return true; 3846 default: 3847 // intentionally blank - do nothing and carry on to run normally 3848 } 3849 } 3850 return false; 3851 } 3852 3853 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3854 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3855 // we support command line thing for easy testing everywhere 3856 // it needs to be called ./app method uri [other args...] 3857 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3858 Cgi cgi = new CustomCgi(args); 3859 scope(exit) cgi.dispose(); 3860 try { 3861 fun(cgi); 3862 cgi.close(); 3863 } catch(AuthorizationRequiredException are) { 3864 cgi.setResponseStatus("401 Authorization Required"); 3865 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 3866 cgi.close(); 3867 } 3868 writeln(); // just to put a blank line before the prompt cuz it annoys me 3869 // FIXME: put in some footers to show what changes happened in the session 3870 // could make the MockSession be some kind of ReflectableSessionObject or something 3871 return true; 3872 } 3873 return false; 3874 } 3875 3876 /++ 3877 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3878 3879 As of version 11 (released August 2023), you can also make things like this: 3880 3881 --- 3882 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3883 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3884 3885 // can also: 3886 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3887 3888 // NOT IMPLEMENTED YET 3889 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3890 3891 foreach(listenSpec; server.listenSpecs) { 3892 // you can check what it actually bound to here and see your assigned ports 3893 } 3894 3895 // NOT IMPLEMENTED YET 3896 // server.start!handler(); // starts and runs in the arsd.core event loop 3897 3898 server.serve!handler(); // blocks the thread until the server exits 3899 --- 3900 3901 History: 3902 Added Sept 26, 2020 (release version 8.5). 3903 3904 The `listenSpec` member was added July 31, 2023. 3905 +/ 3906 struct RequestServer { 3907 /++ 3908 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3909 +/ 3910 string listeningHost = defaultListeningHost(); 3911 /// ditto 3912 ushort listeningPort = defaultListeningPort(); 3913 3914 static struct ListenSpec { 3915 enum Protocol { 3916 http, 3917 https, 3918 scgi 3919 } 3920 Protocol protocol; 3921 3922 enum AddressType { 3923 ip, 3924 unix, 3925 abstract_ 3926 } 3927 AddressType addressType; 3928 3929 string address; 3930 ushort port; 3931 } 3932 3933 /++ 3934 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3935 3936 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3937 3938 `http://localhost:8080` 3939 3940 `http://unix:filename/here` 3941 3942 `scgi://abstract:/name/here` 3943 3944 `http://[::1]:4444` 3945 3946 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3947 3948 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3949 3950 `localhost:8080` serves the default protocol. 3951 3952 `8080` or `:8080` assumes default protocol on localhost. 3953 3954 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3955 3956 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3957 3958 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3959 3960 $(PITFALL 3961 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3962 ) 3963 3964 Bugs: 3965 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3966 3967 History: 3968 Added July 31, 2023 (dub v11.0) 3969 +/ 3970 string[] listenSpec; 3971 3972 /++ 3973 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3974 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3975 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3976 [stop] may not work as well. 3977 3978 History: 3979 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3980 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3981 compatibility. 3982 +/ 3983 bool useFork = cgi_use_fork_default; 3984 3985 /++ 3986 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3987 default based on the number of cpus modified by the server mode. 3988 3989 History: 3990 Added August 12, 2022 (dub v10.9) 3991 +/ 3992 int numberOfThreads = 0; 3993 3994 /++ 3995 Creates a server configured to listen to multiple URLs. 3996 3997 History: 3998 Added July 31, 2023 (dub v11.0) 3999 +/ 4000 this(string[] listenTo) { 4001 this.listenSpec = listenTo; 4002 } 4003 4004 /// Creates a server object configured to listen on a single host and port. 4005 this(string defaultHost, ushort defaultPort) { 4006 this.listeningHost = defaultHost; 4007 this.listeningPort = defaultPort; 4008 } 4009 4010 /// ditto 4011 this(ushort defaultPort) { 4012 listeningPort = defaultPort; 4013 } 4014 4015 /++ 4016 Reads the command line arguments into the values here. 4017 4018 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 4019 4020 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 4021 +/ 4022 void configureFromCommandLine(string[] args) { 4023 bool portOrHostFound = false; 4024 4025 bool foundPort = false; 4026 bool foundHost = false; 4027 bool foundUid = false; 4028 bool foundGid = false; 4029 bool foundListen = false; 4030 foreach(arg; args) { 4031 if(foundPort) { 4032 listeningPort = to!ushort(arg); 4033 portOrHostFound = true; 4034 foundPort = false; 4035 continue; 4036 } 4037 if(foundHost) { 4038 listeningHost = arg; 4039 portOrHostFound = true; 4040 foundHost = false; 4041 continue; 4042 } 4043 if(foundUid) { 4044 privilegesDropToUid = to!uid_t(arg); 4045 foundUid = false; 4046 continue; 4047 } 4048 if(foundGid) { 4049 privilegesDropToGid = to!gid_t(arg); 4050 foundGid = false; 4051 continue; 4052 } 4053 if(foundListen) { 4054 this.listenSpec ~= arg; 4055 foundListen = false; 4056 continue; 4057 } 4058 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 4059 foundHost = true; 4060 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 4061 foundPort = true; 4062 else if(arg == "--uid") 4063 foundUid = true; 4064 else if(arg == "--gid") 4065 foundGid = true; 4066 else if(arg == "--listen") 4067 foundListen = true; 4068 } 4069 4070 if(portOrHostFound && listenSpec.length) { 4071 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 4072 } 4073 } 4074 4075 version(Windows) { 4076 private alias uid_t = int; 4077 private alias gid_t = int; 4078 } 4079 4080 /// user (uid) to drop privileges to 4081 /// 0 … do nothing 4082 uid_t privilegesDropToUid = 0; 4083 /// group (gid) to drop privileges to 4084 /// 0 … do nothing 4085 gid_t privilegesDropToGid = 0; 4086 4087 private void dropPrivileges() { 4088 version(Posix) { 4089 import core.sys.posix.unistd; 4090 4091 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 4092 throw new Exception("Dropping privileges via setgid() failed."); 4093 4094 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 4095 throw new Exception("Dropping privileges via setuid() failed."); 4096 } 4097 else { 4098 // FIXME: Windows? 4099 //pragma(msg, "Dropping privileges is not implemented for this platform"); 4100 } 4101 4102 // done, set zero 4103 privilegesDropToGid = 0; 4104 privilegesDropToUid = 0; 4105 } 4106 4107 /++ 4108 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 4109 4110 History: 4111 Added Oct 10, 2020. 4112 Example: 4113 4114 --- 4115 import arsd.cgi; 4116 void main() { 4117 RequestServer server = RequestServer("127.0.0.1", 6789); 4118 string oauthCode; 4119 string oauthScope; 4120 server.serveHttpOnce!((cgi) { 4121 oauthCode = cgi.request("code"); 4122 oauthScope = cgi.request("scope"); 4123 cgi.write("Thank you, please return to the application."); 4124 }); 4125 // use the code and scope given 4126 } 4127 --- 4128 +/ 4129 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4130 import std.socket; 4131 4132 bool tcp; 4133 void delegate() cleanup; 4134 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4135 auto connection = socket.accept(); 4136 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4137 4138 if(cleanup) 4139 cleanup(); 4140 } 4141 4142 /++ 4143 Starts serving requests according to the current configuration. 4144 +/ 4145 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4146 version(netman_httpd) { 4147 // Obsolete! 4148 4149 import arsd.httpd; 4150 // what about forwarding the other constructor args? 4151 // this probably needs a whole redoing... 4152 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4153 return; 4154 } else 4155 version(embedded_httpd_processes) { 4156 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4157 } else 4158 version(embedded_httpd_threads) { 4159 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4160 } else 4161 version(scgi) { 4162 serveScgi!(fun, CustomCgi, maxContentLength)(); 4163 } else 4164 version(fastcgi) { 4165 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4166 } else 4167 version(stdio_http) { 4168 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4169 } else { 4170 //version=plain_cgi; 4171 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4172 } 4173 } 4174 4175 /++ 4176 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4177 4178 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4179 +/ 4180 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4181 globalStopFlag = false; 4182 static if(__traits(isStaticFunction, fun)) 4183 alias funToUse = fun; 4184 else 4185 void funToUse(CustomCgi cgi) { 4186 static if(__VERSION__ > 2097) 4187 __traits(child, _this, fun)(cgi); 4188 else static assert(0, "Not implemented in your compiler version!"); 4189 } 4190 auto manager = this.listenSpec is null ? 4191 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4192 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4193 manager.listen(); 4194 } 4195 4196 /++ 4197 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4198 +/ 4199 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4200 globalStopFlag = false; 4201 auto manager = this.listenSpec is null ? 4202 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4203 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4204 manager.listen(); 4205 } 4206 4207 /++ 4208 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4209 4210 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4211 4212 History: 4213 Added May 29, 2021 4214 +/ 4215 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4216 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4217 } 4218 4219 /++ 4220 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4221 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4222 +/ 4223 enum ForceStop { 4224 /++ 4225 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4226 +/ 4227 afterQueuedRequestsComplete, 4228 /++ 4229 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4230 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4231 +/ 4232 afterCurrentRequestsComplete, 4233 /++ 4234 Partial response writes will throw an exception, cancelling any streaming response, but complete 4235 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4236 +/ 4237 cancelStreamingRequestsEarly, 4238 /++ 4239 All writes will throw. 4240 +/ 4241 cancelAllRequestsEarly, 4242 /++ 4243 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4244 +/ 4245 forciblyTerminate, 4246 } 4247 4248 version(embedded_httpd_processes) {} else 4249 /++ 4250 Stops serving after the current requests are completed. 4251 4252 Bugs: 4253 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4254 on Windows however). Only partially implemented on non-Linux posix systems. 4255 4256 You might also try SIGINT perhaps. 4257 4258 The stopPriority is not yet fully implemented. 4259 +/ 4260 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4261 globalStopFlag = true; 4262 4263 version(Posix) { 4264 if(cancelfd > 0) { 4265 ulong a = 1; 4266 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4267 } 4268 } 4269 version(Windows) { 4270 if(iocp) { 4271 foreach(i; 0 .. 16) // FIXME 4272 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4273 } 4274 } 4275 } 4276 } 4277 4278 class AuthorizationRequiredException : Exception { 4279 string type; 4280 string realm; 4281 this(string type, string realm, string file, size_t line) { 4282 this.type = type; 4283 this.realm = realm; 4284 4285 super("Authorization Required", file, line); 4286 } 4287 } 4288 4289 private alias AliasSeq(T...) = T; 4290 4291 version(with_breaking_cgi_features) 4292 mixin(q{ 4293 template ThisFor(alias t) { 4294 static if(__traits(isStaticFunction, t)) { 4295 alias ThisFor = AliasSeq!(); 4296 } else { 4297 alias ThisFor = __traits(parent, t); 4298 } 4299 } 4300 }); 4301 else 4302 alias ThisFor(alias t) = AliasSeq!(); 4303 4304 private __gshared bool globalStopFlag = false; 4305 4306 version(embedded_httpd_processes) 4307 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4308 import core.sys.posix.unistd; 4309 import core.sys.posix.sys.socket; 4310 import core.sys.posix.netinet.in_; 4311 //import std.c.linux.socket; 4312 4313 int sock = socket(AF_INET, SOCK_STREAM, 0); 4314 if(sock == -1) 4315 throw new Exception("socket"); 4316 4317 cloexec(sock); 4318 4319 { 4320 4321 sockaddr_in addr; 4322 addr.sin_family = AF_INET; 4323 addr.sin_port = htons(params.listeningPort); 4324 auto lh = params.listeningHost; 4325 if(lh.length) { 4326 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4327 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4328 } else 4329 addr.sin_addr.s_addr = INADDR_ANY; 4330 4331 // HACKISH 4332 int on = 1; 4333 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4334 // end hack 4335 4336 4337 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4338 close(sock); 4339 throw new Exception("bind"); 4340 } 4341 4342 // FIXME: if this queue is full, it will just ignore it 4343 // and wait for the client to retransmit it. This is an 4344 // obnoxious timeout condition there. 4345 if(sock.listen(128) == -1) { 4346 close(sock); 4347 throw new Exception("listen"); 4348 } 4349 params.dropPrivileges(); 4350 } 4351 4352 version(embedded_httpd_processes_accept_after_fork) {} else { 4353 int pipeReadFd; 4354 int pipeWriteFd; 4355 4356 { 4357 int[2] pipeFd; 4358 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4359 import core.stdc.errno; 4360 throw new Exception("pipe failed " ~ to!string(errno)); 4361 } 4362 4363 pipeReadFd = pipeFd[0]; 4364 pipeWriteFd = pipeFd[1]; 4365 } 4366 } 4367 4368 4369 int processCount; 4370 pid_t newPid; 4371 reopen: 4372 while(processCount < processPoolSize) { 4373 newPid = fork(); 4374 if(newPid == 0) { 4375 // start serving on the socket 4376 //ubyte[4096] backingBuffer; 4377 for(;;) { 4378 bool closeConnection; 4379 uint i; 4380 sockaddr addr; 4381 i = addr.sizeof; 4382 version(embedded_httpd_processes_accept_after_fork) { 4383 int s = accept(sock, &addr, &i); 4384 int opt = 1; 4385 import core.sys.posix.netinet.tcp; 4386 // the Cgi class does internal buffering, so disabling this 4387 // helps with latency in many cases... 4388 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4389 cloexec(s); 4390 } else { 4391 int s; 4392 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4393 if(readret != s.sizeof) { 4394 import core.stdc.errno; 4395 throw new Exception("pipe read failed " ~ to!string(errno)); 4396 } 4397 4398 //writeln("process ", getpid(), " got socket ", s); 4399 } 4400 4401 try { 4402 4403 if(s == -1) 4404 throw new Exception("accept"); 4405 4406 scope(failure) close(s); 4407 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4408 auto ir = new BufferedInputRange(s); 4409 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4410 4411 while(!ir.empty) { 4412 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4413 4414 Cgi cgi; 4415 try { 4416 cgi = new CustomCgi(ir, &closeConnection); 4417 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4418 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4419 if(processPoolSize <= 1) 4420 closeConnection = true; 4421 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4422 } catch(HttpVersionNotSupportedException he) { 4423 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4424 closeConnection = true; 4425 break; 4426 } catch(Throwable t) { 4427 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4428 // anyway let's kill the connection 4429 version(CRuntime_Musl) { 4430 // LockingTextWriter fails here 4431 // so working around it 4432 auto estr = t.toString(); 4433 stderr.rawWrite(estr); 4434 stderr.rawWrite("\n"); 4435 } else 4436 stderr.writeln(t.toString()); 4437 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4438 closeConnection = true; 4439 break; 4440 } 4441 assert(cgi !is null); 4442 scope(exit) 4443 cgi.dispose(); 4444 4445 try { 4446 fun(cgi); 4447 cgi.close(); 4448 if(cgi.websocketMode) 4449 closeConnection = true; 4450 4451 } catch(AuthorizationRequiredException are) { 4452 cgi.setResponseStatus("401 Authorization Required"); 4453 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4454 cgi.close(); 4455 } catch(ConnectionException ce) { 4456 closeConnection = true; 4457 } catch(Throwable t) { 4458 // a processing error can be recovered from 4459 version(CRuntime_Musl) { 4460 // LockingTextWriter fails here 4461 // so working around it 4462 auto estr = t.toString(); 4463 stderr.rawWrite(estr); 4464 } else { 4465 stderr.writeln(t.toString); 4466 } 4467 if(!handleException(cgi, t)) 4468 closeConnection = true; 4469 } 4470 4471 if(closeConnection) { 4472 ir.source.close(); 4473 break; 4474 } else { 4475 if(!ir.empty) 4476 ir.popFront(); // get the next 4477 else if(ir.sourceClosed) { 4478 ir.source.close(); 4479 } 4480 } 4481 } 4482 4483 ir.source.close(); 4484 } catch(Throwable t) { 4485 version(CRuntime_Musl) {} else 4486 debug writeln(t); 4487 // most likely cause is a timeout 4488 } 4489 } 4490 } else if(newPid < 0) { 4491 throw new Exception("fork failed"); 4492 } else { 4493 processCount++; 4494 } 4495 } 4496 4497 // the parent should wait for its children... 4498 if(newPid) { 4499 import core.sys.posix.sys.wait; 4500 4501 version(embedded_httpd_processes_accept_after_fork) {} else { 4502 import core.sys.posix.sys.select; 4503 int[] fdQueue; 4504 while(true) { 4505 // writeln("select call"); 4506 int nfds = pipeWriteFd; 4507 if(sock > pipeWriteFd) 4508 nfds = sock; 4509 nfds += 1; 4510 fd_set read_fds; 4511 fd_set write_fds; 4512 FD_ZERO(&read_fds); 4513 FD_ZERO(&write_fds); 4514 FD_SET(sock, &read_fds); 4515 if(fdQueue.length) 4516 FD_SET(pipeWriteFd, &write_fds); 4517 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4518 if(ret == -1) { 4519 import core.stdc.errno; 4520 if(errno == EINTR) 4521 goto try_wait; 4522 else 4523 throw new Exception("wtf select"); 4524 } 4525 4526 int s = -1; 4527 if(FD_ISSET(sock, &read_fds)) { 4528 uint i; 4529 sockaddr addr; 4530 i = addr.sizeof; 4531 s = accept(sock, &addr, &i); 4532 cloexec(s); 4533 import core.sys.posix.netinet.tcp; 4534 int opt = 1; 4535 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4536 } 4537 4538 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4539 if(s == -1 && fdQueue.length) { 4540 s = fdQueue[0]; 4541 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4542 } 4543 write_fd(pipeWriteFd, &s, s.sizeof, s); 4544 close(s); // we are done with it, let the other process take ownership 4545 } else 4546 fdQueue ~= s; 4547 } 4548 } 4549 4550 try_wait: 4551 4552 int status; 4553 while(-1 != wait(&status)) { 4554 version(CRuntime_Musl) {} else { 4555 import std.stdio; writeln("Process died ", status); 4556 } 4557 processCount--; 4558 goto reopen; 4559 } 4560 close(sock); 4561 } 4562 } 4563 4564 version(fastcgi) 4565 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4566 // SetHandler fcgid-script 4567 FCGX_Stream* input, output, error; 4568 FCGX_ParamArray env; 4569 4570 4571 4572 const(ubyte)[] getFcgiChunk() { 4573 const(ubyte)[] ret; 4574 while(FCGX_HasSeenEOF(input) != -1) 4575 ret ~= cast(ubyte) FCGX_GetChar(input); 4576 return ret; 4577 } 4578 4579 void writeFcgi(const(ubyte)[] data) { 4580 FCGX_PutStr(data.ptr, data.length, output); 4581 } 4582 4583 void doARequest() { 4584 string[string] fcgienv; 4585 4586 for(auto e = env; e !is null && *e !is null; e++) { 4587 string cur = to!string(*e); 4588 auto idx = cur.indexOf("="); 4589 string name, value; 4590 if(idx == -1) 4591 name = cur; 4592 else { 4593 name = cur[0 .. idx]; 4594 value = cur[idx + 1 .. $]; 4595 } 4596 4597 fcgienv[name] = value; 4598 } 4599 4600 void flushFcgi() { 4601 FCGX_FFlush(output); 4602 } 4603 4604 Cgi cgi; 4605 try { 4606 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4607 } catch(Throwable t) { 4608 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4609 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4610 return; //continue; 4611 } 4612 assert(cgi !is null); 4613 scope(exit) cgi.dispose(); 4614 try { 4615 fun(cgi); 4616 cgi.close(); 4617 } catch(AuthorizationRequiredException are) { 4618 cgi.setResponseStatus("401 Authorization Required"); 4619 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4620 cgi.close(); 4621 } catch(Throwable t) { 4622 // log it to the error stream 4623 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4624 // handle it for the user, if we can 4625 if(!handleException(cgi, t)) 4626 return; // continue; 4627 } 4628 } 4629 4630 auto lp = params.listeningPort; 4631 auto host = params.listeningHost; 4632 4633 FCGX_Request request; 4634 if(lp || !host.empty) { 4635 // if a listening port was specified on the command line, we want to spawn ourself 4636 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4637 FCGX_Init(); 4638 4639 int sock; 4640 4641 if(host.startsWith("unix:")) { 4642 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4643 } else if(host.startsWith("abstract:")) { 4644 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4645 } else { 4646 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4647 } 4648 4649 if(sock < 0) 4650 throw new Exception("Couldn't listen on the port"); 4651 FCGX_InitRequest(&request, sock, 0); 4652 while(FCGX_Accept_r(&request) >= 0) { 4653 input = request.inStream; 4654 output = request.outStream; 4655 error = request.errStream; 4656 env = request.envp; 4657 doARequest(); 4658 } 4659 } else { 4660 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4661 // using the version with a global variable since we are separate processes anyway 4662 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4663 doARequest(); 4664 } 4665 } 4666 } 4667 4668 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4669 ushort defaultListeningPort() { 4670 version(netman_httpd) 4671 return 8080; 4672 else version(embedded_httpd_processes) 4673 return 8085; 4674 else version(embedded_httpd_threads) 4675 return 8085; 4676 else version(scgi) 4677 return 4000; 4678 else 4679 return 0; 4680 } 4681 4682 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4683 string defaultListeningHost() { 4684 version(netman_httpd) 4685 return null; 4686 else version(embedded_httpd_processes) 4687 return null; 4688 else version(embedded_httpd_threads) 4689 return null; 4690 else version(scgi) 4691 return "127.0.0.1"; 4692 else 4693 return null; 4694 4695 } 4696 4697 /++ 4698 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4699 4700 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4701 4702 Params: 4703 fun = Your request handler 4704 CustomCgi = a subclass of Cgi, if you wise to customize it further 4705 maxContentLength = max POST size you want to allow 4706 args = command-line arguments 4707 4708 History: 4709 Documented Sept 26, 2020. 4710 +/ 4711 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4712 if(tryAddonServers(args)) 4713 return; 4714 4715 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4716 return; 4717 4718 RequestServer server; 4719 // you can change the port here if you like 4720 // server.listeningPort = 9000; 4721 4722 // then call this to let the command line args override your default 4723 server.configureFromCommandLine(args); 4724 4725 // and serve the request(s). 4726 server.serve!(fun, CustomCgi, maxContentLength)(); 4727 } 4728 4729 //version(plain_cgi) 4730 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4731 // standard CGI is the default version 4732 4733 4734 // Set stdin to binary mode if necessary to avoid mangled newlines 4735 // the fact that stdin is global means this could be trouble but standard cgi request 4736 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4737 version(Windows) { 4738 version(Win64) 4739 _setmode(std.stdio.stdin.fileno(), 0x8000); 4740 else 4741 setmode(std.stdio.stdin.fileno(), 0x8000); 4742 } 4743 4744 Cgi cgi; 4745 try { 4746 cgi = new CustomCgi(maxContentLength); 4747 version(Posix) 4748 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4749 else version(Windows) 4750 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4751 else static assert(0); 4752 } catch(Throwable t) { 4753 version(CRuntime_Musl) { 4754 // LockingTextWriter fails here 4755 // so working around it 4756 auto s = t.toString(); 4757 stderr.rawWrite(s); 4758 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4759 } else { 4760 stderr.writeln(t.msg); 4761 // the real http server will probably handle this; 4762 // most likely, this is a bug in Cgi. But, oh well. 4763 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4764 } 4765 return; 4766 } 4767 assert(cgi !is null); 4768 scope(exit) cgi.dispose(); 4769 4770 try { 4771 fun(cgi); 4772 cgi.close(); 4773 } catch(AuthorizationRequiredException are) { 4774 cgi.setResponseStatus("401 Authorization Required"); 4775 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4776 cgi.close(); 4777 } catch (Throwable t) { 4778 version(CRuntime_Musl) { 4779 // LockingTextWriter fails here 4780 // so working around it 4781 auto s = t.msg; 4782 stderr.rawWrite(s); 4783 } else { 4784 stderr.writeln(t.msg); 4785 } 4786 if(!handleException(cgi, t)) 4787 return; 4788 } 4789 } 4790 4791 private __gshared int cancelfd = -1; 4792 4793 /+ 4794 The event loop for embedded_httpd_threads will prolly fiber dispatch 4795 cgi constructors too, so slow posts will not monopolize a worker thread. 4796 4797 May want to provide the worker task system just need to ensure all the fibers 4798 has a big enough stack for real work... would also ideally like to reuse them. 4799 4800 4801 So prolly bir would switch it to nonblocking. If it would block, it epoll 4802 registers one shot with this existing fiber to take it over. 4803 4804 new connection comes in. it picks a fiber off the free list, 4805 or if there is none, it creates a new one. this fiber handles 4806 this connection the whole time. 4807 4808 epoll triggers the fiber when something comes in. it is called by 4809 a random worker thread, it might change at any time. at least during 4810 the constructor. maybe into the main body it will stay tied to a thread 4811 just so TLS stuff doesn't randomly change in the middle. but I could 4812 specify if you yield all bets are off. 4813 4814 when the request is finished, if there's more data buffered, it just 4815 keeps going. if there is no more data buffered, it epoll ctls to 4816 get triggered when more data comes in. all one shot. 4817 4818 when a connection is closed, the fiber returns and is then reset 4819 and added to the free list. if the free list is full, the fiber is 4820 just freed, this means it will balloon to a certain size but not generally 4821 grow beyond that unless the activity keeps going. 4822 4823 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4824 4825 So the fiber has its own magic methods to read and write. if they would block, it registers 4826 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4827 4828 basically you issue the command and it tells you when it is done 4829 4830 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4831 4832 +/ 4833 4834 /++ 4835 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4836 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4837 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4838 4839 History: 4840 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4841 +/ 4842 version(cgi_use_fiber) 4843 __gshared size_t fiberStackSize = 4096 * 100; 4844 4845 version(cgi_use_fiber) 4846 class CgiFiber : Fiber { 4847 private void function(Socket) f_handler; 4848 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4849 f_handler(s); 4850 } 4851 this(void function(Socket) handler) { 4852 this.f_handler = handler; 4853 this(&f_handler_dg); 4854 } 4855 4856 this(void delegate(Socket) handler) { 4857 this.handler = handler; 4858 super(&run, fiberStackSize); 4859 } 4860 4861 Socket connection; 4862 void delegate(Socket) handler; 4863 4864 void run() { 4865 handler(connection); 4866 } 4867 4868 void delegate() postYield; 4869 4870 private void setPostYield(scope void delegate() py) @nogc { 4871 postYield = cast(void delegate()) py; 4872 } 4873 4874 void proceed() { 4875 try { 4876 call(); 4877 auto py = postYield; 4878 postYield = null; 4879 if(py !is null) 4880 py(); 4881 } catch(Exception e) { 4882 if(connection) 4883 connection.close(); 4884 goto terminate; 4885 } 4886 4887 if(state == State.TERM) { 4888 terminate: 4889 import core.memory; 4890 GC.removeRoot(cast(void*) this); 4891 } 4892 } 4893 } 4894 4895 version(cgi_use_fiber) 4896 version(Windows) { 4897 4898 extern(Windows) private { 4899 4900 import core.sys.windows.mswsock; 4901 4902 alias GROUP=uint; 4903 alias LPWSAPROTOCOL_INFOW = void*; 4904 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4905 alias WSASend = arsd.core.WSASend; 4906 alias WSARecv = arsd.core.WSARecv; 4907 alias WSABUF = arsd.core.WSABUF; 4908 4909 /+ 4910 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4911 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4912 4913 struct WSABUF { 4914 ULONG len; 4915 CHAR *buf; 4916 } 4917 +/ 4918 alias LPWSABUF = WSABUF*; 4919 4920 alias WSAOVERLAPPED = OVERLAPPED; 4921 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4922 /+ 4923 4924 alias LPFN_ACCEPTEX = 4925 BOOL 4926 function( 4927 SOCKET sListenSocket, 4928 SOCKET sAcceptSocket, 4929 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4930 void* lpOutputBuffer, 4931 WORD dwReceiveDataLength, 4932 WORD dwLocalAddressLength, 4933 WORD dwRemoteAddressLength, 4934 LPDWORD lpdwBytesReceived, 4935 LPOVERLAPPED lpOverlapped 4936 ); 4937 4938 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4939 +/ 4940 4941 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4942 } 4943 4944 private class PseudoblockingOverlappedSocket : Socket { 4945 SOCKET handle; 4946 4947 CgiFiber fiber; 4948 4949 this(AddressFamily af, SocketType st) { 4950 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4951 if(!handle) 4952 throw new Exception("WSASocketW"); 4953 this.handle = handle; 4954 4955 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4956 4957 if(iocp is null) { 4958 writeln(GetLastError()); 4959 throw new Exception("CreateIoCompletionPort"); 4960 } 4961 4962 super(cast(socket_t) handle, af); 4963 } 4964 this() pure nothrow @trusted { assert(0); } 4965 4966 override void blocking(bool) {} // meaningless to us, just ignore it. 4967 4968 protected override Socket accepting() pure nothrow { 4969 assert(0); 4970 } 4971 4972 bool addressesParsed; 4973 Address la; 4974 Address ra; 4975 4976 private void populateAddresses() { 4977 if(addressesParsed) 4978 return; 4979 addressesParsed = true; 4980 4981 int lalen, ralen; 4982 4983 sockaddr_in* la; 4984 sockaddr_in* ra; 4985 4986 lpfnGetAcceptExSockaddrs( 4987 scratchBuffer.ptr, 4988 0, // same as in the AcceptEx call! 4989 sockaddr_in.sizeof + 16, 4990 sockaddr_in.sizeof + 16, 4991 cast(sockaddr**) &la, 4992 &lalen, 4993 cast(sockaddr**) &ra, 4994 &ralen 4995 ); 4996 4997 if(la) 4998 this.la = new InternetAddress(*la); 4999 if(ra) 5000 this.ra = new InternetAddress(*ra); 5001 5002 } 5003 5004 override @property @trusted Address localAddress() { 5005 populateAddresses(); 5006 return la; 5007 } 5008 override @property @trusted Address remoteAddress() { 5009 populateAddresses(); 5010 return ra; 5011 } 5012 5013 PseudoblockingOverlappedSocket accepted; 5014 5015 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 5016 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 5017 5018 override Socket accept() @trusted { 5019 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 5020 5021 if(lpfnAcceptEx is null) { 5022 DWORD dwBytes; 5023 GUID GuidAcceptEx = WSAID_ACCEPTEX; 5024 5025 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 5026 &GuidAcceptEx, GuidAcceptEx.sizeof, 5027 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 5028 &dwBytes, null, null); 5029 5030 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 5031 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 5032 &GuidAcceptEx, GuidAcceptEx.sizeof, 5033 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 5034 &dwBytes, null, null); 5035 5036 } 5037 5038 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 5039 accepted = pfa; 5040 5041 SOCKET pendingForAccept = pfa.handle; 5042 DWORD ignored; 5043 5044 auto ret = lpfnAcceptEx(handle, 5045 pendingForAccept, 5046 // buffer to receive up front 5047 pfa.scratchBuffer.ptr, 5048 0, 5049 // size of local and remote addresses. normally + 16. 5050 sockaddr_in.sizeof + 16, 5051 sockaddr_in.sizeof + 16, 5052 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 5053 &overlapped 5054 ); 5055 5056 return pfa; 5057 } 5058 5059 override void connect(Address to) { assert(0); } 5060 5061 DWORD lastAnswer; 5062 ubyte[1024] scratchBuffer; 5063 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 5064 5065 WSABUF[1] buffer; 5066 OVERLAPPED overlapped; 5067 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 5068 overlapped = overlapped.init; 5069 buffer[0].len = cast(DWORD) buf.length; 5070 buffer[0].buf = cast(ubyte*) buf.ptr; 5071 fiber.setPostYield( () { 5072 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 5073 if(GetLastError() != 997) { 5074 //throw new Exception("WSASend fail"); 5075 } 5076 } 5077 }); 5078 5079 Fiber.yield(); 5080 return lastAnswer; 5081 } 5082 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 5083 overlapped = overlapped.init; 5084 buffer[0].len = cast(DWORD) buf.length; 5085 buffer[0].buf = cast(ubyte*) buf.ptr; 5086 5087 DWORD flags2 = 0; 5088 5089 fiber.setPostYield(() { 5090 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 5091 if(GetLastError() != 997) { 5092 //writeln("WSARecv ", WSAGetLastError()); 5093 //throw new Exception("WSARecv fail"); 5094 } 5095 } 5096 }); 5097 5098 Fiber.yield(); 5099 return lastAnswer; 5100 } 5101 5102 // I might go back and implement these for udp things. 5103 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 5104 assert(0); 5105 } 5106 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 5107 assert(0); 5108 } 5109 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 5110 assert(0); 5111 } 5112 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 5113 assert(0); 5114 } 5115 5116 // lol overload sets 5117 alias send = typeof(super).send; 5118 alias receive = typeof(super).receive; 5119 alias sendTo = typeof(super).sendTo; 5120 alias receiveFrom = typeof(super).receiveFrom; 5121 5122 } 5123 } 5124 5125 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 5126 assert(connection !is null); 5127 version(cgi_use_fiber) { 5128 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 5129 5130 version(Windows) { 5131 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 5132 } 5133 5134 import core.memory; 5135 GC.addRoot(cast(void*) fiber); 5136 fiber.connection = connection; 5137 fiber.proceed(); 5138 } else { 5139 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 5140 } 5141 } 5142 5143 /+ 5144 5145 /+ 5146 The represents a recyclable per-task arena allocator. The default is to let the GC manage the whole block as a large array, meaning if a reference into it is escaped, it waste memory but is not dangerous. If you don't escape any references to it and don't do anything special, the GC collects it. 5147 5148 But, if you call `cgi.recyclable = true`, the memory is retained for the next request on the thread. If a reference is escaped, it is the user's problem; it can be modified (and break the `immutable` guarantees!) and thus be memory unsafe. They're taking responsibility for doing it right when they call `escape`. But if they do it right and opt into recycling, the memory is all reused to give a potential boost without requiring the GC's involvement. 5149 5150 What if one request used an abnormally large amount of memory though? Will recycling it keep that pinned forever? No, that's why it keeps track of some stats. If a working set was significantly above average and not fully utilized for a while, it will just let the GC have it again despite your suggestion to recycle it. 5151 5152 Be warned that growing the memory block may release the old, smaller block for garbage collection. If you retained references to it, it may not be collectable and lead to some unnecessary memory growth. It is probably best to try to keep the things sized in a continuous block that doesn't have to grow often. 5153 5154 Internally, it is broken up into a few blocks: 5155 * the request block. This holds the incoming request and associated data (parsed headers, variables, etc). 5156 * the scannable block. this holds pointers arrays, classes, etc. associated with this request, so named because the GC scans it. 5157 * the response block. This holds the output buffer. 5158 5159 And I may add more later if I decide to open this up to outside user code. 5160 5161 The scannable block is separate to limit the amount of work the GC has to do; no point asking it to scan that which need not be scanned. 5162 5163 The request and response blocks are separated because they will have different typical sizes, with the request likely being less predictable. Being able to release one to the GC while recycling the other might help, and having them grow independently (if needed) may also prevent some pain. 5164 5165 All of this are internal implementation details subject to change at any time without notice. It is valid for my recycle method to do absolutely nothing; the GC also eventually recycles memory! 5166 5167 Each active task can have its own recyclable memory object. When you recycle it, it is added to a thread-local freelist. If the list is excessively large, entries maybe discarded at random and left for the GC to prevent a temporary burst of activity from leading to a permanent waste of memory. 5168 +/ 5169 struct RecyclableMemory { 5170 private ubyte[] inputBuffer; 5171 private ubyte[] processedRequestBlock; 5172 private void[] scannableBlock; 5173 private ubyte[] outputBuffer; 5174 5175 RecyclableMemory* next; 5176 } 5177 5178 /++ 5179 This emulates the D associative array interface with a different internal implementation. 5180 5181 string s = cgi.get["foo"]; // just does cgi.getArray[x][$-1]; 5182 string[] arr = cgi.getArray["foo"]; 5183 5184 "foo" in cgi.get 5185 5186 foreach(k, v; cgi.get) 5187 5188 cgi.get.toAA // for compatibility 5189 5190 // and this can urldecode lazily tbh... in-place even, since %xx is always longer than a single char thing it turns into... 5191 ... but how does it mark that it has already been processed in-place? it'd have to just add it to the index then. 5192 5193 deprecated alias toAA this; 5194 +/ 5195 struct VariableCollection { 5196 private VariableArrayCollection* vac; 5197 5198 const(char[]) opIndex(scope const char[] key) { 5199 return (*vac)[key][$-1]; 5200 } 5201 5202 const(char[]*) opBinaryRight(string op : "in")(scope const char[] key) { 5203 return key in (*vac); 5204 } 5205 5206 int opApply(int delegate(scope const(char)[] key, scope const(char)[] value) dg) { 5207 foreach(k, v; *vac) { 5208 if(auto res = dg(k, v[$-1])) 5209 return res; 5210 } 5211 return 0; 5212 } 5213 5214 immutable(string[string]) toAA() { 5215 string[string] aa; 5216 foreach(k, v; *vac) 5217 aa[k.idup] = v[$-1].idup; 5218 return aa; 5219 } 5220 5221 deprecated alias toAA this; 5222 } 5223 5224 struct VariableArrayCollection { 5225 /+ 5226 This needs the actual implementation of looking it up. As it pulls data, it should 5227 decode and index for later. 5228 5229 The index will go into a block attached to the cgi object and it should prolly be sorted 5230 something like 5231 5232 [count of names] 5233 [slice to name][count of values][slice to value, decoded in-place, ...] 5234 ... 5235 +/ 5236 private Cgi cgi; 5237 5238 const(char[][]) opIndex(scope const char[] key) { 5239 return null; 5240 } 5241 5242 const(char[][]*) opBinaryRight(string op : "in")(scope const char[] key) { 5243 return null; 5244 } 5245 5246 // int opApply(int delegate(scope const(char)[] key, scope const(char)[][] value) dg) 5247 5248 immutable(string[string]) toAA() { 5249 return null; 5250 } 5251 5252 deprecated alias toAA this; 5253 5254 } 5255 5256 struct HeaderCollection { 5257 5258 } 5259 +/ 5260 5261 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 5262 scope(failure) { 5263 // catch all for other errors 5264 try { 5265 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5266 connection.close(); 5267 } catch(Exception e) {} // swallow it, we're aborting anyway. 5268 } 5269 5270 bool closeConnection = alwaysCloseConnection; 5271 5272 /+ 5273 ubyte[4096] inputBuffer = void; 5274 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5275 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5276 5277 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5278 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5279 ir.__ctor(connection, inputBuffer[], true); 5280 +/ 5281 5282 auto ir = new BufferedInputRange(connection); 5283 5284 while(!ir.empty) { 5285 5286 if(ir.view.length == 0) { 5287 ir.popFront(); 5288 if(ir.sourceClosed) { 5289 connection.close(); 5290 closeConnection = true; 5291 break; 5292 } 5293 } 5294 5295 Cgi cgi; 5296 try { 5297 cgi = new CustomCgi(ir, &closeConnection); 5298 // There's a bunch of these casts around because the type matches up with 5299 // the -version=.... specifiers, just you can also create a RequestServer 5300 // and instantiate the things where the types don't match up. It isn't exactly 5301 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5302 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5303 } catch(ConnectionClosedException ce) { 5304 closeConnection = true; 5305 break; 5306 } catch(ConnectionException ce) { 5307 // broken pipe or something, just abort the connection 5308 closeConnection = true; 5309 break; 5310 } catch(HttpVersionNotSupportedException ve) { 5311 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5312 closeConnection = true; 5313 break; 5314 } catch(Throwable t) { 5315 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5316 // anyway let's kill the connection 5317 version(CRuntime_Musl) { 5318 stderr.rawWrite(t.toString()); 5319 stderr.rawWrite("\n"); 5320 } else { 5321 stderr.writeln(t.toString()); 5322 } 5323 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5324 closeConnection = true; 5325 break; 5326 } 5327 assert(cgi !is null); 5328 scope(exit) 5329 cgi.dispose(); 5330 5331 try { 5332 fun(cgi); 5333 cgi.close(); 5334 if(cgi.websocketMode) 5335 closeConnection = true; 5336 } catch(AuthorizationRequiredException are) { 5337 cgi.setResponseStatus("401 Authorization Required"); 5338 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5339 cgi.close(); 5340 } catch(ConnectionException ce) { 5341 // broken pipe or something, just abort the connection 5342 closeConnection = true; 5343 } catch(ConnectionClosedException ce) { 5344 // broken pipe or something, just abort the connection 5345 closeConnection = true; 5346 } catch(Throwable t) { 5347 // a processing error can be recovered from 5348 version(CRuntime_Musl) {} else 5349 stderr.writeln(t.toString); 5350 if(!handleException(cgi, t)) 5351 closeConnection = true; 5352 } 5353 5354 if(globalStopFlag) 5355 closeConnection = true; 5356 5357 if(closeConnection || alwaysCloseConnection) { 5358 connection.shutdown(SocketShutdown.BOTH); 5359 connection.close(); 5360 ir.dispose(); 5361 closeConnection = false; // don't reclose after loop 5362 break; 5363 } else { 5364 if(ir.front.length) { 5365 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5366 } else if(ir.sourceClosed) { 5367 ir.source.shutdown(SocketShutdown.BOTH); 5368 ir.source.close(); 5369 ir.dispose(); 5370 closeConnection = false; 5371 } else { 5372 continue; 5373 // break; // this was for a keepalive experiment 5374 } 5375 } 5376 } 5377 5378 if(closeConnection) { 5379 connection.shutdown(SocketShutdown.BOTH); 5380 connection.close(); 5381 ir.dispose(); 5382 } 5383 5384 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5385 } 5386 5387 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5388 // and now we can buffer 5389 scope(failure) 5390 connection.close(); 5391 5392 import al = std.algorithm; 5393 5394 size_t size; 5395 5396 string[string] headers; 5397 5398 auto range = new BufferedInputRange(connection); 5399 more_data: 5400 auto chunk = range.front(); 5401 // waiting for colon for header length 5402 auto idx = indexOf(cast(string) chunk, ':'); 5403 if(idx == -1) { 5404 try { 5405 range.popFront(); 5406 } catch(Exception e) { 5407 // it is just closed, no big deal 5408 connection.close(); 5409 return; 5410 } 5411 goto more_data; 5412 } 5413 5414 size = to!size_t(cast(string) chunk[0 .. idx]); 5415 chunk = range.consume(idx + 1); 5416 // reading headers 5417 if(chunk.length < size) 5418 range.popFront(0, size + 1); 5419 // we are now guaranteed to have enough 5420 chunk = range.front(); 5421 assert(chunk.length > size); 5422 5423 idx = 0; 5424 string key; 5425 string value; 5426 foreach(part; al.splitter(chunk, '\0')) { 5427 if(idx & 1) { // odd is value 5428 value = cast(string)(part.idup); 5429 headers[key] = value; // commit 5430 } else 5431 key = cast(string)(part.idup); 5432 idx++; 5433 } 5434 5435 enforce(chunk[size] == ','); // the terminator 5436 5437 range.consume(size + 1); 5438 // reading data 5439 // this will be done by Cgi 5440 5441 const(ubyte)[] getScgiChunk() { 5442 // we are already primed 5443 auto data = range.front(); 5444 if(data.length == 0 && !range.sourceClosed) { 5445 range.popFront(0); 5446 data = range.front(); 5447 } else if (range.sourceClosed) 5448 range.source.close(); 5449 5450 return data; 5451 } 5452 5453 void writeScgi(const(ubyte)[] data) { 5454 sendAll(connection, data); 5455 } 5456 5457 void flushScgi() { 5458 // I don't *think* I have to do anything.... 5459 } 5460 5461 Cgi cgi; 5462 try { 5463 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5464 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5465 } catch(Throwable t) { 5466 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5467 connection.close(); 5468 return; // this connection is dead 5469 } 5470 assert(cgi !is null); 5471 scope(exit) cgi.dispose(); 5472 try { 5473 fun(cgi); 5474 cgi.close(); 5475 connection.close(); 5476 5477 } catch(AuthorizationRequiredException are) { 5478 cgi.setResponseStatus("401 Authorization Required"); 5479 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5480 cgi.close(); 5481 } catch(Throwable t) { 5482 // no std err 5483 if(!handleException(cgi, t)) { 5484 connection.close(); 5485 return; 5486 } else { 5487 connection.close(); 5488 return; 5489 } 5490 } 5491 } 5492 5493 string printDate(DateTime date) { 5494 char[29] buffer = void; 5495 printDateToBuffer(date, buffer[]); 5496 return buffer.idup; 5497 } 5498 5499 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5500 assert(buffer.length >= 29); 5501 // 29 static length ? 5502 5503 static immutable daysOfWeek = [ 5504 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5505 ]; 5506 5507 static immutable months = [ 5508 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5509 ]; 5510 5511 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5512 buffer[3 .. 5] = ", "; 5513 buffer[5] = date.day / 10 + '0'; 5514 buffer[6] = date.day % 10 + '0'; 5515 buffer[7] = ' '; 5516 buffer[8 .. 11] = months[date.month]; 5517 buffer[11] = ' '; 5518 auto y = date.year; 5519 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5520 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5521 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5522 buffer[15] = cast(char) (y + '0'); 5523 buffer[16] = ' '; 5524 buffer[17] = date.hour / 10 + '0'; 5525 buffer[18] = date.hour % 10 + '0'; 5526 buffer[19] = ':'; 5527 buffer[20] = date.minute / 10 + '0'; 5528 buffer[21] = date.minute % 10 + '0'; 5529 buffer[22] = ':'; 5530 buffer[23] = date.second / 10 + '0'; 5531 buffer[24] = date.second % 10 + '0'; 5532 buffer[25 .. $] = " GMT"; 5533 5534 return 29; 5535 } 5536 5537 5538 // Referencing this gigantic typeid seems to remind the compiler 5539 // to actually put the symbol in the object file. I guess the immutable 5540 // assoc array array isn't actually included in druntime 5541 void hackAroundLinkerError() { 5542 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5543 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5544 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5545 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5546 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5547 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5548 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5549 // this is getting kinda ridiculous btw. Moving assoc arrays 5550 // to the library is the pain that keeps on coming. 5551 5552 // eh this broke the build on the work server 5553 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5554 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5555 } 5556 5557 5558 5559 5560 5561 version(fastcgi) { 5562 pragma(lib, "fcgi"); 5563 5564 static if(size_t.sizeof == 8) // 64 bit 5565 alias long c_int; 5566 else 5567 alias int c_int; 5568 5569 extern(C) { 5570 struct FCGX_Stream { 5571 ubyte* rdNext; 5572 ubyte* wrNext; 5573 ubyte* stop; 5574 ubyte* stopUnget; 5575 c_int isReader; 5576 c_int isClosed; 5577 c_int wasFCloseCalled; 5578 c_int FCGI_errno; 5579 void* function(FCGX_Stream* stream) fillBuffProc; 5580 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5581 void* data; 5582 } 5583 5584 // note: this is meant to be opaque, so don't access it directly 5585 struct FCGX_Request { 5586 int requestId; 5587 int role; 5588 FCGX_Stream* inStream; 5589 FCGX_Stream* outStream; 5590 FCGX_Stream* errStream; 5591 char** envp; 5592 void* paramsPtr; 5593 int ipcFd; 5594 int isBeginProcessed; 5595 int keepConnection; 5596 int appStatus; 5597 int nWriters; 5598 int flags; 5599 int listen_sock; 5600 } 5601 5602 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5603 void FCGX_Init(); 5604 5605 int FCGX_Accept_r(FCGX_Request *request); 5606 5607 5608 alias char** FCGX_ParamArray; 5609 5610 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5611 c_int FCGX_GetChar(FCGX_Stream* stream); 5612 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5613 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5614 c_int FCGX_FFlush(FCGX_Stream *stream); 5615 5616 int FCGX_OpenSocket(in char*, int); 5617 } 5618 } 5619 5620 5621 /* This might go int a separate module eventually. It is a network input helper class. */ 5622 5623 import std.socket; 5624 5625 version(cgi_use_fiber) { 5626 import core.thread; 5627 5628 version(linux) { 5629 import core.sys.linux.epoll; 5630 5631 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5632 } else version(Windows) { 5633 // declaring the iocp thing below... 5634 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5635 } 5636 5637 version(Windows) 5638 __gshared HANDLE iocp; 5639 5640 version(cgi_use_fiber) { 5641 version(linux) 5642 private enum WakeupEvent { 5643 Read = EPOLLIN, 5644 Write = EPOLLOUT 5645 } 5646 else version(Windows) 5647 private enum WakeupEvent { 5648 Read, Write 5649 } 5650 else static assert(0); 5651 } 5652 5653 version(cgi_use_fiber) 5654 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5655 5656 // static cast since I know what i have in here and don't want to pay for dynamic cast 5657 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5658 5659 version(linux) { 5660 f.setPostYield = () { 5661 if(*registered) { 5662 // rearm 5663 epoll_event evt; 5664 evt.events = e | EPOLLONESHOT; 5665 evt.data.ptr = cast(void*) f; 5666 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5667 throw new Exception("epoll_ctl"); 5668 } else { 5669 // initial registration 5670 *registered = true ; 5671 int fd = source.handle; 5672 epoll_event evt; 5673 evt.events = e | EPOLLONESHOT; 5674 evt.data.ptr = cast(void*) f; 5675 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5676 throw new Exception("epoll_ctl"); 5677 } 5678 }; 5679 5680 Fiber.yield(); 5681 5682 f.setPostYield(null); 5683 } else version(Windows) { 5684 Fiber.yield(); 5685 } 5686 else static assert(0); 5687 } 5688 5689 version(cgi_use_fiber) 5690 void unregisterSource(Socket s) { 5691 version(linux) { 5692 epoll_event evt; 5693 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5694 } else version(Windows) { 5695 // intentionally blank 5696 } 5697 else static assert(0); 5698 } 5699 5700 // it is a class primarily for reference semantics 5701 // I might change this interface 5702 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5703 class BufferedInputRange { 5704 version(Posix) 5705 this(int source, ubyte[] buffer = null) { 5706 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5707 } 5708 5709 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5710 // if they connect but never send stuff to us, we don't want it wasting the process 5711 // so setting a time out 5712 version(cgi_use_fiber) 5713 source.blocking = false; 5714 else 5715 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5716 5717 this.source = source; 5718 if(buffer is null) { 5719 underlyingBuffer = new ubyte[4096]; 5720 this.allowGrowth = true; 5721 } else { 5722 underlyingBuffer = buffer; 5723 this.allowGrowth = allowGrowth; 5724 } 5725 5726 assert(underlyingBuffer.length); 5727 5728 // we assume view.ptr is always inside underlyingBuffer 5729 view = underlyingBuffer[0 .. 0]; 5730 5731 popFront(); // prime 5732 } 5733 5734 version(cgi_use_fiber) { 5735 bool registered; 5736 } 5737 5738 void dispose() { 5739 version(cgi_use_fiber) { 5740 if(registered) 5741 unregisterSource(source); 5742 } 5743 } 5744 5745 /** 5746 A slight difference from regular ranges is you can give it the maximum 5747 number of bytes to consume. 5748 5749 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5750 consume() yourself and use a regular foreach, it will infinitely loop! 5751 5752 The default is to do what a normal range does, and consume the whole buffer 5753 and wait for additional input. 5754 5755 You can also specify 0, to append to the buffer, or any other number 5756 to remove the front n bytes and wait for more. 5757 */ 5758 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5759 if(sourceClosed) 5760 throw new ConnectionClosedException("can't get any more data from a closed source"); 5761 if(!skipConsume) 5762 consume(maxBytesToConsume); 5763 5764 // we might have to grow the buffer 5765 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5766 if(allowGrowth) { 5767 //import std.stdio; writeln("growth"); 5768 auto viewStart = view.ptr - underlyingBuffer.ptr; 5769 size_t growth = 4096; 5770 // make sure we have enough for what we're being asked for 5771 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5772 growth = minBytesToSettleFor - underlyingBuffer.length; 5773 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5774 underlyingBuffer.length += growth; 5775 view = underlyingBuffer[viewStart .. view.length]; 5776 } else 5777 throw new Exception("No room left in the buffer"); 5778 } 5779 5780 do { 5781 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5782 try_again: 5783 auto ret = source.receive(freeSpace); 5784 if(ret == Socket.ERROR) { 5785 if(wouldHaveBlocked()) { 5786 version(cgi_use_fiber) { 5787 registerEventWakeup(®istered, source, WakeupEvent.Read); 5788 goto try_again; 5789 } else { 5790 // gonna treat a timeout here as a close 5791 sourceClosed = true; 5792 return; 5793 } 5794 } 5795 version(Posix) { 5796 import core.stdc.errno; 5797 if(errno == EINTR || errno == EAGAIN) { 5798 goto try_again; 5799 } 5800 if(errno == ECONNRESET) { 5801 sourceClosed = true; 5802 return; 5803 } 5804 } 5805 throw new Exception(lastSocketError); // FIXME 5806 } 5807 if(ret == 0) { 5808 sourceClosed = true; 5809 return; 5810 } 5811 5812 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5813 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5814 //import std.stdio; writeln(cast(string) view); 5815 } while(view.length < minBytesToSettleFor); 5816 } 5817 5818 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5819 /// You might want to idup the data you are consuming if you store it, since it may 5820 /// be overwritten on the new popFront. 5821 /// 5822 /// You do not need to call this if you always want to wait for more data when you 5823 /// consume some. 5824 ubyte[] consume(size_t bytes) { 5825 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5826 view = view[bytes > $ ? $ : bytes .. $]; 5827 if(view.length == 0) { 5828 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5829 /* 5830 writeln("HERE"); 5831 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5832 writeln(cast(string)front); 5833 writeln("DONE"); 5834 */ 5835 } 5836 return front; 5837 } 5838 5839 bool empty() { 5840 return sourceClosed && view.length == 0; 5841 } 5842 5843 ubyte[] front() { 5844 return view; 5845 } 5846 5847 invariant() { 5848 assert(view.ptr >= underlyingBuffer.ptr); 5849 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5850 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5851 } 5852 5853 ubyte[] underlyingBuffer; 5854 bool allowGrowth; 5855 ubyte[] view; 5856 Socket source; 5857 bool sourceClosed; 5858 } 5859 5860 private class FakeSocketForStdin : Socket { 5861 import std.stdio; 5862 5863 this() { 5864 5865 } 5866 5867 private bool closed; 5868 5869 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5870 if(closed) 5871 throw new Exception("Closed"); 5872 return stdin.rawRead(buffer).length; 5873 } 5874 5875 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5876 if(closed) 5877 throw new Exception("Closed"); 5878 stdout.rawWrite(buffer); 5879 return buffer.length; 5880 } 5881 5882 override void close() @trusted scope { 5883 (cast(void delegate() @nogc nothrow) &realClose)(); 5884 } 5885 5886 override void shutdown(SocketShutdown s) { 5887 // FIXME 5888 } 5889 5890 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5891 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5892 5893 override @property @trusted Address remoteAddress() { return null; } 5894 override @property @trusted Address localAddress() { return null; } 5895 5896 void realClose() { 5897 closed = true; 5898 try { 5899 stdin.close(); 5900 stdout.close(); 5901 } catch(Exception e) { 5902 5903 } 5904 } 5905 } 5906 5907 import core.sync.semaphore; 5908 import core.atomic; 5909 5910 /** 5911 To use this thing: 5912 5913 --- 5914 void handler(Socket s) { do something... } 5915 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5916 manager.listen(); 5917 --- 5918 5919 The 4th parameter is optional. 5920 5921 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5922 comes in, you will get control. You can just continue; though to fetch more. 5923 5924 5925 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5926 */ 5927 class ListeningConnectionManager { 5928 Semaphore semaphore; 5929 Socket[256] queue; 5930 shared(ubyte) nextIndexFront; 5931 ubyte nextIndexBack; 5932 shared(int) queueLength; 5933 5934 Socket acceptCancelable() { 5935 version(Posix) { 5936 import core.sys.posix.sys.select; 5937 fd_set read_fds; 5938 FD_ZERO(&read_fds); 5939 int max = 0; 5940 foreach(listener; listeners) { 5941 FD_SET(listener.handle, &read_fds); 5942 if(listener.handle > max) 5943 max = listener.handle; 5944 } 5945 if(cancelfd != -1) { 5946 FD_SET(cancelfd, &read_fds); 5947 if(cancelfd > max) 5948 max = cancelfd; 5949 } 5950 auto ret = select(max + 1, &read_fds, null, null, null); 5951 if(ret == -1) { 5952 import core.stdc.errno; 5953 if(errno == EINTR) 5954 return null; 5955 else 5956 throw new Exception("wtf select"); 5957 } 5958 5959 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5960 return null; 5961 } 5962 5963 foreach(listener; listeners) { 5964 if(FD_ISSET(listener.handle, &read_fds)) 5965 return listener.accept(); 5966 } 5967 5968 return null; 5969 } else { 5970 5971 auto check = new SocketSet(); 5972 5973 keep_looping: 5974 check.reset(); 5975 foreach(listener; listeners) 5976 check.add(listener); 5977 5978 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5979 auto got = Socket.select(check, null, null, 3.seconds); 5980 if(got > 0) 5981 foreach(listener; listeners) 5982 if(check.isSet(listener)) 5983 return listener.accept(); 5984 if(globalStopFlag) 5985 return null; 5986 else 5987 goto keep_looping; 5988 } 5989 } 5990 5991 int defaultNumberOfThreads() { 5992 import std.parallelism; 5993 version(cgi_use_fiber) { 5994 return totalCPUs * 2 + 1; // still chance some will be pointlessly blocked anyway 5995 } else { 5996 // I times 4 here because there's a good chance some will be blocked on i/o. 5997 return totalCPUs * 4; 5998 } 5999 6000 } 6001 6002 void listen() { 6003 shared(int) loopBroken; 6004 6005 version(Posix) { 6006 import core.sys.posix.signal; 6007 signal(SIGPIPE, SIG_IGN); 6008 } 6009 6010 version(linux) { 6011 if(cancelfd == -1) 6012 cancelfd = eventfd(0, 0); 6013 } 6014 6015 version(cgi_no_threads) { 6016 // NEVER USE THIS 6017 // it exists only for debugging and other special occasions 6018 6019 // the thread mode is faster and less likely to stall the whole 6020 // thing when a request is slow 6021 while(!loopBroken && !globalStopFlag) { 6022 auto sn = acceptCancelable(); 6023 if(sn is null) continue; 6024 cloexec(sn); 6025 try { 6026 handler(sn); 6027 } catch(Exception e) { 6028 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 6029 sn.close(); 6030 } 6031 } 6032 } else { 6033 6034 if(useFork) { 6035 version(linux) { 6036 //asm { int 3; } 6037 fork(); 6038 } 6039 } 6040 6041 version(cgi_use_fiber) { 6042 6043 version(Windows) { 6044 // please note these are overlapped sockets! so the accept just kicks things off 6045 foreach(listener; listeners) 6046 listener.accept(); 6047 } 6048 6049 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 6050 foreach(i, ref thread; threads) { 6051 thread = new WorkerThread(this, handler, cast(int) i); 6052 thread.start(); 6053 } 6054 6055 bool fiber_crash_check() { 6056 bool hasAnyRunning; 6057 foreach(thread; threads) { 6058 if(!thread.isRunning) { 6059 thread.join(); 6060 } else hasAnyRunning = true; 6061 } 6062 6063 return (!hasAnyRunning); 6064 } 6065 6066 6067 while(!globalStopFlag) { 6068 Thread.sleep(1.seconds); 6069 if(fiber_crash_check()) 6070 break; 6071 } 6072 6073 } else { 6074 semaphore = new Semaphore(); 6075 6076 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 6077 foreach(i, ref thread; threads) { 6078 thread = new ConnectionThread(this, handler, cast(int) i); 6079 thread.start(); 6080 } 6081 6082 while(!loopBroken && !globalStopFlag) { 6083 Socket sn; 6084 6085 bool crash_check() { 6086 bool hasAnyRunning; 6087 foreach(thread; threads) { 6088 if(!thread.isRunning) { 6089 thread.join(); 6090 } else hasAnyRunning = true; 6091 } 6092 6093 return (!hasAnyRunning); 6094 } 6095 6096 6097 void accept_new_connection() { 6098 sn = acceptCancelable(); 6099 if(sn is null) return; 6100 cloexec(sn); 6101 if(tcp) { 6102 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6103 // on the socket because we do some buffering internally. I think this helps, 6104 // certainly does for small requests, and I think it does for larger ones too 6105 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6106 6107 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6108 } 6109 } 6110 6111 void existing_connection_new_data() { 6112 // wait until a slot opens up 6113 // int waited = 0; 6114 while(queueLength >= queue.length) { 6115 Thread.sleep(1.msecs); 6116 // waited ++; 6117 } 6118 // if(waited) {import std.stdio; writeln(waited);} 6119 synchronized(this) { 6120 queue[nextIndexBack] = sn; 6121 nextIndexBack++; 6122 atomicOp!"+="(queueLength, 1); 6123 } 6124 semaphore.notify(); 6125 } 6126 6127 6128 accept_new_connection(); 6129 if(sn !is null) 6130 existing_connection_new_data(); 6131 else if(sn is null && globalStopFlag) { 6132 foreach(thread; threads) { 6133 semaphore.notify(); 6134 } 6135 Thread.sleep(50.msecs); 6136 } 6137 6138 if(crash_check()) 6139 break; 6140 } 6141 } 6142 6143 // FIXME: i typically stop this with ctrl+c which never 6144 // actually gets here. i need to do a sigint handler. 6145 if(cleanup) 6146 cleanup(); 6147 } 6148 } 6149 6150 //version(linux) 6151 //int epoll_fd; 6152 6153 bool tcp; 6154 void delegate() cleanup; 6155 6156 private void function(Socket) fhandler; 6157 private void dg_handler(Socket s) { 6158 fhandler(s); 6159 } 6160 6161 6162 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6163 fhandler = handler; 6164 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 6165 } 6166 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6167 string[] host; 6168 ushort[] port; 6169 6170 foreach(spec; listenSpec) { 6171 /+ 6172 The format: 6173 6174 protocol:// 6175 address_spec 6176 6177 Protocol is optional. Must be http, https, scgi, or fastcgi. 6178 6179 address_spec is either: 6180 ipv4 address : port 6181 [ipv6 address] : port 6182 unix:filename 6183 abstract:name 6184 port <which is tcp but on any interface> 6185 +/ 6186 6187 string protocol; 6188 string address_spec; 6189 6190 auto protocolIdx = spec.indexOf("://"); 6191 if(protocolIdx != -1) { 6192 protocol = spec[0 .. protocolIdx]; 6193 address_spec = spec[protocolIdx + "://".length .. $]; 6194 } else { 6195 address_spec = spec; 6196 } 6197 6198 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 6199 host ~= address_spec; 6200 port ~= 0; 6201 } else { 6202 auto idx = address_spec.lastIndexOf(":"); 6203 if(idx == -1) { 6204 host ~= null; 6205 } else { 6206 auto as = address_spec[0 .. idx]; 6207 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 6208 as = as[1 .. $-1]; 6209 host ~= as; 6210 } 6211 port ~= address_spec[idx + 1 .. $].to!ushort; 6212 } 6213 6214 } 6215 6216 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 6217 } 6218 6219 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6220 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6221 } 6222 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6223 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6224 } 6225 6226 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6227 fhandler = handler; 6228 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 6229 } 6230 6231 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6232 assert(host.length == port.length); 6233 6234 this.handler = handler; 6235 this.useFork = useFork; 6236 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 6237 6238 listeners.reserve(host.length); 6239 6240 foreach(i; 0 .. host.length) 6241 if(host[i] == "localhost") { 6242 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 6243 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 6244 } else { 6245 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 6246 } 6247 6248 version(cgi_use_fiber) 6249 if(useFork) { 6250 foreach(listener; listeners) 6251 listener.blocking = false; 6252 } 6253 6254 // this is the UI control thread and thus gets more priority 6255 Thread.getThis.priority = Thread.PRIORITY_MAX; 6256 } 6257 6258 Socket[] listeners; 6259 void delegate(Socket) handler; 6260 6261 immutable bool useFork; 6262 int numberOfThreads; 6263 } 6264 6265 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 6266 Socket listener; 6267 if(host.startsWith("unix:")) { 6268 version(Posix) { 6269 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6270 cloexec(listener); 6271 string filename = host["unix:".length .. $].idup; 6272 listener.bind(new UnixAddress(filename)); 6273 cleanup = delegate() { 6274 listener.close(); 6275 import std.file; 6276 remove(filename); 6277 }; 6278 tcp = false; 6279 } else { 6280 throw new Exception("unix sockets not supported on this system"); 6281 } 6282 } else if(host.startsWith("abstract:")) { 6283 version(linux) { 6284 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6285 cloexec(listener); 6286 string filename = "\0" ~ host["abstract:".length .. $]; 6287 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6288 listener.bind(new UnixAddress(filename)); 6289 tcp = false; 6290 } else { 6291 throw new Exception("abstract unix sockets not supported on this system"); 6292 } 6293 } else { 6294 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6295 version(cgi_use_fiber) { 6296 version(Windows) 6297 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6298 else 6299 listener = new Socket(address.addressFamily, SocketType.STREAM); 6300 } else { 6301 listener = new Socket(address.addressFamily, SocketType.STREAM); 6302 } 6303 cloexec(listener); 6304 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6305 if(address.addressFamily == AddressFamily.INET6) 6306 listener.setOption(SocketOptionLevel.IPV6, SocketOption.IPV6_V6ONLY, true); 6307 listener.bind(address); 6308 cleanup = delegate() { 6309 listener.close(); 6310 }; 6311 tcp = true; 6312 } 6313 6314 listener.listen(backQueue); 6315 6316 if (dropPrivs !is null) // can be null, backwards compatibility 6317 dropPrivs(); 6318 6319 return listener; 6320 } 6321 6322 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6323 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6324 if(data.length == 0) return; 6325 ptrdiff_t amount; 6326 //import std.stdio; writeln("***",cast(string) data,"///"); 6327 do { 6328 amount = s.send(data); 6329 if(amount == Socket.ERROR) { 6330 version(cgi_use_fiber) { 6331 if(wouldHaveBlocked()) { 6332 bool registered = true; 6333 registerEventWakeup(®istered, s, WakeupEvent.Write); 6334 continue; 6335 } 6336 } 6337 throw new ConnectionException(s, lastSocketError, file, line); 6338 } 6339 assert(amount > 0); 6340 6341 data = data[amount .. $]; 6342 } while(data.length); 6343 } 6344 6345 class ConnectionException : Exception { 6346 Socket socket; 6347 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6348 this.socket = s; 6349 super(msg, file, line); 6350 } 6351 } 6352 6353 class HttpVersionNotSupportedException : Exception { 6354 this(string file = __FILE__, size_t line = __LINE__) { 6355 super("HTTP Version Not Supported", file, line); 6356 } 6357 } 6358 6359 alias void delegate(Socket) CMT; 6360 6361 import core.thread; 6362 /+ 6363 cgi.d now uses a hybrid of event i/o and threads at the top level. 6364 6365 Top level thread is responsible for accepting sockets and selecting on them. 6366 6367 It then indicates to a child that a request is pending, and any random worker 6368 thread that is free handles it. It goes into blocking mode and handles that 6369 http request to completion. 6370 6371 At that point, it goes back into the waiting queue. 6372 6373 6374 This concept is only implemented on Linux. On all other systems, it still 6375 uses the worker threads and semaphores (which is perfectly fine for a lot of 6376 things! Just having a great number of keep-alive connections will break that.) 6377 6378 6379 So the algorithm is: 6380 6381 select(accept, event, pending) 6382 if accept -> send socket to free thread, if any. if not, add socket to queue 6383 if event -> send the signaling thread a socket from the queue, if not, mark it free 6384 - event might block until it can be *written* to. it is a fifo sending socket fds! 6385 6386 A worker only does one http request at a time, then signals its availability back to the boss. 6387 6388 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6389 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6390 actual FD will not be kept out here. 6391 6392 So: 6393 queue = sockets we know are ready to read now, but no worker thread is available 6394 idle list = worker threads not doing anything else. they signal back and forth 6395 6396 the workers all read off the event fd. This is the semaphore wait 6397 6398 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6399 it puts it in the queue and writes to the event fd. 6400 6401 The child could put the socket back in the epoll thing itself. 6402 6403 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6404 +/ 6405 class ConnectionThread : Thread { 6406 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6407 this.lcm = lcm; 6408 this.dg = dg; 6409 this.myThreadNumber = myThreadNumber; 6410 super(&run); 6411 } 6412 6413 void run() { 6414 while(true) { 6415 // so if there's a bunch of idle keep-alive connections, it can 6416 // consume all the worker threads... just sitting there. 6417 lcm.semaphore.wait(); 6418 if(globalStopFlag) 6419 return; 6420 Socket socket; 6421 synchronized(lcm) { 6422 auto idx = lcm.nextIndexFront; 6423 socket = lcm.queue[idx]; 6424 lcm.queue[idx] = null; 6425 atomicOp!"+="(lcm.nextIndexFront, 1); 6426 atomicOp!"-="(lcm.queueLength, 1); 6427 } 6428 try { 6429 //import std.stdio; writeln(myThreadNumber, " taking it"); 6430 dg(socket); 6431 /+ 6432 if(socket.isAlive) { 6433 // process it more later 6434 version(linux) { 6435 import core.sys.linux.epoll; 6436 epoll_event ev; 6437 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6438 ev.data.fd = socket.handle; 6439 import std.stdio; writeln("adding"); 6440 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6441 if(errno == EEXIST) { 6442 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6443 ev.data.fd = socket.handle; 6444 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6445 throw new Exception("epoll_ctl " ~ to!string(errno)); 6446 } else 6447 throw new Exception("epoll_ctl " ~ to!string(errno)); 6448 } 6449 //import std.stdio; writeln("keep alive"); 6450 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6451 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6452 } else { 6453 continue; // hope it times out in a reasonable amount of time... 6454 } 6455 } 6456 +/ 6457 } catch(ConnectionClosedException e) { 6458 // can just ignore this, it is fairly normal 6459 socket.close(); 6460 } catch(Throwable e) { 6461 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6462 socket.close(); 6463 } 6464 } 6465 } 6466 6467 ListeningConnectionManager lcm; 6468 CMT dg; 6469 int myThreadNumber; 6470 } 6471 6472 version(cgi_use_fiber) 6473 class WorkerThread : Thread { 6474 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6475 this.lcm = lcm; 6476 this.dg = dg; 6477 this.myThreadNumber = myThreadNumber; 6478 super(&run); 6479 } 6480 6481 version(Windows) 6482 void run() { 6483 auto timeout = INFINITE; 6484 PseudoblockingOverlappedSocket key; 6485 OVERLAPPED* overlapped; 6486 DWORD bytes; 6487 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6488 if(key is null) 6489 continue; 6490 key.lastAnswer = bytes; 6491 if(key.fiber) { 6492 key.fiber.proceed(); 6493 } else { 6494 // we have a new connection, issue the first receive on it and issue the next accept 6495 6496 auto sn = key.accepted; 6497 6498 key.accept(); 6499 6500 cloexec(sn); 6501 if(lcm.tcp) { 6502 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6503 // on the socket because we do some buffering internally. I think this helps, 6504 // certainly does for small requests, and I think it does for larger ones too 6505 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6506 6507 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6508 } 6509 6510 dg(sn); 6511 } 6512 } 6513 //SleepEx(INFINITE, TRUE); 6514 } 6515 6516 version(linux) 6517 void run() { 6518 6519 import core.sys.linux.epoll; 6520 epfd = epoll_create1(EPOLL_CLOEXEC); 6521 if(epfd == -1) 6522 throw new Exception("epoll_create1 " ~ to!string(errno)); 6523 scope(exit) { 6524 import core.sys.posix.unistd; 6525 close(epfd); 6526 } 6527 6528 { 6529 epoll_event ev; 6530 ev.events = EPOLLIN; 6531 ev.data.fd = cancelfd; 6532 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6533 } 6534 6535 foreach(listener; lcm.listeners) { 6536 epoll_event ev; 6537 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6538 ev.data.fd = listener.handle; 6539 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6540 throw new Exception("epoll_ctl " ~ to!string(errno)); 6541 } 6542 6543 6544 6545 while(!globalStopFlag) { 6546 Socket sn; 6547 6548 epoll_event[64] events; 6549 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6550 if(nfds == -1) { 6551 if(errno == EINTR) 6552 continue; 6553 throw new Exception("epoll_wait " ~ to!string(errno)); 6554 } 6555 6556 outer: foreach(idx; 0 .. nfds) { 6557 auto flags = events[idx].events; 6558 6559 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6560 globalStopFlag = true; 6561 //import std.stdio; writeln("exit heard"); 6562 break; 6563 } else { 6564 foreach(listener; lcm.listeners) { 6565 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6566 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6567 // this try/catch is because it is set to non-blocking mode 6568 // and Phobos' stupid api throws an exception instead of returning 6569 // if it would block. Why would it block? because a forked process 6570 // might have beat us to it, but the wakeup event thundered our herds. 6571 try 6572 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6573 catch(SocketAcceptException e) { continue outer; } 6574 6575 cloexec(sn); 6576 if(lcm.tcp) { 6577 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6578 // on the socket because we do some buffering internally. I think this helps, 6579 // certainly does for small requests, and I think it does for larger ones too 6580 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6581 6582 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6583 } 6584 6585 dg(sn); 6586 continue outer; 6587 } else { 6588 // writeln(events[idx].data.ptr); 6589 } 6590 } 6591 6592 if(cast(size_t) events[idx].data.ptr < 1024) { 6593 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6594 } 6595 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6596 fiber.proceed(); 6597 } 6598 } 6599 } 6600 } 6601 6602 ListeningConnectionManager lcm; 6603 CMT dg; 6604 int myThreadNumber; 6605 } 6606 6607 6608 /* Done with network helper */ 6609 6610 /* Helpers for doing temporary files. Used both here and in web.d */ 6611 6612 version(Windows) { 6613 import core.sys.windows.windows; 6614 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6615 alias GetTempPathW GetTempPath; 6616 } 6617 6618 version(Posix) { 6619 static import linux = core.sys.posix.unistd; 6620 } 6621 6622 string getTempDirectory() { 6623 string path; 6624 version(Windows) { 6625 wchar[1024] buffer; 6626 auto len = GetTempPath(1024, buffer.ptr); 6627 if(len == 0) 6628 throw new Exception("couldn't find a temporary path"); 6629 6630 auto b = buffer[0 .. len]; 6631 6632 path = to!string(b); 6633 } else 6634 path = "/tmp/"; 6635 6636 return path; 6637 } 6638 6639 6640 // I like std.date. These functions help keep my old code and data working with phobos changing. 6641 6642 long sysTimeToDTime(in SysTime sysTime) { 6643 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6644 } 6645 6646 long dateTimeToDTime(in DateTime dt) { 6647 return sysTimeToDTime(cast(SysTime) dt); 6648 } 6649 6650 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6651 return sysTimeToDTime(Clock.currTime(UTC())); 6652 } 6653 6654 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6655 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6656 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6657 return SysTime(hnsecs, tz); 6658 } 6659 6660 6661 6662 // this is a helper to read HTTP transfer-encoding: chunked responses 6663 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6664 immutable(ubyte)[] ret; 6665 6666 another_chunk: 6667 // If here, we are at the beginning of a chunk. 6668 auto a = ir.front(); 6669 int chunkSize; 6670 int loc = locationOf(a, "\r\n"); 6671 while(loc == -1) { 6672 ir.popFront(); 6673 a = ir.front(); 6674 loc = locationOf(a, "\r\n"); 6675 } 6676 6677 string hex; 6678 hex = ""; 6679 for(int i = 0; i < loc; i++) { 6680 char c = a[i]; 6681 if(c >= 'A' && c <= 'Z') 6682 c += 0x20; 6683 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6684 hex ~= c; 6685 } else { 6686 break; 6687 } 6688 } 6689 6690 assert(hex.length); 6691 6692 int power = 1; 6693 int size = 0; 6694 foreach(cc1; retro(hex)) { 6695 dchar cc = cc1; 6696 if(cc >= 'a' && cc <= 'z') 6697 cc -= 0x20; 6698 int val = 0; 6699 if(cc >= '0' && cc <= '9') 6700 val = cc - '0'; 6701 else 6702 val = cc - 'A' + 10; 6703 6704 size += power * val; 6705 power *= 16; 6706 } 6707 6708 chunkSize = size; 6709 assert(size >= 0); 6710 6711 if(loc + 2 > a.length) { 6712 ir.popFront(0, a.length + loc + 2); 6713 a = ir.front(); 6714 } 6715 6716 a = ir.consume(loc + 2); 6717 6718 if(chunkSize == 0) { // we're done with the response 6719 // if we got here, will change must be true.... 6720 more_footers: 6721 loc = locationOf(a, "\r\n"); 6722 if(loc == -1) { 6723 ir.popFront(); 6724 a = ir.front; 6725 goto more_footers; 6726 } else { 6727 assert(loc == 0); 6728 ir.consume(loc + 2); 6729 goto finish; 6730 } 6731 } else { 6732 // if we got here, will change must be true.... 6733 if(a.length < chunkSize + 2) { 6734 ir.popFront(0, chunkSize + 2); 6735 a = ir.front(); 6736 } 6737 6738 ret ~= (a[0..chunkSize]); 6739 6740 if(!(a.length > chunkSize + 2)) { 6741 ir.popFront(0, chunkSize + 2); 6742 a = ir.front(); 6743 } 6744 assert(a[chunkSize] == 13); 6745 assert(a[chunkSize+1] == 10); 6746 a = ir.consume(chunkSize + 2); 6747 chunkSize = 0; 6748 goto another_chunk; 6749 } 6750 6751 finish: 6752 return ret; 6753 } 6754 6755 // I want to be able to get data from multiple sources the same way... 6756 interface ByChunkRange { 6757 bool empty(); 6758 void popFront(); 6759 const(ubyte)[] front(); 6760 } 6761 6762 ByChunkRange byChunk(const(ubyte)[] data) { 6763 return new class ByChunkRange { 6764 override bool empty() { 6765 return !data.length; 6766 } 6767 6768 override void popFront() { 6769 if(data.length > 4096) 6770 data = data[4096 .. $]; 6771 else 6772 data = null; 6773 } 6774 6775 override const(ubyte)[] front() { 6776 return data[0 .. $ > 4096 ? 4096 : $]; 6777 } 6778 }; 6779 } 6780 6781 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6782 const(ubyte)[] f; 6783 6784 f = ir.front; 6785 if(f.length > atMost) 6786 f = f[0 .. atMost]; 6787 6788 return new class ByChunkRange { 6789 override bool empty() { 6790 return atMost == 0; 6791 } 6792 6793 override const(ubyte)[] front() { 6794 return f; 6795 } 6796 6797 override void popFront() { 6798 ir.consume(f.length); 6799 atMost -= f.length; 6800 auto a = ir.front(); 6801 6802 if(a.length <= atMost) { 6803 f = a; 6804 atMost -= a.length; 6805 a = ir.consume(a.length); 6806 if(atMost != 0) 6807 ir.popFront(); 6808 if(f.length == 0) { 6809 f = ir.front(); 6810 } 6811 } else { 6812 // we actually have *more* here than we need.... 6813 f = a[0..atMost]; 6814 atMost = 0; 6815 ir.consume(atMost); 6816 } 6817 } 6818 }; 6819 } 6820 6821 version(cgi_with_websocket) { 6822 // http://tools.ietf.org/html/rfc6455 6823 6824 /++ 6825 WEBSOCKET SUPPORT: 6826 6827 Full example: 6828 --- 6829 import arsd.cgi; 6830 6831 void websocketEcho(Cgi cgi) { 6832 if(cgi.websocketRequested()) { 6833 if(cgi.origin != "http://arsdnet.net") 6834 throw new Exception("bad origin"); 6835 auto websocket = cgi.acceptWebsocket(); 6836 6837 websocket.send("hello"); 6838 websocket.send(" world!"); 6839 6840 auto msg = websocket.recv(); 6841 while(msg.opcode != WebSocketOpcode.close) { 6842 if(msg.opcode == WebSocketOpcode.text) { 6843 websocket.send(msg.textData); 6844 } else if(msg.opcode == WebSocketOpcode.binary) { 6845 websocket.send(msg.data); 6846 } 6847 6848 msg = websocket.recv(); 6849 } 6850 6851 websocket.close(); 6852 } else { 6853 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6854 } 6855 } 6856 6857 mixin GenericMain!websocketEcho; 6858 --- 6859 +/ 6860 6861 class WebSocket { 6862 Cgi cgi; 6863 6864 private this(Cgi cgi) { 6865 this.cgi = cgi; 6866 6867 Socket socket = cgi.idlol.source; 6868 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6869 } 6870 6871 // returns true if data available, false if it timed out 6872 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6873 if(!waitForNextMessageWouldBlock()) 6874 return true; 6875 if(isDataPending(timeout)) 6876 return true; // this is kinda a lie. 6877 6878 return false; 6879 } 6880 6881 public bool lowLevelReceive() { 6882 auto bfr = cgi.idlol; 6883 top: 6884 auto got = bfr.front; 6885 if(got.length) { 6886 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6887 receiveBuffer.length += receiveBufferUsedLength + got.length; 6888 6889 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6890 receiveBufferUsedLength += got.length; 6891 bfr.consume(got.length); 6892 6893 return true; 6894 } 6895 6896 if(bfr.sourceClosed) 6897 return false; 6898 6899 bfr.popFront(0); 6900 if(bfr.sourceClosed) 6901 return false; 6902 goto top; 6903 } 6904 6905 6906 bool isDataPending(Duration timeout = 0.seconds) { 6907 Socket socket = cgi.idlol.source; 6908 6909 auto check = new SocketSet(); 6910 check.add(socket); 6911 6912 auto got = Socket.select(check, null, null, timeout); 6913 if(got > 0) 6914 return true; 6915 return false; 6916 } 6917 6918 // note: this blocks 6919 WebSocketFrame recv() { 6920 return waitForNextMessage(); 6921 } 6922 6923 6924 6925 6926 private void llclose() { 6927 cgi.close(); 6928 } 6929 6930 private void llsend(ubyte[] data) { 6931 cgi.write(data); 6932 cgi.flush(); 6933 } 6934 6935 void unregisterActiveSocket(WebSocket) {} 6936 6937 /* copy/paste section { */ 6938 6939 private int readyState_; 6940 private ubyte[] receiveBuffer; 6941 private size_t receiveBufferUsedLength; 6942 6943 private Config config; 6944 6945 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6946 enum OPEN = 1; /// The connection is open and ready to communicate. 6947 enum CLOSING = 2; /// The connection is in the process of closing. 6948 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6949 6950 /++ 6951 6952 +/ 6953 /// Group: foundational 6954 static struct Config { 6955 /++ 6956 These control the size of the receive buffer. 6957 6958 It starts at the initial size, will temporarily 6959 balloon up to the maximum size, and will reuse 6960 a buffer up to the likely size. 6961 6962 Anything larger than the maximum size will cause 6963 the connection to be aborted and an exception thrown. 6964 This is to protect you against a peer trying to 6965 exhaust your memory, while keeping the user-level 6966 processing simple. 6967 +/ 6968 size_t initialReceiveBufferSize = 4096; 6969 size_t likelyReceiveBufferSize = 4096; /// ditto 6970 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6971 6972 /++ 6973 Maximum combined size of a message. 6974 +/ 6975 size_t maximumMessageSize = 10 * 1024 * 1024; 6976 6977 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6978 string origin; /// Origin URL to send with the handshake, if desired. 6979 string protocol; /// the protocol header, if desired. 6980 6981 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6982 } 6983 6984 /++ 6985 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6986 +/ 6987 int readyState() { 6988 return readyState_; 6989 } 6990 6991 /++ 6992 Closes the connection, sending a graceful teardown message to the other side. 6993 +/ 6994 /// Group: foundational 6995 void close(int code = 0, string reason = null) 6996 //in (reason.length < 123) 6997 in { assert(reason.length < 123); } do 6998 { 6999 if(readyState_ != OPEN) 7000 return; // it cool, we done 7001 WebSocketFrame wss; 7002 wss.fin = true; 7003 wss.opcode = WebSocketOpcode.close; 7004 wss.data = cast(ubyte[]) reason.dup; 7005 wss.send(&llsend); 7006 7007 readyState_ = CLOSING; 7008 7009 llclose(); 7010 } 7011 7012 /++ 7013 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 7014 +/ 7015 /// Group: foundational 7016 void ping() { 7017 WebSocketFrame wss; 7018 wss.fin = true; 7019 wss.opcode = WebSocketOpcode.ping; 7020 wss.send(&llsend); 7021 } 7022 7023 // automatically handled.... 7024 void pong() { 7025 WebSocketFrame wss; 7026 wss.fin = true; 7027 wss.opcode = WebSocketOpcode.pong; 7028 wss.send(&llsend); 7029 } 7030 7031 /++ 7032 Sends a text message through the websocket. 7033 +/ 7034 /// Group: foundational 7035 void send(in char[] textData) { 7036 WebSocketFrame wss; 7037 wss.fin = true; 7038 wss.opcode = WebSocketOpcode.text; 7039 wss.data = cast(ubyte[]) textData.dup; 7040 wss.send(&llsend); 7041 } 7042 7043 /++ 7044 Sends a binary message through the websocket. 7045 +/ 7046 /// Group: foundational 7047 void send(in ubyte[] binaryData) { 7048 WebSocketFrame wss; 7049 wss.fin = true; 7050 wss.opcode = WebSocketOpcode.binary; 7051 wss.data = cast(ubyte[]) binaryData.dup; 7052 wss.send(&llsend); 7053 } 7054 7055 /++ 7056 Waits for and returns the next complete message on the socket. 7057 7058 Note that the onmessage function is still called, right before 7059 this returns. 7060 +/ 7061 /// Group: blocking_api 7062 public WebSocketFrame waitForNextMessage() { 7063 do { 7064 auto m = processOnce(); 7065 if(m.populated) 7066 return m; 7067 } while(lowLevelReceive()); 7068 7069 throw new ConnectionClosedException("Websocket receive timed out"); 7070 //return WebSocketFrame.init; // FIXME? maybe. 7071 } 7072 7073 /++ 7074 Tells if [waitForNextMessage] would block. 7075 +/ 7076 /// Group: blocking_api 7077 public bool waitForNextMessageWouldBlock() { 7078 checkAgain: 7079 if(isMessageBuffered()) 7080 return false; 7081 if(!isDataPending()) 7082 return true; 7083 while(isDataPending()) { 7084 if(lowLevelReceive() == false) 7085 throw new ConnectionClosedException("Connection closed in middle of message"); 7086 } 7087 goto checkAgain; 7088 } 7089 7090 /++ 7091 Is there a message in the buffer already? 7092 If `true`, [waitForNextMessage] is guaranteed to return immediately. 7093 If `false`, check [isDataPending] as the next step. 7094 +/ 7095 /// Group: blocking_api 7096 public bool isMessageBuffered() { 7097 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 7098 auto s = d; 7099 if(d.length) { 7100 auto orig = d; 7101 auto m = WebSocketFrame.read(d); 7102 // that's how it indicates that it needs more data 7103 if(d !is orig) 7104 return true; 7105 } 7106 7107 return false; 7108 } 7109 7110 private ubyte continuingType; 7111 private ubyte[] continuingData; 7112 //private size_t continuingDataLength; 7113 7114 private WebSocketFrame processOnce() { 7115 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 7116 auto s = d; 7117 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 7118 WebSocketFrame m; 7119 if(d.length) { 7120 auto orig = d; 7121 m = WebSocketFrame.read(d); 7122 // that's how it indicates that it needs more data 7123 if(d is orig) 7124 return WebSocketFrame.init; 7125 m.unmaskInPlace(); 7126 switch(m.opcode) { 7127 case WebSocketOpcode.continuation: 7128 if(continuingData.length + m.data.length > config.maximumMessageSize) 7129 throw new Exception("message size exceeded"); 7130 7131 continuingData ~= m.data; 7132 if(m.fin) { 7133 if(ontextmessage) 7134 ontextmessage(cast(char[]) continuingData); 7135 if(onbinarymessage) 7136 onbinarymessage(continuingData); 7137 7138 continuingData = null; 7139 } 7140 break; 7141 case WebSocketOpcode.text: 7142 if(m.fin) { 7143 if(ontextmessage) 7144 ontextmessage(m.textData); 7145 } else { 7146 continuingType = m.opcode; 7147 //continuingDataLength = 0; 7148 continuingData = null; 7149 continuingData ~= m.data; 7150 } 7151 break; 7152 case WebSocketOpcode.binary: 7153 if(m.fin) { 7154 if(onbinarymessage) 7155 onbinarymessage(m.data); 7156 } else { 7157 continuingType = m.opcode; 7158 //continuingDataLength = 0; 7159 continuingData = null; 7160 continuingData ~= m.data; 7161 } 7162 break; 7163 case WebSocketOpcode.close: 7164 readyState_ = CLOSED; 7165 if(onclose) 7166 onclose(); 7167 7168 unregisterActiveSocket(this); 7169 break; 7170 case WebSocketOpcode.ping: 7171 pong(); 7172 break; 7173 case WebSocketOpcode.pong: 7174 // just really references it is still alive, nbd. 7175 break; 7176 default: // ignore though i could and perhaps should throw too 7177 } 7178 } 7179 7180 // the recv thing can be invalidated so gotta copy it over ugh 7181 if(d.length) { 7182 m.data = m.data.dup(); 7183 } 7184 7185 import core.stdc.string; 7186 memmove(receiveBuffer.ptr, d.ptr, d.length); 7187 receiveBufferUsedLength = d.length; 7188 7189 return m; 7190 } 7191 7192 private void autoprocess() { 7193 // FIXME 7194 do { 7195 processOnce(); 7196 } while(lowLevelReceive()); 7197 } 7198 7199 7200 void delegate() onclose; /// 7201 void delegate() onerror; /// 7202 void delegate(in char[]) ontextmessage; /// 7203 void delegate(in ubyte[]) onbinarymessage; /// 7204 void delegate() onopen; /// 7205 7206 /++ 7207 7208 +/ 7209 /// Group: browser_api 7210 void onmessage(void delegate(in char[]) dg) { 7211 ontextmessage = dg; 7212 } 7213 7214 /// ditto 7215 void onmessage(void delegate(in ubyte[]) dg) { 7216 onbinarymessage = dg; 7217 } 7218 7219 /* } end copy/paste */ 7220 7221 7222 } 7223 7224 /++ 7225 Returns true if the request headers are asking for a websocket upgrade. 7226 7227 If this returns true, and you want to accept it, call [acceptWebsocket]. 7228 +/ 7229 bool websocketRequested(Cgi cgi) { 7230 return 7231 "sec-websocket-key" in cgi.requestHeaders 7232 && 7233 "connection" in cgi.requestHeaders && 7234 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 7235 && 7236 "upgrade" in cgi.requestHeaders && 7237 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 7238 ; 7239 } 7240 7241 /++ 7242 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 7243 +/ 7244 WebSocket acceptWebsocket(Cgi cgi) { 7245 assert(!cgi.closed); 7246 assert(!cgi.outputtedResponseData); 7247 cgi.setResponseStatus("101 Switching Protocols"); 7248 cgi.header("Upgrade: WebSocket"); 7249 cgi.header("Connection: upgrade"); 7250 7251 string key = cgi.requestHeaders["sec-websocket-key"]; 7252 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 7253 7254 import std.digest.sha; 7255 auto hash = sha1Of(key); 7256 auto accept = Base64.encode(hash); 7257 7258 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 7259 7260 cgi.websocketMode = true; 7261 cgi.write(""); 7262 7263 cgi.flush(); 7264 7265 return new WebSocket(cgi); 7266 } 7267 7268 // FIXME get websocket to work on other modes, not just embedded_httpd 7269 7270 /* copy/paste in http2.d { */ 7271 enum WebSocketOpcode : ubyte { 7272 continuation = 0, 7273 text = 1, 7274 binary = 2, 7275 // 3, 4, 5, 6, 7 RESERVED 7276 close = 8, 7277 ping = 9, 7278 pong = 10, 7279 // 11,12,13,14,15 RESERVED 7280 } 7281 7282 public struct WebSocketFrame { 7283 private bool populated; 7284 bool fin; 7285 bool rsv1; 7286 bool rsv2; 7287 bool rsv3; 7288 WebSocketOpcode opcode; // 4 bits 7289 bool masked; 7290 ubyte lengthIndicator; // don't set this when building one to send 7291 ulong realLength; // don't use when sending 7292 ubyte[4] maskingKey; // don't set this when sending 7293 ubyte[] data; 7294 7295 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7296 WebSocketFrame msg; 7297 msg.fin = true; 7298 msg.opcode = opcode; 7299 msg.data = cast(ubyte[]) data.dup; 7300 7301 return msg; 7302 } 7303 7304 private void send(scope void delegate(ubyte[]) llsend) { 7305 ubyte[64] headerScratch; 7306 int headerScratchPos = 0; 7307 7308 realLength = data.length; 7309 7310 { 7311 ubyte b1; 7312 b1 |= cast(ubyte) opcode; 7313 b1 |= rsv3 ? (1 << 4) : 0; 7314 b1 |= rsv2 ? (1 << 5) : 0; 7315 b1 |= rsv1 ? (1 << 6) : 0; 7316 b1 |= fin ? (1 << 7) : 0; 7317 7318 headerScratch[0] = b1; 7319 headerScratchPos++; 7320 } 7321 7322 { 7323 headerScratchPos++; // we'll set header[1] at the end of this 7324 auto rlc = realLength; 7325 ubyte b2; 7326 b2 |= masked ? (1 << 7) : 0; 7327 7328 assert(headerScratchPos == 2); 7329 7330 if(realLength > 65535) { 7331 // use 64 bit length 7332 b2 |= 0x7f; 7333 7334 // FIXME: double check endinaness 7335 foreach(i; 0 .. 8) { 7336 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7337 rlc >>>= 8; 7338 } 7339 7340 headerScratchPos += 8; 7341 } else if(realLength > 125) { 7342 // use 16 bit length 7343 b2 |= 0x7e; 7344 7345 // FIXME: double check endinaness 7346 foreach(i; 0 .. 2) { 7347 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7348 rlc >>>= 8; 7349 } 7350 7351 headerScratchPos += 2; 7352 } else { 7353 // use 7 bit length 7354 b2 |= realLength & 0b_0111_1111; 7355 } 7356 7357 headerScratch[1] = b2; 7358 } 7359 7360 //assert(!masked, "masking key not properly implemented"); 7361 if(masked) { 7362 // FIXME: randomize this 7363 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7364 headerScratchPos += 4; 7365 7366 // we'll just mask it in place... 7367 int keyIdx = 0; 7368 foreach(i; 0 .. data.length) { 7369 data[i] = data[i] ^ maskingKey[keyIdx]; 7370 if(keyIdx == 3) 7371 keyIdx = 0; 7372 else 7373 keyIdx++; 7374 } 7375 } 7376 7377 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7378 llsend(headerScratch[0 .. headerScratchPos]); 7379 llsend(data); 7380 } 7381 7382 static WebSocketFrame read(ref ubyte[] d) { 7383 WebSocketFrame msg; 7384 7385 auto orig = d; 7386 7387 WebSocketFrame needsMoreData() { 7388 d = orig; 7389 return WebSocketFrame.init; 7390 } 7391 7392 if(d.length < 2) 7393 return needsMoreData(); 7394 7395 ubyte b = d[0]; 7396 7397 msg.populated = true; 7398 7399 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7400 b >>= 4; 7401 msg.rsv3 = b & 0x01; 7402 b >>= 1; 7403 msg.rsv2 = b & 0x01; 7404 b >>= 1; 7405 msg.rsv1 = b & 0x01; 7406 b >>= 1; 7407 msg.fin = b & 0x01; 7408 7409 b = d[1]; 7410 msg.masked = (b & 0b1000_0000) ? true : false; 7411 msg.lengthIndicator = b & 0b0111_1111; 7412 7413 d = d[2 .. $]; 7414 7415 if(msg.lengthIndicator == 0x7e) { 7416 // 16 bit length 7417 msg.realLength = 0; 7418 7419 if(d.length < 2) return needsMoreData(); 7420 7421 foreach(i; 0 .. 2) { 7422 msg.realLength |= d[0] << ((1-i) * 8); 7423 d = d[1 .. $]; 7424 } 7425 } else if(msg.lengthIndicator == 0x7f) { 7426 // 64 bit length 7427 msg.realLength = 0; 7428 7429 if(d.length < 8) return needsMoreData(); 7430 7431 foreach(i; 0 .. 8) { 7432 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7433 d = d[1 .. $]; 7434 } 7435 } else { 7436 // 7 bit length 7437 msg.realLength = msg.lengthIndicator; 7438 } 7439 7440 if(msg.masked) { 7441 7442 if(d.length < 4) return needsMoreData(); 7443 7444 msg.maskingKey = d[0 .. 4]; 7445 d = d[4 .. $]; 7446 } 7447 7448 if(msg.realLength > d.length) { 7449 return needsMoreData(); 7450 } 7451 7452 msg.data = d[0 .. cast(size_t) msg.realLength]; 7453 d = d[cast(size_t) msg.realLength .. $]; 7454 7455 return msg; 7456 } 7457 7458 void unmaskInPlace() { 7459 if(this.masked) { 7460 int keyIdx = 0; 7461 foreach(i; 0 .. this.data.length) { 7462 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7463 if(keyIdx == 3) 7464 keyIdx = 0; 7465 else 7466 keyIdx++; 7467 } 7468 } 7469 } 7470 7471 char[] textData() { 7472 return cast(char[]) data; 7473 } 7474 } 7475 /* } */ 7476 } 7477 7478 7479 version(Windows) 7480 { 7481 version(CRuntime_DigitalMars) 7482 { 7483 extern(C) int setmode(int, int) nothrow @nogc; 7484 } 7485 else version(CRuntime_Microsoft) 7486 { 7487 extern(C) int _setmode(int, int) nothrow @nogc; 7488 alias setmode = _setmode; 7489 } 7490 else static assert(0); 7491 } 7492 7493 version(Posix) { 7494 import core.sys.posix.unistd; 7495 version(CRuntime_Musl) {} else { 7496 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7497 } 7498 } 7499 7500 7501 // FIXME: these aren't quite public yet. 7502 //private: 7503 7504 // template for laziness 7505 void startAddonServer()(string arg) { 7506 version(OSX) { 7507 assert(0, "Not implemented"); 7508 } else version(linux) { 7509 import core.sys.posix.unistd; 7510 pid_t pid; 7511 const(char)*[16] args; 7512 args[0] = "ARSD_CGI_ADDON_SERVER"; 7513 args[1] = arg.ptr; 7514 posix_spawn(&pid, "/proc/self/exe", 7515 null, 7516 null, 7517 args.ptr, 7518 null // env 7519 ); 7520 } else version(Windows) { 7521 wchar[2048] filename; 7522 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7523 if(len == 0 || len == filename.length) 7524 throw new Exception("could not get process name to start helper server"); 7525 7526 STARTUPINFOW startupInfo; 7527 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7528 PROCESS_INFORMATION processInfo; 7529 7530 import std.utf; 7531 7532 // I *MIGHT* need to run it as a new job or a service... 7533 auto ret = CreateProcessW( 7534 filename.ptr, 7535 toUTF16z(arg), 7536 null, // process attributes 7537 null, // thread attributes 7538 false, // inherit handles 7539 0, // creation flags 7540 null, // environment 7541 null, // working directory 7542 &startupInfo, 7543 &processInfo 7544 ); 7545 7546 if(!ret) 7547 throw new Exception("create process failed"); 7548 7549 // when done with those, if we set them 7550 /* 7551 CloseHandle(hStdInput); 7552 CloseHandle(hStdOutput); 7553 CloseHandle(hStdError); 7554 */ 7555 7556 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7557 } 7558 7559 // template for laziness 7560 /* 7561 The websocket server is a single-process, single-thread, event 7562 I/O thing. It is passed websockets from other CGI processes 7563 and is then responsible for handling their messages and responses. 7564 Note that the CGI process is responsible for websocket setup, 7565 including authentication, etc. 7566 7567 It also gets data sent to it by other processes and is responsible 7568 for distributing that, as necessary. 7569 */ 7570 void runWebsocketServer()() { 7571 assert(0, "not implemented"); 7572 } 7573 7574 void sendToWebsocketServer(WebSocket ws, string group) { 7575 assert(0, "not implemented"); 7576 } 7577 7578 void sendToWebsocketServer(string content, string group) { 7579 assert(0, "not implemented"); 7580 } 7581 7582 7583 void runEventServer()() { 7584 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7585 } 7586 7587 void runTimerServer()() { 7588 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7589 } 7590 7591 version(Posix) { 7592 alias LocalServerConnectionHandle = int; 7593 alias CgiConnectionHandle = int; 7594 alias SocketConnectionHandle = int; 7595 7596 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7597 } else version(Windows) { 7598 alias LocalServerConnectionHandle = HANDLE; 7599 version(embedded_httpd_threads) { 7600 alias CgiConnectionHandle = SOCKET; 7601 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7602 } else version(fastcgi) { 7603 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7604 enum INVALID_CGI_CONNECTION_HANDLE = null; 7605 } else version(scgi) { 7606 alias CgiConnectionHandle = SOCKET; 7607 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7608 } else { /* version(plain_cgi) */ 7609 alias CgiConnectionHandle = HANDLE; 7610 enum INVALID_CGI_CONNECTION_HANDLE = null; 7611 } 7612 alias SocketConnectionHandle = SOCKET; 7613 } 7614 7615 version(with_addon_servers_connections) 7616 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7617 version(Posix) { 7618 import core.sys.posix.unistd; 7619 import core.sys.posix.sys.un; 7620 7621 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7622 if(sock == -1) 7623 throw new Exception("socket " ~ to!string(errno)); 7624 7625 scope(failure) 7626 close(sock); 7627 7628 cloexec(sock); 7629 7630 // add-on server processes are assumed to be local, and thus will 7631 // use unix domain sockets. Besides, I want to pass sockets to them, 7632 // so it basically must be local (except for the session server, but meh). 7633 sockaddr_un addr; 7634 addr.sun_family = AF_UNIX; 7635 version(linux) { 7636 // on linux, we will use the abstract namespace 7637 addr.sun_path[0] = 0; 7638 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7639 } else { 7640 // but otherwise, just use a file cuz we must. 7641 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7642 } 7643 7644 bool alreadyTried; 7645 7646 try_again: 7647 7648 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7649 if(!alreadyTried && errno == ECONNREFUSED) { 7650 // try auto-spawning the server, then attempt connection again 7651 startAddonServer(arg); 7652 import core.thread; 7653 Thread.sleep(50.msecs); 7654 alreadyTried = true; 7655 goto try_again; 7656 } else 7657 throw new Exception("connect " ~ to!string(errno)); 7658 } 7659 7660 return sock; 7661 } else version(Windows) { 7662 return null; // FIXME 7663 } 7664 } 7665 7666 version(with_addon_servers_connections) 7667 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7668 version(Posix) { 7669 import core.sys.posix.unistd; 7670 close(handle); 7671 } else version(Windows) 7672 CloseHandle(handle); 7673 } 7674 7675 void runSessionServer()() { 7676 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7677 } 7678 7679 import core.stdc.errno; 7680 7681 struct IoOp { 7682 @disable this(); 7683 @disable this(this); 7684 7685 /* 7686 So we want to be able to eventually handle generic sockets too. 7687 */ 7688 7689 enum Read = 1; 7690 enum Write = 2; 7691 enum Accept = 3; 7692 enum ReadSocketHandle = 4; 7693 7694 // Your handler may be called in a different thread than the one that initiated the IO request! 7695 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7696 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7697 private void delegate(IoOp*) closeHandler; 7698 private void delegate(IoOp*) completeHandler; 7699 private int internalFd; 7700 private int operation; 7701 private int bufferLengthAllocated; 7702 private int bufferLengthUsed; 7703 private ubyte[1] internalBuffer; // it can be overallocated! 7704 7705 ubyte[] allocatedBuffer() return { 7706 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7707 } 7708 7709 ubyte[] usedBuffer() return { 7710 return allocatedBuffer[0 .. bufferLengthUsed]; 7711 } 7712 7713 void reset() { 7714 bufferLengthUsed = 0; 7715 } 7716 7717 int fd() { 7718 return internalFd; 7719 } 7720 } 7721 7722 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7723 import core.stdc.stdlib; 7724 7725 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7726 if(ptr is null) 7727 assert(0); // out of memory! 7728 7729 auto op = cast(IoOp*) ptr; 7730 7731 op.handler = handler; 7732 op.internalFd = fd; 7733 op.operation = operation; 7734 op.bufferLengthAllocated = bufferSize; 7735 op.bufferLengthUsed = 0; 7736 7737 import core.memory; 7738 7739 GC.addRoot(ptr); 7740 7741 return op; 7742 } 7743 7744 void freeIoOp(ref IoOp* ptr) { 7745 7746 import core.memory; 7747 GC.removeRoot(ptr); 7748 7749 import core.stdc.stdlib; 7750 free(ptr); 7751 ptr = null; 7752 } 7753 7754 version(Posix) 7755 version(with_addon_servers_connections) 7756 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7757 7758 //import std.stdio : writeln; writeln(cast(string) data); 7759 7760 import core.sys.posix.unistd; 7761 7762 auto ret = write(connection, data.ptr, data.length); 7763 if(ret != data.length) { 7764 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7765 // the file is closed, remove it 7766 eis.fileClosed(connection); 7767 } else 7768 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7769 } 7770 } 7771 version(Windows) 7772 version(with_addon_servers_connections) 7773 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7774 // FIXME 7775 } 7776 7777 bool isInvalidHandle(CgiConnectionHandle h) { 7778 return h == INVALID_CGI_CONNECTION_HANDLE; 7779 } 7780 7781 /+ 7782 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7783 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7784 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7785 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7786 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7787 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7788 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7789 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7790 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7791 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7792 7793 +/ 7794 7795 /++ 7796 You can customize your server by subclassing the appropriate server. Then, register your 7797 subclass at compile time with the [registerEventIoServer] template, or implement your own 7798 main function and call it yourself. 7799 7800 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7801 +/ 7802 version(with_addon_servers_connections) 7803 interface EventIoServer { 7804 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7805 void handleLocalConnectionClose(IoOp* op); 7806 void handleLocalConnectionComplete(IoOp* op); 7807 void wait_timeout(); 7808 void fileClosed(int fd); 7809 7810 void epoll_fd(int fd); 7811 } 7812 7813 // the sink should buffer it 7814 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7815 static if(is(T == struct)) { 7816 foreach(member; __traits(allMembers, T)) 7817 serialize(sink, __traits(getMember, t, member)); 7818 } else static if(is(T : int)) { 7819 // no need to think of endianness just because this is only used 7820 // for local, same-machine stuff anyway. thanks private lol 7821 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7822 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7823 // these are common enough to optimize 7824 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7825 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7826 sink(cast(ubyte[]) t[]); 7827 } else static if(is(T : A[], A)) { 7828 // generic array is less optimal but still prolly ok 7829 int len = cast(int) t.length; 7830 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7831 foreach(item; t) 7832 serialize(sink, item); 7833 } else static assert(0, T.stringof); 7834 } 7835 7836 // all may be stack buffers, so use cautio 7837 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7838 static if(is(T == struct)) { 7839 T t; 7840 foreach(member; __traits(allMembers, T)) 7841 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7842 dg(t); 7843 } else static if(is(T : int)) { 7844 // no need to think of endianness just because this is only used 7845 // for local, same-machine stuff anyway. thanks private lol 7846 T t; 7847 auto data = get(t.sizeof); 7848 t = (cast(T[]) data)[0]; 7849 dg(t); 7850 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7851 // these are common enough to optimize 7852 int len; 7853 auto data = get(len.sizeof); 7854 len = (cast(int[]) data)[0]; 7855 7856 /* 7857 typeof(T[0])[2000] stackBuffer; 7858 T buffer; 7859 7860 if(len < stackBuffer.length) 7861 buffer = stackBuffer[0 .. len]; 7862 else 7863 buffer = new T(len); 7864 7865 data = get(len * typeof(T[0]).sizeof); 7866 */ 7867 7868 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7869 7870 dg(t); 7871 } else static if(is(T == E[], E)) { 7872 T t; 7873 int len; 7874 auto data = get(len.sizeof); 7875 len = (cast(int[]) data)[0]; 7876 t.length = len; 7877 foreach(ref e; t) { 7878 deserialize!E(get, (ele) { e = ele; }); 7879 } 7880 dg(t); 7881 } else static assert(0, T.stringof); 7882 } 7883 7884 unittest { 7885 serialize((ubyte[] b) { 7886 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7887 }, 1); 7888 serialize((ubyte[] b) { 7889 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7890 }, 56674); 7891 ubyte[1000] buffer; 7892 int bufferPoint; 7893 void add(scope ubyte[] b) { 7894 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7895 bufferPoint += b.length; 7896 } 7897 ubyte[] get(int sz) { 7898 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7899 bufferPoint += sz; 7900 return b; 7901 } 7902 serialize(&add, "test here"); 7903 bufferPoint = 0; 7904 deserialize!string(&get, (t) { assert(t == "test here"); }); 7905 bufferPoint = 0; 7906 7907 struct Foo { 7908 int a; 7909 ubyte c; 7910 string d; 7911 } 7912 serialize(&add, Foo(403, 37, "amazing")); 7913 bufferPoint = 0; 7914 deserialize!Foo(&get, (t) { 7915 assert(t.a == 403); 7916 assert(t.c == 37); 7917 assert(t.d == "amazing"); 7918 }); 7919 bufferPoint = 0; 7920 } 7921 7922 /* 7923 Here's the way the RPC interface works: 7924 7925 You define the interface that lists the functions you can call on the remote process. 7926 The interface may also have static methods for convenience. These forward to a singleton 7927 instance of an auto-generated class, which actually sends the args over the pipe. 7928 7929 An impl class actually implements it. A receiving server deserializes down the pipe and 7930 calls methods on the class. 7931 7932 I went with the interface to get some nice compiler checking and documentation stuff. 7933 7934 I could have skipped the interface and just implemented it all from the server class definition 7935 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7936 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7937 7938 7939 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7940 7941 Realistically though the bodies would just be 7942 connection.call(this.mangleof, args...) sooooo. 7943 7944 FIXME: overloads aren't supported 7945 */ 7946 7947 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7948 interface SessionObject {} 7949 7950 private immutable void delegate(string[])[string] scheduledJobHandlers; 7951 private immutable void delegate(string[])[string] websocketServers; 7952 7953 version(with_breaking_cgi_features) 7954 mixin(q{ 7955 7956 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7957 static import std.traits; 7958 7959 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7960 static foreach(idx, member; __traits(derivedMembers, T)) { 7961 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7962 mixin( q{ 7963 std.traits.ReturnType!(__traits(getMember, T, member)) 7964 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7965 { 7966 SerializationBuffer buffer; 7967 auto i = cast(ushort) idx; 7968 serialize(&buffer.sink, i); 7969 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7970 foreach(param; params) 7971 serialize(&buffer.sink, param); 7972 7973 auto sendable = buffer.sendable; 7974 7975 version(Posix) {{ 7976 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7977 7978 if(ret == -1) { 7979 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7980 } else if(ret == 0) { 7981 throw new Exception("Connection to addon server lost"); 7982 } if(ret < sendable.length) 7983 throw new Exception("Send failed to send all"); 7984 assert(ret == sendable.length); 7985 }} // FIXME Windows impl 7986 7987 static if(!is(typeof(return) == void)) { 7988 // there is a return value; we need to wait for it too 7989 version(Posix) { 7990 ubyte[3000] revBuffer; 7991 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7992 auto got = revBuffer[0 .. ret]; 7993 7994 int dataLocation; 7995 ubyte[] grab(int sz) { 7996 auto dataLocation1 = dataLocation; 7997 dataLocation += sz; 7998 return got[dataLocation1 .. dataLocation]; 7999 } 8000 8001 typeof(return) retu; 8002 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 8003 return retu; 8004 } else { 8005 // FIXME Windows impl 8006 return typeof(return).init; 8007 } 8008 8009 } 8010 }}); 8011 } 8012 8013 private static typeof(this) singletonInstance; 8014 private LocalServerConnectionHandle connectionHandle; 8015 8016 static typeof(this) connection() { 8017 if(singletonInstance is null) { 8018 singletonInstance = new typeof(this)(); 8019 singletonInstance.connect(); 8020 } 8021 return singletonInstance; 8022 } 8023 8024 void connect() { 8025 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 8026 } 8027 8028 void disconnect() { 8029 closeLocalServerConnection(connectionHandle); 8030 } 8031 } 8032 8033 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 8034 ushort calledIdx; 8035 string calledFunction; 8036 8037 int dataLocation; 8038 ubyte[] grab(int sz) { 8039 if(sz == 0) assert(0); 8040 auto d = data[dataLocation .. dataLocation + sz]; 8041 dataLocation += sz; 8042 return d; 8043 } 8044 8045 again: 8046 8047 deserialize!ushort(&grab, (a) { calledIdx = a; }); 8048 deserialize!string(&grab, (a) { calledFunction = a; }); 8049 8050 import std.traits; 8051 8052 sw: switch(calledIdx) { 8053 foreach(idx, memberName; __traits(derivedMembers, Interface)) 8054 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 8055 case idx: 8056 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 8057 8058 Parameters!(__traits(getMember, Interface, memberName)) params; 8059 foreach(ref param; params) 8060 deserialize!(typeof(param))(&grab, (a) { param = a; }); 8061 8062 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 8063 __traits(getMember, this_, memberName)(params); 8064 } else { 8065 auto ret = __traits(getMember, this_, memberName)(params); 8066 SerializationBuffer buffer; 8067 serialize(&buffer.sink, ret); 8068 8069 auto sendable = buffer.sendable; 8070 8071 version(Posix) { 8072 auto r = send(fd, sendable.ptr, sendable.length, 0); 8073 if(r == -1) { 8074 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 8075 } else if(r == 0) { 8076 throw new Exception("Connection to addon client lost"); 8077 } if(r < sendable.length) 8078 throw new Exception("Send failed to send all"); 8079 8080 } // FIXME Windows impl 8081 } 8082 break sw; 8083 } 8084 default: assert(0); 8085 } 8086 8087 if(dataLocation != data.length) 8088 goto again; 8089 } 8090 8091 8092 private struct SerializationBuffer { 8093 ubyte[2048] bufferBacking; 8094 int bufferLocation; 8095 void sink(scope ubyte[] data) { 8096 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 8097 bufferLocation += data.length; 8098 } 8099 8100 ubyte[] sendable() return { 8101 return bufferBacking[0 .. bufferLocation]; 8102 } 8103 } 8104 8105 /* 8106 FIXME: 8107 add a version command line arg 8108 version data in the library 8109 management gui as external program 8110 8111 at server with event_fd for each run 8112 use .mangleof in the at function name 8113 8114 i think the at server will have to: 8115 pipe args to the child 8116 collect child output for logging 8117 get child return value for logging 8118 8119 on windows timers work differently. idk how to best combine with the io stuff. 8120 8121 will have to have dump and restore too, so i can restart without losing stuff. 8122 */ 8123 8124 /++ 8125 A convenience object for talking to the [BasicDataServer] from a higher level. 8126 See: [Cgi.getSessionObject]. 8127 8128 You pass it a `Data` struct describing the data you want saved in the session. 8129 Then, this class will generate getter and setter properties that allow access 8130 to that data. 8131 8132 Note that each load and store will be done as-accessed; it doesn't front-load 8133 mutable data nor does it batch updates out of fear of read-modify-write race 8134 conditions. (In fact, right now it does this for everything, but in the future, 8135 I might batch load `immutable` members of the Data struct.) 8136 8137 At some point in the future, I might also let it do different backends, like 8138 a client-side cookie store too, but idk. 8139 8140 Note that the plain-old-data members of your `Data` struct are wrapped by this 8141 interface via a static foreach to make property functions. 8142 8143 See_Also: [MockSession] 8144 +/ 8145 interface Session(Data) : SessionObject { 8146 @property string sessionId() const; 8147 8148 /++ 8149 Starts a new session. Note that a session is also 8150 implicitly started as soon as you write data to it, 8151 so if you need to alter these parameters from their 8152 defaults, be sure to explicitly call this BEFORE doing 8153 any writes to session data. 8154 8155 Params: 8156 idleLifetime = How long, in seconds, the session 8157 should remain in memory when not being read from 8158 or written to. The default is one day. 8159 8160 NOT IMPLEMENTED 8161 8162 useExtendedLifetimeCookie = The session ID is always 8163 stored in a HTTP cookie, and by default, that cookie 8164 is discarded when the user closes their browser. 8165 8166 But if you set this to true, it will use a non-perishable 8167 cookie for the given idleLifetime. 8168 8169 NOT IMPLEMENTED 8170 +/ 8171 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 8172 8173 /++ 8174 Regenerates the session ID and updates the associated 8175 cookie. 8176 8177 This is also your chance to change immutable data 8178 (not yet implemented). 8179 +/ 8180 void regenerateId(); 8181 8182 /++ 8183 Terminates this session, deleting all saved data. 8184 +/ 8185 void terminate(); 8186 8187 /++ 8188 Plain-old-data members of your `Data` struct are wrapped here via 8189 the property getters and setters. 8190 8191 If the member is a non-string array, it returns a magical array proxy 8192 object which allows for atomic appends and replaces via overloaded operators. 8193 You can slice this to get a range representing a $(B const) view of the array. 8194 This is to protect you against read-modify-write race conditions. 8195 +/ 8196 static foreach(memberName; __traits(allMembers, Data)) 8197 static if(is(typeof(__traits(getMember, Data, memberName)))) 8198 mixin(q{ 8199 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 8200 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 8201 }); 8202 8203 } 8204 8205 /++ 8206 An implementation of [Session] that works on real cgi connections utilizing the 8207 [BasicDataServer]. 8208 8209 As opposed to a [MockSession] which is made for testing purposes. 8210 8211 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 8212 +/ 8213 class BasicDataServerSession(Data) : Session!Data { 8214 private Cgi cgi; 8215 private string sessionId_; 8216 8217 public @property string sessionId() const { 8218 return sessionId_; 8219 } 8220 8221 protected @property string sessionId(string s) { 8222 return this.sessionId_ = s; 8223 } 8224 8225 private this(Cgi cgi) { 8226 this.cgi = cgi; 8227 if(auto ptr = "sessionId" in cgi.cookies) 8228 sessionId = (*ptr).length ? *ptr : null; 8229 } 8230 8231 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 8232 assert(sessionId is null); 8233 8234 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 8235 8236 import std.random, std.conv; 8237 sessionId = to!string(uniform(1, long.max)); 8238 8239 BasicDataServer.connection.createSession(sessionId, idleLifetime); 8240 setCookie(); 8241 } 8242 8243 protected void setCookie() { 8244 cgi.setCookie( 8245 "sessionId", sessionId, 8246 0 /* expiration */, 8247 "/" /* path */, 8248 null /* domain */, 8249 true /* http only */, 8250 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 8251 } 8252 8253 void regenerateId() { 8254 if(sessionId is null) { 8255 start(); 8256 return; 8257 } 8258 import std.random, std.conv; 8259 auto oldSessionId = sessionId; 8260 sessionId = to!string(uniform(1, long.max)); 8261 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 8262 setCookie(); 8263 } 8264 8265 void terminate() { 8266 BasicDataServer.connection.destroySession(sessionId); 8267 sessionId = null; 8268 setCookie(); 8269 } 8270 8271 static foreach(memberName; __traits(allMembers, Data)) 8272 static if(is(typeof(__traits(getMember, Data, memberName)))) 8273 mixin(q{ 8274 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8275 if(sessionId is null) 8276 return typeof(return).init; 8277 8278 import std.traits; 8279 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8280 if(v.length == 0) 8281 return typeof(return).init; 8282 import std.conv; 8283 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8284 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8285 // basically. Assuming the session is POD this should be fine. 8286 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8287 } 8288 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8289 if(sessionId is null) 8290 start(); 8291 import std.conv; 8292 import std.traits; 8293 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8294 return value; 8295 } 8296 }); 8297 } 8298 8299 /++ 8300 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8301 Simply stores the data in its instance members. 8302 +/ 8303 class MockSession(Data) : Session!Data { 8304 pure { 8305 @property string sessionId() const { return "mock"; } 8306 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8307 void regenerateId() {} 8308 void terminate() {} 8309 8310 private Data store_; 8311 8312 static foreach(memberName; __traits(allMembers, Data)) 8313 static if(is(typeof(__traits(getMember, Data, memberName)))) 8314 mixin(q{ 8315 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8316 return __traits(getMember, store_, memberName); 8317 } 8318 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8319 return __traits(getMember, store_, memberName) = value; 8320 } 8321 }); 8322 } 8323 } 8324 8325 /++ 8326 Direct interface to the basic data add-on server. You can 8327 typically use [Cgi.getSessionObject] as a more convenient interface. 8328 +/ 8329 version(with_addon_servers_connections) 8330 interface BasicDataServer { 8331 /// 8332 void createSession(string sessionId, int lifetime); 8333 /// 8334 void renewSession(string sessionId, int lifetime); 8335 /// 8336 void destroySession(string sessionId); 8337 /// 8338 void renameSession(string oldSessionId, string newSessionId); 8339 8340 /// 8341 void setSessionData(string sessionId, string dataKey, string dataValue); 8342 /// 8343 string getSessionData(string sessionId, string dataKey); 8344 8345 /// 8346 static BasicDataServerConnection connection() { 8347 return BasicDataServerConnection.connection(); 8348 } 8349 } 8350 8351 version(with_addon_servers_connections) 8352 class BasicDataServerConnection : BasicDataServer { 8353 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8354 } 8355 8356 version(with_addon_servers) 8357 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8358 8359 void createSession(string sessionId, int lifetime) { 8360 sessions[sessionId.idup] = Session(lifetime); 8361 } 8362 void destroySession(string sessionId) { 8363 sessions.remove(sessionId); 8364 } 8365 void renewSession(string sessionId, int lifetime) { 8366 sessions[sessionId].lifetime = lifetime; 8367 } 8368 void renameSession(string oldSessionId, string newSessionId) { 8369 sessions[newSessionId.idup] = sessions[oldSessionId]; 8370 sessions.remove(oldSessionId); 8371 } 8372 void setSessionData(string sessionId, string dataKey, string dataValue) { 8373 if(sessionId !in sessions) 8374 createSession(sessionId, 3600); // FIXME? 8375 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8376 } 8377 string getSessionData(string sessionId, string dataKey) { 8378 if(auto session = sessionId in sessions) { 8379 if(auto data = dataKey in (*session).values) 8380 return *data; 8381 else 8382 return null; // no such data 8383 8384 } else { 8385 return null; // no session 8386 } 8387 } 8388 8389 8390 protected: 8391 8392 struct Session { 8393 int lifetime; 8394 8395 string[string] values; 8396 } 8397 8398 Session[string] sessions; 8399 8400 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8401 auto data = op.usedBuffer; 8402 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8403 return false; 8404 } 8405 8406 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8407 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8408 void wait_timeout() {} 8409 void fileClosed(int fd) {} // stateless so irrelevant 8410 void epoll_fd(int fd) {} 8411 } 8412 8413 /++ 8414 See [schedule] to make one of these. You then call one of the methods here to set it up: 8415 8416 --- 8417 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8418 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8419 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8420 --- 8421 +/ 8422 version(with_addon_servers_connections) 8423 struct ScheduledJobHelper { 8424 private string func; 8425 private string[] args; 8426 private bool consumed; 8427 8428 private this(string func, string[] args) { 8429 this.func = func; 8430 this.args = args; 8431 } 8432 8433 ~this() { 8434 assert(consumed); 8435 } 8436 8437 /++ 8438 Schedules the job to be run at the given time. 8439 +/ 8440 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8441 consumed = true; 8442 8443 auto conn = ScheduledJobServerConnection.connection; 8444 import std.file; 8445 auto st = SysTime(when, timezone); 8446 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8447 } 8448 8449 /++ 8450 Schedules the job to run at least after the specified delay. 8451 +/ 8452 void delay(Duration delay) { 8453 consumed = true; 8454 8455 auto conn = ScheduledJobServerConnection.connection; 8456 import std.file; 8457 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8458 } 8459 8460 /++ 8461 Runs the job in the background ASAP. 8462 8463 $(NOTE It may run in a background thread. Don't segfault!) 8464 +/ 8465 void asap() { 8466 consumed = true; 8467 8468 auto conn = ScheduledJobServerConnection.connection; 8469 import std.file; 8470 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8471 } 8472 8473 /+ 8474 /++ 8475 Schedules the job to recur on the given pattern. 8476 +/ 8477 void recur(string spec) { 8478 8479 } 8480 +/ 8481 } 8482 8483 /++ 8484 First step to schedule a job on the scheduled job server. 8485 8486 The scheduled job needs to be a top-level function that doesn't read any 8487 variables from outside its arguments because it may be run in a new process, 8488 without any context existing later. 8489 8490 You MUST set details on the returned object to actually do anything! 8491 +/ 8492 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8493 /// 8494 ScheduledJobHelper schedule(T args) { 8495 // this isn't meant to ever be called, but instead just to 8496 // get the compiler to type check the arguments passed for us 8497 auto sample = delegate() { 8498 fn(args); 8499 }; 8500 string[] sargs; 8501 foreach(arg; args) 8502 sargs ~= to!string(arg); 8503 return ScheduledJobHelper(fn.mangleof, sargs); 8504 } 8505 8506 shared static this() { 8507 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8508 import std.traits; 8509 Parameters!fn args; 8510 foreach(idx, ref arg; args) 8511 arg = to!(typeof(arg))(sargs[idx]); 8512 fn(args); 8513 }; 8514 } 8515 } 8516 8517 /// 8518 interface ScheduledJobServer { 8519 /// Use the [schedule] function for a higher-level interface. 8520 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8521 /// 8522 void cancelJob(int jobId); 8523 } 8524 8525 version(with_addon_servers_connections) 8526 class ScheduledJobServerConnection : ScheduledJobServer { 8527 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8528 } 8529 8530 version(with_addon_servers) 8531 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8532 // FIXME: we need to handle SIGCHLD in this somehow 8533 // whenIs is 0 for relative, 1 for absolute 8534 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8535 auto nj = nextJobId; 8536 nextJobId++; 8537 8538 version(linux) { 8539 import core.sys.linux.timerfd; 8540 import core.sys.linux.epoll; 8541 import core.sys.posix.unistd; 8542 8543 8544 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8545 if(fd == -1) 8546 throw new Exception("fd timer create failed"); 8547 8548 foreach(ref arg; args) 8549 arg = arg.idup; 8550 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8551 8552 itimerspec value; 8553 value.it_value.tv_sec = when; 8554 value.it_value.tv_nsec = 0; 8555 8556 value.it_interval.tv_sec = 0; 8557 value.it_interval.tv_nsec = 0; 8558 8559 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8560 throw new Exception("couldn't set fd timer"); 8561 8562 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8563 jobs.remove(nj); 8564 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8565 close(fd); 8566 8567 8568 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8569 8570 return true; 8571 }); 8572 scope(failure) 8573 freeIoOp(op); 8574 8575 epoll_event ev; 8576 ev.events = EPOLLIN | EPOLLET; 8577 ev.data.ptr = op; 8578 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8579 throw new Exception("epoll_ctl " ~ to!string(errno)); 8580 8581 jobs[nj] = job; 8582 return nj; 8583 } else assert(0); 8584 } 8585 8586 protected void cancelJob(int jobId) { 8587 version(linux) { 8588 auto job = jobId in jobs; 8589 if(job is null) 8590 return; 8591 8592 jobs.remove(jobId); 8593 8594 version(linux) { 8595 import core.sys.linux.timerfd; 8596 import core.sys.linux.epoll; 8597 import core.sys.posix.unistd; 8598 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8599 close(job.timerfd); 8600 } 8601 } 8602 jobs.remove(jobId); 8603 } 8604 8605 int nextJobId = 1; 8606 static struct Job { 8607 string executable; 8608 string func; 8609 string[] args; 8610 int timerfd; 8611 int id; 8612 } 8613 Job[int] jobs; 8614 8615 8616 // event io server methods below 8617 8618 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8619 auto data = op.usedBuffer; 8620 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8621 return false; 8622 } 8623 8624 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8625 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8626 void wait_timeout() {} 8627 void fileClosed(int fd) {} // stateless so irrelevant 8628 8629 int epoll_fd_; 8630 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8631 int epoll_fd() { return epoll_fd_; } 8632 } 8633 8634 /++ 8635 History: 8636 Added January 6, 2019 8637 +/ 8638 version(with_addon_servers_connections) 8639 interface EventSourceServer { 8640 /++ 8641 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8642 8643 See_Also: 8644 [sendEvent] 8645 8646 Bugs: 8647 Not implemented on Windows! 8648 8649 History: 8650 Officially stabilised on November 23, 2023 (dub v11.4). It actually worked pretty well in its original design. 8651 +/ 8652 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8653 /* 8654 If lastEventId is missing or empty, you just get new events as they come. 8655 8656 If it is set from something else, it sends all since then (that are still alive) 8657 down the pipe immediately. 8658 8659 The reason it can come from the header is that's what the standard defines for 8660 browser reconnects. The reason it can come from a query string is just convenience 8661 in catching up in a user-defined manner. 8662 8663 The reason the header overrides the query string is if the browser tries to reconnect, 8664 it will send the header AND the query (it reconnects to the same url), so we just 8665 want to do the restart thing. 8666 8667 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8668 */ 8669 string lastEventId = cgi.lastEventId; 8670 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8671 lastEventId = cgi.get["lastEventId"]; 8672 8673 cgi.setResponseContentType("text/event-stream"); 8674 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8675 cgi.flush(); 8676 8677 cgi.closed = true; 8678 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8679 scope(exit) 8680 closeLocalServerConnection(s); 8681 8682 version(fastcgi) 8683 throw new Exception("sending fcgi connections not supported"); 8684 else { 8685 auto fd = cgi.getOutputFileHandle(); 8686 if(isInvalidHandle(fd)) 8687 throw new Exception("bad fd from cgi!"); 8688 8689 EventSourceServerImplementation.SendableEventConnection sec; 8690 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8691 8692 version(Posix) { 8693 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8694 assert(res == sec.sizeof); 8695 } else version(Windows) { 8696 // FIXME 8697 } 8698 } 8699 } 8700 8701 /++ 8702 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8703 8704 Params: 8705 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8706 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8707 data = the event data. Available in JS as `event.data`. 8708 lifetime = the amount of time to keep this event for replaying on the event server. 8709 8710 Bugs: 8711 Not implemented on Windows! 8712 8713 History: 8714 Officially stabilised on November 23, 2023 (dub v11.4). It actually worked pretty well in its original design. 8715 +/ 8716 public static void sendEvent(string url, string event, string data, int lifetime) { 8717 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8718 scope(exit) 8719 closeLocalServerConnection(s); 8720 8721 EventSourceServerImplementation.SendableEvent sev; 8722 sev.populate(url, event, data, lifetime); 8723 8724 version(Posix) { 8725 auto ret = send(s, &sev, sev.sizeof, 0); 8726 assert(ret == sev.sizeof); 8727 } else version(Windows) { 8728 // FIXME 8729 } 8730 } 8731 8732 /++ 8733 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8734 8735 See_Also: [disconnect] 8736 +/ 8737 void connect(string url, string forwardUrl); 8738 8739 /++ 8740 Disconnects `forwardUrl` from `url` 8741 8742 See_Also: [connect] 8743 +/ 8744 void disconnect(string url, string forwardUrl); 8745 } 8746 8747 /// 8748 version(with_addon_servers) 8749 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8750 8751 protected: 8752 8753 void connect(string url, string forwardUrl) { 8754 pipes[url] ~= forwardUrl; 8755 } 8756 void disconnect(string url, string forwardUrl) { 8757 auto t = url in pipes; 8758 if(t is null) 8759 return; 8760 foreach(idx, n; (*t)) 8761 if(n == forwardUrl) { 8762 (*t)[idx] = (*t)[$-1]; 8763 (*t) = (*t)[0 .. $-1]; 8764 break; 8765 } 8766 } 8767 8768 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8769 if(receivedFd != -1) { 8770 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8771 8772 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8773 8774 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8775 8776 auto url = got.url.idup; 8777 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8778 8779 // FIXME: catch up on past messages here 8780 } else { 8781 auto data = op.usedBuffer; 8782 auto event = cast(SendableEvent*) data.ptr; 8783 8784 if(event.magic == 0xdeadbeef) { 8785 handleInputEvent(event); 8786 8787 if(event.url in pipes) 8788 foreach(pipe; pipes[event.url]) { 8789 event.url = pipe; 8790 handleInputEvent(event); 8791 } 8792 } else { 8793 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8794 } 8795 } 8796 return false; 8797 } 8798 void handleLocalConnectionClose(IoOp* op) { 8799 fileClosed(op.fd); 8800 } 8801 void handleLocalConnectionComplete(IoOp* op) {} 8802 8803 void wait_timeout() { 8804 // just keeping alive 8805 foreach(url, connections; eventConnectionsByUrl) 8806 foreach(connection; connections) 8807 if(connection.needsChunking) 8808 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8809 else 8810 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8811 } 8812 8813 void fileClosed(int fd) { 8814 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8815 foreach(idx, conn; connections) { 8816 if(fd == conn.fd) { 8817 connections[idx] = connections[$-1]; 8818 connections = connections[0 .. $ - 1]; 8819 continue outer; 8820 } 8821 } 8822 } 8823 } 8824 8825 void epoll_fd(int fd) {} 8826 8827 8828 private: 8829 8830 8831 struct SendableEventConnection { 8832 ubyte responseChunked; 8833 8834 int urlLength; 8835 char[256] urlBuffer = 0; 8836 8837 int lastEventIdLength; 8838 char[32] lastEventIdBuffer = 0; 8839 8840 char[] url() return { 8841 return urlBuffer[0 .. urlLength]; 8842 } 8843 void url(in char[] u) { 8844 urlBuffer[0 .. u.length] = u[]; 8845 urlLength = cast(int) u.length; 8846 } 8847 char[] lastEventId() return { 8848 return lastEventIdBuffer[0 .. lastEventIdLength]; 8849 } 8850 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8851 in { 8852 assert(url.length < this.urlBuffer.length); 8853 assert(lastEventId.length < this.lastEventIdBuffer.length); 8854 } 8855 do { 8856 this.responseChunked = responseChunked ? 1 : 0; 8857 this.urlLength = cast(int) url.length; 8858 this.lastEventIdLength = cast(int) lastEventId.length; 8859 8860 this.urlBuffer[0 .. url.length] = url[]; 8861 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8862 } 8863 } 8864 8865 struct SendableEvent { 8866 int magic = 0xdeadbeef; 8867 int urlLength; 8868 char[256] urlBuffer = 0; 8869 int typeLength; 8870 char[32] typeBuffer = 0; 8871 int messageLength; 8872 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8873 int _lifetime; 8874 8875 char[] message() return { 8876 return messageBuffer[0 .. messageLength]; 8877 } 8878 char[] type() return { 8879 return typeBuffer[0 .. typeLength]; 8880 } 8881 char[] url() return { 8882 return urlBuffer[0 .. urlLength]; 8883 } 8884 void url(in char[] u) { 8885 urlBuffer[0 .. u.length] = u[]; 8886 urlLength = cast(int) u.length; 8887 } 8888 int lifetime() { 8889 return _lifetime; 8890 } 8891 8892 /// 8893 void populate(string url, string type, string message, int lifetime) 8894 in { 8895 assert(url.length < this.urlBuffer.length); 8896 assert(type.length < this.typeBuffer.length); 8897 assert(message.length < this.messageBuffer.length); 8898 } 8899 do { 8900 this.urlLength = cast(int) url.length; 8901 this.typeLength = cast(int) type.length; 8902 this.messageLength = cast(int) message.length; 8903 this._lifetime = lifetime; 8904 8905 this.urlBuffer[0 .. url.length] = url[]; 8906 this.typeBuffer[0 .. type.length] = type[]; 8907 this.messageBuffer[0 .. message.length] = message[]; 8908 } 8909 } 8910 8911 struct EventConnection { 8912 int fd; 8913 bool needsChunking; 8914 } 8915 8916 private EventConnection[][string] eventConnectionsByUrl; 8917 private string[][string] pipes; 8918 8919 private void handleInputEvent(scope SendableEvent* event) { 8920 static int eventId; 8921 8922 static struct StoredEvent { 8923 int id; 8924 string type; 8925 string message; 8926 int lifetimeRemaining; 8927 } 8928 8929 StoredEvent[][string] byUrl; 8930 8931 int thisId = ++eventId; 8932 8933 if(event.lifetime) 8934 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8935 8936 auto connectionsPtr = event.url in eventConnectionsByUrl; 8937 EventConnection[] connections; 8938 if(connectionsPtr is null) 8939 return; 8940 else 8941 connections = *connectionsPtr; 8942 8943 char[4096] buffer; 8944 char[] formattedMessage; 8945 8946 void append(const char[] a) { 8947 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8948 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8949 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8950 } 8951 8952 import std.algorithm.iteration; 8953 8954 if(connections.length) { 8955 append("id: "); 8956 append(to!string(thisId)); 8957 append("\n"); 8958 8959 append("event: "); 8960 append(event.type); 8961 append("\n"); 8962 8963 foreach(line; event.message.splitter("\n")) { 8964 append("data: "); 8965 append(line); 8966 append("\n"); 8967 } 8968 8969 append("\n"); 8970 } 8971 8972 // chunk it for HTTP! 8973 auto len = toHex(formattedMessage.length); 8974 buffer[4 .. 6] = "\r\n"[]; 8975 buffer[4 - len.length .. 4] = len[]; 8976 buffer[6 + formattedMessage.length] = '\r'; 8977 buffer[6 + formattedMessage.length + 1] = '\n'; 8978 8979 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8980 // done 8981 8982 // FIXME: send back requests when needed 8983 // FIXME: send a single ":\n" every 15 seconds to keep alive 8984 8985 foreach(connection; connections) { 8986 if(connection.needsChunking) { 8987 nonBlockingWrite(this, connection.fd, chunkedMessage); 8988 } else { 8989 nonBlockingWrite(this, connection.fd, formattedMessage); 8990 } 8991 } 8992 } 8993 } 8994 8995 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8996 version(Posix) { 8997 8998 import core.sys.posix.unistd; 8999 import core.sys.posix.fcntl; 9000 import core.sys.posix.sys.un; 9001 9002 import core.sys.posix.signal; 9003 signal(SIGPIPE, SIG_IGN); 9004 9005 static extern(C) void sigchldhandler(int) { 9006 int status; 9007 import w = core.sys.posix.sys.wait; 9008 w.wait(&status); 9009 } 9010 signal(SIGCHLD, &sigchldhandler); 9011 9012 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 9013 if(sock == -1) 9014 throw new Exception("socket " ~ to!string(errno)); 9015 9016 scope(failure) 9017 close(sock); 9018 9019 cloexec(sock); 9020 9021 // add-on server processes are assumed to be local, and thus will 9022 // use unix domain sockets. Besides, I want to pass sockets to them, 9023 // so it basically must be local (except for the session server, but meh). 9024 sockaddr_un addr; 9025 addr.sun_family = AF_UNIX; 9026 version(linux) { 9027 // on linux, we will use the abstract namespace 9028 addr.sun_path[0] = 0; 9029 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 9030 } else { 9031 // but otherwise, just use a file cuz we must. 9032 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 9033 } 9034 9035 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 9036 throw new Exception("bind " ~ to!string(errno)); 9037 9038 if(listen(sock, 128) == -1) 9039 throw new Exception("listen " ~ to!string(errno)); 9040 9041 makeNonBlocking(sock); 9042 9043 version(linux) { 9044 import core.sys.linux.epoll; 9045 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 9046 if(epoll_fd == -1) 9047 throw new Exception("epoll_create1 " ~ to!string(errno)); 9048 scope(failure) 9049 close(epoll_fd); 9050 } else { 9051 import core.sys.posix.poll; 9052 } 9053 9054 version(linux) 9055 eis.epoll_fd = epoll_fd; 9056 9057 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 9058 scope(exit) 9059 freeIoOp(acceptOp); 9060 9061 version(linux) { 9062 epoll_event ev; 9063 ev.events = EPOLLIN | EPOLLET; 9064 ev.data.ptr = acceptOp; 9065 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 9066 throw new Exception("epoll_ctl " ~ to!string(errno)); 9067 9068 epoll_event[64] events; 9069 } else { 9070 pollfd[] pollfds; 9071 IoOp*[int] ioops; 9072 pollfds ~= pollfd(sock, POLLIN); 9073 ioops[sock] = acceptOp; 9074 } 9075 9076 import core.time : MonoTime, seconds; 9077 9078 MonoTime timeout = MonoTime.currTime + 15.seconds; 9079 9080 while(true) { 9081 9082 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 9083 9084 int timeout_milliseconds = 0; // -1; // infinite 9085 9086 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 9087 if(timeout_milliseconds < 0) 9088 timeout_milliseconds = 0; 9089 9090 //writeln("waiting for ", name); 9091 9092 version(linux) { 9093 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 9094 if(nfds == -1) { 9095 if(errno == EINTR) 9096 continue; 9097 throw new Exception("epoll_wait " ~ to!string(errno)); 9098 } 9099 } else { 9100 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 9101 size_t lastIdx = 0; 9102 } 9103 9104 if(nfds == 0) { 9105 eis.wait_timeout(); 9106 timeout += 15.seconds; 9107 } 9108 9109 foreach(idx; 0 .. nfds) { 9110 version(linux) { 9111 auto flags = events[idx].events; 9112 auto ioop = cast(IoOp*) events[idx].data.ptr; 9113 } else { 9114 IoOp* ioop; 9115 foreach(tidx, thing; pollfds[lastIdx .. $]) { 9116 if(thing.revents) { 9117 ioop = ioops[thing.fd]; 9118 lastIdx += tidx + 1; 9119 break; 9120 } 9121 } 9122 } 9123 9124 //writeln(flags, " ", ioop.fd); 9125 9126 void newConnection() { 9127 // on edge triggering, it is important that we get it all 9128 while(true) { 9129 auto size = cast(socklen_t) addr.sizeof; 9130 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 9131 if(ns == -1) { 9132 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9133 // all done, got it all 9134 break; 9135 } 9136 throw new Exception("accept " ~ to!string(errno)); 9137 } 9138 cloexec(ns); 9139 9140 makeNonBlocking(ns); 9141 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 9142 niop.closeHandler = &eis.handleLocalConnectionClose; 9143 niop.completeHandler = &eis.handleLocalConnectionComplete; 9144 scope(failure) freeIoOp(niop); 9145 9146 version(linux) { 9147 epoll_event nev; 9148 nev.events = EPOLLIN | EPOLLET; 9149 nev.data.ptr = niop; 9150 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 9151 throw new Exception("epoll_ctl " ~ to!string(errno)); 9152 } else { 9153 bool found = false; 9154 foreach(ref pfd; pollfds) { 9155 if(pfd.fd < 0) { 9156 pfd.fd = ns; 9157 found = true; 9158 } 9159 } 9160 if(!found) 9161 pollfds ~= pollfd(ns, POLLIN); 9162 ioops[ns] = niop; 9163 } 9164 } 9165 } 9166 9167 bool newConnectionCondition() { 9168 version(linux) 9169 return ioop.fd == sock && (flags & EPOLLIN); 9170 else 9171 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 9172 } 9173 9174 if(newConnectionCondition()) { 9175 newConnection(); 9176 } else if(ioop.operation == IoOp.ReadSocketHandle) { 9177 while(true) { 9178 int in_fd; 9179 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 9180 if(got == -1) { 9181 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9182 // all done, got it all 9183 if(ioop.completeHandler) 9184 ioop.completeHandler(ioop); 9185 break; 9186 } 9187 throw new Exception("recv " ~ to!string(errno)); 9188 } 9189 9190 if(got == 0) { 9191 if(ioop.closeHandler) { 9192 ioop.closeHandler(ioop); 9193 version(linux) {} // nothing needed 9194 else { 9195 foreach(ref pfd; pollfds) { 9196 if(pfd.fd == ioop.fd) 9197 pfd.fd = -1; 9198 } 9199 } 9200 } 9201 close(ioop.fd); 9202 freeIoOp(ioop); 9203 break; 9204 } 9205 9206 ioop.bufferLengthUsed = cast(int) got; 9207 ioop.handler(ioop, in_fd); 9208 } 9209 } else if(ioop.operation == IoOp.Read) { 9210 while(true) { 9211 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 9212 if(got == -1) { 9213 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9214 // all done, got it all 9215 if(ioop.completeHandler) 9216 ioop.completeHandler(ioop); 9217 break; 9218 } 9219 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 9220 } 9221 9222 if(got == 0) { 9223 if(ioop.closeHandler) 9224 ioop.closeHandler(ioop); 9225 close(ioop.fd); 9226 freeIoOp(ioop); 9227 break; 9228 } 9229 9230 ioop.bufferLengthUsed = cast(int) got; 9231 if(ioop.handler(ioop, ioop.fd)) { 9232 close(ioop.fd); 9233 freeIoOp(ioop); 9234 break; 9235 } 9236 } 9237 } 9238 9239 // EPOLLHUP? 9240 } 9241 } 9242 } else version(Windows) { 9243 9244 // set up a named pipe 9245 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 9246 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 9247 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 9248 9249 } else static assert(0); 9250 } 9251 9252 9253 version(with_sendfd) 9254 // copied from the web and ported from C 9255 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 9256 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 9257 msghdr msg; 9258 iovec[1] iov; 9259 9260 version(OSX) { 9261 //msg.msg_accrights = cast(cattr_t) &sendfd; 9262 //msg.msg_accrightslen = int.sizeof; 9263 } else version(Android) { 9264 } else { 9265 union ControlUnion { 9266 cmsghdr cm; 9267 char[CMSG_SPACE(int.sizeof)] control; 9268 } 9269 9270 ControlUnion control_un; 9271 cmsghdr* cmptr; 9272 9273 msg.msg_control = control_un.control.ptr; 9274 msg.msg_controllen = control_un.control.length; 9275 9276 cmptr = CMSG_FIRSTHDR(&msg); 9277 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 9278 cmptr.cmsg_level = SOL_SOCKET; 9279 cmptr.cmsg_type = SCM_RIGHTS; 9280 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 9281 } 9282 9283 msg.msg_name = null; 9284 msg.msg_namelen = 0; 9285 9286 iov[0].iov_base = ptr; 9287 iov[0].iov_len = nbytes; 9288 msg.msg_iov = iov.ptr; 9289 msg.msg_iovlen = 1; 9290 9291 return sendmsg(fd, &msg, 0); 9292 } 9293 9294 version(with_sendfd) 9295 // copied from the web and ported from C 9296 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9297 msghdr msg; 9298 iovec[1] iov; 9299 ssize_t n; 9300 int newfd; 9301 9302 version(OSX) { 9303 //msg.msg_accrights = cast(cattr_t) recvfd; 9304 //msg.msg_accrightslen = int.sizeof; 9305 } else version(Android) { 9306 } else { 9307 union ControlUnion { 9308 cmsghdr cm; 9309 char[CMSG_SPACE(int.sizeof)] control; 9310 } 9311 ControlUnion control_un; 9312 cmsghdr* cmptr; 9313 9314 msg.msg_control = control_un.control.ptr; 9315 msg.msg_controllen = control_un.control.length; 9316 } 9317 9318 msg.msg_name = null; 9319 msg.msg_namelen = 0; 9320 9321 iov[0].iov_base = ptr; 9322 iov[0].iov_len = nbytes; 9323 msg.msg_iov = iov.ptr; 9324 msg.msg_iovlen = 1; 9325 9326 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9327 return n; 9328 9329 version(OSX) { 9330 //if(msg.msg_accrightslen != int.sizeof) 9331 //*recvfd = -1; 9332 } else version(Android) { 9333 } else { 9334 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9335 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9336 if (cmptr.cmsg_level != SOL_SOCKET) 9337 throw new Exception("control level != SOL_SOCKET"); 9338 if (cmptr.cmsg_type != SCM_RIGHTS) 9339 throw new Exception("control type != SCM_RIGHTS"); 9340 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9341 } else 9342 *recvfd = -1; /* descriptor was not passed */ 9343 } 9344 9345 return n; 9346 } 9347 /* end read_fd */ 9348 9349 9350 /* 9351 Event source stuff 9352 9353 The api is: 9354 9355 sendEvent(string url, string type, string data, int timeout = 60*10); 9356 9357 attachEventListener(string url, int fd, lastId) 9358 9359 9360 It just sends to all attached listeners, and stores it until the timeout 9361 for replaying via lastEventId. 9362 */ 9363 9364 /* 9365 Session process stuff 9366 9367 it stores it all. the cgi object has a session object that can grab it 9368 9369 session may be done in the same process if possible, there is a version 9370 switch to choose if you want to override. 9371 */ 9372 9373 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9374 alias handler = dispatchHandler; 9375 string urlPrefix; 9376 bool rejectFurther; 9377 immutable(DispatcherDetails) details; 9378 } 9379 9380 private string urlify(string name) pure { 9381 return beautify(name, '-', true); 9382 } 9383 9384 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9385 if(name == "id") 9386 return allLowerCase ? name : "ID"; 9387 9388 char[160] buffer; 9389 int bufferIndex = 0; 9390 bool shouldCap = true; 9391 bool shouldSpace; 9392 bool lastWasCap; 9393 foreach(idx, char ch; name) { 9394 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9395 9396 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9397 if(lastWasCap) { 9398 // two caps in a row, don't change. Prolly acronym. 9399 } else { 9400 if(idx) 9401 shouldSpace = true; // new word, add space 9402 } 9403 9404 lastWasCap = true; 9405 } else { 9406 lastWasCap = false; 9407 } 9408 9409 if(shouldSpace) { 9410 buffer[bufferIndex++] = space; 9411 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9412 shouldSpace = false; 9413 } 9414 if(shouldCap) { 9415 if(ch >= 'a' && ch <= 'z') 9416 ch -= 32; 9417 shouldCap = false; 9418 } 9419 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9420 ch += 32; 9421 buffer[bufferIndex++] = ch; 9422 } 9423 return buffer[0 .. bufferIndex].idup; 9424 } 9425 9426 /* 9427 string urlFor(alias func)() { 9428 return __traits(identifier, func); 9429 } 9430 */ 9431 9432 /++ 9433 UDA: The name displayed to the user in auto-generated HTML. 9434 9435 Default is `beautify(identifier)`. 9436 +/ 9437 struct DisplayName { 9438 string name; 9439 } 9440 9441 /++ 9442 UDA: The name used in the URL or web parameter. 9443 9444 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9445 +/ 9446 struct UrlName { 9447 string name; 9448 } 9449 9450 /++ 9451 UDA: default format to respond for this method 9452 +/ 9453 struct DefaultFormat { string value; } 9454 9455 class MissingArgumentException : Exception { 9456 string functionName; 9457 string argumentName; 9458 string argumentType; 9459 9460 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9461 this.functionName = functionName; 9462 this.argumentName = argumentName; 9463 this.argumentType = argumentType; 9464 9465 super("Missing Argument: " ~ this.argumentName, file, line, next); 9466 } 9467 } 9468 9469 /++ 9470 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9471 9472 History: 9473 Added December 15, 2021 (dub v10.5) 9474 +/ 9475 class ResourceNotFoundException : Exception { 9476 string resourceType; 9477 string resourceId; 9478 9479 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9480 this.resourceType = resourceType; 9481 this.resourceId = resourceId; 9482 9483 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9484 } 9485 9486 } 9487 9488 /++ 9489 This can be attached to any constructor or function called from the cgi system. 9490 9491 If it is present, the function argument can NOT be set from web params, but instead 9492 is set to the return value of the given `func`. 9493 9494 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9495 the current request. Otherwise, it must take zero arguments. 9496 9497 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9498 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9499 the cgi object. 9500 9501 const arguments are also supported. 9502 +/ 9503 struct ifCalledFromWeb(alias func) {} 9504 9505 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9506 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9507 9508 // FIXME: any array of structs should also be settable or gettable from csv as well. 9509 9510 // FIXME: think more about checkboxes and bools. 9511 9512 import std.traits; 9513 9514 Parameters!method params; 9515 alias idents = ParameterIdentifierTuple!method; 9516 alias defaults = ParameterDefaults!method; 9517 9518 const(string)[] names; 9519 const(string)[] values; 9520 9521 // first, check for missing arguments and initialize to defaults if necessary 9522 9523 static if(is(typeof(method) P == __parameters)) 9524 foreach(idx, param; P) {{ 9525 // see: mustNotBeSetFromWebParams 9526 static if(is(param : Cgi)) { 9527 static assert(!is(param == immutable)); 9528 cast() params[idx] = cgi; 9529 } else static if(is(param == Session!D, D)) { 9530 static assert(!is(param == immutable)); 9531 cast() params[idx] = cgi.getSessionObject!D(); 9532 } else { 9533 bool populated; 9534 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9535 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9536 static if(is(typeof(func(cgi)))) 9537 params[idx] = func(cgi); 9538 else 9539 params[idx] = func(); 9540 9541 populated = true; 9542 } 9543 } 9544 9545 if(!populated) { 9546 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9547 params[idx] = param.getAutomaticallyForCgi(cgi); 9548 populated = true; 9549 } 9550 } 9551 9552 if(!populated) { 9553 auto ident = idents[idx]; 9554 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9555 if(ident !in cgi.get) { 9556 static if(is(defaults[idx] == void)) { 9557 static if(is(param == bool)) 9558 params[idx] = false; 9559 else 9560 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9561 } else 9562 params[idx] = defaults[idx]; 9563 } 9564 } else { 9565 if(ident !in cgi.post) { 9566 static if(is(defaults[idx] == void)) { 9567 static if(is(param == bool)) 9568 params[idx] = false; 9569 else 9570 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9571 } else 9572 params[idx] = defaults[idx]; 9573 } 9574 } 9575 } 9576 } 9577 }} 9578 9579 // second, parse the arguments in order to build up arrays, etc. 9580 9581 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9582 static if(is(T == struct)) { 9583 if(name == paramName) { 9584 *what = T.init; 9585 return true; 9586 } else { 9587 // could be a child. gonna allow either obj.field OR obj[field] 9588 9589 string afterName; 9590 9591 if(name[paramName.length] == '[') { 9592 int count = 1; 9593 auto idx = paramName.length + 1; 9594 while(idx < name.length && count > 0) { 9595 if(name[idx] == '[') 9596 count++; 9597 else if(name[idx] == ']') { 9598 count--; 9599 if(count == 0) break; 9600 } 9601 idx++; 9602 } 9603 9604 if(idx == name.length) 9605 return false; // malformed 9606 9607 auto insideBrackets = name[paramName.length + 1 .. idx]; 9608 afterName = name[idx + 1 .. $]; 9609 9610 name = name[0 .. paramName.length]; 9611 9612 paramName = insideBrackets; 9613 9614 } else if(name[paramName.length] == '.') { 9615 paramName = name[paramName.length + 1 .. $]; 9616 name = paramName; 9617 int p = 0; 9618 foreach(ch; paramName) { 9619 if(ch == '.' || ch == '[') 9620 break; 9621 p++; 9622 } 9623 9624 afterName = paramName[p .. $]; 9625 paramName = paramName[0 .. p]; 9626 } else { 9627 return false; 9628 } 9629 9630 if(paramName.length) 9631 // set the child member 9632 switch(paramName) { 9633 foreach(idx, memberName; __traits(allMembers, T)) 9634 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9635 // data member! 9636 case memberName: 9637 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9638 } 9639 default: 9640 // ok, not a member 9641 } 9642 } 9643 9644 return false; 9645 } else static if(is(T == enum)) { 9646 *what = to!T(value); 9647 return true; 9648 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9649 *what = to!T(value); 9650 return true; 9651 } else static if(is(T == bool)) { 9652 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9653 return true; 9654 } else static if(is(T == K[], K)) { 9655 K tmp; 9656 if(name == paramName) { 9657 // direct - set and append 9658 if(setVariable(name, paramName, &tmp, value)) { 9659 (*what) ~= tmp; 9660 return true; 9661 } else { 9662 return false; 9663 } 9664 } else { 9665 // child, append to last element 9666 // FIXME: what about range violations??? 9667 auto ptr = &(*what)[(*what).length - 1]; 9668 return setVariable(name, paramName, ptr, value); 9669 9670 } 9671 } else static if(is(T == V[K], K, V)) { 9672 // assoc array, name[key] is valid 9673 if(name == paramName) { 9674 // no action necessary 9675 return true; 9676 } else if(name[paramName.length] == '[') { 9677 int count = 1; 9678 auto idx = paramName.length + 1; 9679 while(idx < name.length && count > 0) { 9680 if(name[idx] == '[') 9681 count++; 9682 else if(name[idx] == ']') { 9683 count--; 9684 if(count == 0) break; 9685 } 9686 idx++; 9687 } 9688 if(idx == name.length) 9689 return false; // malformed 9690 9691 auto insideBrackets = name[paramName.length + 1 .. idx]; 9692 auto afterName = name[idx + 1 .. $]; 9693 9694 auto k = to!K(insideBrackets); 9695 V v; 9696 if(auto ptr = k in *what) 9697 v = *ptr; 9698 9699 name = name[0 .. paramName.length]; 9700 //writeln(name, afterName, " ", paramName); 9701 9702 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9703 if(ret) { 9704 (*what)[k] = v; 9705 return true; 9706 } 9707 } 9708 9709 return false; 9710 } else { 9711 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9712 } 9713 9714 //return false; 9715 } 9716 9717 void setArgument(string name, string value) { 9718 int p; 9719 foreach(ch; name) { 9720 if(ch == '.' || ch == '[') 9721 break; 9722 p++; 9723 } 9724 9725 auto paramName = name[0 .. p]; 9726 9727 sw: switch(paramName) { 9728 static if(is(typeof(method) P == __parameters)) 9729 foreach(idx, param; P) { 9730 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9731 // cannot be set from the outside 9732 } else { 9733 case idents[idx]: 9734 static if(is(param == Cgi.UploadedFile)) { 9735 params[idx] = cgi.files[name]; 9736 } else static if(is(param : const Cgi.UploadedFile[])) { 9737 (cast() params[idx]) = cgi.filesArray[name]; 9738 } else { 9739 setVariable(name, paramName, ¶ms[idx], value); 9740 } 9741 break sw; 9742 } 9743 } 9744 default: 9745 // ignore; not relevant argument 9746 } 9747 } 9748 9749 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9750 names = cgi.allGetNamesInOrder; 9751 values = cgi.allGetValuesInOrder; 9752 } else { 9753 names = cgi.allPostNamesInOrder; 9754 values = cgi.allPostValuesInOrder; 9755 } 9756 9757 foreach(idx, name; names) { 9758 setArgument(name, values[idx]); 9759 } 9760 9761 static if(is(ReturnType!method == void)) { 9762 typeof(null) ret; 9763 dg(params); 9764 } else { 9765 auto ret = dg(params); 9766 } 9767 9768 // FIXME: format return values 9769 // options are: json, html, csv. 9770 // also may need to wrap in envelope format: none, html, or json. 9771 return ret; 9772 } 9773 9774 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9775 static if(is(T : const(Cgi))) { 9776 return true; 9777 } else static if(is(T : const(Session!D), D)) { 9778 return true; 9779 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9780 return true; 9781 } else { 9782 foreach(uda; attrs) 9783 static if(is(uda == ifCalledFromWeb!func, alias func)) 9784 return true; 9785 return false; 9786 } 9787 } 9788 9789 private bool hasIfCalledFromWeb(attrs...)() { 9790 foreach(uda; attrs) 9791 static if(is(uda == ifCalledFromWeb!func, alias func)) 9792 return true; 9793 return false; 9794 } 9795 9796 /++ 9797 Implies POST path for the thing itself, then GET will get the automatic form. 9798 9799 The given customizer, if present, will be called as a filter on the Form object. 9800 9801 History: 9802 Added December 27, 2020 9803 +/ 9804 template AutomaticForm(alias customizer) { } 9805 9806 /++ 9807 This is meant to be returned by a function that takes a form POST submission. You 9808 want to set the url of the new resource it created, which is set as the http 9809 Location header for a "201 Created" result, and you can also set a separate 9810 destination for browser users, which it sets via a "Refresh" header. 9811 9812 The `resourceRepresentation` should generally be the thing you just created, and 9813 it will be the body of the http response when formatted through the presenter. 9814 The exact thing is up to you - it could just return an id, or the whole object, or 9815 perhaps a partial object. 9816 9817 Examples: 9818 --- 9819 class Test : WebObject { 9820 @(Cgi.RequestMethod.POST) 9821 CreatedResource!int makeThing(string value) { 9822 return CreatedResource!int(value.to!int, "/resources/id"); 9823 } 9824 } 9825 --- 9826 9827 History: 9828 Added December 18, 2021 9829 +/ 9830 struct CreatedResource(T) { 9831 static if(!is(T == void)) 9832 T resourceRepresentation; 9833 string resourceUrl; 9834 string refreshUrl; 9835 } 9836 9837 /+ 9838 /++ 9839 This can be attached as a UDA to a handler to add a http Refresh header on a 9840 successful run. (It will not be attached if the function throws an exception.) 9841 This will refresh the browser the given number of seconds after the page loads, 9842 to the url returned by `urlFunc`, which can be either a static function or a 9843 member method of the current handler object. 9844 9845 You might use this for a POST handler that is normally used from ajax, but you 9846 want it to degrade gracefully to a temporarily flashed message before reloading 9847 the main page. 9848 9849 History: 9850 Added December 18, 2021 9851 +/ 9852 struct Refresh(alias urlFunc) { 9853 int waitInSeconds; 9854 9855 string url() { 9856 static if(__traits(isStaticFunction, urlFunc)) 9857 return urlFunc(); 9858 else static if(is(urlFunc : string)) 9859 return urlFunc; 9860 } 9861 } 9862 +/ 9863 9864 /+ 9865 /++ 9866 Sets a filter to be run before 9867 9868 A before function can do validations of params and log and stop the function from running. 9869 +/ 9870 template Before(alias b) {} 9871 template After(alias b) {} 9872 +/ 9873 9874 /+ 9875 Argument conversions: for the most part, it is to!Thing(string). 9876 9877 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9878 they are passed 9879 9880 arr=foo&arr=bar <-- notice the same name. 9881 9882 Structs are first declared with an empty thing, then have their members set individually, 9883 with dot notation. The members are not required, just the initial declaration. 9884 9885 struct Foo { 9886 int a; 9887 string b; 9888 } 9889 void test(Foo foo){} 9890 9891 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9892 9893 Arrays of structs use this declaration. 9894 9895 void test(Foo[] foo) {} 9896 9897 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9898 9899 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9900 declaration is ignored. 9901 9902 Mind that order matters! The declaration MUST come first in the string. 9903 9904 Arrays of struct members follow this rule recursively. 9905 9906 struct Foo { 9907 int[] a; 9908 } 9909 9910 foo&foo.a=1&foo.a=2&foo&foo.a=1 9911 9912 9913 Associative arrays are formatted with brackets, after a declaration, like structs: 9914 9915 foo&foo[key]=value&foo[other_key]=value 9916 9917 9918 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9919 do not support the strict ordering requirements to work with these struct protocols. 9920 9921 FIXME: also perhaps accept application/json to better work with outside trash. 9922 9923 9924 Return values are also auto-formatted according to user-requested type: 9925 for json, it loops over and converts. 9926 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9927 +/ 9928 9929 /++ 9930 A web presenter is responsible for rendering things to HTML to be usable 9931 in a web browser. 9932 9933 They are passed as template arguments to the base classes of [WebObject] 9934 9935 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9936 and override it. Use forwarding and specialization to customize it. 9937 9938 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9939 class templates and your overridden templates work with each other. 9940 9941 --- 9942 class MyPresenter : WebPresenter!(MyPresenter) { 9943 @Override 9944 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9945 // present the CustomType 9946 } 9947 @Override 9948 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9949 // handle everything else via the super class, which will call 9950 // back to your class when appropriate 9951 super.presentSuccessfulReturnAsHtml(cgi, ret); 9952 } 9953 } 9954 --- 9955 9956 The meta argument in there can be overridden by your own facility. 9957 9958 +/ 9959 class WebPresenter(CRTP) { 9960 9961 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9962 /// If you override a plain method, use `override`. If a template, use `@Override`. 9963 enum Override; 9964 9965 string script() { 9966 return ` 9967 `; 9968 } 9969 9970 string style() { 9971 return ` 9972 :root { 9973 --mild-border: #ccc; 9974 --middle-border: #999; 9975 --accent-color: #f2f2f2; 9976 --sidebar-color: #fefefe; 9977 } 9978 ` ~ genericFormStyling() ~ genericSiteStyling(); 9979 } 9980 9981 string genericFormStyling() { 9982 return 9983 q"css 9984 table.automatic-data-display { 9985 border-collapse: collapse; 9986 border: solid 1px var(--mild-border); 9987 } 9988 9989 table.automatic-data-display td { 9990 vertical-align: top; 9991 border: solid 1px var(--mild-border); 9992 padding: 2px 4px; 9993 } 9994 9995 table.automatic-data-display th { 9996 border: solid 1px var(--mild-border); 9997 border-bottom: solid 1px var(--middle-border); 9998 padding: 2px 4px; 9999 } 10000 10001 ol.automatic-data-display { 10002 margin: 0px; 10003 list-style-position: inside; 10004 padding: 0px; 10005 } 10006 10007 dl.automatic-data-display { 10008 10009 } 10010 10011 .automatic-form { 10012 max-width: 600px; 10013 } 10014 10015 .form-field { 10016 margin: 0.5em; 10017 padding-left: 0.5em; 10018 } 10019 10020 .label-text { 10021 display: block; 10022 font-weight: bold; 10023 margin-left: -0.5em; 10024 } 10025 10026 .submit-button-holder { 10027 padding-left: 2em; 10028 } 10029 10030 .add-array-button { 10031 10032 } 10033 css"; 10034 } 10035 10036 string genericSiteStyling() { 10037 return 10038 q"css 10039 * { box-sizing: border-box; } 10040 html, body { margin: 0px; } 10041 body { 10042 font-family: sans-serif; 10043 } 10044 header { 10045 background: var(--accent-color); 10046 height: 64px; 10047 } 10048 footer { 10049 background: var(--accent-color); 10050 height: 64px; 10051 } 10052 #site-container { 10053 display: flex; 10054 flex-wrap: wrap; 10055 } 10056 main { 10057 flex: 1 1 auto; 10058 order: 2; 10059 min-height: calc(100vh - 64px - 64px); 10060 min-width: 80ch; 10061 padding: 4px; 10062 padding-left: 1em; 10063 } 10064 #sidebar { 10065 flex: 0 0 16em; 10066 order: 1; 10067 background: var(--sidebar-color); 10068 } 10069 css"; 10070 } 10071 10072 import arsd.dom; 10073 Element htmlContainer() { 10074 auto document = new Document(q"html 10075 <!DOCTYPE html> 10076 <html class="no-script"> 10077 <head> 10078 <script>document.documentElement.classList.remove("no-script");</script> 10079 <style>.no-script requires-script { display: none; }</style> 10080 <title>D Application</title> 10081 <meta name="viewport" content="initial-scale=1, width=device-width" /> 10082 <link rel="stylesheet" href="style.css" /> 10083 </head> 10084 <body> 10085 <header></header> 10086 <div id="site-container"> 10087 <main></main> 10088 <div id="sidebar"></div> 10089 </div> 10090 <footer></footer> 10091 <script src="script.js"></script> 10092 </body> 10093 </html> 10094 html", true, true); 10095 10096 return document.requireSelector("main"); 10097 } 10098 10099 /// Renders a response as an HTTP error with associated html body 10100 void renderBasicError(Cgi cgi, int httpErrorCode) { 10101 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 10102 auto c = htmlContainer(); 10103 c.innerText = getHttpCodeText(httpErrorCode); 10104 cgi.setResponseContentType("text/html; charset=utf-8"); 10105 cgi.write(c.parentDocument.toString(), true); 10106 } 10107 10108 template methodMeta(alias method) { 10109 enum methodMeta = null; 10110 } 10111 10112 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10113 switch(format) { 10114 case "html": 10115 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 10116 break; 10117 case "json": 10118 import arsd.jsvar; 10119 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 10120 var json; 10121 foreach(index, type; Types) { 10122 if(ret.contains == index) 10123 json = ret.payload[index]; 10124 } 10125 } else { 10126 var json = ret; 10127 } 10128 var envelope = json; // var.emptyObject; 10129 /* 10130 envelope.success = true; 10131 envelope.result = json; 10132 envelope.error = null; 10133 */ 10134 cgi.setResponseContentType("application/json"); 10135 cgi.write(envelope.toJson(), true); 10136 break; 10137 default: 10138 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 10139 } 10140 } 10141 10142 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 10143 /// in the default presenter - allowing the function to have full low-level control over the 10144 /// response. 10145 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 10146 // nothing intentionally! 10147 } 10148 10149 /// Redirections are forwarded to [Cgi.setResponseLocation] 10150 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10151 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 10152 } 10153 10154 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 10155 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 10156 cgi.setResponseStatus(getHttpCodeText(201)); 10157 if(ret.resourceUrl.length) 10158 cgi.header("Location: " ~ ret.resourceUrl); 10159 if(ret.refreshUrl.length) 10160 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 10161 static if(!is(R == void)) 10162 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 10163 } 10164 10165 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 10166 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 10167 bool outputted = false; 10168 foreach(index, type; Types) { 10169 if(ret.contains == index) { 10170 assert(!outputted); 10171 outputted = true; 10172 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 10173 } 10174 } 10175 if(!outputted) 10176 assert(0); 10177 } 10178 10179 /++ 10180 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 10181 +/ 10182 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10183 cgi.setCache(true); // not necessarily true but meh 10184 if(auto fn = ret.filename()) { 10185 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 10186 } 10187 cgi.setResponseContentType(ret.contentType); 10188 cgi.write(ret.getData(), true); 10189 } 10190 10191 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 10192 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 10193 auto container = this.htmlContainer(); 10194 container.appendChild(formatReturnValueAsHtml(ret)); 10195 cgi.write(container.parentDocument.toString(), true); 10196 } 10197 10198 /++ 10199 10200 History: 10201 Added January 23, 2023 (dub v11.0) 10202 +/ 10203 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 10204 switch(format) { 10205 case "html": 10206 presentExceptionAsHtml(cgi, t, meta); 10207 break; 10208 case "json": 10209 presentExceptionAsJsonImpl(cgi, t); 10210 break; 10211 default: 10212 } 10213 } 10214 10215 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 10216 cgi.setResponseStatus("500 Internal Server Error"); 10217 cgi.setResponseContentType("application/json"); 10218 import arsd.jsvar; 10219 var v = var.emptyObject; 10220 v.type = typeid(t).toString; 10221 v.msg = t.msg; 10222 v.fullString = t.toString(); 10223 cgi.write(v.toJson(), true); 10224 } 10225 10226 10227 /++ 10228 If you override this, you will need to cast the exception type `t` dynamically, 10229 but can then use the template arguments here to refer back to the function. 10230 10231 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 10232 method on the live object. You could, in theory, change arguments and retry, but I 10233 provide that information mostly with the expectation that you will use them to make 10234 useful forms or richer error messages for the user. 10235 10236 History: 10237 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 10238 I removed this in favor of a `Meta` param. 10239 10240 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 10241 10242 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 10243 10244 If you used the func for something, move that something into your `methodMeta` template. 10245 10246 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 10247 enabling an easier implementation of [presentExceptionalReturn]. 10248 +/ 10249 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 10250 Form af; 10251 /+ 10252 foreach(attr; __traits(getAttributes, func)) { 10253 static if(__traits(isSame, attr, AutomaticForm)) { 10254 af = createAutomaticFormForFunction!(func)(dg); 10255 } 10256 } 10257 +/ 10258 presentExceptionAsHtmlImpl(cgi, t, af); 10259 } 10260 10261 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 10262 if(auto e = cast(ResourceNotFoundException) t) { 10263 auto container = this.htmlContainer(); 10264 10265 container.addChild("p", e.msg); 10266 10267 if(!cgi.outputtedResponseData) 10268 cgi.setResponseStatus("404 Not Found"); 10269 cgi.write(container.parentDocument.toString(), true); 10270 } else if(auto mae = cast(MissingArgumentException) t) { 10271 if(automaticForm is null) 10272 goto generic; 10273 auto container = this.htmlContainer(); 10274 if(cgi.requestMethod == Cgi.RequestMethod.POST) 10275 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 10276 container.appendChild(automaticForm); 10277 10278 cgi.write(container.parentDocument.toString(), true); 10279 } else { 10280 generic: 10281 auto container = this.htmlContainer(); 10282 10283 // import std.stdio; writeln(t.toString()); 10284 10285 container.appendChild(exceptionToElement(t)); 10286 10287 container.addChild("h4", "GET"); 10288 foreach(k, v; cgi.get) { 10289 auto deets = container.addChild("details"); 10290 deets.addChild("summary", k); 10291 deets.addChild("div", v); 10292 } 10293 10294 container.addChild("h4", "POST"); 10295 foreach(k, v; cgi.post) { 10296 auto deets = container.addChild("details"); 10297 deets.addChild("summary", k); 10298 deets.addChild("div", v); 10299 } 10300 10301 10302 if(!cgi.outputtedResponseData) 10303 cgi.setResponseStatus("500 Internal Server Error"); 10304 cgi.write(container.parentDocument.toString(), true); 10305 } 10306 } 10307 10308 Element exceptionToElement(Throwable t) { 10309 auto div = Element.make("div"); 10310 div.addClass("exception-display"); 10311 10312 div.addChild("p", t.msg); 10313 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10314 10315 auto pre = div.addChild("pre"); 10316 string s; 10317 s = t.toString(); 10318 Element currentBox; 10319 bool on = false; 10320 foreach(line; s.splitLines) { 10321 if(!on && line.startsWith("-----")) 10322 on = true; 10323 if(!on) continue; 10324 if(line.indexOf("arsd/") != -1) { 10325 if(currentBox is null) { 10326 currentBox = pre.addChild("details"); 10327 currentBox.addChild("summary", "Framework code"); 10328 } 10329 currentBox.addChild("span", line ~ "\n"); 10330 } else { 10331 pre.addChild("span", line ~ "\n"); 10332 currentBox = null; 10333 } 10334 } 10335 10336 return div; 10337 } 10338 10339 /++ 10340 Returns an element for a particular type 10341 +/ 10342 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10343 import std.traits; 10344 10345 auto div = Element.make("div"); 10346 div.addClass("form-field"); 10347 10348 static if(is(T : const Cgi.UploadedFile)) { 10349 Element lbl; 10350 if(displayName !is null) { 10351 lbl = div.addChild("label"); 10352 lbl.addChild("span", displayName, "label-text"); 10353 lbl.appendText(" "); 10354 } else { 10355 lbl = div; 10356 } 10357 auto i = lbl.addChild("input", name); 10358 i.attrs.name = name; 10359 i.attrs.type = "file"; 10360 i.attrs.multiple = "multiple"; 10361 } else static if(is(T == Cgi.UploadedFile)) { 10362 Element lbl; 10363 if(displayName !is null) { 10364 lbl = div.addChild("label"); 10365 lbl.addChild("span", displayName, "label-text"); 10366 lbl.appendText(" "); 10367 } else { 10368 lbl = div; 10369 } 10370 auto i = lbl.addChild("input", name); 10371 i.attrs.name = name; 10372 i.attrs.type = "file"; 10373 } else static if(is(T == enum)) { 10374 Element lbl; 10375 if(displayName !is null) { 10376 lbl = div.addChild("label"); 10377 lbl.addChild("span", displayName, "label-text"); 10378 lbl.appendText(" "); 10379 } else { 10380 lbl = div; 10381 } 10382 auto i = lbl.addChild("select", name); 10383 i.attrs.name = name; 10384 10385 foreach(memberName; __traits(allMembers, T)) 10386 i.addChild("option", memberName); 10387 10388 } else static if(is(T == struct)) { 10389 if(displayName !is null) 10390 div.addChild("span", displayName, "label-text"); 10391 auto fieldset = div.addChild("fieldset"); 10392 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10393 fieldset.addChild("input", name); 10394 foreach(idx, memberName; __traits(allMembers, T)) 10395 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10396 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10397 } 10398 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10399 Element lbl; 10400 if(displayName !is null) { 10401 lbl = div.addChild("label"); 10402 lbl.addChild("span", displayName, "label-text"); 10403 lbl.appendText(" "); 10404 } else { 10405 lbl = div; 10406 } 10407 Element i; 10408 if(udaSuggestion) { 10409 i = udaSuggestion(); 10410 lbl.appendChild(i); 10411 } else { 10412 i = lbl.addChild("input", name); 10413 } 10414 i.attrs.name = name; 10415 static if(isSomeString!T) 10416 i.attrs.type = "text"; 10417 else 10418 i.attrs.type = "number"; 10419 if(i.tagName == "textarea") 10420 i.textContent = to!string(T.init); 10421 else 10422 i.attrs.value = to!string(T.init); 10423 } else static if(is(T == bool)) { 10424 Element lbl; 10425 if(displayName !is null) { 10426 lbl = div.addChild("label"); 10427 lbl.addChild("span", displayName, "label-text"); 10428 lbl.appendText(" "); 10429 } else { 10430 lbl = div; 10431 } 10432 auto i = lbl.addChild("input", name); 10433 i.attrs.type = "checkbox"; 10434 i.attrs.value = "true"; 10435 i.attrs.name = name; 10436 } else static if(is(T == K[], K)) { 10437 auto templ = div.addChild("template"); 10438 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10439 if(displayName !is null) 10440 div.addChild("span", displayName, "label-text"); 10441 auto btn = div.addChild("button"); 10442 btn.addClass("add-array-button"); 10443 btn.attrs.type = "button"; 10444 btn.innerText = "Add"; 10445 btn.attrs.onclick = q{ 10446 var a = document.importNode(this.parentNode.firstChild.content, true); 10447 this.parentNode.insertBefore(a, this); 10448 }; 10449 } else static if(is(T == V[K], K, V)) { 10450 div.innerText = "assoc array not implemented for automatic form at this time"; 10451 } else { 10452 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10453 } 10454 10455 10456 return div; 10457 } 10458 10459 /// creates a form for gathering the function's arguments 10460 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10461 10462 auto form = cast(Form) Element.make("form"); 10463 10464 form.method = "POST"; // FIXME 10465 10466 form.addClass("automatic-form"); 10467 10468 string formDisplayName = beautify(__traits(identifier, method)); 10469 foreach(attr; __traits(getAttributes, method)) 10470 static if(is(typeof(attr) == DisplayName)) 10471 formDisplayName = attr.name; 10472 form.addChild("h3", formDisplayName); 10473 10474 import std.traits; 10475 10476 //Parameters!method params; 10477 //alias idents = ParameterIdentifierTuple!method; 10478 //alias defaults = ParameterDefaults!method; 10479 10480 static if(is(typeof(method) P == __parameters)) 10481 foreach(idx, _; P) {{ 10482 10483 alias param = P[idx .. idx + 1]; 10484 10485 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10486 string displayName = beautify(__traits(identifier, param)); 10487 Element function() element; 10488 foreach(attr; __traits(getAttributes, param)) { 10489 static if(is(typeof(attr) == DisplayName)) 10490 displayName = attr.name; 10491 else static if(is(typeof(attr) : typeof(element))) { 10492 element = attr; 10493 } 10494 } 10495 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10496 if(i.querySelector("input[type=file]") !is null) 10497 form.setAttribute("enctype", "multipart/form-data"); 10498 } 10499 }} 10500 10501 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10502 10503 return form; 10504 } 10505 10506 /// creates a form for gathering object members (for the REST object thing right now) 10507 Form createAutomaticFormForObject(T)(T obj) { 10508 auto form = cast(Form) Element.make("form"); 10509 10510 form.addClass("automatic-form"); 10511 10512 form.addChild("h3", beautify(__traits(identifier, T))); 10513 10514 import std.traits; 10515 10516 //Parameters!method params; 10517 //alias idents = ParameterIdentifierTuple!method; 10518 //alias defaults = ParameterDefaults!method; 10519 10520 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10521 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10522 string displayName = beautify(memberName); 10523 Element function() element; 10524 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10525 static if(is(typeof(attr) == DisplayName)) 10526 displayName = attr.name; 10527 else static if(is(typeof(attr) : typeof(element))) 10528 element = attr; 10529 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10530 10531 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10532 }}} 10533 10534 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10535 10536 return form; 10537 } 10538 10539 /// 10540 Element formatReturnValueAsHtml(T)(T t) { 10541 import std.traits; 10542 10543 static if(is(T == typeof(null))) { 10544 return Element.make("span"); 10545 } else static if(is(T : Element)) { 10546 return t; 10547 } else static if(is(T == MultipleResponses!Types, Types...)) { 10548 foreach(index, type; Types) { 10549 if(t.contains == index) 10550 return formatReturnValueAsHtml(t.payload[index]); 10551 } 10552 assert(0); 10553 } else static if(is(T == Paginated!E, E)) { 10554 auto e = Element.make("div").addClass("paginated-result"); 10555 e.appendChild(formatReturnValueAsHtml(t.items)); 10556 if(t.nextPageUrl.length) 10557 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10558 return e; 10559 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10560 return Element.make("span", to!string(t), "automatic-data-display"); 10561 } else static if(is(T == V[K], K, V)) { 10562 auto dl = Element.make("dl"); 10563 dl.addClass("automatic-data-display associative-array"); 10564 foreach(k, v; t) { 10565 dl.addChild("dt", to!string(k)); 10566 dl.addChild("dd", formatReturnValueAsHtml(v)); 10567 } 10568 return dl; 10569 } else static if(is(T == struct)) { 10570 auto dl = Element.make("dl"); 10571 dl.addClass("automatic-data-display struct"); 10572 10573 foreach(idx, memberName; __traits(allMembers, T)) 10574 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10575 dl.addChild("dt", beautify(memberName)); 10576 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10577 } 10578 10579 return dl; 10580 } else static if(is(T == bool)) { 10581 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10582 } else static if(is(T == E[], E)) { 10583 static if(is(E : RestObject!Proxy, Proxy)) { 10584 // treat RestObject similar to struct 10585 auto table = cast(Table) Element.make("table"); 10586 table.addClass("automatic-data-display"); 10587 string[] names; 10588 foreach(idx, memberName; __traits(derivedMembers, E)) 10589 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10590 names ~= beautify(memberName); 10591 } 10592 table.appendHeaderRow(names); 10593 10594 foreach(l; t) { 10595 auto tr = table.appendRow(); 10596 foreach(idx, memberName; __traits(derivedMembers, E)) 10597 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10598 static if(memberName == "id") { 10599 string val = to!string(__traits(getMember, l, memberName)); 10600 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10601 } else { 10602 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10603 } 10604 } 10605 } 10606 10607 return table; 10608 } else static if(is(E == struct)) { 10609 // an array of structs is kinda special in that I like 10610 // having those formatted as tables. 10611 auto table = cast(Table) Element.make("table"); 10612 table.addClass("automatic-data-display"); 10613 string[] names; 10614 foreach(idx, memberName; __traits(allMembers, E)) 10615 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10616 names ~= beautify(memberName); 10617 } 10618 table.appendHeaderRow(names); 10619 10620 foreach(l; t) { 10621 auto tr = table.appendRow(); 10622 foreach(idx, memberName; __traits(allMembers, E)) 10623 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10624 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10625 } 10626 } 10627 10628 return table; 10629 } else { 10630 // otherwise, I will just make a list. 10631 auto ol = Element.make("ol"); 10632 ol.addClass("automatic-data-display"); 10633 foreach(e; t) 10634 ol.addChild("li", formatReturnValueAsHtml(e)); 10635 return ol; 10636 } 10637 } else static if(is(T : Object)) { 10638 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10639 return Element.make("div", t.toHtml()); 10640 else 10641 return Element.make("div", t.toString()); 10642 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10643 10644 assert(0); 10645 } 10646 10647 } 10648 10649 /++ 10650 The base class for the [dispatcher] function and object support. 10651 +/ 10652 class WebObject { 10653 //protected Cgi cgi; 10654 10655 protected void initialize(Cgi cgi) { 10656 //this.cgi = cgi; 10657 } 10658 } 10659 10660 /++ 10661 Can return one of the given types, decided at runtime. The syntax 10662 is to declare all the possible types in the return value, then you 10663 can `return typeof(return)(...value...)` to construct it. 10664 10665 It has an auto-generated constructor for each value it can hold. 10666 10667 --- 10668 MultipleResponses!(Redirection, string) getData(int how) { 10669 if(how & 1) 10670 return typeof(return)(Redirection("http://dpldocs.info/")); 10671 else 10672 return typeof(return)("hi there!"); 10673 } 10674 --- 10675 10676 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10677 +/ 10678 struct MultipleResponses(T...) { 10679 private size_t contains; 10680 private union { 10681 private T payload; 10682 } 10683 10684 static foreach(index, type; T) 10685 public this(type t) { 10686 contains = index; 10687 payload[index] = t; 10688 } 10689 10690 /++ 10691 This is primarily for testing. It is your way of getting to the response. 10692 10693 Let's say you wanted to test that one holding a Redirection and a string actually 10694 holds a string, by name of "test": 10695 10696 --- 10697 auto valueToTest = your_test_function(); 10698 10699 valueToTest.visit( 10700 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10701 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10702 ); 10703 --- 10704 10705 History: 10706 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10707 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10708 +/ 10709 void visit(Handlers...)(Handlers handlers) { 10710 template findHandler(type, int count, HandlersToCheck...) { 10711 static if(HandlersToCheck.length == 0) 10712 enum findHandler = -1; 10713 else { 10714 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10715 enum findHandler = count; 10716 else 10717 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10718 } 10719 } 10720 foreach(index, type; T) { 10721 enum handlerIndex = findHandler!(type, 0, Handlers); 10722 static if(handlerIndex == -1) 10723 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10724 else { 10725 if(index == this.contains) 10726 handlers[handlerIndex](this.payload[index]); 10727 } 10728 } 10729 } 10730 10731 /+ 10732 auto toArsdJsvar()() { 10733 import arsd.jsvar; 10734 return var(null); 10735 } 10736 +/ 10737 } 10738 10739 // FIXME: implement this somewhere maybe 10740 struct RawResponse { 10741 int code; 10742 string[] headers; 10743 const(ubyte)[] responseBody; 10744 } 10745 10746 /++ 10747 You can return this from [WebObject] subclasses for redirections. 10748 10749 (though note the static types means that class must ALWAYS redirect if 10750 you return this directly. You might want to return [MultipleResponses] if it 10751 can be conditional) 10752 +/ 10753 struct Redirection { 10754 string to; /// The URL to redirect to. 10755 int code = 303; /// The HTTP code to return. 10756 } 10757 10758 /++ 10759 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10760 10761 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10762 the presenter in the dispatcher. 10763 10764 FIXME: explain this better 10765 10766 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10767 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10768 the runtime result of that is undefined. 10769 10770 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10771 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10772 10773 $(WARNING 10774 --- 10775 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10776 // even if you put different URL udas on it, the current code ignores them. 10777 void foo(int a) {} 10778 void foo(string a) {} 10779 --- 10780 ) 10781 10782 See_Also: [serveRestObject], [serveStaticFile] 10783 +/ 10784 auto serveApi(T)(string urlPrefix) { 10785 assert(urlPrefix[$ - 1] == '/'); 10786 return serveApiInternal!T(urlPrefix); 10787 } 10788 10789 private string nextPieceFromSlash(ref string remainingUrl) { 10790 if(remainingUrl.length == 0) 10791 return remainingUrl; 10792 int slash = 0; 10793 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10794 slash++; 10795 10796 // I am specifically passing `null` to differentiate it vs empty string 10797 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10798 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10799 // so if it is the last item, the dot can be used to load an alternative view 10800 // otherwise tho the dot is considered part of the identifier 10801 // FIXME 10802 10803 // again notice "" vs null here! 10804 if(slash == remainingUrl.length) 10805 remainingUrl = null; 10806 else 10807 remainingUrl = remainingUrl[slash + 1 .. $]; 10808 10809 return ident; 10810 } 10811 10812 /++ 10813 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10814 +/ 10815 enum AddTrailingSlash; 10816 /// ditto 10817 enum RemoveTrailingSlash; 10818 10819 private auto serveApiInternal(T)(string urlPrefix) { 10820 10821 import arsd.dom; 10822 import arsd.jsvar; 10823 10824 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10825 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10826 10827 try { 10828 // see duplicated code below by searching subresource_ctor 10829 // also see mustNotBeSetFromWebParams 10830 10831 static if(is(typeof(T.__ctor) P == __parameters)) { 10832 P params; 10833 10834 foreach(pidx, param; P) { 10835 static if(is(param : Cgi)) { 10836 static assert(!is(param == immutable)); 10837 cast() params[pidx] = cgi; 10838 } else static if(is(param == Session!D, D)) { 10839 static assert(!is(param == immutable)); 10840 cast() params[pidx] = cgi.getSessionObject!D(); 10841 10842 } else { 10843 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10844 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10845 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10846 static if(is(typeof(func(cgi)))) 10847 params[pidx] = func(cgi); 10848 else 10849 params[pidx] = func(); 10850 } 10851 } 10852 } else { 10853 10854 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10855 params[pidx] = param.getAutomaticallyForCgi(cgi); 10856 } else static if(is(param == string)) { 10857 auto ident = nextPieceFromSlash(remainingUrl); 10858 params[pidx] = ident; 10859 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10860 } 10861 } 10862 } 10863 10864 auto obj = new T(params); 10865 } else { 10866 auto obj = new T(); 10867 } 10868 10869 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10870 } catch(Throwable t) { 10871 switch(cgi.request("format", "html")) { 10872 case "html": 10873 static void dummy() {} 10874 presenter.presentExceptionAsHtml(cgi, t, null); 10875 return true; 10876 case "json": 10877 var envelope = var.emptyObject; 10878 envelope.success = false; 10879 envelope.result = null; 10880 envelope.error = t.toString(); 10881 cgi.setResponseContentType("application/json"); 10882 cgi.write(envelope.toJson(), true); 10883 return true; 10884 default: 10885 throw t; 10886 // return true; 10887 } 10888 // return true; 10889 } 10890 10891 assert(0); 10892 } 10893 10894 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10895 10896 obj.initialize(cgi); 10897 10898 /+ 10899 Overload rules: 10900 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10901 statically. 10902 10903 Moreover, some args vs no args can be overloaded dynamically. 10904 +/ 10905 10906 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10907 /+ 10908 auto orig = remainingUrl; 10909 assert(0, 10910 (orig is null ? "__null" : orig) 10911 ~ " .. " ~ 10912 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10913 +/ 10914 10915 if(methodNameFromUrl is null) 10916 methodNameFromUrl = "__null"; 10917 10918 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10919 10920 if(remainingUrl.length) 10921 hack ~= "/"; 10922 10923 switch(hack) { 10924 foreach(methodName; __traits(derivedMembers, T)) 10925 static if(methodName != "__ctor") 10926 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10927 static if(is(typeof(overload) P == __parameters)) 10928 static if(is(typeof(overload) R == return)) 10929 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10930 { 10931 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10932 case urlNameForMethod: 10933 10934 static if(is(R : WebObject)) { 10935 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10936 10937 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10938 10939 // subresource_ctor 10940 // also see mustNotBeSetFromWebParams 10941 10942 P params; 10943 10944 string ident; 10945 10946 foreach(pidx, param; P) { 10947 static if(is(param : Cgi)) { 10948 static assert(!is(param == immutable)); 10949 cast() params[pidx] = cgi; 10950 } else static if(is(param == typeof(presenter))) { 10951 cast() param[pidx] = presenter; 10952 } else static if(is(param == Session!D, D)) { 10953 static assert(!is(param == immutable)); 10954 cast() params[pidx] = cgi.getSessionObject!D(); 10955 } else { 10956 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10957 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10958 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10959 static if(is(typeof(func(cgi)))) 10960 params[pidx] = func(cgi); 10961 else 10962 params[pidx] = func(); 10963 } 10964 } 10965 } else { 10966 10967 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10968 params[pidx] = param.getAutomaticallyForCgi(cgi); 10969 } else static if(is(param == string)) { 10970 ident = nextPieceFromSlash(remainingUrl); 10971 if(ident is null) { 10972 // trailing slash mandated on subresources 10973 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10974 return true; 10975 } else { 10976 params[pidx] = ident; 10977 } 10978 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10979 } 10980 } 10981 } 10982 10983 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10984 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10985 } else { 10986 // 404 it if any url left - not a subresource means we don't get to play with that! 10987 if(remainingUrl.length) 10988 return false; 10989 10990 bool automaticForm; 10991 10992 foreach(attr; __traits(getAttributes, overload)) 10993 static if(is(attr == AddTrailingSlash)) { 10994 if(remainingUrl is null) { 10995 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10996 return true; 10997 } 10998 } else static if(is(attr == RemoveTrailingSlash)) { 10999 if(remainingUrl !is null) { 11000 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 11001 return true; 11002 } 11003 11004 } else static if(__traits(isSame, AutomaticForm, attr)) { 11005 automaticForm = true; 11006 } 11007 11008 /+ 11009 int zeroArgOverload = -1; 11010 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 11011 bool calledWithZeroArgs = true; 11012 foreach(k, v; cgi.get) 11013 if(k != "format") { 11014 calledWithZeroArgs = false; 11015 break; 11016 } 11017 foreach(k, v; cgi.post) 11018 if(k != "format") { 11019 calledWithZeroArgs = false; 11020 break; 11021 } 11022 11023 // first, we need to go through and see if there is an empty one, since that 11024 // changes inside. But otherwise, all the stuff I care about can be done via 11025 // simple looping (other improper overloads might be flagged for runtime semantic check) 11026 // 11027 // an argument of type Cgi is ignored for these purposes 11028 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 11029 static if(is(typeof(overload) P == __parameters)) 11030 static if(P.length == 0) 11031 zeroArgOverload = cast(int) idx; 11032 else static if(P.length == 1 && is(P[0] : Cgi)) 11033 zeroArgOverload = cast(int) idx; 11034 }} 11035 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 11036 bool overloadHasBeenCalled = false; 11037 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 11038 bool callFunction = true; 11039 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 11040 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 11041 callFunction = false; 11042 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 11043 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 11044 callFunction = false; 11045 11046 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 11047 11048 bool hadAnyMethodRestrictions = false; 11049 bool foundAcceptableMethod = false; 11050 foreach(attr; __traits(getAttributes, overload)) { 11051 static if(is(typeof(attr) == Cgi.RequestMethod)) { 11052 hadAnyMethodRestrictions = true; 11053 if(attr == cgi.requestMethod) 11054 foundAcceptableMethod = true; 11055 } 11056 } 11057 11058 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 11059 callFunction = false; 11060 11061 /+ 11062 The overloads we really want to allow are the sane ones 11063 from the web perspective. Which is likely on HTTP verbs, 11064 for the most part, but might also be potentially based on 11065 some args vs zero args, or on argument names. Can't really 11066 do argument types very reliable through the web though; those 11067 should probably be different URLs. 11068 11069 Even names I feel is better done inside the function, so I'm not 11070 going to support that here. But the HTTP verbs and zero vs some 11071 args makes sense - it lets you define custom forms pretty easily. 11072 11073 Moreover, I'm of the opinion that empty overload really only makes 11074 sense on GET for this case. On a POST, it is just a missing argument 11075 exception and that should be handled by the presenter. But meh, I'll 11076 let the user define that, D only allows one empty arg thing anyway 11077 so the method UDAs are irrelevant. 11078 +/ 11079 if(callFunction) 11080 +/ 11081 11082 auto format = cgi.request("format", defaultFormat!overload()); 11083 auto wantsFormFormat = format.startsWith("form-"); 11084 11085 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 11086 // Should I still show the form on a json thing? idk... 11087 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 11088 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 11089 return true; 11090 } 11091 11092 try { 11093 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 11094 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 11095 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 11096 } catch(Throwable t) { 11097 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 11098 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 11099 } 11100 return true; 11101 //}} 11102 11103 //cgi.header("Accept: POST"); // FIXME list the real thing 11104 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 11105 //return true; 11106 } 11107 } 11108 } 11109 case "GET script.js": 11110 cgi.setResponseContentType("text/javascript"); 11111 cgi.gzipResponse = true; 11112 cgi.write(presenter.script(), true); 11113 return true; 11114 case "GET style.css": 11115 cgi.setResponseContentType("text/css"); 11116 cgi.gzipResponse = true; 11117 cgi.write(presenter.style(), true); 11118 return true; 11119 default: 11120 return false; 11121 } 11122 11123 assert(0); 11124 } 11125 return DispatcherDefinition!internalHandler(urlPrefix, false); 11126 } 11127 11128 string defaultFormat(alias method)() { 11129 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 11130 foreach(attr; __traits(getAttributes, method)) { 11131 static if(is(typeof(attr) == DefaultFormat)) { 11132 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 11133 return attr.value; 11134 } 11135 } 11136 return "html"; 11137 } 11138 11139 struct Paginated(T) { 11140 T[] items; 11141 string nextPageUrl; 11142 } 11143 11144 template urlNamesForMethod(alias method, string default_) { 11145 string[] helper() { 11146 auto verb = Cgi.RequestMethod.GET; 11147 bool foundVerb = false; 11148 bool foundNoun = false; 11149 11150 string def = default_; 11151 11152 bool hasAutomaticForm = false; 11153 11154 foreach(attr; __traits(getAttributes, method)) { 11155 static if(is(typeof(attr) == Cgi.RequestMethod)) { 11156 verb = attr; 11157 if(foundVerb) 11158 assert(0, "Multiple http verbs on one function is not currently supported"); 11159 foundVerb = true; 11160 } 11161 static if(is(typeof(attr) == UrlName)) { 11162 if(foundNoun) 11163 assert(0, "Multiple url names on one function is not currently supported"); 11164 foundNoun = true; 11165 def = attr.name; 11166 } 11167 static if(__traits(isSame, attr, AutomaticForm)) { 11168 hasAutomaticForm = true; 11169 } 11170 } 11171 11172 if(def is null) 11173 def = "__null"; 11174 11175 string[] ret; 11176 11177 static if(is(typeof(method) R == return)) { 11178 static if(is(R : WebObject)) { 11179 def ~= "/"; 11180 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 11181 ret ~= v ~ " " ~ def; 11182 } else { 11183 if(hasAutomaticForm) { 11184 ret ~= "GET " ~ def; 11185 ret ~= "POST " ~ def; 11186 } else { 11187 ret ~= to!string(verb) ~ " " ~ def; 11188 } 11189 } 11190 } else static assert(0); 11191 11192 return ret; 11193 } 11194 enum urlNamesForMethod = helper(); 11195 } 11196 11197 11198 enum AccessCheck { 11199 allowed, 11200 denied, 11201 nonExistant, 11202 } 11203 11204 enum Operation { 11205 show, 11206 create, 11207 replace, 11208 remove, 11209 update 11210 } 11211 11212 enum UpdateResult { 11213 accessDenied, 11214 noSuchResource, 11215 success, 11216 failure, 11217 unnecessary 11218 } 11219 11220 enum ValidationResult { 11221 valid, 11222 invalid 11223 } 11224 11225 11226 /++ 11227 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 11228 11229 WARNING: this is not stable. 11230 +/ 11231 class RestObject(CRTP) : WebObject { 11232 11233 import arsd.dom; 11234 import arsd.jsvar; 11235 11236 /// Prepare the object to be shown. 11237 void show() {} 11238 /// ditto 11239 void show(string urlId) { 11240 load(urlId); 11241 show(); 11242 } 11243 11244 /// Override this to provide access control to this object. 11245 AccessCheck accessCheck(string urlId, Operation operation) { 11246 return AccessCheck.allowed; 11247 } 11248 11249 ValidationResult validate() { 11250 // FIXME 11251 return ValidationResult.valid; 11252 } 11253 11254 string getUrlSlug() { 11255 import std.conv; 11256 static if(is(typeof(CRTP.id))) 11257 return to!string((cast(CRTP) this).id); 11258 else 11259 return null; 11260 } 11261 11262 // The functions with more arguments are the low-level ones, 11263 // they forward to the ones with fewer arguments by default. 11264 11265 // POST on a parent collection - this is called from a collection class after the members are updated 11266 /++ 11267 Given a populated object, this creates a new entry. Returns the url identifier 11268 of the new object. 11269 +/ 11270 string create(scope void delegate() applyChanges) { 11271 applyChanges(); 11272 save(); 11273 return getUrlSlug(); 11274 } 11275 11276 void replace() { 11277 save(); 11278 } 11279 void replace(string urlId, scope void delegate() applyChanges) { 11280 load(urlId); 11281 applyChanges(); 11282 replace(); 11283 } 11284 11285 void update(string[] fieldList) { 11286 save(); 11287 } 11288 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11289 load(urlId); 11290 applyChanges(); 11291 update(fieldList); 11292 } 11293 11294 void remove() {} 11295 11296 void remove(string urlId) { 11297 load(urlId); 11298 remove(); 11299 } 11300 11301 abstract void load(string urlId); 11302 abstract void save(); 11303 11304 Element toHtml(Presenter)(Presenter presenter) { 11305 import arsd.dom; 11306 import std.conv; 11307 auto obj = cast(CRTP) this; 11308 auto div = Element.make("div"); 11309 div.addClass("Dclass_" ~ CRTP.stringof); 11310 div.dataset.url = getUrlSlug(); 11311 bool first = true; 11312 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11313 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11314 if(!first) div.addChild("br"); else first = false; 11315 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11316 } 11317 return div; 11318 } 11319 11320 var toJson() { 11321 import arsd.jsvar; 11322 var v = var.emptyObject(); 11323 auto obj = cast(CRTP) this; 11324 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11325 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11326 v[memberName] = __traits(getMember, obj, memberName); 11327 } 11328 return v; 11329 } 11330 11331 /+ 11332 auto structOf(this This) { 11333 11334 } 11335 +/ 11336 } 11337 11338 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11339 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11340 11341 /++ 11342 Base class for REST collections. 11343 +/ 11344 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11345 /// You might subclass this and use the cgi object's query params 11346 /// to implement a search filter, for example. 11347 /// 11348 /// FIXME: design a way to auto-generate that form 11349 /// (other than using the WebObject thing above lol 11350 // it'll prolly just be some searchParams UDA or maybe an enum. 11351 // 11352 // pagination too perhaps. 11353 // 11354 // and sorting too 11355 IndexResult index() { return IndexResult.init; } 11356 11357 string[] sortableFields() { return null; } 11358 string[] searchableFields() { return null; } 11359 11360 struct IndexResult { 11361 Obj[] results; 11362 11363 string[] sortableFields; 11364 11365 string previousPageIdentifier; 11366 string nextPageIdentifier; 11367 string firstPageIdentifier; 11368 string lastPageIdentifier; 11369 11370 int numberOfPages; 11371 } 11372 11373 override string create(scope void delegate() applyChanges) { assert(0); } 11374 override void load(string urlId) { assert(0); } 11375 override void save() { assert(0); } 11376 override void show() { 11377 index(); 11378 } 11379 override void show(string urlId) { 11380 show(); 11381 } 11382 11383 /// Proxy POST requests (create calls) to the child collection 11384 alias PostProxy = Obj; 11385 } 11386 11387 /++ 11388 Serves a REST object, similar to a Ruby on Rails resource. 11389 11390 You put data members in your class. cgi.d will automatically make something out of those. 11391 11392 It will call your constructor with the ID from the URL. This may be null. 11393 It will then populate the data members from the request. 11394 It will then call a method, if present, telling what happened. You don't need to write these! 11395 It finally returns a reply. 11396 11397 Your methods are passed a list of fields it actually set. 11398 11399 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11400 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11401 with relative linking. But meh.) 11402 11403 GET /items -> index. all values not set. 11404 GET /items/id -> get. only ID will be set, other params ignored. 11405 POST /items -> create. values set as given 11406 PUT /items/id -> replace. values set as given 11407 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11408 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11409 PATCH /items/id -> update. values set as given, list of changed fields passed 11410 or POST /items/id with cgi.post["_method"] == "PATCH" 11411 DELETE /items/id -> destroy. only ID guaranteed to be set 11412 or POST /items/id with cgi.post["_method"] == "DELETE" 11413 11414 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11415 redirect you away from it. 11416 11417 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11418 11419 I will also let you change the default, if you must. 11420 11421 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11422 11423 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11424 that follow the same thing. They may be individual resources or collections themselves. 11425 11426 Your class is expected to have at least the following methods: 11427 11428 FIXME: i kinda wanna add a routes object to the initialize call 11429 11430 create 11431 Create returns the new address on success, some code on failure. 11432 show 11433 index 11434 update 11435 remove 11436 11437 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11438 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11439 11440 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11441 11442 NOT IMPLEMENTED 11443 11444 11445 Really, a collection is a resource with a bunch of subresources. 11446 11447 GET /items 11448 index because it is GET on the top resource 11449 11450 GET /items/foo 11451 item but different than items? 11452 11453 class Items { 11454 11455 } 11456 11457 ... but meh, a collection can be automated. not worth making it 11458 a separate thing, let's look at a real example. Users has many 11459 items and a virtual one, /users/current. 11460 11461 the individual users have properties and two sub-resources: 11462 session, which is just one, and comments, a collection. 11463 11464 class User : RestObject!() { // no parent 11465 int id; 11466 string name; 11467 11468 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11469 // but you can override them to do it differently. 11470 11471 // any member which is of type RestObject can be linked automatically via href btw. 11472 11473 void show() {} 11474 void show(string urlId) {} // automated! GET of this specific thing 11475 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11476 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11477 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11478 void remove(string urlId) {} // DELETE 11479 11480 void load(string urlId) {} // the default implementation of show() populates the id, then 11481 11482 this() {} 11483 11484 mixin Subresource!Session; 11485 mixin Subresource!Comment; 11486 } 11487 11488 class Session : RestObject!() { 11489 // the parent object may not be fully constructed/loaded 11490 this(User parent) {} 11491 11492 } 11493 11494 class Comment : CollectionOf!Comment { 11495 this(User parent) {} 11496 } 11497 11498 class Users : CollectionOf!User { 11499 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11500 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11501 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11502 } 11503 11504 +/ 11505 auto serveRestObject(T)(string urlPrefix) { 11506 assert(urlPrefix[0] == '/'); 11507 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11508 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11509 string url = cgi.pathInfo[urlPrefix.length .. $]; 11510 11511 if(url.length && url[$ - 1] == '/') { 11512 // remove the final slash... 11513 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11514 return true; 11515 } 11516 11517 return restObjectServeHandler!T(cgi, presenter, url); 11518 } 11519 return DispatcherDefinition!internalHandler(urlPrefix, false); 11520 } 11521 11522 /+ 11523 /// Convenience method for serving a collection. It will be named the same 11524 /// as type T, just with an s at the end. If you need any further, just 11525 /// write the class yourself. 11526 auto serveRestCollectionOf(T)(string urlPrefix) { 11527 assert(urlPrefix[0] == '/'); 11528 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11529 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11530 } 11531 +/ 11532 11533 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11534 string urlId = null; 11535 if(url.length && url[0] == '/') { 11536 // asking for a subobject 11537 urlId = url[1 .. $]; 11538 foreach(idx, ch; urlId) { 11539 if(ch == '/') { 11540 urlId = urlId[0 .. idx]; 11541 break; 11542 } 11543 } 11544 } 11545 11546 // FIXME handle other subresources 11547 11548 static if(is(T : CollectionOf!(C), C)) { 11549 if(urlId !is null) { 11550 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11551 } 11552 } 11553 11554 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11555 11556 auto obj = new T(); 11557 obj.initialize(cgi); 11558 // FIXME: populate reflection info delegates 11559 11560 11561 // FIXME: I am not happy with this. 11562 switch(urlId) { 11563 case "script.js": 11564 cgi.setResponseContentType("text/javascript"); 11565 cgi.gzipResponse = true; 11566 cgi.write(presenter.script(), true); 11567 return true; 11568 case "style.css": 11569 cgi.setResponseContentType("text/css"); 11570 cgi.gzipResponse = true; 11571 cgi.write(presenter.style(), true); 11572 return true; 11573 default: 11574 // intentionally blank 11575 } 11576 11577 11578 11579 11580 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11581 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11582 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11583 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11584 } 11585 } 11586 void applyChanges() { 11587 applyChangesTemplate(cgi, obj); 11588 } 11589 11590 string[] modifiedList; 11591 11592 void writeObject(bool addFormLinks) { 11593 if(cgi.request("format") == "json") { 11594 cgi.setResponseContentType("application/json"); 11595 cgi.write(obj.toJson().toString, true); 11596 } else { 11597 auto container = presenter.htmlContainer(); 11598 if(addFormLinks) { 11599 static if(is(T : CollectionOf!(C), C)) 11600 container.appendHtml(` 11601 <form> 11602 <button type="submit" name="_method" value="POST">Create New</button> 11603 </form> 11604 `); 11605 else 11606 container.appendHtml(` 11607 <a href="..">Back</a> 11608 <form> 11609 <button type="submit" name="_method" value="PATCH">Edit</button> 11610 <button type="submit" name="_method" value="DELETE">Delete</button> 11611 </form> 11612 `); 11613 } 11614 container.appendChild(obj.toHtml(presenter)); 11615 cgi.write(container.parentDocument.toString, true); 11616 } 11617 } 11618 11619 // FIXME: I think I need a set type in here.... 11620 // it will be nice to pass sets of members. 11621 11622 try 11623 switch(cgi.requestMethod) { 11624 case Cgi.RequestMethod.GET: 11625 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11626 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11627 11628 // automatic forms here for usable basic auto site from browser. 11629 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11630 switch(cgi.request("_method", "GET")) { 11631 case "GET": 11632 static if(is(T : CollectionOf!(C), C)) { 11633 auto results = obj.index(); 11634 if(cgi.request("format", "html") == "html") { 11635 auto container = presenter.htmlContainer(); 11636 auto html = presenter.formatReturnValueAsHtml(results.results); 11637 container.appendHtml(` 11638 <form> 11639 <button type="submit" name="_method" value="POST">Create New</button> 11640 </form> 11641 `); 11642 11643 container.appendChild(html); 11644 cgi.write(container.parentDocument.toString, true); 11645 } else { 11646 cgi.setResponseContentType("application/json"); 11647 import arsd.jsvar; 11648 var json = var.emptyArray; 11649 foreach(r; results.results) { 11650 var o = var.emptyObject; 11651 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11652 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11653 o[memberName] = __traits(getMember, r, memberName); 11654 } 11655 11656 json ~= o; 11657 } 11658 cgi.write(json.toJson(), true); 11659 } 11660 } else { 11661 obj.show(urlId); 11662 writeObject(true); 11663 } 11664 break; 11665 case "PATCH": 11666 obj.load(urlId); 11667 goto case; 11668 case "PUT": 11669 case "POST": 11670 // an editing form for the object 11671 auto container = presenter.htmlContainer(); 11672 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11673 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11674 } else { 11675 auto form = presenter.createAutomaticFormForObject(obj); 11676 } 11677 form.attrs.method = "POST"; 11678 form.setValue("_method", cgi.request("_method", "GET")); 11679 container.appendChild(form); 11680 cgi.write(container.parentDocument.toString(), true); 11681 break; 11682 case "DELETE": 11683 // FIXME: a delete form for the object (can be phrased "are you sure?") 11684 auto container = presenter.htmlContainer(); 11685 container.appendHtml(` 11686 <form method="POST"> 11687 Are you sure you want to delete this item? 11688 <input type="hidden" name="_method" value="DELETE" /> 11689 <input type="submit" value="Yes, Delete It" /> 11690 </form> 11691 11692 `); 11693 cgi.write(container.parentDocument.toString(), true); 11694 break; 11695 default: 11696 cgi.write("bad method\n", true); 11697 } 11698 break; 11699 case Cgi.RequestMethod.POST: 11700 // this is to allow compatibility with HTML forms 11701 switch(cgi.request("_method", "POST")) { 11702 case "PUT": 11703 goto PUT; 11704 case "PATCH": 11705 goto PATCH; 11706 case "DELETE": 11707 goto DELETE; 11708 case "POST": 11709 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11710 auto p = new obj.PostProxy(); 11711 void specialApplyChanges() { 11712 applyChangesTemplate(cgi, p); 11713 } 11714 string n = p.create(&specialApplyChanges); 11715 } else { 11716 string n = obj.create(&applyChanges); 11717 } 11718 11719 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11720 cgi.setResponseLocation(newUrl); 11721 cgi.setResponseStatus("201 Created"); 11722 cgi.write(`The object has been created.`); 11723 break; 11724 default: 11725 cgi.write("bad method\n", true); 11726 } 11727 // FIXME this should be valid on the collection, but not the child.... 11728 // 303 See Other 11729 break; 11730 case Cgi.RequestMethod.PUT: 11731 PUT: 11732 obj.replace(urlId, &applyChanges); 11733 writeObject(false); 11734 break; 11735 case Cgi.RequestMethod.PATCH: 11736 PATCH: 11737 obj.update(urlId, &applyChanges, modifiedList); 11738 writeObject(false); 11739 break; 11740 case Cgi.RequestMethod.DELETE: 11741 DELETE: 11742 obj.remove(urlId); 11743 cgi.setResponseStatus("204 No Content"); 11744 break; 11745 default: 11746 // FIXME: OPTIONS, HEAD 11747 } 11748 catch(Throwable t) { 11749 presenter.presentExceptionAsHtml(cgi, t); 11750 } 11751 11752 return true; 11753 } 11754 11755 /+ 11756 struct SetOfFields(T) { 11757 private void[0][string] storage; 11758 void set(string what) { 11759 //storage[what] = 11760 } 11761 void unset(string what) {} 11762 void setAll() {} 11763 void unsetAll() {} 11764 bool isPresent(string what) { return false; } 11765 } 11766 +/ 11767 11768 /+ 11769 enum readonly; 11770 enum hideonindex; 11771 +/ 11772 11773 /++ 11774 Returns true if I recommend gzipping content of this type. You might 11775 want to call it from your Presenter classes before calling cgi.write. 11776 11777 --- 11778 cgi.setResponseContentType(yourContentType); 11779 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11780 cgi.write(yourData, true); 11781 --- 11782 11783 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11784 11785 11786 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11787 11788 History: 11789 Added January 28, 2023 (dub v11.0) 11790 +/ 11791 bool gzipRecommendedForContentType(string contentType) { 11792 if(contentType.startsWith("text/")) 11793 return true; 11794 if(contentType.startsWith("application/javascript")) 11795 return true; 11796 11797 return false; 11798 } 11799 11800 /++ 11801 Serves a static file. To be used with [dispatcher]. 11802 11803 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11804 +/ 11805 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11806 // https://baus.net/on-tcp_cork/ 11807 // man 2 sendfile 11808 assert(urlPrefix[0] == '/'); 11809 if(filename is null) 11810 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11811 if(contentType is null) { 11812 contentType = contentTypeFromFileExtension(filename); 11813 } 11814 11815 static struct DispatcherDetails { 11816 string filename; 11817 string contentType; 11818 } 11819 11820 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11821 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11822 cgi.setCache(true); 11823 cgi.setResponseContentType(details.contentType); 11824 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11825 cgi.write(std.file.read(details.filename), true); 11826 return true; 11827 } 11828 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11829 } 11830 11831 /++ 11832 Serves static data. To be used with [dispatcher]. 11833 11834 History: 11835 Added October 31, 2021 11836 +/ 11837 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11838 assert(urlPrefix[0] == '/'); 11839 if(contentType is null) { 11840 contentType = contentTypeFromFileExtension(urlPrefix); 11841 } 11842 11843 static struct DispatcherDetails { 11844 immutable(void)[] data; 11845 string contentType; 11846 } 11847 11848 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11849 cgi.setCache(true); 11850 cgi.setResponseContentType(details.contentType); 11851 cgi.write(details.data, true); 11852 return true; 11853 } 11854 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11855 } 11856 11857 string contentTypeFromFileExtension(string filename) { 11858 if(filename.endsWith(".png")) 11859 return "image/png"; 11860 if(filename.endsWith(".apng")) 11861 return "image/apng"; 11862 if(filename.endsWith(".svg")) 11863 return "image/svg+xml"; 11864 if(filename.endsWith(".jpg")) 11865 return "image/jpeg"; 11866 if(filename.endsWith(".html")) 11867 return "text/html"; 11868 if(filename.endsWith(".css")) 11869 return "text/css"; 11870 if(filename.endsWith(".js")) 11871 return "application/javascript"; 11872 if(filename.endsWith(".wasm")) 11873 return "application/wasm"; 11874 if(filename.endsWith(".mp3")) 11875 return "audio/mpeg"; 11876 if(filename.endsWith(".pdf")) 11877 return "application/pdf"; 11878 return null; 11879 } 11880 11881 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11882 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11883 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11884 assert(urlPrefix[0] == '/'); 11885 assert(urlPrefix[$-1] == '/'); 11886 11887 static struct DispatcherDetails { 11888 string directory; 11889 bool recursive; 11890 } 11891 11892 if(directory is null) 11893 directory = urlPrefix[1 .. $]; 11894 11895 if(directory.length == 0) 11896 directory = "./"; 11897 11898 assert(directory[$-1] == '/'); 11899 11900 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11901 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11902 11903 if(details.recursive) { 11904 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11905 if(file.indexOf("\\") != -1) 11906 return false; 11907 11908 import std.path; 11909 11910 file = std.path.buildNormalizedPath(file); 11911 enum upOneDir = ".." ~ std.path.dirSeparator; 11912 11913 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11914 if(file == ".." || file.startsWith(upOneDir)) 11915 return false; 11916 if(std.path.isAbsolute(file)) 11917 return false; 11918 11919 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11920 11921 // once it passes these filters it is probably ok. 11922 } else { 11923 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11924 return false; 11925 } 11926 11927 if(file.length == 0) 11928 return false; 11929 11930 auto contentType = contentTypeFromFileExtension(file); 11931 11932 auto fn = details.directory ~ file; 11933 if(std.file.exists(fn)) { 11934 //if(contentType.indexOf("image/") == 0) 11935 //cgi.setCache(true); 11936 //else if(contentType.indexOf("audio/") == 0) 11937 cgi.setCache(true); 11938 cgi.setResponseContentType(contentType); 11939 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11940 cgi.write(std.file.read(fn), true); 11941 return true; 11942 } else { 11943 return false; 11944 } 11945 } 11946 11947 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11948 } 11949 11950 /++ 11951 Redirects one url to another 11952 11953 See_Also: [dispatcher], [serveStaticFile] 11954 +/ 11955 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11956 assert(urlPrefix[0] == '/'); 11957 static struct DispatcherDetails { 11958 string redirectTo; 11959 string code; 11960 } 11961 11962 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11963 cgi.setResponseLocation(details.redirectTo, true, details.code); 11964 return true; 11965 } 11966 11967 11968 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11969 } 11970 11971 /// Used exclusively with `dispatchTo` 11972 struct DispatcherData(Presenter) { 11973 Cgi cgi; /// You can use this cgi object. 11974 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11975 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11976 } 11977 11978 /++ 11979 Dispatches the URL to a specific function. 11980 +/ 11981 auto handleWith(alias handler)(string urlPrefix) { 11982 // cuz I'm too lazy to do it better right now 11983 static class Hack : WebObject { 11984 static import std.traits; 11985 @UrlName("") 11986 auto handle(std.traits.Parameters!handler args) { 11987 return handler(args); 11988 } 11989 } 11990 11991 return urlPrefix.serveApiInternal!Hack; 11992 } 11993 11994 /++ 11995 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11996 11997 --- 11998 bool other(DD)(DD dd) { 11999 return dd.dispatcher!( 12000 "/whatever".serveRedirect("/success"), 12001 "/api/".serveApi!MyClass 12002 ); 12003 } 12004 --- 12005 12006 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 12007 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 12008 Or, of course, you could just use the exact type in your own code. 12009 12010 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 12011 good job. 12012 12013 12014 +/ 12015 auto dispatchTo(alias handler)(string urlPrefix) { 12016 assert(urlPrefix[0] == '/'); 12017 assert(urlPrefix[$-1] != '/'); 12018 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 12019 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 12020 } 12021 12022 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 12023 } 12024 12025 /++ 12026 See [serveStaticFile] if you want to serve a file off disk. 12027 12028 History: 12029 Added January 28, 2023 (dub v11.0) 12030 +/ 12031 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 12032 assert(urlPrefix[0] == '/'); 12033 12034 static struct DispatcherDetails { 12035 immutable(ubyte)[] data; 12036 string contentType; 12037 string filenameToSuggestAsDownload; 12038 } 12039 12040 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 12041 cgi.setCache(true); 12042 cgi.setResponseContentType(details.contentType); 12043 if(details.filenameToSuggestAsDownload.length) 12044 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 12045 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 12046 cgi.write(details.data, true); 12047 return true; 12048 } 12049 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 12050 } 12051 12052 /++ 12053 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 12054 12055 History: 12056 Added January 28, 2023 (dub v11.0) 12057 +/ 12058 alias KeepExistingPresenter = typeof(null); 12059 12060 /++ 12061 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 12062 this issues the given errorCode and stops processing. 12063 12064 --- 12065 bool hasAdminPermissions(Cgi cgi) { 12066 return true; 12067 } 12068 12069 mixin DispatcherMain!( 12070 "/admin".dispatchSubsection!( 12071 passFilterOrIssueError!(hasAdminPermissions, 403), 12072 KeepExistingPresenter, 12073 "/".serveApi!AdminFunctions 12074 ) 12075 ); 12076 --- 12077 12078 History: 12079 Added January 28, 2023 (dub v11.0) 12080 +/ 12081 template passFilterOrIssueError(alias filter, int errorCode) { 12082 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 12083 if(filter(dd.cgi)) 12084 return true; 12085 dd.presenter.renderBasicError(dd.cgi, errorCode); 12086 return false; 12087 } 12088 } 12089 12090 /++ 12091 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 12092 and then be dispatched to their own handlers. 12093 12094 --- 12095 /+ 12096 // a long-form filter function 12097 bool permissionCheck(DispatcherData)(DispatcherData dd) { 12098 // you are permitted to call mutable methods on the Cgi object 12099 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 12100 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 12101 12102 if(checkPermissionOnRequest(dd.cgi)) { 12103 return true; // OK, allow processing to continue 12104 } else { 12105 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 12106 return false; // and stop further processing into this subsection 12107 } 12108 } 12109 +/ 12110 12111 // but you can also do short-form filters: 12112 12113 bool permissionCheck(Cgi cgi) { 12114 return ("ok" in cgi.get) !is null; 12115 } 12116 12117 // handler for the subsection 12118 class AdminClass : WebObject { 12119 int foo() { return 5; } 12120 } 12121 12122 // handler for the main site 12123 class TheMainSite : WebObject {} 12124 12125 mixin DispatcherMain!( 12126 "/admin".dispatchSubsection!( 12127 // converts our short-form filter into a long-form filter 12128 passFilterOrIssueError!(permissionCheck, 403), 12129 // can use a new presenter if wanted for the subsection 12130 KeepExistingPresenter, 12131 // and then provide child route dispatchers 12132 "/".serveApi!AdminClass 12133 ), 12134 // and back to the top level 12135 "/".serveApi!TheMainSite 12136 ); 12137 --- 12138 12139 Note you can encapsulate sections in files like this: 12140 12141 --- 12142 auto adminDispatcher(string urlPrefix) { 12143 return urlPrefix.dispatchSubsection!( 12144 .... 12145 ); 12146 } 12147 12148 mixin DispatcherMain!( 12149 "/admin".adminDispatcher, 12150 // and so on 12151 ) 12152 --- 12153 12154 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 12155 12156 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 12157 12158 12159 History: 12160 Added January 28, 2023 (dub v11.0) 12161 +/ 12162 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 12163 assert(urlPrefix[0] == '/'); 12164 assert(urlPrefix[$-1] != '/'); 12165 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 12166 static if(!is(PreRequestFilter == typeof(null))) { 12167 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 12168 return true; // we handled it by rejecting it 12169 } 12170 12171 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 12172 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 12173 } else { 12174 auto newPresenter = new NewPresenter(); 12175 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 12176 } 12177 } 12178 12179 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 12180 } 12181 12182 /++ 12183 A URL dispatcher. 12184 12185 --- 12186 if(cgi.dispatcher!( 12187 "/api/".serveApi!MyApiClass, 12188 "/objects/lol".serveRestObject!MyRestObject, 12189 "/file.js".serveStaticFile, 12190 "/admin/".dispatchTo!adminHandler 12191 )) return; 12192 --- 12193 12194 12195 You define a series of url prefixes followed by handlers. 12196 12197 You may want to do different pre- and post- processing there, for example, 12198 an authorization check and different page layout. You can use different 12199 presenters and different function chains. See [dispatchSubsection] for details. 12200 12201 [dispatchTo] will send the request to another function for handling. 12202 +/ 12203 template dispatcher(definitions...) { 12204 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 12205 static if(is(Presenter == typeof(null))) { 12206 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 12207 auto presenter = new GenericWebPresenter(); 12208 } else 12209 alias presenter = presenterArg; 12210 12211 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 12212 } 12213 12214 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 12215 // I can prolly make this more efficient later but meh. 12216 foreach(definition; definitions) { 12217 if(definition.rejectFurther) { 12218 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 12219 auto ret = definition.handler( 12220 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12221 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12222 if(ret) 12223 return true; 12224 } 12225 } else if( 12226 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 12227 // cgi.d dispatcher urls must be complete or have a /; 12228 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 12229 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 12230 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 12231 ) { 12232 auto ret = definition.handler( 12233 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12234 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12235 if(ret) 12236 return true; 12237 } 12238 } 12239 return false; 12240 } 12241 } 12242 12243 }); 12244 12245 private struct StackBuffer { 12246 char[1024] initial = void; 12247 char[] buffer; 12248 size_t position; 12249 12250 this(int a) { 12251 buffer = initial[]; 12252 position = 0; 12253 } 12254 12255 void add(in char[] what) { 12256 if(position + what.length > buffer.length) 12257 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 12258 buffer[position .. position + what.length] = what[]; 12259 position += what.length; 12260 } 12261 12262 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 12263 add(w1); 12264 add(w2); 12265 add(w3); 12266 } 12267 12268 void add(long v) { 12269 char[16] buffer = void; 12270 auto pos = buffer.length; 12271 bool negative; 12272 if(v < 0) { 12273 negative = true; 12274 v = -v; 12275 } 12276 do { 12277 buffer[--pos] = cast(char) (v % 10 + '0'); 12278 v /= 10; 12279 } while(v); 12280 12281 if(negative) 12282 buffer[--pos] = '-'; 12283 12284 auto res = buffer[pos .. $]; 12285 12286 add(res[]); 12287 } 12288 12289 char[] get() @nogc { 12290 return buffer[0 .. position]; 12291 } 12292 } 12293 12294 // duplicated in http2.d 12295 private static string getHttpCodeText(int code) pure nothrow @nogc { 12296 switch(code) { 12297 case 200: return "200 OK"; 12298 case 201: return "201 Created"; 12299 case 202: return "202 Accepted"; 12300 case 203: return "203 Non-Authoritative Information"; 12301 case 204: return "204 No Content"; 12302 case 205: return "205 Reset Content"; 12303 case 206: return "206 Partial Content"; 12304 // 12305 case 300: return "300 Multiple Choices"; 12306 case 301: return "301 Moved Permanently"; 12307 case 302: return "302 Found"; 12308 case 303: return "303 See Other"; 12309 case 304: return "304 Not Modified"; 12310 case 305: return "305 Use Proxy"; 12311 case 307: return "307 Temporary Redirect"; 12312 case 308: return "308 Permanent Redirect"; 12313 12314 // 12315 case 400: return "400 Bad Request"; 12316 case 401: return "401 Unauthorized"; 12317 case 402: return "402 Payment Required"; 12318 case 403: return "403 Forbidden"; 12319 case 404: return "404 Not Found"; 12320 case 405: return "405 Method Not Allowed"; 12321 case 406: return "406 Not Acceptable"; 12322 case 407: return "407 Proxy Authentication Required"; 12323 case 408: return "408 Request Timeout"; 12324 case 409: return "409 Conflict"; 12325 case 410: return "410 Gone"; 12326 case 411: return "411 Length Required"; 12327 case 412: return "412 Precondition Failed"; 12328 case 413: return "413 Payload Too Large"; 12329 case 414: return "414 URI Too Long"; 12330 case 415: return "415 Unsupported Media Type"; 12331 case 416: return "416 Range Not Satisfiable"; 12332 case 417: return "417 Expectation Failed"; 12333 case 418: return "418 I'm a teapot"; 12334 case 421: return "421 Misdirected Request"; 12335 case 422: return "422 Unprocessable Entity (WebDAV)"; 12336 case 423: return "423 Locked (WebDAV)"; 12337 case 424: return "424 Failed Dependency (WebDAV)"; 12338 case 425: return "425 Too Early"; 12339 case 426: return "426 Upgrade Required"; 12340 case 428: return "428 Precondition Required"; 12341 case 431: return "431 Request Header Fields Too Large"; 12342 case 451: return "451 Unavailable For Legal Reasons"; 12343 12344 case 500: return "500 Internal Server Error"; 12345 case 501: return "501 Not Implemented"; 12346 case 502: return "502 Bad Gateway"; 12347 case 503: return "503 Service Unavailable"; 12348 case 504: return "504 Gateway Timeout"; 12349 case 505: return "505 HTTP Version Not Supported"; 12350 case 506: return "506 Variant Also Negotiates"; 12351 case 507: return "507 Insufficient Storage (WebDAV)"; 12352 case 508: return "508 Loop Detected (WebDAV)"; 12353 case 510: return "510 Not Extended"; 12354 case 511: return "511 Network Authentication Required"; 12355 // 12356 default: assert(0, "Unsupported http code"); 12357 } 12358 } 12359 12360 12361 /+ 12362 /++ 12363 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12364 12365 It relies on jsvar.d and dom.d. 12366 12367 12368 You can get javascript out of it to call. The generated functions need to look 12369 like 12370 12371 function name(a,b,c,d,e) { 12372 return _call("name", {"realName":a,"sds":b}); 12373 } 12374 12375 And _call returns an object you can call or set up or whatever. 12376 +/ 12377 bool apiDispatcher()(Cgi cgi) { 12378 import arsd.jsvar; 12379 import arsd.dom; 12380 } 12381 +/ 12382 version(linux) 12383 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12384 /* 12385 Copyright: Adam D. Ruppe, 2008 - 2023 12386 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12387 Authors: Adam D. Ruppe 12388 12389 Copyright Adam D. Ruppe 2008 - 2023. 12390 Distributed under the Boost Software License, Version 1.0. 12391 (See accompanying file LICENSE_1_0.txt or copy at 12392 http://www.boost.org/LICENSE_1_0.txt) 12393 */