1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 import arsd.core; 490 491 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 492 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 493 494 /++ 495 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 496 +/ 497 unittest { 498 import arsd.cgi; 499 500 mixin DispatcherMain!( 501 "/".serveStaticFileDirectory(null, true) 502 ); 503 } 504 505 /++ 506 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 507 +/ 508 unittest { 509 import arsd.cgi; 510 511 void requestHandler(Cgi cgi) { 512 cgi.dispatcher!( 513 "/".serveStaticFileDirectory(null, true) 514 ); 515 } 516 517 // mixin GenericMain!requestHandler would add this function: 518 void main(string[] args) { 519 // this is all the content of [cgiMainImpl] which you can also call 520 521 // cgi.d embeds a few add on functions like real time event forwarders 522 // and session servers it can run in other processes. this spawns them, if needed. 523 if(tryAddonServers(args)) 524 return; 525 526 // cgi.d allows you to easily simulate http requests from the command line, 527 // without actually starting a server. this function will do that. 528 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 529 return; 530 531 RequestServer server; 532 // you can change the default port here if you like 533 // server.listeningPort = 9000; 534 535 // then call this to let the command line args override your default 536 server.configureFromCommandLine(args); 537 538 // here is where you could print out the listeningPort to the user if you wanted 539 540 // and serve the request(s) according to the compile configuration 541 server.serve!(requestHandler)(); 542 543 // or you could explicitly choose a serve mode like this: 544 // server.serveEmbeddedHttp!requestHandler(); 545 } 546 } 547 548 /++ 549 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 550 otherwise run through the rest of the internal mechanisms to call your functions without actually 551 spinning up a server. 552 +/ 553 unittest { 554 import arsd.cgi; 555 556 void requestHandler(Cgi cgi) { 557 558 } 559 560 // D doesn't let me embed a unittest inside an example unittest 561 // so this is a function, but you can do it however in your real program 562 /* unittest */ void runTests() { 563 auto tester = new CgiTester(&requestHandler); 564 565 auto response = tester.GET("/"); 566 assert(response.code == 200); 567 } 568 } 569 570 static import std.file; 571 572 // for a single thread, linear request thing, use: 573 // -version=embedded_httpd_threads -version=cgi_no_threads 574 575 version(Posix) { 576 version(CRuntime_Musl) { 577 578 } else version(minimal) { 579 580 } else { 581 version(FreeBSD) { 582 // I never implemented the fancy stuff there either 583 } else { 584 version=with_breaking_cgi_features; 585 version=with_sendfd; 586 version=with_addon_servers; 587 } 588 } 589 } 590 591 version(Windows) { 592 version(minimal) { 593 594 } else { 595 // not too concerned about gdc here since the mingw version is fairly new as well 596 version=with_breaking_cgi_features; 597 } 598 } 599 600 void cloexec(int fd) { 601 version(Posix) { 602 import core.sys.posix.fcntl; 603 fcntl(fd, F_SETFD, FD_CLOEXEC); 604 } 605 } 606 607 void cloexec(Socket s) { 608 version(Posix) { 609 import core.sys.posix.fcntl; 610 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 611 } 612 } 613 614 version(embedded_httpd_hybrid) { 615 version=embedded_httpd_threads; 616 version(cgi_no_fork) {} else version(Posix) 617 version=cgi_use_fork; 618 version=cgi_use_fiber; 619 } 620 621 version(cgi_use_fork) 622 enum cgi_use_fork_default = true; 623 else 624 enum cgi_use_fork_default = false; 625 626 // the servers must know about the connections to talk to them; the interfaces are vital 627 version(with_addon_servers) 628 version=with_addon_servers_connections; 629 630 version(embedded_httpd) { 631 version(linux) 632 version=embedded_httpd_processes; 633 else { 634 version=embedded_httpd_threads; 635 } 636 637 /* 638 version(with_openssl) { 639 pragma(lib, "crypto"); 640 pragma(lib, "ssl"); 641 } 642 */ 643 } 644 645 version(embedded_httpd_processes) 646 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 647 648 version(embedded_httpd_threads) { 649 // unless the user overrides the default.. 650 version(cgi_session_server_process) 651 {} 652 else 653 version=cgi_embedded_sessions; 654 } 655 version(scgi) { 656 // unless the user overrides the default.. 657 version(cgi_session_server_process) 658 {} 659 else 660 version=cgi_embedded_sessions; 661 } 662 663 // fall back if the other is not defined so we can cleanly version it below 664 version(cgi_embedded_sessions) {} 665 else version=cgi_session_server_process; 666 667 668 version=cgi_with_websocket; 669 670 enum long defaultMaxContentLength = 5_000_000; 671 672 /* 673 674 To do a file download offer in the browser: 675 676 cgi.setResponseContentType("text/csv"); 677 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 678 */ 679 680 // FIXME: the location header is supposed to be an absolute url I guess. 681 682 // FIXME: would be cool to flush part of a dom document before complete 683 // somehow in here and dom.d. 684 685 686 // these are public so you can mixin GenericMain. 687 // FIXME: use a function level import instead! 688 public import std.string; 689 public import std.stdio; 690 public import std.conv; 691 import std.uri; 692 import std.uni; 693 import std.algorithm.comparison; 694 import std.algorithm.searching; 695 import std.exception; 696 import std.base64; 697 static import std.algorithm; 698 import std.datetime; 699 import std.range; 700 701 import std.process; 702 703 import std.zlib; 704 705 706 T[] consume(T)(T[] range, int count) { 707 if(count > range.length) 708 count = range.length; 709 return range[count..$]; 710 } 711 712 int locationOf(T)(T[] data, string item) { 713 const(ubyte[]) d = cast(const(ubyte[])) data; 714 const(ubyte[]) i = cast(const(ubyte[])) item; 715 716 // this is a vague sanity check to ensure we aren't getting insanely 717 // sized input that will infinite loop below. it should never happen; 718 // even huge file uploads ought to come in smaller individual pieces. 719 if(d.length > (int.max/2)) 720 throw new Exception("excessive block of input"); 721 722 for(int a = 0; a < d.length; a++) { 723 if(a + i.length > d.length) 724 return -1; 725 if(d[a..a+i.length] == i) 726 return a; 727 } 728 729 return -1; 730 } 731 732 /// If you are doing a custom cgi class, mixing this in can take care of 733 /// the required constructors for you 734 mixin template ForwardCgiConstructors() { 735 this(long maxContentLength = defaultMaxContentLength, 736 string[string] env = null, 737 const(ubyte)[] delegate() readdata = null, 738 void delegate(const(ubyte)[]) _rawDataOutput = null, 739 void delegate() _flush = null 740 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 741 742 this(string[] args) { super(args); } 743 744 this( 745 BufferedInputRange inputData, 746 string address, ushort _port, 747 int pathInfoStarts = 0, 748 bool _https = false, 749 void delegate(const(ubyte)[]) _rawDataOutput = null, 750 void delegate() _flush = null, 751 // this pointer tells if the connection is supposed to be closed after we handle this 752 bool* closeConnection = null) 753 { 754 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 755 } 756 757 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 758 } 759 760 /// thrown when a connection is closed remotely while we waiting on data from it 761 class ConnectionClosedException : Exception { 762 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 763 super(message, file, line, next); 764 } 765 } 766 767 768 version(Windows) { 769 // FIXME: ugly hack to solve stdin exception problems on Windows: 770 // reading stdin results in StdioException (Bad file descriptor) 771 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 772 private struct stdin { 773 struct ByChunk { // Replicates std.stdio.ByChunk 774 private: 775 ubyte[] chunk_; 776 public: 777 this(size_t size) 778 in { 779 assert(size, "size must be larger than 0"); 780 } 781 do { 782 chunk_ = new ubyte[](size); 783 popFront(); 784 } 785 786 @property bool empty() const { 787 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 788 } 789 @property nothrow ubyte[] front() { return chunk_; } 790 void popFront() { 791 enforce(!empty, "Cannot call popFront on empty range"); 792 chunk_ = stdin.rawRead(chunk_); 793 } 794 } 795 796 import core.sys.windows.windows; 797 static: 798 799 T[] rawRead(T)(T[] buf) { 800 uint bytesRead; 801 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 802 803 if (!result) { 804 auto err = GetLastError(); 805 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 806 return buf[0..0]; 807 // Some other error, throw it 808 809 char* buffer; 810 scope(exit) LocalFree(buffer); 811 812 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 813 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 814 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 815 throw new Exception(to!string(buffer)); 816 } 817 enforce(!(bytesRead % T.sizeof), "I/O error"); 818 return buf[0..bytesRead / T.sizeof]; 819 } 820 821 auto byChunk(size_t sz) { return ByChunk(sz); } 822 823 void close() { 824 std.stdio.stdin.close; 825 } 826 } 827 } 828 829 /// The main interface with the web request 830 class Cgi { 831 public: 832 /// the methods a request can be 833 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 834 // these are defined in the standard, but idk if they are useful for anything 835 OPTIONS, TRACE, CONNECT, 836 // These seem new, I have only recently seen them 837 PATCH, MERGE, 838 // this is an extension for when the method is not specified and you want to assume 839 CommandLine } 840 841 842 /+ 843 /++ 844 Cgi provides a per-request memory pool 845 846 +/ 847 void[] allocateMemory(size_t nBytes) { 848 849 } 850 851 /// ditto 852 void[] reallocateMemory(void[] old, size_t nBytes) { 853 854 } 855 856 /// ditto 857 void freeMemory(void[] memory) { 858 859 } 860 +/ 861 862 863 /* 864 import core.runtime; 865 auto args = Runtime.args(); 866 867 we can call the app a few ways: 868 869 1) set up the environment variables and call the app (manually simulating CGI) 870 2) simulate a call automatically: 871 ./app method 'uri' 872 873 for example: 874 ./app get /path?arg arg2=something 875 876 Anything on the uri is treated as query string etc 877 878 on get method, further args are appended to the query string (encoded automatically) 879 on post method, further args are done as post 880 881 882 @name means import from file "name". if name == -, it uses stdin 883 (so info=@- means set info to the value of stdin) 884 885 886 Other arguments include: 887 --cookie name=value (these are all concated together) 888 --header 'X-Something: cool' 889 --referrer 'something' 890 --port 80 891 --remote-address some.ip.address.here 892 --https yes 893 --user-agent 'something' 894 --userpass 'user:pass' 895 --authorization 'Basic base64encoded_user:pass' 896 --accept 'content' // FIXME: better example 897 --last-event-id 'something' 898 --host 'something.com' 899 900 Non-simulation arguments: 901 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 902 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 903 904 */ 905 906 /** Initializes it with command line arguments (for easy testing) */ 907 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 908 rawDataOutput = _rawDataOutput; 909 // these are all set locally so the loop works 910 // without triggering errors in dmd 2.064 911 // we go ahead and set them at the end of it to the this version 912 int port; 913 string referrer; 914 string remoteAddress; 915 string userAgent; 916 string authorization; 917 string origin; 918 string accept; 919 string lastEventId; 920 bool https; 921 string host; 922 RequestMethod requestMethod; 923 string requestUri; 924 string pathInfo; 925 string queryString; 926 927 bool lookingForMethod; 928 bool lookingForUri; 929 string nextArgIs; 930 931 string _cookie; 932 string _queryString; 933 string[][string] _post; 934 string[string] _headers; 935 936 string[] breakUp(string s) { 937 string k, v; 938 auto idx = s.indexOf("="); 939 if(idx == -1) { 940 k = s; 941 } else { 942 k = s[0 .. idx]; 943 v = s[idx + 1 .. $]; 944 } 945 946 return [k, v]; 947 } 948 949 lookingForMethod = true; 950 951 scriptName = args[0]; 952 scriptFileName = args[0]; 953 954 environmentVariables = cast(const) environment.toAA; 955 956 foreach(arg; args[1 .. $]) { 957 if(arg.startsWith("--")) { 958 nextArgIs = arg[2 .. $]; 959 } else if(nextArgIs.length) { 960 if (nextArgIs == "cookie") { 961 auto info = breakUp(arg); 962 if(_cookie.length) 963 _cookie ~= "; "; 964 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 965 } 966 else if (nextArgIs == "port") { 967 port = to!int(arg); 968 } 969 else if (nextArgIs == "referrer") { 970 referrer = arg; 971 } 972 else if (nextArgIs == "remote-address") { 973 remoteAddress = arg; 974 } 975 else if (nextArgIs == "user-agent") { 976 userAgent = arg; 977 } 978 else if (nextArgIs == "authorization") { 979 authorization = arg; 980 } 981 else if (nextArgIs == "userpass") { 982 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 983 } 984 else if (nextArgIs == "origin") { 985 origin = arg; 986 } 987 else if (nextArgIs == "accept") { 988 accept = arg; 989 } 990 else if (nextArgIs == "last-event-id") { 991 lastEventId = arg; 992 } 993 else if (nextArgIs == "https") { 994 if(arg == "yes") 995 https = true; 996 } 997 else if (nextArgIs == "header") { 998 string thing, other; 999 auto idx = arg.indexOf(":"); 1000 if(idx == -1) 1001 throw new Exception("need a colon in a http header"); 1002 thing = arg[0 .. idx]; 1003 other = arg[idx + 1.. $]; 1004 _headers[thing.strip.toLower()] = other.strip; 1005 } 1006 else if (nextArgIs == "host") { 1007 host = arg; 1008 } 1009 // else 1010 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1011 1012 nextArgIs = null; 1013 } else if(lookingForMethod) { 1014 lookingForMethod = false; 1015 lookingForUri = true; 1016 1017 if(arg.asLowerCase().equal("commandline")) 1018 requestMethod = RequestMethod.CommandLine; 1019 else 1020 requestMethod = to!RequestMethod(arg.toUpper()); 1021 } else if(lookingForUri) { 1022 lookingForUri = false; 1023 1024 requestUri = arg; 1025 1026 auto idx = arg.indexOf("?"); 1027 if(idx == -1) 1028 pathInfo = arg; 1029 else { 1030 pathInfo = arg[0 .. idx]; 1031 _queryString = arg[idx + 1 .. $]; 1032 } 1033 } else { 1034 // it is an argument of some sort 1035 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1036 auto parts = breakUp(arg); 1037 _post[parts[0]] ~= parts[1]; 1038 allPostNamesInOrder ~= parts[0]; 1039 allPostValuesInOrder ~= parts[1]; 1040 } else { 1041 if(_queryString.length) 1042 _queryString ~= "&"; 1043 auto parts = breakUp(arg); 1044 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1045 } 1046 } 1047 } 1048 1049 acceptsGzip = false; 1050 keepAliveRequested = false; 1051 requestHeaders = cast(immutable) _headers; 1052 1053 cookie = _cookie; 1054 cookiesArray = getCookieArray(); 1055 cookies = keepLastOf(cookiesArray); 1056 1057 queryString = _queryString; 1058 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1059 get = keepLastOf(getArray); 1060 1061 postArray = cast(immutable) _post; 1062 post = keepLastOf(_post); 1063 1064 // FIXME 1065 filesArray = null; 1066 files = null; 1067 1068 isCalledWithCommandLineArguments = true; 1069 1070 this.port = port; 1071 this.referrer = referrer; 1072 this.remoteAddress = remoteAddress; 1073 this.userAgent = userAgent; 1074 this.authorization = authorization; 1075 this.origin = origin; 1076 this.accept = accept; 1077 this.lastEventId = lastEventId; 1078 this.https = https; 1079 this.host = host; 1080 this.requestMethod = requestMethod; 1081 this.requestUri = requestUri; 1082 this.pathInfo = pathInfo; 1083 this.queryString = queryString; 1084 this.postBody = null; 1085 } 1086 1087 private { 1088 string[] allPostNamesInOrder; 1089 string[] allPostValuesInOrder; 1090 string[] allGetNamesInOrder; 1091 string[] allGetValuesInOrder; 1092 } 1093 1094 CgiConnectionHandle getOutputFileHandle() { 1095 return _outputFileHandle; 1096 } 1097 1098 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1099 1100 /** Initializes it using a CGI or CGI-like interface */ 1101 this(long maxContentLength = defaultMaxContentLength, 1102 // use this to override the environment variable listing 1103 in string[string] env = null, 1104 // and this should return a chunk of data. return empty when done 1105 const(ubyte)[] delegate() readdata = null, 1106 // finally, use this to do custom output if needed 1107 void delegate(const(ubyte)[]) _rawDataOutput = null, 1108 // to flush teh custom output 1109 void delegate() _flush = null 1110 ) 1111 { 1112 1113 // these are all set locally so the loop works 1114 // without triggering errors in dmd 2.064 1115 // we go ahead and set them at the end of it to the this version 1116 int port; 1117 string referrer; 1118 string remoteAddress; 1119 string userAgent; 1120 string authorization; 1121 string origin; 1122 string accept; 1123 string lastEventId; 1124 bool https; 1125 string host; 1126 RequestMethod requestMethod; 1127 string requestUri; 1128 string pathInfo; 1129 string queryString; 1130 1131 1132 1133 isCalledWithCommandLineArguments = false; 1134 rawDataOutput = _rawDataOutput; 1135 flushDelegate = _flush; 1136 auto getenv = delegate string(string var) { 1137 if(env is null) 1138 return std.process.environment.get(var); 1139 auto e = var in env; 1140 if(e is null) 1141 return null; 1142 return *e; 1143 }; 1144 1145 environmentVariables = env is null ? 1146 cast(const) environment.toAA : 1147 env; 1148 1149 // fetching all the request headers 1150 string[string] requestHeadersHere; 1151 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1152 if(k.startsWith("HTTP_")) { 1153 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1154 } 1155 } 1156 1157 this.requestHeaders = assumeUnique(requestHeadersHere); 1158 1159 requestUri = getenv("REQUEST_URI"); 1160 1161 cookie = getenv("HTTP_COOKIE"); 1162 cookiesArray = getCookieArray(); 1163 cookies = keepLastOf(cookiesArray); 1164 1165 referrer = getenv("HTTP_REFERER"); 1166 userAgent = getenv("HTTP_USER_AGENT"); 1167 remoteAddress = getenv("REMOTE_ADDR"); 1168 host = getenv("HTTP_HOST"); 1169 pathInfo = getenv("PATH_INFO"); 1170 1171 queryString = getenv("QUERY_STRING"); 1172 scriptName = getenv("SCRIPT_NAME"); 1173 { 1174 import core.runtime; 1175 auto sfn = getenv("SCRIPT_FILENAME"); 1176 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1177 } 1178 1179 bool iis = false; 1180 1181 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1182 if(requestUri.length == 0) { 1183 // IIS sometimes includes the script name as part of the path info - we don't want that 1184 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1185 pathInfo = pathInfo[scriptName.length .. $]; 1186 1187 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1188 1189 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1190 1191 // FIXME: this works for apache and iis... but what about others? 1192 } 1193 1194 1195 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1196 getArray = assumeUnique(ugh); 1197 get = keepLastOf(getArray); 1198 1199 1200 // NOTE: on shitpache, you need to specifically forward this 1201 authorization = getenv("HTTP_AUTHORIZATION"); 1202 // this is a hack because Apache is a shitload of fuck and 1203 // refuses to send the real header to us. Compatible 1204 // programs should send both the standard and X- versions 1205 1206 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1207 // unnecessary with mod_rewrite, so it is commented 1208 1209 //if(authorization.length == 0) // if the std is there, use it 1210 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1211 1212 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1213 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1214 port = to!int(getenv("SERVER_PORT")); 1215 else 1216 port = 0; // this was probably called from the command line 1217 1218 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1219 if(ae.length && ae.indexOf("gzip") != -1) 1220 acceptsGzip = true; 1221 1222 accept = getenv("HTTP_ACCEPT"); 1223 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1224 1225 auto ka = getenv("HTTP_CONNECTION"); 1226 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1227 keepAliveRequested = true; 1228 1229 auto or = getenv("HTTP_ORIGIN"); 1230 origin = or; 1231 1232 auto rm = getenv("REQUEST_METHOD"); 1233 if(rm.length) 1234 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1235 else 1236 requestMethod = RequestMethod.CommandLine; 1237 1238 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1239 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1240 1241 // FIXME: DOCUMENT_ROOT? 1242 1243 // FIXME: what about PUT? 1244 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1245 version(preserveData) // a hack to make forwarding simpler 1246 immutable(ubyte)[] data; 1247 size_t amountReceived = 0; 1248 auto contentType = getenv("CONTENT_TYPE"); 1249 1250 // FIXME: is this ever not going to be set? I guess it depends 1251 // on if the server de-chunks and buffers... seems like it has potential 1252 // to be slow if they did that. The spec says it is always there though. 1253 // And it has worked reliably for me all year in the live environment, 1254 // but some servers might be different. 1255 auto cls = getenv("CONTENT_LENGTH"); 1256 auto contentLength = to!size_t(cls.length ? cls : "0"); 1257 1258 immutable originalContentLength = contentLength; 1259 if(contentLength) { 1260 if(maxContentLength > 0 && contentLength > maxContentLength) { 1261 setResponseStatus("413 Request entity too large"); 1262 write("You tried to upload a file that is too large."); 1263 close(); 1264 throw new Exception("POST too large"); 1265 } 1266 prepareForIncomingDataChunks(contentType, contentLength); 1267 1268 1269 int processChunk(in ubyte[] chunk) { 1270 if(chunk.length > contentLength) { 1271 handleIncomingDataChunk(chunk[0..contentLength]); 1272 amountReceived += contentLength; 1273 contentLength = 0; 1274 return 1; 1275 } else { 1276 handleIncomingDataChunk(chunk); 1277 contentLength -= chunk.length; 1278 amountReceived += chunk.length; 1279 } 1280 if(contentLength == 0) 1281 return 1; 1282 1283 onRequestBodyDataReceived(amountReceived, originalContentLength); 1284 return 0; 1285 } 1286 1287 1288 if(readdata is null) { 1289 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1290 if(processChunk(chunk)) 1291 break; 1292 } else { 1293 // we have a custom data source.. 1294 auto chunk = readdata(); 1295 while(chunk.length) { 1296 if(processChunk(chunk)) 1297 break; 1298 chunk = readdata(); 1299 } 1300 } 1301 1302 onRequestBodyDataReceived(amountReceived, originalContentLength); 1303 postArray = assumeUnique(pps._post); 1304 filesArray = assumeUnique(pps._files); 1305 files = keepLastOf(filesArray); 1306 post = keepLastOf(postArray); 1307 this.postBody = pps.postBody; 1308 cleanUpPostDataState(); 1309 } 1310 1311 version(preserveData) 1312 originalPostData = data; 1313 } 1314 // fixme: remote_user script name 1315 1316 1317 this.port = port; 1318 this.referrer = referrer; 1319 this.remoteAddress = remoteAddress; 1320 this.userAgent = userAgent; 1321 this.authorization = authorization; 1322 this.origin = origin; 1323 this.accept = accept; 1324 this.lastEventId = lastEventId; 1325 this.https = https; 1326 this.host = host; 1327 this.requestMethod = requestMethod; 1328 this.requestUri = requestUri; 1329 this.pathInfo = pathInfo; 1330 this.queryString = queryString; 1331 } 1332 1333 /// Cleans up any temporary files. Do not use the object 1334 /// after calling this. 1335 /// 1336 /// NOTE: it is called automatically by GenericMain 1337 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1338 void dispose() { 1339 foreach(file; files) { 1340 if(!file.contentInMemory) 1341 if(std.file.exists(file.contentFilename)) 1342 std.file.remove(file.contentFilename); 1343 } 1344 } 1345 1346 private { 1347 struct PostParserState { 1348 string contentType; 1349 string boundary; 1350 string localBoundary; // the ones used at the end or something lol 1351 bool isMultipart; 1352 bool needsSavedBody; 1353 1354 ulong expectedLength; 1355 ulong contentConsumed; 1356 immutable(ubyte)[] buffer; 1357 1358 // multipart parsing state 1359 int whatDoWeWant; 1360 bool weHaveAPart; 1361 string[] thisOnesHeaders; 1362 immutable(ubyte)[] thisOnesData; 1363 1364 string postBody; 1365 1366 UploadedFile piece; 1367 bool isFile = false; 1368 1369 size_t memoryCommitted; 1370 1371 // do NOT keep mutable references to these anywhere! 1372 // I assume they are unique in the constructor once we're all done getting data. 1373 string[][string] _post; 1374 UploadedFile[][string] _files; 1375 } 1376 1377 PostParserState pps; 1378 } 1379 1380 /// This represents a file the user uploaded via a POST request. 1381 static struct UploadedFile { 1382 /// If you want to create one of these structs for yourself from some data, 1383 /// use this function. 1384 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1385 Cgi.UploadedFile f; 1386 f.filename = name; 1387 f.content = cast(immutable(ubyte)[]) data; 1388 f.contentInMemory = true; 1389 return f; 1390 } 1391 1392 string name; /// The name of the form element. 1393 string filename; /// The filename the user set. 1394 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1395 1396 /** 1397 For small files, cgi.d will buffer the uploaded file in memory, and make it 1398 directly accessible to you through the content member. I find this very convenient 1399 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1400 often want to inspect and modify the file anyway!) 1401 1402 I find the file is very large, it is undesirable to eat that much memory just 1403 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1404 to the constructor so they are accepted, cgi.d will write the content to a temporary 1405 file that you can re-read later. 1406 1407 You can override this behavior by subclassing Cgi and overriding the protected 1408 handlePostChunk method. Note that the object is not initialized when you 1409 write that method - the http headers are available, but the cgi.post method 1410 is not. You may parse the file as it streams in using this method. 1411 1412 1413 Anyway, if the file is small enough to be in memory, contentInMemory will be 1414 set to true, and the content is available in the content member. 1415 1416 If not, contentInMemory will be set to false, and the content saved in a file, 1417 whose name will be available in the contentFilename member. 1418 1419 1420 Tip: if you know you are always dealing with small files, and want the convenience 1421 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1422 a large file comes in, it simply throws an exception (and HTTP error response) 1423 instead of trying to handle it. 1424 1425 The default value of maxContentLength in the constructor is for small files. 1426 */ 1427 bool contentInMemory = true; // the default ought to always be true 1428 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1429 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1430 1431 /// 1432 ulong fileSize() { 1433 if(contentInMemory) 1434 return content.length; 1435 import std.file; 1436 return std.file.getSize(contentFilename); 1437 1438 } 1439 1440 /// 1441 void writeToFile(string filenameToSaveTo) const { 1442 import std.file; 1443 if(contentInMemory) 1444 std.file.write(filenameToSaveTo, content); 1445 else 1446 std.file.rename(contentFilename, filenameToSaveTo); 1447 } 1448 } 1449 1450 // given a content type and length, decide what we're going to do with the data.. 1451 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1452 pps.expectedLength = contentLength; 1453 1454 auto terminator = contentType.indexOf(";"); 1455 if(terminator == -1) 1456 terminator = contentType.length; 1457 1458 pps.contentType = contentType[0 .. terminator]; 1459 auto b = contentType[terminator .. $]; 1460 if(b.length) { 1461 auto idx = b.indexOf("boundary="); 1462 if(idx != -1) { 1463 pps.boundary = b[idx + "boundary=".length .. $]; 1464 pps.localBoundary = "\r\n--" ~ pps.boundary; 1465 } 1466 } 1467 1468 // while a content type SHOULD be sent according to the RFC, it is 1469 // not required. We're told we SHOULD guess by looking at the content 1470 // but it seems to me that this only happens when it is urlencoded. 1471 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1472 pps.isMultipart = false; 1473 pps.needsSavedBody = false; 1474 } else if(pps.contentType == "multipart/form-data") { 1475 pps.isMultipart = true; 1476 enforce(pps.boundary.length, "no boundary"); 1477 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1478 // save the body so the application can handle it 1479 pps.isMultipart = false; 1480 pps.needsSavedBody = true; 1481 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1482 // save the body so the application can handle it 1483 pps.needsSavedBody = true; 1484 pps.isMultipart = false; 1485 } else { 1486 // the rest is 100% handled by the application. just save the body and send it to them 1487 pps.needsSavedBody = true; 1488 pps.isMultipart = false; 1489 } 1490 } 1491 1492 // handles streaming POST data. If you handle some other content type, you should 1493 // override this. If the data isn't the content type you want, you ought to call 1494 // super.handleIncomingDataChunk so regular forms and files still work. 1495 1496 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1497 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1498 // input anyway, so I'm not going to get too worked up about it right now. 1499 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1500 if(chunk.length == 0) 1501 return; 1502 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1503 // if we're passed big chunks, it might throw unnecessarily. 1504 // just pass it smaller chunks at a time. 1505 if(pps.isMultipart) { 1506 // multipart/form-data 1507 1508 1509 // FIXME: this might want to be factored out and factorized 1510 // need to make sure the stream hooks actually work. 1511 void pieceHasNewContent() { 1512 // we just grew the piece's buffer. Do we have to switch to file backing? 1513 if(pps.piece.contentInMemory) { 1514 if(pps.piece.content.length <= 10 * 1024 * 1024) 1515 // meh, I'm ok with it. 1516 return; 1517 else { 1518 // this is too big. 1519 if(!pps.isFile) 1520 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1521 else { 1522 // a file this large is probably acceptable though... let's use a backing file. 1523 pps.piece.contentInMemory = false; 1524 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1525 1526 int count = 0; 1527 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1528 // odds are this loop will never be entered, but we want it just in case. 1529 while(std.file.exists(pps.piece.contentFilename)) { 1530 count++; 1531 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1532 } 1533 // I hope this creates the file pretty quickly, or the loop might be useless... 1534 // FIXME: maybe I should write some kind of custom transaction here. 1535 std.file.write(pps.piece.contentFilename, pps.piece.content); 1536 1537 pps.piece.content = null; 1538 } 1539 } 1540 } else { 1541 // it's already in a file, so just append it to what we have 1542 if(pps.piece.content.length) { 1543 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1544 std.file.append(pps.piece.contentFilename, pps.piece.content); 1545 pps.piece.content = null; 1546 } 1547 } 1548 } 1549 1550 1551 void commitPart() { 1552 if(!pps.weHaveAPart) 1553 return; 1554 1555 pieceHasNewContent(); // be sure the new content is handled every time 1556 1557 if(pps.isFile) { 1558 // I'm not sure if other environments put files in post or not... 1559 // I used to not do it, but I think I should, since it is there... 1560 pps._post[pps.piece.name] ~= pps.piece.filename; 1561 pps._files[pps.piece.name] ~= pps.piece; 1562 1563 allPostNamesInOrder ~= pps.piece.name; 1564 allPostValuesInOrder ~= pps.piece.filename; 1565 } else { 1566 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1567 1568 allPostNamesInOrder ~= pps.piece.name; 1569 allPostValuesInOrder ~= cast(string) pps.piece.content; 1570 } 1571 1572 /* 1573 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1574 pps.piece.content.length < 1000 1575 ? 1576 to!string(pps.piece.content) 1577 : 1578 "too long"); 1579 */ 1580 1581 // FIXME: the limit here 1582 pps.memoryCommitted += pps.piece.content.length; 1583 1584 pps.weHaveAPart = false; 1585 pps.whatDoWeWant = 1; 1586 pps.thisOnesHeaders = null; 1587 pps.thisOnesData = null; 1588 1589 pps.piece = UploadedFile.init; 1590 pps.isFile = false; 1591 } 1592 1593 void acceptChunk() { 1594 pps.buffer ~= chunk; 1595 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1596 } 1597 1598 immutable(ubyte)[] consume(size_t howMuch) { 1599 pps.contentConsumed += howMuch; 1600 auto ret = pps.buffer[0 .. howMuch]; 1601 pps.buffer = pps.buffer[howMuch .. $]; 1602 return ret; 1603 } 1604 1605 dataConsumptionLoop: do { 1606 switch(pps.whatDoWeWant) { 1607 default: assert(0); 1608 case 0: 1609 acceptChunk(); 1610 // the format begins with two extra leading dashes, then we should be at the boundary 1611 if(pps.buffer.length < 2) 1612 return; 1613 assert(pps.buffer[0] == '-', "no leading dash"); 1614 consume(1); 1615 assert(pps.buffer[0] == '-', "no second leading dash"); 1616 consume(1); 1617 1618 pps.whatDoWeWant = 1; 1619 goto case 1; 1620 /* fallthrough */ 1621 case 1: // looking for headers 1622 // here, we should be lined up right at the boundary, which is followed by a \r\n 1623 1624 // want to keep the buffer under control in case we're under attack 1625 //stderr.writeln("here once"); 1626 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1627 // throw new Exception("wtf is up with the huge mime part headers"); 1628 1629 acceptChunk(); 1630 1631 if(pps.buffer.length < pps.boundary.length) 1632 return; // not enough data, since there should always be a boundary here at least 1633 1634 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1635 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1636 // we *should* be at the end here! 1637 assert(pps.buffer[0] == '-'); 1638 consume(1); 1639 assert(pps.buffer[0] == '-'); 1640 consume(1); 1641 1642 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1643 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1644 "not lined up on boundary " ~ pps.boundary); 1645 consume(pps.boundary.length); 1646 1647 assert(pps.buffer[0] == '-'); 1648 consume(1); 1649 assert(pps.buffer[0] == '-'); 1650 consume(1); 1651 1652 assert(pps.buffer[0] == '\r'); 1653 consume(1); 1654 assert(pps.buffer[0] == '\n'); 1655 consume(1); 1656 1657 assert(pps.buffer.length == 0); 1658 assert(pps.contentConsumed == pps.expectedLength); 1659 break dataConsumptionLoop; // we're done! 1660 } else { 1661 // we're not done yet. We should be lined up on a boundary. 1662 1663 // But, we want to ensure the headers are here before we consume anything! 1664 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1665 if(headerEndLocation == -1) 1666 return; // they *should* all be here, so we can handle them all at once. 1667 1668 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1669 "not lined up on boundary " ~ pps.boundary); 1670 1671 consume(pps.boundary.length); 1672 // the boundary is always followed by a \r\n 1673 assert(pps.buffer[0] == '\r'); 1674 consume(1); 1675 assert(pps.buffer[0] == '\n'); 1676 consume(1); 1677 } 1678 1679 // re-running since by consuming the boundary, we invalidate the old index. 1680 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1681 assert(headerEndLocation >= 0, "no header"); 1682 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1683 1684 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1685 1686 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1687 1688 // now we'll parse the headers 1689 foreach(h; pps.thisOnesHeaders) { 1690 auto p = h.indexOf(":"); 1691 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1692 string hn = h[0..p]; 1693 string hv = h[p+2..$]; 1694 1695 switch(hn.toLower) { 1696 default: assert(0); 1697 case "content-disposition": 1698 auto info = hv.split("; "); 1699 foreach(i; info[1..$]) { // skipping the form-data 1700 auto o = i.split("="); // FIXME 1701 string pn = o[0]; 1702 string pv = o[1][1..$-1]; 1703 1704 if(pn == "name") { 1705 pps.piece.name = pv; 1706 } else if (pn == "filename") { 1707 pps.piece.filename = pv; 1708 pps.isFile = true; 1709 } 1710 } 1711 break; 1712 case "content-type": 1713 pps.piece.contentType = hv; 1714 break; 1715 } 1716 } 1717 1718 pps.whatDoWeWant++; // move to the next step - the data 1719 break; 1720 case 2: 1721 // when we get here, pps.buffer should contain our first chunk of data 1722 1723 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1724 throw new Exception("wtf is up with the huge mime part buffer"); 1725 1726 acceptChunk(); 1727 1728 // so the trick is, we want to process all the data up to the boundary, 1729 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1730 // want to wait for the next chunk. We start by looking for the whole boundary 1731 // in the buffer somewhere. 1732 1733 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1734 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1735 if(boundaryLocation != -1) { 1736 // this is easy - we can see it in it's entirety! 1737 1738 pps.piece.content ~= consume(boundaryLocation); 1739 1740 assert(pps.buffer[0] == '\r'); 1741 consume(1); 1742 assert(pps.buffer[0] == '\n'); 1743 consume(1); 1744 assert(pps.buffer[0] == '-'); 1745 consume(1); 1746 assert(pps.buffer[0] == '-'); 1747 consume(1); 1748 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1749 pps.weHaveAPart = true; 1750 pps.whatDoWeWant = 1; // back to getting headers for the next part 1751 1752 commitPart(); // we're done here 1753 } else { 1754 // we can't see the whole thing, but what if there's a partial boundary? 1755 1756 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1757 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1758 bool potentialBoundaryFound = false; 1759 1760 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1761 // we grow the boundary a bit each time. If we think it looks the 1762 // same, better pull another chunk to be sure it's not the end. 1763 // Starting small because exiting the loop early is desirable, since 1764 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1765 // the best we can do. 1766 if(a > pps.buffer.length) 1767 break; // FIXME: is this right? 1768 assert(a <= pps.buffer.length); 1769 assert(a > 0); 1770 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1771 // ok, there *might* be a boundary here, so let's 1772 // not treat the end as data yet. The rest is good to 1773 // use though, since if there was a boundary there, we'd 1774 // have handled it up above after locationOf. 1775 1776 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1777 consume(pps.buffer.length - a); 1778 pieceHasNewContent(); 1779 potentialBoundaryFound = true; 1780 break boundaryCheck; 1781 } 1782 } 1783 1784 if(!potentialBoundaryFound) { 1785 // we can consume the whole thing 1786 pps.piece.content ~= pps.buffer; 1787 pieceHasNewContent(); 1788 consume(pps.buffer.length); 1789 } else { 1790 // we found a possible boundary, but there was 1791 // insufficient data to be sure. 1792 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1793 1794 return; // wait for the next chunk. 1795 } 1796 } 1797 } 1798 } while(pps.buffer.length); 1799 1800 // btw all boundaries except the first should have a \r\n before them 1801 } else { 1802 // application/x-www-form-urlencoded and application/json 1803 1804 // not using maxContentLength because that might be cranked up to allow 1805 // large file uploads. We can handle them, but a huge post[] isn't any good. 1806 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1807 throw new Exception("wtf is up with such a gigantic form submission????"); 1808 1809 pps.buffer ~= chunk; 1810 1811 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1812 if(pps.buffer.length == pps.expectedLength) { 1813 if(pps.needsSavedBody) 1814 pps.postBody = cast(string) pps.buffer; 1815 else 1816 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1817 version(preserveData) 1818 originalPostData = pps.buffer; 1819 } else { 1820 // just for debugging 1821 } 1822 } 1823 } 1824 1825 protected void cleanUpPostDataState() { 1826 pps = PostParserState.init; 1827 } 1828 1829 /// you can override this function to somehow react 1830 /// to an upload in progress. 1831 /// 1832 /// Take note that parts of the CGI object is not yet 1833 /// initialized! Stuff from HTTP headers, including get[], is usable. 1834 /// But, none of post[] is usable, and you cannot write here. That's 1835 /// why this method is const - mutating the object won't do much anyway. 1836 /// 1837 /// My idea here was so you can output a progress bar or 1838 /// something to a cooperative client (see arsd.rtud for a potential helper) 1839 /// 1840 /// The default is to do nothing. Subclass cgi and use the 1841 /// CustomCgiMain mixin to do something here. 1842 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1843 // This space intentionally left blank. 1844 } 1845 1846 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1847 /// *closeConnection will be set to true if you should close the connection after handling this request 1848 this(BufferedInputRange ir, bool* closeConnection) { 1849 isCalledWithCommandLineArguments = false; 1850 import al = std.algorithm; 1851 1852 immutable(ubyte)[] data; 1853 1854 void rdo(const(ubyte)[] d) { 1855 //import std.stdio; writeln(d); 1856 sendAll(ir.source, d); 1857 } 1858 1859 auto ira = ir.source.remoteAddress(); 1860 auto irLocalAddress = ir.source.localAddress(); 1861 1862 ushort port = 80; 1863 if(auto ia = cast(InternetAddress) irLocalAddress) { 1864 port = ia.port; 1865 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1866 port = ia.port; 1867 } 1868 1869 // that check for UnixAddress is to work around a Phobos bug 1870 // see: https://github.com/dlang/phobos/pull/7383 1871 // but this might be more useful anyway tbh for this case 1872 version(Posix) 1873 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1874 else 1875 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1876 } 1877 1878 /** 1879 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1880 1881 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1882 1883 Params: 1884 inputData = the incoming data, including headers and other raw http data. 1885 When the constructor exits, it will leave this range exactly at the start of 1886 the next request on the connection (if there is one). 1887 1888 address = the IP address of the remote user 1889 _port = the port number of the connection 1890 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1891 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1892 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1893 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1894 closeConnection = if the request asks to close the connection, *closeConnection == true. 1895 */ 1896 this( 1897 BufferedInputRange inputData, 1898 // string[] headers, immutable(ubyte)[] data, 1899 string address, ushort _port, 1900 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1901 bool _https = false, 1902 void delegate(const(ubyte)[]) _rawDataOutput = null, 1903 void delegate() _flush = null, 1904 // this pointer tells if the connection is supposed to be closed after we handle this 1905 bool* closeConnection = null) 1906 { 1907 // these are all set locally so the loop works 1908 // without triggering errors in dmd 2.064 1909 // we go ahead and set them at the end of it to the this version 1910 int port; 1911 string referrer; 1912 string remoteAddress; 1913 string userAgent; 1914 string authorization; 1915 string origin; 1916 string accept; 1917 string lastEventId; 1918 bool https; 1919 string host; 1920 RequestMethod requestMethod; 1921 string requestUri; 1922 string pathInfo; 1923 string queryString; 1924 string scriptName; 1925 string[string] get; 1926 string[][string] getArray; 1927 bool keepAliveRequested; 1928 bool acceptsGzip; 1929 string cookie; 1930 1931 1932 1933 environmentVariables = cast(const) environment.toAA; 1934 1935 idlol = inputData; 1936 1937 isCalledWithCommandLineArguments = false; 1938 1939 https = _https; 1940 port = _port; 1941 1942 rawDataOutput = _rawDataOutput; 1943 flushDelegate = _flush; 1944 nph = true; 1945 1946 remoteAddress = address; 1947 1948 // streaming parser 1949 import al = std.algorithm; 1950 1951 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1952 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1953 while(idx == -1) { 1954 inputData.popFront(0); 1955 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1956 } 1957 1958 assert(idx != -1); 1959 1960 1961 string contentType = ""; 1962 string[string] requestHeadersHere; 1963 1964 size_t contentLength; 1965 1966 bool isChunked; 1967 1968 { 1969 import core.runtime; 1970 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 1971 } 1972 1973 1974 int headerNumber = 0; 1975 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 1976 if(line.length) { 1977 headerNumber++; 1978 auto header = cast(string) line.idup; 1979 if(headerNumber == 1) { 1980 // request line 1981 auto parts = al.splitter(header, " "); 1982 requestMethod = to!RequestMethod(parts.front); 1983 parts.popFront(); 1984 requestUri = parts.front; 1985 1986 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 1987 scriptName = requestUri[0 .. pathInfoStarts]; 1988 1989 auto question = requestUri.indexOf("?"); 1990 if(question == -1) { 1991 queryString = ""; 1992 // FIXME: double check, this might be wrong since it could be url encoded 1993 pathInfo = requestUri[pathInfoStarts..$]; 1994 } else { 1995 queryString = requestUri[question+1..$]; 1996 pathInfo = requestUri[pathInfoStarts..question]; 1997 } 1998 1999 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2000 getArray = cast(string[][string]) assumeUnique(ugh); 2001 2002 if(header.indexOf("HTTP/1.0") != -1) { 2003 http10 = true; 2004 autoBuffer = true; 2005 if(closeConnection) { 2006 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2007 *closeConnection = true; 2008 } 2009 } 2010 } else { 2011 // other header 2012 auto colon = header.indexOf(":"); 2013 if(colon == -1) 2014 throw new Exception("HTTP headers should have a colon!"); 2015 string name = header[0..colon].toLower; 2016 string value = header[colon+2..$]; // skip the colon and the space 2017 2018 requestHeadersHere[name] = value; 2019 2020 if (name == "accept") { 2021 accept = value; 2022 } 2023 else if (name == "origin") { 2024 origin = value; 2025 } 2026 else if (name == "connection") { 2027 if(value == "close" && closeConnection) 2028 *closeConnection = true; 2029 if(value.asLowerCase().canFind("keep-alive")) { 2030 keepAliveRequested = true; 2031 2032 // on http 1.0, the connection is closed by default, 2033 // but not if they request keep-alive. then we don't close 2034 // anymore - undoing the set above 2035 if(http10 && closeConnection) { 2036 *closeConnection = false; 2037 } 2038 } 2039 } 2040 else if (name == "transfer-encoding") { 2041 if(value == "chunked") 2042 isChunked = true; 2043 } 2044 else if (name == "last-event-id") { 2045 lastEventId = value; 2046 } 2047 else if (name == "authorization") { 2048 authorization = value; 2049 } 2050 else if (name == "content-type") { 2051 contentType = value; 2052 } 2053 else if (name == "content-length") { 2054 contentLength = to!size_t(value); 2055 } 2056 else if (name == "x-forwarded-for") { 2057 remoteAddress = value; 2058 } 2059 else if (name == "x-forwarded-host" || name == "host") { 2060 if(name != "host" || host is null) 2061 host = value; 2062 } 2063 // FIXME: https://tools.ietf.org/html/rfc7239 2064 else if (name == "accept-encoding") { 2065 if(value.indexOf("gzip") != -1) 2066 acceptsGzip = true; 2067 } 2068 else if (name == "user-agent") { 2069 userAgent = value; 2070 } 2071 else if (name == "referer") { 2072 referrer = value; 2073 } 2074 else if (name == "cookie") { 2075 cookie ~= value; 2076 } else if(name == "expect") { 2077 if(value == "100-continue") { 2078 // FIXME we should probably give user code a chance 2079 // to process and reject but that needs to be virtual, 2080 // perhaps part of the CGI redesign. 2081 2082 // FIXME: if size is > max content length it should 2083 // also fail at this point. 2084 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2085 2086 // FIXME: let the user write out 103 early hints too 2087 } 2088 } 2089 // else 2090 // ignore it 2091 2092 } 2093 } 2094 2095 inputData.consume(idx + 4); 2096 // done 2097 2098 requestHeaders = assumeUnique(requestHeadersHere); 2099 2100 ByChunkRange dataByChunk; 2101 2102 // reading Content-Length type data 2103 // We need to read up the data we have, and write it out as a chunk. 2104 if(!isChunked) { 2105 dataByChunk = byChunk(inputData, contentLength); 2106 } else { 2107 // chunked requests happen, but not every day. Since we need to know 2108 // the content length (for now, maybe that should change), we'll buffer 2109 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2110 auto data = dechunk(inputData); 2111 2112 // set the range here 2113 dataByChunk = byChunk(data); 2114 contentLength = data.length; 2115 } 2116 2117 assert(dataByChunk !is null); 2118 2119 if(contentLength) { 2120 prepareForIncomingDataChunks(contentType, contentLength); 2121 foreach(dataChunk; dataByChunk) { 2122 handleIncomingDataChunk(dataChunk); 2123 } 2124 postArray = assumeUnique(pps._post); 2125 filesArray = assumeUnique(pps._files); 2126 files = keepLastOf(filesArray); 2127 post = keepLastOf(postArray); 2128 postBody = pps.postBody; 2129 cleanUpPostDataState(); 2130 } 2131 2132 this.port = port; 2133 this.referrer = referrer; 2134 this.remoteAddress = remoteAddress; 2135 this.userAgent = userAgent; 2136 this.authorization = authorization; 2137 this.origin = origin; 2138 this.accept = accept; 2139 this.lastEventId = lastEventId; 2140 this.https = https; 2141 this.host = host; 2142 this.requestMethod = requestMethod; 2143 this.requestUri = requestUri; 2144 this.pathInfo = pathInfo; 2145 this.queryString = queryString; 2146 2147 this.scriptName = scriptName; 2148 this.get = keepLastOf(getArray); 2149 this.getArray = cast(immutable) getArray; 2150 this.keepAliveRequested = keepAliveRequested; 2151 this.acceptsGzip = acceptsGzip; 2152 this.cookie = cookie; 2153 2154 cookiesArray = getCookieArray(); 2155 cookies = keepLastOf(cookiesArray); 2156 2157 } 2158 BufferedInputRange idlol; 2159 2160 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2161 string[string] ca; 2162 foreach(k, v; arr) 2163 ca[k] = v[$-1]; 2164 2165 return assumeUnique(ca); 2166 } 2167 2168 // FIXME duplication 2169 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2170 UploadedFile[string] ca; 2171 foreach(k, v; arr) 2172 ca[k] = v[$-1]; 2173 2174 return assumeUnique(ca); 2175 } 2176 2177 2178 private immutable(string[][string]) getCookieArray() { 2179 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2180 return assumeUnique(forTheLoveOfGod); 2181 } 2182 2183 /// Very simple method to require a basic auth username and password. 2184 /// If the http request doesn't include the required credentials, it throws a 2185 /// HTTP 401 error, and an exception. 2186 /// 2187 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 2188 /// the user's credentials are sent in plain text on every request. 2189 /// 2190 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2191 /// application. Either use Apache's built in methods for basic authentication, or add 2192 /// something along these lines to your server configuration: 2193 /// 2194 /// RewriteEngine On 2195 /// RewriteCond %{HTTP:Authorization} ^(.*) 2196 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2197 /// 2198 /// To ensure the necessary data is available to cgi.d. 2199 void requireBasicAuth(string user, string pass, string message = null) { 2200 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2201 setResponseStatus("401 Authorization Required"); 2202 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 2203 close(); 2204 throw new Exception("Not authorized; got " ~ authorization); 2205 } 2206 } 2207 2208 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2209 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2210 /// Use setResponseExpires and updateResponseExpires for more control 2211 void setCache(bool allowCaching) { 2212 noCache = !allowCaching; 2213 } 2214 2215 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2216 /// who can accept it 2217 bool gzipResponse; 2218 2219 immutable bool acceptsGzip; 2220 immutable bool keepAliveRequested; 2221 2222 /// Set to true if and only if this was initialized with command line arguments 2223 immutable bool isCalledWithCommandLineArguments; 2224 2225 /// This gets a full url for the current request, including port, protocol, host, path, and query 2226 string getCurrentCompleteUri() const { 2227 ushort defaultPort = https ? 443 : 80; 2228 2229 string uri = "http"; 2230 if(https) 2231 uri ~= "s"; 2232 uri ~= "://"; 2233 uri ~= host; 2234 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2235 version(none) 2236 if(!(!port || port == defaultPort)) { 2237 uri ~= ":"; 2238 uri ~= to!string(port); 2239 } 2240 +/ 2241 uri ~= requestUri; 2242 return uri; 2243 } 2244 2245 /// You can override this if your site base url isn't the same as the script name 2246 string logicalScriptName() const { 2247 return scriptName; 2248 } 2249 2250 /++ 2251 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2252 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2253 Note setResponseStatus() must be called *before* you write() any data to the output. 2254 2255 History: 2256 The `int` overload was added on January 11, 2021. 2257 +/ 2258 void setResponseStatus(string status) { 2259 assert(!outputtedResponseData); 2260 responseStatus = status; 2261 } 2262 /// ditto 2263 void setResponseStatus(int statusCode) { 2264 setResponseStatus(getHttpCodeText(statusCode)); 2265 } 2266 private string responseStatus = null; 2267 2268 /// Returns true if it is still possible to output headers 2269 bool canOutputHeaders() { 2270 return !isClosed && !outputtedResponseData; 2271 } 2272 2273 /// Sets the location header, which the browser will redirect the user to automatically. 2274 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2275 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2276 void setResponseLocation(string uri, bool important = true, string status = null) { 2277 if(!important && isCurrentResponseLocationImportant) 2278 return; // important redirects always override unimportant ones 2279 2280 if(uri is null) { 2281 responseStatus = "200 OK"; 2282 responseLocation = null; 2283 isCurrentResponseLocationImportant = important; 2284 return; // this just cancels the redirect 2285 } 2286 2287 assert(!outputtedResponseData); 2288 if(status is null) 2289 responseStatus = "302 Found"; 2290 else 2291 responseStatus = status; 2292 2293 responseLocation = uri.strip; 2294 isCurrentResponseLocationImportant = important; 2295 } 2296 protected string responseLocation = null; 2297 private bool isCurrentResponseLocationImportant = false; 2298 2299 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2300 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2301 /// Note: the when parameter is different than setCookie's expire parameter. 2302 void setResponseExpires(long when, bool isPublic = false) { 2303 responseExpires = when; 2304 setCache(true); // need to enable caching so the date has meaning 2305 2306 responseIsPublic = isPublic; 2307 responseExpiresRelative = false; 2308 } 2309 2310 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2311 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2312 responseExpires = whenFromNow; 2313 setCache(true); // need to enable caching so the date has meaning 2314 2315 responseIsPublic = isPublic; 2316 responseExpiresRelative = true; 2317 } 2318 private long responseExpires = long.min; 2319 private bool responseIsPublic = false; 2320 private bool responseExpiresRelative = false; 2321 2322 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2323 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2324 /// output as a whole is as cacheable as the least cachable part in the chain. 2325 2326 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2327 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2328 void updateResponseExpires(long when, bool isPublic) { 2329 if(responseExpires == long.min) 2330 setResponseExpires(when, isPublic); 2331 else if(when < responseExpires) 2332 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2333 } 2334 2335 /* 2336 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2337 /// Should generally be false if the user is logged in. It assumes private cache only. 2338 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2339 void setPublicCaching(bool allowPublicCaches) { 2340 publicCaching = allowPublicCaches; 2341 } 2342 private bool publicCaching = false; 2343 */ 2344 2345 /++ 2346 History: 2347 Added January 11, 2021 2348 +/ 2349 enum SameSitePolicy { 2350 Lax, 2351 Strict, 2352 None 2353 } 2354 2355 /++ 2356 Sets an HTTP cookie, automatically encoding the data to the correct string. 2357 expiresIn is how many milliseconds in the future the cookie will expire. 2358 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2359 Note setCookie() must be called *before* you write() any data to the output. 2360 2361 History: 2362 Parameter `sameSitePolicy` was added on January 11, 2021. 2363 +/ 2364 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2365 assert(!outputtedResponseData); 2366 string cookie = std.uri.encodeComponent(name) ~ "="; 2367 cookie ~= std.uri.encodeComponent(data); 2368 if(path !is null) 2369 cookie ~= "; path=" ~ path; 2370 // FIXME: should I just be using max-age here? (also in cache below) 2371 if(expiresIn != 0) 2372 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2373 if(domain !is null) 2374 cookie ~= "; domain=" ~ domain; 2375 if(secure == true) 2376 cookie ~= "; Secure"; 2377 if(httpOnly == true ) 2378 cookie ~= "; HttpOnly"; 2379 final switch(sameSitePolicy) { 2380 case SameSitePolicy.Lax: 2381 cookie ~= "; SameSite=Lax"; 2382 break; 2383 case SameSitePolicy.Strict: 2384 cookie ~= "; SameSite=Strict"; 2385 break; 2386 case SameSitePolicy.None: 2387 cookie ~= "; SameSite=None"; 2388 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2389 break; 2390 } 2391 2392 if(auto idx = name in cookieIndexes) { 2393 responseCookies[*idx] = cookie; 2394 } else { 2395 cookieIndexes[name] = responseCookies.length; 2396 responseCookies ~= cookie; 2397 } 2398 } 2399 private string[] responseCookies; 2400 private size_t[string] cookieIndexes; 2401 2402 /// Clears a previously set cookie with the given name, path, and domain. 2403 void clearCookie(string name, string path = null, string domain = null) { 2404 assert(!outputtedResponseData); 2405 setCookie(name, "", 1, path, domain); 2406 } 2407 2408 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2409 void setResponseContentType(string ct) { 2410 assert(!outputtedResponseData); 2411 responseContentType = ct; 2412 } 2413 private string responseContentType = null; 2414 2415 /// Adds a custom header. It should be the name: value, but without any line terminator. 2416 /// For example: header("X-My-Header: Some value"); 2417 /// Note you should use the specialized functions in this object if possible to avoid 2418 /// duplicates in the output. 2419 void header(string h) { 2420 customHeaders ~= h; 2421 } 2422 2423 /++ 2424 I named the original function `header` after PHP, but this pattern more fits 2425 the rest of the Cgi object. 2426 2427 Either name are allowed. 2428 2429 History: 2430 Alias added June 17, 2022. 2431 +/ 2432 alias setResponseHeader = header; 2433 2434 private string[] customHeaders; 2435 private bool websocketMode; 2436 2437 void flushHeaders(const(void)[] t, bool isAll = false) { 2438 StackBuffer buffer = StackBuffer(0); 2439 2440 prepHeaders(t, isAll, &buffer); 2441 2442 if(rawDataOutput !is null) 2443 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2444 else { 2445 stdout.rawWrite(buffer.get()); 2446 } 2447 } 2448 2449 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2450 string terminator = "\n"; 2451 if(rawDataOutput !is null) 2452 terminator = "\r\n"; 2453 2454 if(responseStatus !is null) { 2455 if(nph) { 2456 if(http10) 2457 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2458 else 2459 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2460 } else 2461 buffer.add("Status: ", responseStatus, terminator); 2462 } else if (nph) { 2463 if(http10) 2464 buffer.add("HTTP/1.0 200 OK", terminator); 2465 else 2466 buffer.add("HTTP/1.1 200 OK", terminator); 2467 } 2468 2469 if(websocketMode) 2470 goto websocket; 2471 2472 if(nph) { // we're responsible for setting the date too according to http 1.1 2473 char[29] db = void; 2474 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2475 buffer.add("Date: ", db[], terminator); 2476 } 2477 2478 // FIXME: what if the user wants to set his own content-length? 2479 // The custom header function can do it, so maybe that's best. 2480 // Or we could reuse the isAll param. 2481 if(responseLocation !is null) { 2482 buffer.add("Location: ", responseLocation, terminator); 2483 } 2484 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2485 if(responseExpiresRelative) { 2486 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2487 buffer.add(responseExpires); 2488 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2489 } else { 2490 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2491 char[29] db = void; 2492 printDateToBuffer(cast(DateTime) expires, db[]); 2493 buffer.add("Expires: ", db[], terminator); 2494 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2495 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2496 buffer.add(terminator); 2497 } 2498 } 2499 if(responseCookies !is null && responseCookies.length > 0) { 2500 foreach(c; responseCookies) 2501 buffer.add("Set-Cookie: ", c, terminator); 2502 } 2503 if(noCache) { // we specifically do not want caching (this is actually the default) 2504 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2505 buffer.add("Expires: 0", terminator); 2506 buffer.add("Pragma: no-cache", terminator); 2507 } else { 2508 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2509 buffer.add("Cache-Control: public", terminator); 2510 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2511 } 2512 } 2513 if(responseContentType !is null) { 2514 buffer.add("Content-Type: ", responseContentType, terminator); 2515 } else 2516 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2517 2518 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2519 buffer.add("Content-Encoding: gzip", terminator); 2520 } 2521 2522 2523 if(!isAll) { 2524 if(nph && !http10) { 2525 buffer.add("Transfer-Encoding: chunked", terminator); 2526 responseChunked = true; 2527 } 2528 } else { 2529 buffer.add("Content-Length: "); 2530 buffer.add(t.length); 2531 buffer.add(terminator); 2532 if(nph && keepAliveRequested) { 2533 buffer.add("Connection: Keep-Alive", terminator); 2534 } 2535 } 2536 2537 websocket: 2538 2539 foreach(hd; customHeaders) 2540 buffer.add(hd, terminator); 2541 2542 // FIXME: what about duplicated headers? 2543 2544 // end of header indicator 2545 buffer.add(terminator); 2546 2547 outputtedResponseData = true; 2548 } 2549 2550 /// Writes the data to the output, flushing headers if they have not yet been sent. 2551 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2552 assert(!closed, "Output has already been closed"); 2553 2554 StackBuffer buffer = StackBuffer(0); 2555 2556 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2557 // actually gzip the data here 2558 2559 auto c = new Compress(HeaderFormat.gzip); // want gzip 2560 2561 auto data = c.compress(t); 2562 data ~= c.flush(); 2563 2564 // std.file.write("/tmp/last-item", data); 2565 2566 t = data; 2567 } 2568 2569 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2570 prepHeaders(t, isAll, &buffer); 2571 } 2572 2573 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2574 if (autoBuffer && !isAll) { 2575 outputBuffer ~= cast(ubyte[]) t; 2576 } 2577 if(!autoBuffer || isAll) { 2578 if(rawDataOutput !is null) 2579 if(nph && responseChunked) { 2580 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2581 // we're making the chunk here instead of in a function 2582 // to avoid unneeded gc pressure 2583 buffer.add(toHex(t.length)); 2584 buffer.add("\r\n"); 2585 buffer.add(cast(char[]) t, "\r\n"); 2586 } else { 2587 buffer.add(cast(char[]) t); 2588 } 2589 else 2590 buffer.add(cast(char[]) t); 2591 } 2592 } 2593 2594 if(rawDataOutput !is null) 2595 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2596 else 2597 stdout.rawWrite(buffer.get()); 2598 2599 if(maybeAutoClose && isAll) 2600 close(); // if you say it is all, that means we're definitely done 2601 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2602 } 2603 2604 /++ 2605 Convenience method to set content type to json and write the string as the complete response. 2606 2607 History: 2608 Added January 16, 2020 2609 +/ 2610 void writeJson(string json) { 2611 this.setResponseContentType("application/json"); 2612 this.write(json, true); 2613 } 2614 2615 /// Flushes the pending buffer, leaving the connection open so you can send more. 2616 void flush() { 2617 if(rawDataOutput is null) 2618 stdout.flush(); 2619 else if(flushDelegate !is null) 2620 flushDelegate(); 2621 } 2622 2623 version(autoBuffer) 2624 bool autoBuffer = true; 2625 else 2626 bool autoBuffer = false; 2627 ubyte[] outputBuffer; 2628 2629 /// Flushes the buffers to the network, signifying that you are done. 2630 /// You should always call this explicitly when you are done outputting data. 2631 void close() { 2632 if(closed) 2633 return; // don't double close 2634 2635 if(!outputtedResponseData) 2636 write("", true, false); 2637 2638 // writing auto buffered data 2639 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2640 if(!nph) 2641 stdout.rawWrite(outputBuffer); 2642 else 2643 write(outputBuffer, true, false); // tell it this is everything 2644 } 2645 2646 // closing the last chunk... 2647 if(nph && rawDataOutput !is null && responseChunked) 2648 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2649 2650 if(flushDelegate) 2651 flushDelegate(); 2652 2653 closed = true; 2654 } 2655 2656 // Closes without doing anything, shouldn't be used often 2657 void rawClose() { 2658 closed = true; 2659 } 2660 2661 /++ 2662 Gets a request variable as a specific type, or the default value of it isn't there 2663 or isn't convertible to the request type. 2664 2665 Checks both GET and POST variables, preferring the POST variable, if available. 2666 2667 A nice trick is using the default value to choose the type: 2668 2669 --- 2670 /* 2671 The return value will match the type of the default. 2672 Here, I gave 10 as a default, so the return value will 2673 be an int. 2674 2675 If the user-supplied value cannot be converted to the 2676 requested type, you will get the default value back. 2677 */ 2678 int a = cgi.request("number", 10); 2679 2680 if(cgi.get["number"] == "11") 2681 assert(a == 11); // conversion succeeds 2682 2683 if("number" !in cgi.get) 2684 assert(a == 10); // no value means you can't convert - give the default 2685 2686 if(cgi.get["number"] == "twelve") 2687 assert(a == 10); // conversion from string to int would fail, so we get the default 2688 --- 2689 2690 You can use an enum as an easy whitelist, too: 2691 2692 --- 2693 enum Operations { 2694 add, remove, query 2695 } 2696 2697 auto op = cgi.request("op", Operations.query); 2698 2699 if(cgi.get["op"] == "add") 2700 assert(op == Operations.add); 2701 if(cgi.get["op"] == "remove") 2702 assert(op == Operations.remove); 2703 if(cgi.get["op"] == "query") 2704 assert(op == Operations.query); 2705 2706 if(cgi.get["op"] == "random string") 2707 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2708 --- 2709 +/ 2710 T request(T = string)(in string name, in T def = T.init) const nothrow { 2711 try { 2712 return 2713 (name in post) ? to!T(post[name]) : 2714 (name in get) ? to!T(get[name]) : 2715 def; 2716 } catch(Exception e) { return def; } 2717 } 2718 2719 /// Is the output already closed? 2720 bool isClosed() const { 2721 return closed; 2722 } 2723 2724 /++ 2725 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2726 +/ 2727 Session!Data getSessionObject(Data)() { 2728 if(testInProcess !is null) { 2729 // test mode 2730 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2731 if(obj !is null) 2732 return cast(typeof(return)) obj; 2733 else { 2734 auto o = new MockSession!Data(); 2735 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2736 return o; 2737 } 2738 } else { 2739 // normal operation 2740 return new BasicDataServerSession!Data(this); 2741 } 2742 } 2743 2744 // if it is in test mode; triggers mock sessions. Used by CgiTester 2745 version(with_breaking_cgi_features) 2746 private CgiTester testInProcess; 2747 2748 /* Hooks for redirecting input and output */ 2749 private void delegate(const(ubyte)[]) rawDataOutput = null; 2750 private void delegate() flushDelegate = null; 2751 2752 /* This info is used when handling a more raw HTTP protocol */ 2753 private bool nph; 2754 private bool http10; 2755 private bool closed; 2756 private bool responseChunked = false; 2757 2758 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2759 immutable(ubyte)[] originalPostData; 2760 2761 /++ 2762 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2763 2764 It is intended to be used for JSON and XML request content types, but also may be used 2765 for other content types your application can handle. But it will NOT be populated 2766 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2767 parsed into the post and postArray members. 2768 2769 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2770 will be discarded to the client with an error. This helps keep this array from being exploded in size 2771 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2772 client in certain build modes.) 2773 2774 History: 2775 Added January 5, 2021 2776 Documented February 21, 2023 (dub v11.0) 2777 +/ 2778 public immutable string postBody; 2779 alias postJson = postBody; // old name 2780 2781 /* Internal state flags */ 2782 private bool outputtedResponseData; 2783 private bool noCache = true; 2784 2785 const(string[string]) environmentVariables; 2786 2787 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2788 partially because it logically is (your code doesn't change what the user requested...) 2789 and partially because I hate how bad programs in PHP change those superglobals to do 2790 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2791 2792 For some of these, you'll want to refer to the http or cgi specs for more details. 2793 */ 2794 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2795 2796 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2797 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2798 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2799 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2800 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2801 immutable(char[]) scriptFileName; /// The physical filename of your script 2802 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2803 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2804 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2805 2806 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2807 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2808 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2809 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2810 2811 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2812 */ 2813 immutable(char[]) referrer; 2814 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2815 2816 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2817 2818 immutable bool https; /// Was the request encrypted via https? 2819 immutable int port; /// On what TCP port number did the server receive the request? 2820 2821 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2822 2823 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2824 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2825 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2826 2827 /** 2828 Represents user uploaded files. 2829 2830 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2831 */ 2832 immutable(UploadedFile[][string]) filesArray; 2833 immutable(UploadedFile[string]) files; 2834 2835 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2836 /// the order of the arrays is the order the data arrives 2837 immutable(string[][string]) getArray; /// like get, but an array of values per name 2838 immutable(string[][string]) postArray; /// ditto for post 2839 immutable(string[][string]) cookiesArray; /// ditto for cookies 2840 2841 // convenience function for appending to a uri without extra ? 2842 // matches the name and effect of javascript's location.search property 2843 string search() const { 2844 if(queryString.length) 2845 return "?" ~ queryString; 2846 return ""; 2847 } 2848 2849 // FIXME: what about multiple files with the same name? 2850 private: 2851 //RequestMethod _requestMethod; 2852 } 2853 2854 /// use this for testing or other isolated things when you want it to be no-ops 2855 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2856 // we want to ignore, not use stdout 2857 if(outputSink is null) 2858 outputSink = delegate void(const(ubyte)[]) { }; 2859 2860 string[string] env; 2861 env["REQUEST_METHOD"] = to!string(method); 2862 env["CONTENT_LENGTH"] = to!string(data.length); 2863 2864 auto cgi = new Cgi( 2865 0, 2866 env, 2867 { return data; }, 2868 outputSink, 2869 null); 2870 2871 return cgi; 2872 } 2873 2874 /++ 2875 A helper test class for request handler unittests. 2876 +/ 2877 version(with_breaking_cgi_features) 2878 class CgiTester { 2879 private { 2880 SessionObject[TypeInfo] mockSessions; 2881 SessionObject getSessionOverride(TypeInfo ti) { 2882 if(auto o = ti in mockSessions) 2883 return *o; 2884 else 2885 return null; 2886 } 2887 void setSessionOverride(TypeInfo ti, SessionObject so) { 2888 mockSessions[ti] = so; 2889 } 2890 } 2891 2892 /++ 2893 Gets (and creates if necessary) a mock session object for this test. Note 2894 it will be the same one used for any test operations through this CgiTester instance. 2895 +/ 2896 Session!Data getSessionObject(Data)() { 2897 auto obj = getSessionOverride(typeid(typeof(return))); 2898 if(obj !is null) 2899 return cast(typeof(return)) obj; 2900 else { 2901 auto o = new MockSession!Data(); 2902 setSessionOverride(typeid(typeof(return)), o); 2903 return o; 2904 } 2905 } 2906 2907 /++ 2908 Pass a reference to your request handler when creating the tester. 2909 +/ 2910 this(void function(Cgi) requestHandler) { 2911 this.requestHandler = requestHandler; 2912 } 2913 2914 /++ 2915 You can check response information with these methods after you call the request handler. 2916 +/ 2917 struct Response { 2918 int code; 2919 string[string] headers; 2920 string responseText; 2921 ubyte[] responseBody; 2922 } 2923 2924 /++ 2925 Executes a test request on your request handler, and returns the response. 2926 2927 Params: 2928 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2929 args = additional arguments. Same format as cgi's command line handler. 2930 +/ 2931 Response GET(string url, string[] args = null) { 2932 return executeTest("GET", url, args); 2933 } 2934 /// ditto 2935 Response POST(string url, string[] args = null) { 2936 return executeTest("POST", url, args); 2937 } 2938 2939 /// ditto 2940 Response executeTest(string method, string url, string[] args) { 2941 ubyte[] outputtedRawData; 2942 void outputSink(const(ubyte)[] data) { 2943 outputtedRawData ~= data; 2944 } 2945 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2946 cgi.testInProcess = this; 2947 scope(exit) cgi.dispose(); 2948 2949 requestHandler(cgi); 2950 2951 cgi.close(); 2952 2953 Response response; 2954 2955 if(outputtedRawData.length) { 2956 enum LINE = "\r\n"; 2957 2958 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2959 assert(idx != -1, to!string(outputtedRawData)); 2960 auto headers = cast(string) outputtedRawData[0 .. idx]; 2961 response.code = 200; 2962 while(headers.length) { 2963 auto i = headers.locationOf(LINE); 2964 if(i == -1) i = cast(int) headers.length; 2965 2966 auto header = headers[0 .. i]; 2967 2968 auto c = header.locationOf(":"); 2969 if(c != -1) { 2970 auto name = header[0 .. c]; 2971 auto value = header[c + 2 ..$]; 2972 2973 if(name == "Status") 2974 response.code = value[0 .. value.locationOf(" ")].to!int; 2975 2976 response.headers[name] = value; 2977 } else { 2978 assert(0); 2979 } 2980 2981 if(i != headers.length) 2982 i += 2; 2983 headers = headers[i .. $]; 2984 } 2985 response.responseBody = outputtedRawData[idx + 4 .. $]; 2986 response.responseText = cast(string) response.responseBody; 2987 } 2988 2989 return response; 2990 } 2991 2992 private void function(Cgi) requestHandler; 2993 } 2994 2995 2996 // should this be a separate module? Probably, but that's a hassle. 2997 2998 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 2999 string makeDataUrl(string mimeType, in void[] data) { 3000 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3001 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3002 } 3003 3004 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3005 /// Represents a url that can be broken down or built up through properties 3006 struct Uri { 3007 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3008 3009 // scheme//userinfo@host:port/path?query#fragment 3010 3011 string scheme; /// e.g. "http" in "http://example.com/" 3012 string userinfo; /// the username (and possibly a password) in the uri 3013 string host; /// the domain name 3014 int port; /// port number, if given. Will be zero if a port was not explicitly given 3015 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3016 string query; /// the stuff after the ? in a uri 3017 string fragment; /// the stuff after the # in a uri. 3018 3019 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3020 // the decode ones need to keep different names anyway because we can't overload on return values... 3021 static string encode(string s) { return std.uri.encodeComponent(s); } 3022 static string encode(string[string] s) { return encodeVariables(s); } 3023 static string encode(string[][string] s) { return encodeVariables(s); } 3024 3025 /// Breaks down a uri string to its components 3026 this(string uri) { 3027 reparse(uri); 3028 } 3029 3030 private void reparse(string uri) { 3031 // from RFC 3986 3032 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3033 // it was a nice experiment but just not worth it. 3034 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3035 /* 3036 Captures: 3037 0 = whole url 3038 1 = scheme, with : 3039 2 = scheme, no : 3040 3 = authority, with // 3041 4 = authority, no // 3042 5 = path 3043 6 = query string, with ? 3044 7 = query string, no ? 3045 8 = anchor, with # 3046 9 = anchor, no # 3047 */ 3048 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3049 // instead, I will DIY and cut that down to 0.6s on the same computer. 3050 /* 3051 3052 Note that authority is 3053 user:password@domain:port 3054 where the user:password@ part is optional, and the :port is optional. 3055 3056 Regex translation: 3057 3058 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3059 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3060 Path cannot have any ? or # in it. It is optional. 3061 Query must start with ? and must not have # in it. It is optional. 3062 Anchor must start with # and can have anything else in it to end of string. It is optional. 3063 */ 3064 3065 this = Uri.init; // reset all state 3066 3067 // empty uri = nothing special 3068 if(uri.length == 0) { 3069 return; 3070 } 3071 3072 size_t idx; 3073 3074 scheme_loop: foreach(char c; uri[idx .. $]) { 3075 switch(c) { 3076 case ':': 3077 case '/': 3078 case '?': 3079 case '#': 3080 break scheme_loop; 3081 default: 3082 } 3083 idx++; 3084 } 3085 3086 if(idx == 0 && uri[idx] == ':') { 3087 // this is actually a path! we skip way ahead 3088 goto path_loop; 3089 } 3090 3091 if(idx == uri.length) { 3092 // the whole thing is a path, apparently 3093 path = uri; 3094 return; 3095 } 3096 3097 if(idx > 0 && uri[idx] == ':') { 3098 scheme = uri[0 .. idx]; 3099 idx++; 3100 } else { 3101 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3102 idx = 0; 3103 } 3104 3105 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3106 // we have an authority.... 3107 idx += 2; 3108 3109 auto authority_start = idx; 3110 authority_loop: foreach(char c; uri[idx .. $]) { 3111 switch(c) { 3112 case '/': 3113 case '?': 3114 case '#': 3115 break authority_loop; 3116 default: 3117 } 3118 idx++; 3119 } 3120 3121 auto authority = uri[authority_start .. idx]; 3122 3123 auto idx2 = authority.indexOf("@"); 3124 if(idx2 != -1) { 3125 userinfo = authority[0 .. idx2]; 3126 authority = authority[idx2 + 1 .. $]; 3127 } 3128 3129 if(authority.length && authority[0] == '[') { 3130 // ipv6 address special casing 3131 idx2 = authority.indexOf(']'); 3132 if(idx2 != -1) { 3133 auto end = authority[idx2 + 1 .. $]; 3134 if(end.length && end[0] == ':') 3135 idx2 = idx2 + 1; 3136 else 3137 idx2 = -1; 3138 } 3139 } else { 3140 idx2 = authority.indexOf(":"); 3141 } 3142 3143 if(idx2 == -1) { 3144 port = 0; // 0 means not specified; we should use the default for the scheme 3145 host = authority; 3146 } else { 3147 host = authority[0 .. idx2]; 3148 if(idx2 + 1 < authority.length) 3149 port = to!int(authority[idx2 + 1 .. $]); 3150 else 3151 port = 0; 3152 } 3153 } 3154 3155 path_loop: 3156 auto path_start = idx; 3157 3158 foreach(char c; uri[idx .. $]) { 3159 if(c == '?' || c == '#') 3160 break; 3161 idx++; 3162 } 3163 3164 path = uri[path_start .. idx]; 3165 3166 if(idx == uri.length) 3167 return; // nothing more to examine... 3168 3169 if(uri[idx] == '?') { 3170 idx++; 3171 auto query_start = idx; 3172 foreach(char c; uri[idx .. $]) { 3173 if(c == '#') 3174 break; 3175 idx++; 3176 } 3177 query = uri[query_start .. idx]; 3178 } 3179 3180 if(idx < uri.length && uri[idx] == '#') { 3181 idx++; 3182 fragment = uri[idx .. $]; 3183 } 3184 3185 // uriInvalidated = false; 3186 } 3187 3188 private string rebuildUri() const { 3189 string ret; 3190 if(scheme.length) 3191 ret ~= scheme ~ ":"; 3192 if(userinfo.length || host.length) 3193 ret ~= "//"; 3194 if(userinfo.length) 3195 ret ~= userinfo ~ "@"; 3196 if(host.length) 3197 ret ~= host; 3198 if(port) 3199 ret ~= ":" ~ to!string(port); 3200 3201 ret ~= path; 3202 3203 if(query.length) 3204 ret ~= "?" ~ query; 3205 3206 if(fragment.length) 3207 ret ~= "#" ~ fragment; 3208 3209 // uri = ret; 3210 // uriInvalidated = false; 3211 return ret; 3212 } 3213 3214 /// Converts the broken down parts back into a complete string 3215 string toString() const { 3216 // if(uriInvalidated) 3217 return rebuildUri(); 3218 } 3219 3220 /// Returns a new absolute Uri given a base. It treats this one as 3221 /// relative where possible, but absolute if not. (If protocol, domain, or 3222 /// other info is not set, the new one inherits it from the base.) 3223 /// 3224 /// Browsers use a function like this to figure out links in html. 3225 Uri basedOn(in Uri baseUrl) const { 3226 Uri n = this; // copies 3227 if(n.scheme == "data") 3228 return n; 3229 // n.uriInvalidated = true; // make sure we regenerate... 3230 3231 // userinfo is not inherited... is this wrong? 3232 3233 // if anything is given in the existing url, we don't use the base anymore. 3234 if(n.scheme.empty) { 3235 n.scheme = baseUrl.scheme; 3236 if(n.host.empty) { 3237 n.host = baseUrl.host; 3238 if(n.port == 0) { 3239 n.port = baseUrl.port; 3240 if(n.path.length > 0 && n.path[0] != '/') { 3241 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3242 if(b.length == 0) 3243 b = "/"; 3244 n.path = b ~ n.path; 3245 } else if(n.path.length == 0) { 3246 n.path = baseUrl.path; 3247 } 3248 } 3249 } 3250 } 3251 3252 n.removeDots(); 3253 3254 return n; 3255 } 3256 3257 void removeDots() { 3258 auto parts = this.path.split("/"); 3259 string[] toKeep; 3260 foreach(part; parts) { 3261 if(part == ".") { 3262 continue; 3263 } else if(part == "..") { 3264 //if(toKeep.length > 1) 3265 toKeep = toKeep[0 .. $-1]; 3266 //else 3267 //toKeep = [""]; 3268 continue; 3269 } else { 3270 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3271 //continue; // skip a `//` situation 3272 toKeep ~= part; 3273 } 3274 } 3275 3276 auto path = toKeep.join("/"); 3277 if(path.length && path[0] != '/') 3278 path = "/" ~ path; 3279 3280 this.path = path; 3281 } 3282 3283 unittest { 3284 auto uri = Uri("test.html"); 3285 assert(uri.path == "test.html"); 3286 uri = Uri("path/1/lol"); 3287 assert(uri.path == "path/1/lol"); 3288 uri = Uri("http://me@example.com"); 3289 assert(uri.scheme == "http"); 3290 assert(uri.userinfo == "me"); 3291 assert(uri.host == "example.com"); 3292 uri = Uri("http://example.com/#a"); 3293 assert(uri.scheme == "http"); 3294 assert(uri.host == "example.com"); 3295 assert(uri.fragment == "a"); 3296 uri = Uri("#foo"); 3297 assert(uri.fragment == "foo"); 3298 uri = Uri("?lol"); 3299 assert(uri.query == "lol"); 3300 uri = Uri("#foo?lol"); 3301 assert(uri.fragment == "foo?lol"); 3302 uri = Uri("?lol#foo"); 3303 assert(uri.fragment == "foo"); 3304 assert(uri.query == "lol"); 3305 3306 uri = Uri("http://127.0.0.1/"); 3307 assert(uri.host == "127.0.0.1"); 3308 assert(uri.port == 0); 3309 3310 uri = Uri("http://127.0.0.1:123/"); 3311 assert(uri.host == "127.0.0.1"); 3312 assert(uri.port == 123); 3313 3314 uri = Uri("http://[ff:ff::0]/"); 3315 assert(uri.host == "[ff:ff::0]"); 3316 3317 uri = Uri("http://[ff:ff::0]:123/"); 3318 assert(uri.host == "[ff:ff::0]"); 3319 assert(uri.port == 123); 3320 } 3321 3322 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3323 // the possibilities. 3324 unittest { 3325 auto url = Uri("cool.html"); // checking relative links 3326 3327 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3328 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3329 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3330 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3331 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3332 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3333 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3334 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3335 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3336 3337 url = Uri("/something/cool.html"); // same server, different path 3338 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3339 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3340 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3341 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3342 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3343 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3344 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3345 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3346 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3347 3348 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3349 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3350 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3351 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3352 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3353 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3354 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3355 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3356 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3357 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3358 3359 url = Uri("/test/bar"); 3360 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3361 assert(Uri("../").basedOn(url) == "/"); 3362 3363 url = Uri("http://example.com/"); 3364 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3365 3366 //auto uriBefore = url; 3367 url = Uri("#anchor"); // everything should remain the same except the anchor 3368 //uriBefore.anchor = "anchor"); 3369 //assert(url == uriBefore); 3370 3371 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3372 3373 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3374 3375 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3376 3377 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3378 3379 // FIXME: add something for port too 3380 } 3381 3382 // these are like javascript's location.search and location.hash 3383 string search() const { 3384 return query.length ? ("?" ~ query) : ""; 3385 } 3386 string hash() const { 3387 return fragment.length ? ("#" ~ fragment) : ""; 3388 } 3389 } 3390 3391 3392 /* 3393 for session, see web.d 3394 */ 3395 3396 /// breaks down a url encoded string 3397 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3398 auto vars = data.split(separator); 3399 string[][string] _get; 3400 foreach(var; vars) { 3401 auto equal = var.indexOf("="); 3402 string name; 3403 string value; 3404 if(equal == -1) { 3405 name = decodeComponent(var); 3406 value = ""; 3407 } else { 3408 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3409 // stupid + -> space conversion. 3410 name = decodeComponent(var[0..equal].replace("+", " ")); 3411 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3412 } 3413 3414 _get[name] ~= value; 3415 if(namesInOrder) 3416 (*namesInOrder) ~= name; 3417 if(valuesInOrder) 3418 (*valuesInOrder) ~= value; 3419 } 3420 return _get; 3421 } 3422 3423 /// breaks down a url encoded string, but only returns the last value of any array 3424 string[string] decodeVariablesSingle(string data) { 3425 string[string] va; 3426 auto varArray = decodeVariables(data); 3427 foreach(k, v; varArray) 3428 va[k] = v[$-1]; 3429 3430 return va; 3431 } 3432 3433 /// url encodes the whole string 3434 string encodeVariables(in string[string] data) { 3435 string ret; 3436 3437 bool outputted = false; 3438 foreach(k, v; data) { 3439 if(outputted) 3440 ret ~= "&"; 3441 else 3442 outputted = true; 3443 3444 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3445 } 3446 3447 return ret; 3448 } 3449 3450 /// url encodes a whole string 3451 string encodeVariables(in string[][string] data) { 3452 string ret; 3453 3454 bool outputted = false; 3455 foreach(k, arr; data) { 3456 foreach(v; arr) { 3457 if(outputted) 3458 ret ~= "&"; 3459 else 3460 outputted = true; 3461 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3462 } 3463 } 3464 3465 return ret; 3466 } 3467 3468 /// Encodes all but the explicitly unreserved characters per rfc 3986 3469 /// Alphanumeric and -_.~ are the only ones left unencoded 3470 /// name is borrowed from php 3471 string rawurlencode(in char[] data) { 3472 string ret; 3473 ret.reserve(data.length * 2); 3474 foreach(char c; data) { 3475 if( 3476 (c >= 'a' && c <= 'z') || 3477 (c >= 'A' && c <= 'Z') || 3478 (c >= '0' && c <= '9') || 3479 c == '-' || c == '_' || c == '.' || c == '~') 3480 { 3481 ret ~= c; 3482 } else { 3483 ret ~= '%'; 3484 // since we iterate on char, this should give us the octets of the full utf8 string 3485 ret ~= toHexUpper(c); 3486 } 3487 } 3488 3489 return ret; 3490 } 3491 3492 3493 // http helper functions 3494 3495 // for chunked responses (which embedded http does whenever possible) 3496 version(none) // this is moved up above to avoid making a copy of the data 3497 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3498 const(ubyte)[] ret; 3499 3500 ret = cast(const(ubyte)[]) toHex(data.length); 3501 ret ~= cast(const(ubyte)[]) "\r\n"; 3502 ret ~= data; 3503 ret ~= cast(const(ubyte)[]) "\r\n"; 3504 3505 return ret; 3506 } 3507 3508 string toHex(long num) { 3509 string ret; 3510 while(num) { 3511 int v = num % 16; 3512 num /= 16; 3513 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3514 ret ~= d; 3515 } 3516 3517 return to!string(array(ret.retro)); 3518 } 3519 3520 string toHexUpper(long num) { 3521 string ret; 3522 while(num) { 3523 int v = num % 16; 3524 num /= 16; 3525 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3526 ret ~= d; 3527 } 3528 3529 if(ret.length == 1) 3530 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3531 3532 return to!string(array(ret.retro)); 3533 } 3534 3535 3536 // the generic mixins 3537 3538 /++ 3539 Use this instead of writing your own main 3540 3541 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3542 +/ 3543 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3544 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3545 } 3546 3547 /++ 3548 Boilerplate mixin for a main function that uses the [dispatcher] function. 3549 3550 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3551 3552 History: 3553 Added July 9, 2021 3554 +/ 3555 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3556 /++ 3557 Handler to the generated presenter you can use from your objects, etc. 3558 +/ 3559 Presenter activePresenter; 3560 3561 /++ 3562 Request handler that creates the presenter then forwards to the [dispatcher] function. 3563 Renders 404 if the dispatcher did not handle the request. 3564 3565 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3566 +/ 3567 void handler(Cgi cgi) { 3568 auto presenter = new Presenter; 3569 activePresenter = presenter; 3570 scope(exit) activePresenter = null; 3571 3572 if(cgi.dispatcher!DispatcherArgs(presenter)) 3573 return; 3574 3575 switch(cgi.pathInfo) { 3576 case "/style.css": 3577 cgi.setCache(true); 3578 cgi.setResponseContentType("text/css"); 3579 cgi.write(presenter.style(), true); 3580 break; 3581 case "/script.js": 3582 cgi.setCache(true); 3583 cgi.setResponseContentType("application/javascript"); 3584 cgi.write(presenter.script(), true); 3585 break; 3586 default: 3587 presenter.renderBasicError(cgi, 404); 3588 } 3589 } 3590 mixin GenericMain!handler; 3591 } 3592 3593 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3594 class GenericPresenter : WebPresenter!GenericPresenter {} 3595 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3596 } 3597 3598 private string simpleHtmlEncode(string s) { 3599 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3600 } 3601 3602 string messageFromException(Throwable t) { 3603 string message; 3604 if(t !is null) { 3605 debug message = t.toString(); 3606 else message = "An unexpected error has occurred."; 3607 } else { 3608 message = "Unknown error"; 3609 } 3610 return message; 3611 } 3612 3613 string plainHttpError(bool isCgi, string type, Throwable t) { 3614 auto message = messageFromException(t); 3615 message = simpleHtmlEncode(message); 3616 3617 return format("%s %s\r\nContent-Length: %s\r\n\r\n%s", 3618 isCgi ? "Status:" : "HTTP/1.0", 3619 type, message.length, message); 3620 } 3621 3622 // returns true if we were able to recover reasonably 3623 bool handleException(Cgi cgi, Throwable t) { 3624 if(cgi.isClosed) { 3625 // if the channel has been explicitly closed, we can't handle it here 3626 return true; 3627 } 3628 3629 if(cgi.outputtedResponseData) { 3630 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3631 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3632 } else { 3633 // no headers are sent, we can send a full blown error and recover 3634 cgi.setCache(false); 3635 cgi.setResponseContentType("text/html"); 3636 cgi.setResponseLocation(null); // cancel the redirect 3637 cgi.setResponseStatus("500 Internal Server Error"); 3638 cgi.write(simpleHtmlEncode(messageFromException(t))); 3639 cgi.close(); 3640 return true; 3641 } 3642 } 3643 3644 bool isCgiRequestMethod(string s) { 3645 s = s.toUpper(); 3646 if(s == "COMMANDLINE") 3647 return true; 3648 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3649 if(s == member) 3650 return true; 3651 return false; 3652 } 3653 3654 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3655 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3656 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3657 void main(string[] args) { 3658 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3659 } 3660 } 3661 3662 version(embedded_httpd_processes) 3663 __gshared int processPoolSize = 8; 3664 3665 // Returns true if run. You should exit the program after that. 3666 bool tryAddonServers(string[] args) { 3667 if(args.length > 1) { 3668 // run the special separate processes if needed 3669 switch(args[1]) { 3670 case "--websocket-server": 3671 version(with_addon_servers) 3672 websocketServers[args[2]](args[3 .. $]); 3673 else 3674 printf("Add-on servers not compiled in.\n"); 3675 return true; 3676 case "--websocket-servers": 3677 import core.demangle; 3678 version(with_addon_servers_connections) 3679 foreach(k, v; websocketServers) 3680 writeln(k, "\t", demangle(k)); 3681 return true; 3682 case "--session-server": 3683 version(with_addon_servers) 3684 runSessionServer(); 3685 else 3686 printf("Add-on servers not compiled in.\n"); 3687 return true; 3688 case "--event-server": 3689 version(with_addon_servers) 3690 runEventServer(); 3691 else 3692 printf("Add-on servers not compiled in.\n"); 3693 return true; 3694 case "--timer-server": 3695 version(with_addon_servers) 3696 runTimerServer(); 3697 else 3698 printf("Add-on servers not compiled in.\n"); 3699 return true; 3700 case "--timed-jobs": 3701 import core.demangle; 3702 version(with_addon_servers_connections) 3703 foreach(k, v; scheduledJobHandlers) 3704 writeln(k, "\t", demangle(k)); 3705 return true; 3706 case "--timed-job": 3707 scheduledJobHandlers[args[2]](args[3 .. $]); 3708 return true; 3709 default: 3710 // intentionally blank - do nothing and carry on to run normally 3711 } 3712 } 3713 return false; 3714 } 3715 3716 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3717 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3718 // we support command line thing for easy testing everywhere 3719 // it needs to be called ./app method uri [other args...] 3720 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3721 Cgi cgi = new CustomCgi(args); 3722 scope(exit) cgi.dispose(); 3723 fun(cgi); 3724 cgi.close(); 3725 return true; 3726 } 3727 return false; 3728 } 3729 3730 /++ 3731 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] to an example of how you can use it. 3732 3733 History: 3734 Added Sept 26, 2020 (release version 8.5). 3735 +/ 3736 struct RequestServer { 3737 /// 3738 string listeningHost = defaultListeningHost(); 3739 /// 3740 ushort listeningPort = defaultListeningPort(); 3741 3742 /++ 3743 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3744 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3745 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3746 [stop] may not work as well. 3747 3748 History: 3749 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3750 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3751 compatibility. 3752 +/ 3753 bool useFork = cgi_use_fork_default; 3754 3755 /++ 3756 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3757 default based on the number of cpus modified by the server mode. 3758 3759 History: 3760 Added August 12, 2022 (dub v10.9) 3761 +/ 3762 int numberOfThreads = 0; 3763 3764 /// 3765 this(string defaultHost, ushort defaultPort) { 3766 this.listeningHost = defaultHost; 3767 this.listeningPort = defaultPort; 3768 } 3769 3770 /// 3771 this(ushort defaultPort) { 3772 listeningPort = defaultPort; 3773 } 3774 3775 /++ 3776 Reads the command line arguments into the values here. 3777 3778 Possible arguments are `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3779 +/ 3780 void configureFromCommandLine(string[] args) { 3781 bool foundPort = false; 3782 bool foundHost = false; 3783 bool foundUid = false; 3784 bool foundGid = false; 3785 foreach(arg; args) { 3786 if(foundPort) { 3787 listeningPort = to!ushort(arg); 3788 foundPort = false; 3789 } 3790 if(foundHost) { 3791 listeningHost = arg; 3792 foundHost = false; 3793 } 3794 if(foundUid) { 3795 privilegesDropToUid = to!uid_t(arg); 3796 foundUid = false; 3797 } 3798 if(foundGid) { 3799 privilegesDropToGid = to!gid_t(arg); 3800 foundGid = false; 3801 } 3802 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3803 foundHost = true; 3804 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3805 foundPort = true; 3806 else if(arg == "--uid") 3807 foundUid = true; 3808 else if(arg == "--gid") 3809 foundGid = true; 3810 } 3811 } 3812 3813 version(Windows) { 3814 private alias uid_t = int; 3815 private alias gid_t = int; 3816 } 3817 3818 /// user (uid) to drop privileges to 3819 /// 0 … do nothing 3820 uid_t privilegesDropToUid = 0; 3821 /// group (gid) to drop privileges to 3822 /// 0 … do nothing 3823 gid_t privilegesDropToGid = 0; 3824 3825 private void dropPrivileges() { 3826 version(Posix) { 3827 import core.sys.posix.unistd; 3828 3829 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 3830 throw new Exception("Dropping privileges via setgid() failed."); 3831 3832 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 3833 throw new Exception("Dropping privileges via setuid() failed."); 3834 } 3835 else { 3836 // FIXME: Windows? 3837 //pragma(msg, "Dropping privileges is not implemented for this platform"); 3838 } 3839 3840 // done, set zero 3841 privilegesDropToGid = 0; 3842 privilegesDropToUid = 0; 3843 } 3844 3845 /++ 3846 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 3847 3848 History: 3849 Added Oct 10, 2020. 3850 Example: 3851 3852 --- 3853 import arsd.cgi; 3854 void main() { 3855 RequestServer server = RequestServer("127.0.0.1", 6789); 3856 string oauthCode; 3857 string oauthScope; 3858 server.serveHttpOnce!((cgi) { 3859 oauthCode = cgi.request("code"); 3860 oauthScope = cgi.request("scope"); 3861 cgi.write("Thank you, please return to the application."); 3862 }); 3863 // use the code and scope given 3864 } 3865 --- 3866 +/ 3867 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3868 import std.socket; 3869 3870 bool tcp; 3871 void delegate() cleanup; 3872 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 3873 auto connection = socket.accept(); 3874 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 3875 3876 if(cleanup) 3877 cleanup(); 3878 } 3879 3880 /++ 3881 Starts serving requests according to the current configuration. 3882 +/ 3883 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3884 version(netman_httpd) { 3885 // Obsolete! 3886 3887 import arsd.httpd; 3888 // what about forwarding the other constructor args? 3889 // this probably needs a whole redoing... 3890 serveHttp!CustomCgi(&fun, listeningPort);//5005); 3891 return; 3892 } else 3893 version(embedded_httpd_processes) { 3894 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 3895 } else 3896 version(embedded_httpd_threads) { 3897 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 3898 } else 3899 version(scgi) { 3900 serveScgi!(fun, CustomCgi, maxContentLength)(); 3901 } else 3902 version(fastcgi) { 3903 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 3904 } else 3905 version(stdio_http) { 3906 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 3907 } else { 3908 //version=plain_cgi; 3909 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 3910 } 3911 } 3912 3913 /++ 3914 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 3915 3916 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 3917 +/ 3918 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 3919 globalStopFlag = false; 3920 static if(__traits(isStaticFunction, fun)) 3921 alias funToUse = fun; 3922 else 3923 void funToUse(CustomCgi cgi) { 3924 static if(__VERSION__ > 2097) 3925 __traits(child, _this, fun)(cgi); 3926 else static assert(0, "Not implemented in your compiler version!"); 3927 } 3928 auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 3929 manager.listen(); 3930 } 3931 3932 /++ 3933 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 3934 +/ 3935 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3936 globalStopFlag = false; 3937 auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 3938 manager.listen(); 3939 } 3940 3941 /++ 3942 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 3943 3944 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 3945 3946 History: 3947 Added May 29, 2021 3948 +/ 3949 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3950 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 3951 } 3952 3953 /++ 3954 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 3955 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 3956 +/ 3957 enum ForceStop { 3958 /++ 3959 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 3960 +/ 3961 afterQueuedRequestsComplete, 3962 /++ 3963 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 3964 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 3965 +/ 3966 afterCurrentRequestsComplete, 3967 /++ 3968 Partial response writes will throw an exception, cancelling any streaming response, but complete 3969 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 3970 +/ 3971 cancelStreamingRequestsEarly, 3972 /++ 3973 All writes will throw. 3974 +/ 3975 cancelAllRequestsEarly, 3976 /++ 3977 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 3978 +/ 3979 forciblyTerminate, 3980 } 3981 3982 version(embedded_httpd_processes) {} else 3983 /++ 3984 Stops serving after the current requests are completed. 3985 3986 Bugs: 3987 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 3988 on Windows however). Only partially implemented on non-Linux posix systems. 3989 3990 You might also try SIGINT perhaps. 3991 3992 The stopPriority is not yet fully implemented. 3993 +/ 3994 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 3995 globalStopFlag = true; 3996 3997 version(Posix) { 3998 if(cancelfd > 0) { 3999 ulong a = 1; 4000 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4001 } 4002 } 4003 version(Windows) { 4004 if(iocp) { 4005 foreach(i; 0 .. 16) // FIXME 4006 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4007 } 4008 } 4009 } 4010 } 4011 4012 private alias AliasSeq(T...) = T; 4013 4014 version(with_breaking_cgi_features) 4015 mixin(q{ 4016 template ThisFor(alias t) { 4017 static if(__traits(isStaticFunction, t)) { 4018 alias ThisFor = AliasSeq!(); 4019 } else { 4020 alias ThisFor = __traits(parent, t); 4021 } 4022 } 4023 }); 4024 else 4025 alias ThisFor(alias t) = AliasSeq!(); 4026 4027 private __gshared bool globalStopFlag = false; 4028 4029 version(embedded_httpd_processes) 4030 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4031 import core.sys.posix.unistd; 4032 import core.sys.posix.sys.socket; 4033 import core.sys.posix.netinet.in_; 4034 //import std.c.linux.socket; 4035 4036 int sock = socket(AF_INET, SOCK_STREAM, 0); 4037 if(sock == -1) 4038 throw new Exception("socket"); 4039 4040 cloexec(sock); 4041 4042 { 4043 4044 sockaddr_in addr; 4045 addr.sin_family = AF_INET; 4046 addr.sin_port = htons(params.listeningPort); 4047 auto lh = params.listeningHost; 4048 if(lh.length) { 4049 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4050 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4051 } else 4052 addr.sin_addr.s_addr = INADDR_ANY; 4053 4054 // HACKISH 4055 int on = 1; 4056 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4057 // end hack 4058 4059 4060 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4061 close(sock); 4062 throw new Exception("bind"); 4063 } 4064 4065 // FIXME: if this queue is full, it will just ignore it 4066 // and wait for the client to retransmit it. This is an 4067 // obnoxious timeout condition there. 4068 if(sock.listen(128) == -1) { 4069 close(sock); 4070 throw new Exception("listen"); 4071 } 4072 params.dropPrivileges(); 4073 } 4074 4075 version(embedded_httpd_processes_accept_after_fork) {} else { 4076 int pipeReadFd; 4077 int pipeWriteFd; 4078 4079 { 4080 int[2] pipeFd; 4081 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4082 import core.stdc.errno; 4083 throw new Exception("pipe failed " ~ to!string(errno)); 4084 } 4085 4086 pipeReadFd = pipeFd[0]; 4087 pipeWriteFd = pipeFd[1]; 4088 } 4089 } 4090 4091 4092 int processCount; 4093 pid_t newPid; 4094 reopen: 4095 while(processCount < processPoolSize) { 4096 newPid = fork(); 4097 if(newPid == 0) { 4098 // start serving on the socket 4099 //ubyte[4096] backingBuffer; 4100 for(;;) { 4101 bool closeConnection; 4102 uint i; 4103 sockaddr addr; 4104 i = addr.sizeof; 4105 version(embedded_httpd_processes_accept_after_fork) { 4106 int s = accept(sock, &addr, &i); 4107 int opt = 1; 4108 import core.sys.posix.netinet.tcp; 4109 // the Cgi class does internal buffering, so disabling this 4110 // helps with latency in many cases... 4111 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4112 cloexec(s); 4113 } else { 4114 int s; 4115 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4116 if(readret != s.sizeof) { 4117 import core.stdc.errno; 4118 throw new Exception("pipe read failed " ~ to!string(errno)); 4119 } 4120 4121 //writeln("process ", getpid(), " got socket ", s); 4122 } 4123 4124 try { 4125 4126 if(s == -1) 4127 throw new Exception("accept"); 4128 4129 scope(failure) close(s); 4130 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4131 auto ir = new BufferedInputRange(s); 4132 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4133 4134 while(!ir.empty) { 4135 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4136 4137 Cgi cgi; 4138 try { 4139 cgi = new CustomCgi(ir, &closeConnection); 4140 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4141 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4142 if(processPoolSize <= 1) 4143 closeConnection = true; 4144 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4145 } catch(Throwable t) { 4146 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4147 // anyway let's kill the connection 4148 version(CRuntime_Musl) { 4149 // LockingTextWriter fails here 4150 // so working around it 4151 auto estr = t.toString(); 4152 stderr.rawWrite(estr); 4153 stderr.rawWrite("\n"); 4154 } else 4155 stderr.writeln(t.toString()); 4156 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4157 closeConnection = true; 4158 break; 4159 } 4160 assert(cgi !is null); 4161 scope(exit) 4162 cgi.dispose(); 4163 4164 try { 4165 fun(cgi); 4166 cgi.close(); 4167 if(cgi.websocketMode) 4168 closeConnection = true; 4169 } catch(ConnectionException ce) { 4170 closeConnection = true; 4171 } catch(Throwable t) { 4172 // a processing error can be recovered from 4173 version(CRuntime_Musl) { 4174 // LockingTextWriter fails here 4175 // so working around it 4176 auto estr = t.toString(); 4177 stderr.rawWrite(estr); 4178 } else { 4179 stderr.writeln(t.toString); 4180 } 4181 if(!handleException(cgi, t)) 4182 closeConnection = true; 4183 } 4184 4185 if(closeConnection) { 4186 ir.source.close(); 4187 break; 4188 } else { 4189 if(!ir.empty) 4190 ir.popFront(); // get the next 4191 else if(ir.sourceClosed) { 4192 ir.source.close(); 4193 } 4194 } 4195 } 4196 4197 ir.source.close(); 4198 } catch(Throwable t) { 4199 version(CRuntime_Musl) {} else 4200 debug writeln(t); 4201 // most likely cause is a timeout 4202 } 4203 } 4204 } else if(newPid < 0) { 4205 throw new Exception("fork failed"); 4206 } else { 4207 processCount++; 4208 } 4209 } 4210 4211 // the parent should wait for its children... 4212 if(newPid) { 4213 import core.sys.posix.sys.wait; 4214 4215 version(embedded_httpd_processes_accept_after_fork) {} else { 4216 import core.sys.posix.sys.select; 4217 int[] fdQueue; 4218 while(true) { 4219 // writeln("select call"); 4220 int nfds = pipeWriteFd; 4221 if(sock > pipeWriteFd) 4222 nfds = sock; 4223 nfds += 1; 4224 fd_set read_fds; 4225 fd_set write_fds; 4226 FD_ZERO(&read_fds); 4227 FD_ZERO(&write_fds); 4228 FD_SET(sock, &read_fds); 4229 if(fdQueue.length) 4230 FD_SET(pipeWriteFd, &write_fds); 4231 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4232 if(ret == -1) { 4233 import core.stdc.errno; 4234 if(errno == EINTR) 4235 goto try_wait; 4236 else 4237 throw new Exception("wtf select"); 4238 } 4239 4240 int s = -1; 4241 if(FD_ISSET(sock, &read_fds)) { 4242 uint i; 4243 sockaddr addr; 4244 i = addr.sizeof; 4245 s = accept(sock, &addr, &i); 4246 cloexec(s); 4247 import core.sys.posix.netinet.tcp; 4248 int opt = 1; 4249 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4250 } 4251 4252 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4253 if(s == -1 && fdQueue.length) { 4254 s = fdQueue[0]; 4255 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4256 } 4257 write_fd(pipeWriteFd, &s, s.sizeof, s); 4258 close(s); // we are done with it, let the other process take ownership 4259 } else 4260 fdQueue ~= s; 4261 } 4262 } 4263 4264 try_wait: 4265 4266 int status; 4267 while(-1 != wait(&status)) { 4268 version(CRuntime_Musl) {} else { 4269 import std.stdio; writeln("Process died ", status); 4270 } 4271 processCount--; 4272 goto reopen; 4273 } 4274 close(sock); 4275 } 4276 } 4277 4278 version(fastcgi) 4279 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4280 // SetHandler fcgid-script 4281 FCGX_Stream* input, output, error; 4282 FCGX_ParamArray env; 4283 4284 4285 4286 const(ubyte)[] getFcgiChunk() { 4287 const(ubyte)[] ret; 4288 while(FCGX_HasSeenEOF(input) != -1) 4289 ret ~= cast(ubyte) FCGX_GetChar(input); 4290 return ret; 4291 } 4292 4293 void writeFcgi(const(ubyte)[] data) { 4294 FCGX_PutStr(data.ptr, data.length, output); 4295 } 4296 4297 void doARequest() { 4298 string[string] fcgienv; 4299 4300 for(auto e = env; e !is null && *e !is null; e++) { 4301 string cur = to!string(*e); 4302 auto idx = cur.indexOf("="); 4303 string name, value; 4304 if(idx == -1) 4305 name = cur; 4306 else { 4307 name = cur[0 .. idx]; 4308 value = cur[idx + 1 .. $]; 4309 } 4310 4311 fcgienv[name] = value; 4312 } 4313 4314 void flushFcgi() { 4315 FCGX_FFlush(output); 4316 } 4317 4318 Cgi cgi; 4319 try { 4320 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4321 } catch(Throwable t) { 4322 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4323 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4324 return; //continue; 4325 } 4326 assert(cgi !is null); 4327 scope(exit) cgi.dispose(); 4328 try { 4329 fun(cgi); 4330 cgi.close(); 4331 } catch(Throwable t) { 4332 // log it to the error stream 4333 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4334 // handle it for the user, if we can 4335 if(!handleException(cgi, t)) 4336 return; // continue; 4337 } 4338 } 4339 4340 auto lp = params.listeningPort; 4341 auto host = params.listeningHost; 4342 4343 FCGX_Request request; 4344 if(lp || !host.empty) { 4345 // if a listening port was specified on the command line, we want to spawn ourself 4346 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4347 FCGX_Init(); 4348 4349 int sock; 4350 4351 if(host.startsWith("unix:")) { 4352 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4353 } else if(host.startsWith("abstract:")) { 4354 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4355 } else { 4356 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4357 } 4358 4359 if(sock < 0) 4360 throw new Exception("Couldn't listen on the port"); 4361 FCGX_InitRequest(&request, sock, 0); 4362 while(FCGX_Accept_r(&request) >= 0) { 4363 input = request.inStream; 4364 output = request.outStream; 4365 error = request.errStream; 4366 env = request.envp; 4367 doARequest(); 4368 } 4369 } else { 4370 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4371 // using the version with a global variable since we are separate processes anyway 4372 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4373 doARequest(); 4374 } 4375 } 4376 } 4377 4378 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4379 ushort defaultListeningPort() { 4380 version(netman_httpd) 4381 return 8080; 4382 else version(embedded_httpd_processes) 4383 return 8085; 4384 else version(embedded_httpd_threads) 4385 return 8085; 4386 else version(scgi) 4387 return 4000; 4388 else 4389 return 0; 4390 } 4391 4392 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4393 string defaultListeningHost() { 4394 version(netman_httpd) 4395 return null; 4396 else version(embedded_httpd_processes) 4397 return null; 4398 else version(embedded_httpd_threads) 4399 return null; 4400 else version(scgi) 4401 return "127.0.0.1"; 4402 else 4403 return null; 4404 4405 } 4406 4407 /++ 4408 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4409 4410 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4411 4412 Params: 4413 fun = Your request handler 4414 CustomCgi = a subclass of Cgi, if you wise to customize it further 4415 maxContentLength = max POST size you want to allow 4416 args = command-line arguments 4417 4418 History: 4419 Documented Sept 26, 2020. 4420 +/ 4421 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4422 if(tryAddonServers(args)) 4423 return; 4424 4425 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4426 return; 4427 4428 RequestServer server; 4429 // you can change the port here if you like 4430 // server.listeningPort = 9000; 4431 4432 // then call this to let the command line args override your default 4433 server.configureFromCommandLine(args); 4434 4435 // and serve the request(s). 4436 server.serve!(fun, CustomCgi, maxContentLength)(); 4437 } 4438 4439 //version(plain_cgi) 4440 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4441 // standard CGI is the default version 4442 4443 4444 // Set stdin to binary mode if necessary to avoid mangled newlines 4445 // the fact that stdin is global means this could be trouble but standard cgi request 4446 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4447 version(Windows) { 4448 version(Win64) 4449 _setmode(std.stdio.stdin.fileno(), 0x8000); 4450 else 4451 setmode(std.stdio.stdin.fileno(), 0x8000); 4452 } 4453 4454 Cgi cgi; 4455 try { 4456 cgi = new CustomCgi(maxContentLength); 4457 version(Posix) 4458 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4459 else version(Windows) 4460 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4461 else static assert(0); 4462 } catch(Throwable t) { 4463 version(CRuntime_Musl) { 4464 // LockingTextWriter fails here 4465 // so working around it 4466 auto s = t.toString(); 4467 stderr.rawWrite(s); 4468 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4469 } else { 4470 stderr.writeln(t.msg); 4471 // the real http server will probably handle this; 4472 // most likely, this is a bug in Cgi. But, oh well. 4473 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4474 } 4475 return; 4476 } 4477 assert(cgi !is null); 4478 scope(exit) cgi.dispose(); 4479 4480 try { 4481 fun(cgi); 4482 cgi.close(); 4483 } catch (Throwable t) { 4484 version(CRuntime_Musl) { 4485 // LockingTextWriter fails here 4486 // so working around it 4487 auto s = t.msg; 4488 stderr.rawWrite(s); 4489 } else { 4490 stderr.writeln(t.msg); 4491 } 4492 if(!handleException(cgi, t)) 4493 return; 4494 } 4495 } 4496 4497 private __gshared int cancelfd = -1; 4498 4499 /+ 4500 The event loop for embedded_httpd_threads will prolly fiber dispatch 4501 cgi constructors too, so slow posts will not monopolize a worker thread. 4502 4503 May want to provide the worker task system just need to ensure all the fibers 4504 has a big enough stack for real work... would also ideally like to reuse them. 4505 4506 4507 So prolly bir would switch it to nonblocking. If it would block, it epoll 4508 registers one shot with this existing fiber to take it over. 4509 4510 new connection comes in. it picks a fiber off the free list, 4511 or if there is none, it creates a new one. this fiber handles 4512 this connection the whole time. 4513 4514 epoll triggers the fiber when something comes in. it is called by 4515 a random worker thread, it might change at any time. at least during 4516 the constructor. maybe into the main body it will stay tied to a thread 4517 just so TLS stuff doesn't randomly change in the middle. but I could 4518 specify if you yield all bets are off. 4519 4520 when the request is finished, if there's more data buffered, it just 4521 keeps going. if there is no more data buffered, it epoll ctls to 4522 get triggered when more data comes in. all one shot. 4523 4524 when a connection is closed, the fiber returns and is then reset 4525 and added to the free list. if the free list is full, the fiber is 4526 just freed, this means it will balloon to a certain size but not generally 4527 grow beyond that unless the activity keeps going. 4528 4529 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4530 4531 So the fiber has its own magic methods to read and write. if they would block, it registers 4532 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4533 4534 basically you issue the command and it tells you when it is done 4535 4536 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4537 4538 +/ 4539 4540 /++ 4541 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4542 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4543 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4544 4545 History: 4546 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4547 +/ 4548 version(cgi_use_fiber) 4549 __gshared size_t fiberStackSize = 4096 * 100; 4550 4551 version(cgi_use_fiber) 4552 class CgiFiber : Fiber { 4553 private void function(Socket) f_handler; 4554 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4555 f_handler(s); 4556 } 4557 this(void function(Socket) handler) { 4558 this.f_handler = handler; 4559 this(&f_handler_dg); 4560 } 4561 4562 this(void delegate(Socket) handler) { 4563 this.handler = handler; 4564 super(&run, fiberStackSize); 4565 } 4566 4567 Socket connection; 4568 void delegate(Socket) handler; 4569 4570 void run() { 4571 handler(connection); 4572 } 4573 4574 void delegate() postYield; 4575 4576 private void setPostYield(scope void delegate() py) @nogc { 4577 postYield = cast(void delegate()) py; 4578 } 4579 4580 void proceed() { 4581 try { 4582 call(); 4583 auto py = postYield; 4584 postYield = null; 4585 if(py !is null) 4586 py(); 4587 } catch(Exception e) { 4588 if(connection) 4589 connection.close(); 4590 goto terminate; 4591 } 4592 4593 if(state == State.TERM) { 4594 terminate: 4595 import core.memory; 4596 GC.removeRoot(cast(void*) this); 4597 } 4598 } 4599 } 4600 4601 version(cgi_use_fiber) 4602 version(Windows) { 4603 4604 extern(Windows) private { 4605 4606 import core.sys.windows.mswsock; 4607 4608 alias GROUP=uint; 4609 alias LPWSAPROTOCOL_INFOW = void*; 4610 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4611 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4612 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4613 4614 struct WSABUF { 4615 ULONG len; 4616 CHAR *buf; 4617 } 4618 alias LPWSABUF = WSABUF*; 4619 4620 alias WSAOVERLAPPED = OVERLAPPED; 4621 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4622 /+ 4623 4624 alias LPFN_ACCEPTEX = 4625 BOOL 4626 function( 4627 SOCKET sListenSocket, 4628 SOCKET sAcceptSocket, 4629 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4630 void* lpOutputBuffer, 4631 WORD dwReceiveDataLength, 4632 WORD dwLocalAddressLength, 4633 WORD dwRemoteAddressLength, 4634 LPDWORD lpdwBytesReceived, 4635 LPOVERLAPPED lpOverlapped 4636 ); 4637 4638 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4639 +/ 4640 4641 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4642 } 4643 4644 private class PseudoblockingOverlappedSocket : Socket { 4645 SOCKET handle; 4646 4647 CgiFiber fiber; 4648 4649 this(AddressFamily af, SocketType st) { 4650 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4651 if(!handle) 4652 throw new Exception("WSASocketW"); 4653 this.handle = handle; 4654 4655 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4656 4657 if(iocp is null) { 4658 writeln(GetLastError()); 4659 throw new Exception("CreateIoCompletionPort"); 4660 } 4661 4662 super(cast(socket_t) handle, af); 4663 } 4664 this() pure nothrow @trusted { assert(0); } 4665 4666 override void blocking(bool) {} // meaningless to us, just ignore it. 4667 4668 protected override Socket accepting() pure nothrow { 4669 assert(0); 4670 } 4671 4672 bool addressesParsed; 4673 Address la; 4674 Address ra; 4675 4676 private void populateAddresses() { 4677 if(addressesParsed) 4678 return; 4679 addressesParsed = true; 4680 4681 int lalen, ralen; 4682 4683 sockaddr_in* la; 4684 sockaddr_in* ra; 4685 4686 lpfnGetAcceptExSockaddrs( 4687 scratchBuffer.ptr, 4688 0, // same as in the AcceptEx call! 4689 sockaddr_in.sizeof + 16, 4690 sockaddr_in.sizeof + 16, 4691 cast(sockaddr**) &la, 4692 &lalen, 4693 cast(sockaddr**) &ra, 4694 &ralen 4695 ); 4696 4697 if(la) 4698 this.la = new InternetAddress(*la); 4699 if(ra) 4700 this.ra = new InternetAddress(*ra); 4701 4702 } 4703 4704 override @property @trusted Address localAddress() { 4705 populateAddresses(); 4706 return la; 4707 } 4708 override @property @trusted Address remoteAddress() { 4709 populateAddresses(); 4710 return ra; 4711 } 4712 4713 PseudoblockingOverlappedSocket accepted; 4714 4715 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4716 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4717 4718 override Socket accept() @trusted { 4719 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4720 4721 if(lpfnAcceptEx is null) { 4722 DWORD dwBytes; 4723 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4724 4725 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4726 &GuidAcceptEx, GuidAcceptEx.sizeof, 4727 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4728 &dwBytes, null, null); 4729 4730 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4731 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4732 &GuidAcceptEx, GuidAcceptEx.sizeof, 4733 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4734 &dwBytes, null, null); 4735 4736 } 4737 4738 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4739 accepted = pfa; 4740 4741 SOCKET pendingForAccept = pfa.handle; 4742 DWORD ignored; 4743 4744 auto ret = lpfnAcceptEx(handle, 4745 pendingForAccept, 4746 // buffer to receive up front 4747 pfa.scratchBuffer.ptr, 4748 0, 4749 // size of local and remote addresses. normally + 16. 4750 sockaddr_in.sizeof + 16, 4751 sockaddr_in.sizeof + 16, 4752 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4753 &overlapped 4754 ); 4755 4756 return pfa; 4757 } 4758 4759 override void connect(Address to) { assert(0); } 4760 4761 DWORD lastAnswer; 4762 ubyte[1024] scratchBuffer; 4763 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4764 4765 WSABUF[1] buffer; 4766 OVERLAPPED overlapped; 4767 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4768 overlapped = overlapped.init; 4769 buffer[0].len = cast(DWORD) buf.length; 4770 buffer[0].buf = cast(CHAR*) buf.ptr; 4771 fiber.setPostYield( () { 4772 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4773 if(GetLastError() != 997) { 4774 //throw new Exception("WSASend fail"); 4775 } 4776 } 4777 }); 4778 4779 Fiber.yield(); 4780 return lastAnswer; 4781 } 4782 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4783 overlapped = overlapped.init; 4784 buffer[0].len = cast(DWORD) buf.length; 4785 buffer[0].buf = cast(CHAR*) buf.ptr; 4786 4787 DWORD flags2 = 0; 4788 4789 fiber.setPostYield(() { 4790 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4791 if(GetLastError() != 997) { 4792 //writeln("WSARecv ", WSAGetLastError()); 4793 //throw new Exception("WSARecv fail"); 4794 } 4795 } 4796 }); 4797 4798 Fiber.yield(); 4799 return lastAnswer; 4800 } 4801 4802 // I might go back and implement these for udp things. 4803 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4804 assert(0); 4805 } 4806 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4807 assert(0); 4808 } 4809 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4810 assert(0); 4811 } 4812 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4813 assert(0); 4814 } 4815 4816 // lol overload sets 4817 alias send = typeof(super).send; 4818 alias receive = typeof(super).receive; 4819 alias sendTo = typeof(super).sendTo; 4820 alias receiveFrom = typeof(super).receiveFrom; 4821 4822 } 4823 } 4824 4825 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 4826 assert(connection !is null); 4827 version(cgi_use_fiber) { 4828 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 4829 4830 version(Windows) { 4831 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 4832 } 4833 4834 import core.memory; 4835 GC.addRoot(cast(void*) fiber); 4836 fiber.connection = connection; 4837 fiber.proceed(); 4838 } else { 4839 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 4840 } 4841 } 4842 4843 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 4844 scope(failure) { 4845 // catch all for other errors 4846 try { 4847 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 4848 connection.close(); 4849 } catch(Exception e) {} // swallow it, we're aborting anyway. 4850 } 4851 4852 bool closeConnection = alwaysCloseConnection; 4853 4854 /+ 4855 ubyte[4096] inputBuffer = void; 4856 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 4857 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 4858 4859 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 4860 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 4861 ir.__ctor(connection, inputBuffer[], true); 4862 +/ 4863 4864 auto ir = new BufferedInputRange(connection); 4865 4866 while(!ir.empty) { 4867 4868 if(ir.view.length == 0) { 4869 ir.popFront(); 4870 if(ir.sourceClosed) { 4871 connection.close(); 4872 closeConnection = true; 4873 break; 4874 } 4875 } 4876 4877 Cgi cgi; 4878 try { 4879 cgi = new CustomCgi(ir, &closeConnection); 4880 // There's a bunch of these casts around because the type matches up with 4881 // the -version=.... specifiers, just you can also create a RequestServer 4882 // and instantiate the things where the types don't match up. It isn't exactly 4883 // correct but I also don't care rn. Might FIXME and either remove it later or something. 4884 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 4885 } catch(ConnectionClosedException ce) { 4886 closeConnection = true; 4887 break; 4888 } catch(ConnectionException ce) { 4889 // broken pipe or something, just abort the connection 4890 closeConnection = true; 4891 break; 4892 } catch(Throwable t) { 4893 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4894 // anyway let's kill the connection 4895 version(CRuntime_Musl) { 4896 stderr.rawWrite(t.toString()); 4897 stderr.rawWrite("\n"); 4898 } else { 4899 stderr.writeln(t.toString()); 4900 } 4901 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 4902 closeConnection = true; 4903 break; 4904 } 4905 assert(cgi !is null); 4906 scope(exit) 4907 cgi.dispose(); 4908 4909 try { 4910 fun(cgi); 4911 cgi.close(); 4912 if(cgi.websocketMode) 4913 closeConnection = true; 4914 } catch(ConnectionException ce) { 4915 // broken pipe or something, just abort the connection 4916 closeConnection = true; 4917 } catch(ConnectionClosedException ce) { 4918 // broken pipe or something, just abort the connection 4919 closeConnection = true; 4920 } catch(Throwable t) { 4921 // a processing error can be recovered from 4922 version(CRuntime_Musl) {} else 4923 stderr.writeln(t.toString); 4924 if(!handleException(cgi, t)) 4925 closeConnection = true; 4926 } 4927 4928 if(globalStopFlag) 4929 closeConnection = true; 4930 4931 if(closeConnection || alwaysCloseConnection) { 4932 connection.shutdown(SocketShutdown.BOTH); 4933 connection.close(); 4934 ir.dispose(); 4935 closeConnection = false; // don't reclose after loop 4936 break; 4937 } else { 4938 if(ir.front.length) { 4939 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 4940 } else if(ir.sourceClosed) { 4941 ir.source.shutdown(SocketShutdown.BOTH); 4942 ir.source.close(); 4943 ir.dispose(); 4944 closeConnection = false; 4945 } else { 4946 continue; 4947 // break; // this was for a keepalive experiment 4948 } 4949 } 4950 } 4951 4952 if(closeConnection) { 4953 connection.shutdown(SocketShutdown.BOTH); 4954 connection.close(); 4955 ir.dispose(); 4956 } 4957 4958 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 4959 } 4960 4961 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 4962 // and now we can buffer 4963 scope(failure) 4964 connection.close(); 4965 4966 import al = std.algorithm; 4967 4968 size_t size; 4969 4970 string[string] headers; 4971 4972 auto range = new BufferedInputRange(connection); 4973 more_data: 4974 auto chunk = range.front(); 4975 // waiting for colon for header length 4976 auto idx = indexOf(cast(string) chunk, ':'); 4977 if(idx == -1) { 4978 try { 4979 range.popFront(); 4980 } catch(Exception e) { 4981 // it is just closed, no big deal 4982 connection.close(); 4983 return; 4984 } 4985 goto more_data; 4986 } 4987 4988 size = to!size_t(cast(string) chunk[0 .. idx]); 4989 chunk = range.consume(idx + 1); 4990 // reading headers 4991 if(chunk.length < size) 4992 range.popFront(0, size + 1); 4993 // we are now guaranteed to have enough 4994 chunk = range.front(); 4995 assert(chunk.length > size); 4996 4997 idx = 0; 4998 string key; 4999 string value; 5000 foreach(part; al.splitter(chunk, '\0')) { 5001 if(idx & 1) { // odd is value 5002 value = cast(string)(part.idup); 5003 headers[key] = value; // commit 5004 } else 5005 key = cast(string)(part.idup); 5006 idx++; 5007 } 5008 5009 enforce(chunk[size] == ','); // the terminator 5010 5011 range.consume(size + 1); 5012 // reading data 5013 // this will be done by Cgi 5014 5015 const(ubyte)[] getScgiChunk() { 5016 // we are already primed 5017 auto data = range.front(); 5018 if(data.length == 0 && !range.sourceClosed) { 5019 range.popFront(0); 5020 data = range.front(); 5021 } else if (range.sourceClosed) 5022 range.source.close(); 5023 5024 return data; 5025 } 5026 5027 void writeScgi(const(ubyte)[] data) { 5028 sendAll(connection, data); 5029 } 5030 5031 void flushScgi() { 5032 // I don't *think* I have to do anything.... 5033 } 5034 5035 Cgi cgi; 5036 try { 5037 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5038 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5039 } catch(Throwable t) { 5040 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5041 connection.close(); 5042 return; // this connection is dead 5043 } 5044 assert(cgi !is null); 5045 scope(exit) cgi.dispose(); 5046 try { 5047 fun(cgi); 5048 cgi.close(); 5049 connection.close(); 5050 } catch(Throwable t) { 5051 // no std err 5052 if(!handleException(cgi, t)) { 5053 connection.close(); 5054 return; 5055 } else { 5056 connection.close(); 5057 return; 5058 } 5059 } 5060 } 5061 5062 string printDate(DateTime date) { 5063 char[29] buffer = void; 5064 printDateToBuffer(date, buffer[]); 5065 return buffer.idup; 5066 } 5067 5068 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5069 assert(buffer.length >= 29); 5070 // 29 static length ? 5071 5072 static immutable daysOfWeek = [ 5073 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5074 ]; 5075 5076 static immutable months = [ 5077 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5078 ]; 5079 5080 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5081 buffer[3 .. 5] = ", "; 5082 buffer[5] = date.day / 10 + '0'; 5083 buffer[6] = date.day % 10 + '0'; 5084 buffer[7] = ' '; 5085 buffer[8 .. 11] = months[date.month]; 5086 buffer[11] = ' '; 5087 auto y = date.year; 5088 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5089 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5090 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5091 buffer[15] = cast(char) (y + '0'); 5092 buffer[16] = ' '; 5093 buffer[17] = date.hour / 10 + '0'; 5094 buffer[18] = date.hour % 10 + '0'; 5095 buffer[19] = ':'; 5096 buffer[20] = date.minute / 10 + '0'; 5097 buffer[21] = date.minute % 10 + '0'; 5098 buffer[22] = ':'; 5099 buffer[23] = date.second / 10 + '0'; 5100 buffer[24] = date.second % 10 + '0'; 5101 buffer[25 .. $] = " GMT"; 5102 5103 return 29; 5104 } 5105 5106 5107 // Referencing this gigantic typeid seems to remind the compiler 5108 // to actually put the symbol in the object file. I guess the immutable 5109 // assoc array array isn't actually included in druntime 5110 void hackAroundLinkerError() { 5111 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5112 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5113 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5114 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5115 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5116 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5117 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5118 // this is getting kinda ridiculous btw. Moving assoc arrays 5119 // to the library is the pain that keeps on coming. 5120 5121 // eh this broke the build on the work server 5122 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5123 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5124 } 5125 5126 5127 5128 5129 5130 version(fastcgi) { 5131 pragma(lib, "fcgi"); 5132 5133 static if(size_t.sizeof == 8) // 64 bit 5134 alias long c_int; 5135 else 5136 alias int c_int; 5137 5138 extern(C) { 5139 struct FCGX_Stream { 5140 ubyte* rdNext; 5141 ubyte* wrNext; 5142 ubyte* stop; 5143 ubyte* stopUnget; 5144 c_int isReader; 5145 c_int isClosed; 5146 c_int wasFCloseCalled; 5147 c_int FCGI_errno; 5148 void* function(FCGX_Stream* stream) fillBuffProc; 5149 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5150 void* data; 5151 } 5152 5153 // note: this is meant to be opaque, so don't access it directly 5154 struct FCGX_Request { 5155 int requestId; 5156 int role; 5157 FCGX_Stream* inStream; 5158 FCGX_Stream* outStream; 5159 FCGX_Stream* errStream; 5160 char** envp; 5161 void* paramsPtr; 5162 int ipcFd; 5163 int isBeginProcessed; 5164 int keepConnection; 5165 int appStatus; 5166 int nWriters; 5167 int flags; 5168 int listen_sock; 5169 } 5170 5171 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5172 void FCGX_Init(); 5173 5174 int FCGX_Accept_r(FCGX_Request *request); 5175 5176 5177 alias char** FCGX_ParamArray; 5178 5179 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5180 c_int FCGX_GetChar(FCGX_Stream* stream); 5181 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5182 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5183 c_int FCGX_FFlush(FCGX_Stream *stream); 5184 5185 int FCGX_OpenSocket(in char*, int); 5186 } 5187 } 5188 5189 5190 /* This might go int a separate module eventually. It is a network input helper class. */ 5191 5192 import std.socket; 5193 5194 version(cgi_use_fiber) { 5195 import core.thread; 5196 5197 version(linux) { 5198 import core.sys.linux.epoll; 5199 5200 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5201 } else version(Windows) { 5202 // declaring the iocp thing below... 5203 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5204 } 5205 5206 version(Windows) 5207 __gshared HANDLE iocp; 5208 5209 version(cgi_use_fiber) { 5210 version(linux) 5211 private enum WakeupEvent { 5212 Read = EPOLLIN, 5213 Write = EPOLLOUT 5214 } 5215 else version(Windows) 5216 private enum WakeupEvent { 5217 Read, Write 5218 } 5219 else static assert(0); 5220 } 5221 5222 version(cgi_use_fiber) 5223 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5224 5225 // static cast since I know what i have in here and don't want to pay for dynamic cast 5226 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5227 5228 version(linux) { 5229 f.setPostYield = () { 5230 if(*registered) { 5231 // rearm 5232 epoll_event evt; 5233 evt.events = e | EPOLLONESHOT; 5234 evt.data.ptr = cast(void*) f; 5235 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5236 throw new Exception("epoll_ctl"); 5237 } else { 5238 // initial registration 5239 *registered = true ; 5240 int fd = source.handle; 5241 epoll_event evt; 5242 evt.events = e | EPOLLONESHOT; 5243 evt.data.ptr = cast(void*) f; 5244 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5245 throw new Exception("epoll_ctl"); 5246 } 5247 }; 5248 5249 Fiber.yield(); 5250 5251 f.setPostYield(null); 5252 } else version(Windows) { 5253 Fiber.yield(); 5254 } 5255 else static assert(0); 5256 } 5257 5258 version(cgi_use_fiber) 5259 void unregisterSource(Socket s) { 5260 version(linux) { 5261 epoll_event evt; 5262 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5263 } else version(Windows) { 5264 // intentionally blank 5265 } 5266 else static assert(0); 5267 } 5268 5269 // it is a class primarily for reference semantics 5270 // I might change this interface 5271 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5272 class BufferedInputRange { 5273 version(Posix) 5274 this(int source, ubyte[] buffer = null) { 5275 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5276 } 5277 5278 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5279 // if they connect but never send stuff to us, we don't want it wasting the process 5280 // so setting a time out 5281 version(cgi_use_fiber) 5282 source.blocking = false; 5283 else 5284 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5285 5286 this.source = source; 5287 if(buffer is null) { 5288 underlyingBuffer = new ubyte[4096]; 5289 this.allowGrowth = true; 5290 } else { 5291 underlyingBuffer = buffer; 5292 this.allowGrowth = allowGrowth; 5293 } 5294 5295 assert(underlyingBuffer.length); 5296 5297 // we assume view.ptr is always inside underlyingBuffer 5298 view = underlyingBuffer[0 .. 0]; 5299 5300 popFront(); // prime 5301 } 5302 5303 version(cgi_use_fiber) { 5304 bool registered; 5305 } 5306 5307 void dispose() { 5308 version(cgi_use_fiber) { 5309 if(registered) 5310 unregisterSource(source); 5311 } 5312 } 5313 5314 /** 5315 A slight difference from regular ranges is you can give it the maximum 5316 number of bytes to consume. 5317 5318 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5319 consume() yourself and use a regular foreach, it will infinitely loop! 5320 5321 The default is to do what a normal range does, and consume the whole buffer 5322 and wait for additional input. 5323 5324 You can also specify 0, to append to the buffer, or any other number 5325 to remove the front n bytes and wait for more. 5326 */ 5327 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5328 if(sourceClosed) 5329 throw new ConnectionClosedException("can't get any more data from a closed source"); 5330 if(!skipConsume) 5331 consume(maxBytesToConsume); 5332 5333 // we might have to grow the buffer 5334 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5335 if(allowGrowth) { 5336 //import std.stdio; writeln("growth"); 5337 auto viewStart = view.ptr - underlyingBuffer.ptr; 5338 size_t growth = 4096; 5339 // make sure we have enough for what we're being asked for 5340 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5341 growth = minBytesToSettleFor - underlyingBuffer.length; 5342 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5343 underlyingBuffer.length += growth; 5344 view = underlyingBuffer[viewStart .. view.length]; 5345 } else 5346 throw new Exception("No room left in the buffer"); 5347 } 5348 5349 do { 5350 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5351 try_again: 5352 auto ret = source.receive(freeSpace); 5353 if(ret == Socket.ERROR) { 5354 if(wouldHaveBlocked()) { 5355 version(cgi_use_fiber) { 5356 registerEventWakeup(®istered, source, WakeupEvent.Read); 5357 goto try_again; 5358 } else { 5359 // gonna treat a timeout here as a close 5360 sourceClosed = true; 5361 return; 5362 } 5363 } 5364 version(Posix) { 5365 import core.stdc.errno; 5366 if(errno == EINTR || errno == EAGAIN) { 5367 goto try_again; 5368 } 5369 if(errno == ECONNRESET) { 5370 sourceClosed = true; 5371 return; 5372 } 5373 } 5374 throw new Exception(lastSocketError); // FIXME 5375 } 5376 if(ret == 0) { 5377 sourceClosed = true; 5378 return; 5379 } 5380 5381 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5382 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5383 //import std.stdio; writeln(cast(string) view); 5384 } while(view.length < minBytesToSettleFor); 5385 } 5386 5387 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5388 /// You might want to idup the data you are consuming if you store it, since it may 5389 /// be overwritten on the new popFront. 5390 /// 5391 /// You do not need to call this if you always want to wait for more data when you 5392 /// consume some. 5393 ubyte[] consume(size_t bytes) { 5394 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5395 view = view[bytes > $ ? $ : bytes .. $]; 5396 if(view.length == 0) { 5397 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5398 /* 5399 writeln("HERE"); 5400 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5401 writeln(cast(string)front); 5402 writeln("DONE"); 5403 */ 5404 } 5405 return front; 5406 } 5407 5408 bool empty() { 5409 return sourceClosed && view.length == 0; 5410 } 5411 5412 ubyte[] front() { 5413 return view; 5414 } 5415 5416 invariant() { 5417 assert(view.ptr >= underlyingBuffer.ptr); 5418 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5419 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5420 } 5421 5422 ubyte[] underlyingBuffer; 5423 bool allowGrowth; 5424 ubyte[] view; 5425 Socket source; 5426 bool sourceClosed; 5427 } 5428 5429 private class FakeSocketForStdin : Socket { 5430 import std.stdio; 5431 5432 this() { 5433 5434 } 5435 5436 private bool closed; 5437 5438 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5439 if(closed) 5440 throw new Exception("Closed"); 5441 return stdin.rawRead(buffer).length; 5442 } 5443 5444 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5445 if(closed) 5446 throw new Exception("Closed"); 5447 stdout.rawWrite(buffer); 5448 return buffer.length; 5449 } 5450 5451 override void close() @trusted scope { 5452 (cast(void delegate() @nogc nothrow) &realClose)(); 5453 } 5454 5455 override void shutdown(SocketShutdown s) { 5456 // FIXME 5457 } 5458 5459 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5460 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5461 5462 override @property @trusted Address remoteAddress() { return null; } 5463 override @property @trusted Address localAddress() { return null; } 5464 5465 void realClose() { 5466 closed = true; 5467 try { 5468 stdin.close(); 5469 stdout.close(); 5470 } catch(Exception e) { 5471 5472 } 5473 } 5474 } 5475 5476 import core.sync.semaphore; 5477 import core.atomic; 5478 5479 /** 5480 To use this thing: 5481 5482 --- 5483 void handler(Socket s) { do something... } 5484 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5485 manager.listen(); 5486 --- 5487 5488 The 4th parameter is optional. 5489 5490 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5491 comes in, you will get control. You can just continue; though to fetch more. 5492 5493 5494 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5495 */ 5496 class ListeningConnectionManager { 5497 Semaphore semaphore; 5498 Socket[256] queue; 5499 shared(ubyte) nextIndexFront; 5500 ubyte nextIndexBack; 5501 shared(int) queueLength; 5502 5503 Socket acceptCancelable() { 5504 version(Posix) { 5505 import core.sys.posix.sys.select; 5506 fd_set read_fds; 5507 FD_ZERO(&read_fds); 5508 FD_SET(listener.handle, &read_fds); 5509 if(cancelfd != -1) 5510 FD_SET(cancelfd, &read_fds); 5511 auto max = listener.handle > cancelfd ? listener.handle : cancelfd; 5512 auto ret = select(max + 1, &read_fds, null, null, null); 5513 if(ret == -1) { 5514 import core.stdc.errno; 5515 if(errno == EINTR) 5516 return null; 5517 else 5518 throw new Exception("wtf select"); 5519 } 5520 5521 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5522 return null; 5523 } 5524 5525 if(FD_ISSET(listener.handle, &read_fds)) 5526 return listener.accept(); 5527 5528 return null; 5529 } else { 5530 5531 Socket socket = listener; 5532 5533 auto check = new SocketSet(); 5534 5535 keep_looping: 5536 check.reset(); 5537 check.add(socket); 5538 5539 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5540 auto got = Socket.select(check, null, null, 3.seconds); 5541 if(got > 0) 5542 return listener.accept(); 5543 if(globalStopFlag) 5544 return null; 5545 else 5546 goto keep_looping; 5547 } 5548 } 5549 5550 int defaultNumberOfThreads() { 5551 import std.parallelism; 5552 version(cgi_use_fiber) { 5553 return totalCPUs * 1 + 1; 5554 } else { 5555 // I times 4 here because there's a good chance some will be blocked on i/o. 5556 return totalCPUs * 4; 5557 } 5558 5559 } 5560 5561 void listen() { 5562 shared(int) loopBroken; 5563 5564 version(Posix) { 5565 import core.sys.posix.signal; 5566 signal(SIGPIPE, SIG_IGN); 5567 } 5568 5569 version(linux) { 5570 if(cancelfd == -1) 5571 cancelfd = eventfd(0, 0); 5572 } 5573 5574 version(cgi_no_threads) { 5575 // NEVER USE THIS 5576 // it exists only for debugging and other special occasions 5577 5578 // the thread mode is faster and less likely to stall the whole 5579 // thing when a request is slow 5580 while(!loopBroken && !globalStopFlag) { 5581 auto sn = acceptCancelable(); 5582 if(sn is null) continue; 5583 cloexec(sn); 5584 try { 5585 handler(sn); 5586 } catch(Exception e) { 5587 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5588 sn.close(); 5589 } 5590 } 5591 } else { 5592 5593 if(useFork) { 5594 version(linux) { 5595 //asm { int 3; } 5596 fork(); 5597 } 5598 } 5599 5600 version(cgi_use_fiber) { 5601 5602 version(Windows) { 5603 listener.accept(); 5604 } 5605 5606 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5607 foreach(i, ref thread; threads) { 5608 thread = new WorkerThread(this, handler, cast(int) i); 5609 thread.start(); 5610 } 5611 5612 bool fiber_crash_check() { 5613 bool hasAnyRunning; 5614 foreach(thread; threads) { 5615 if(!thread.isRunning) { 5616 thread.join(); 5617 } else hasAnyRunning = true; 5618 } 5619 5620 return (!hasAnyRunning); 5621 } 5622 5623 5624 while(!globalStopFlag) { 5625 Thread.sleep(1.seconds); 5626 if(fiber_crash_check()) 5627 break; 5628 } 5629 5630 } else { 5631 semaphore = new Semaphore(); 5632 5633 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5634 foreach(i, ref thread; threads) { 5635 thread = new ConnectionThread(this, handler, cast(int) i); 5636 thread.start(); 5637 } 5638 5639 while(!loopBroken && !globalStopFlag) { 5640 Socket sn; 5641 5642 bool crash_check() { 5643 bool hasAnyRunning; 5644 foreach(thread; threads) { 5645 if(!thread.isRunning) { 5646 thread.join(); 5647 } else hasAnyRunning = true; 5648 } 5649 5650 return (!hasAnyRunning); 5651 } 5652 5653 5654 void accept_new_connection() { 5655 sn = acceptCancelable(); 5656 if(sn is null) return; 5657 cloexec(sn); 5658 if(tcp) { 5659 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5660 // on the socket because we do some buffering internally. I think this helps, 5661 // certainly does for small requests, and I think it does for larger ones too 5662 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5663 5664 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5665 } 5666 } 5667 5668 void existing_connection_new_data() { 5669 // wait until a slot opens up 5670 //int waited = 0; 5671 while(queueLength >= queue.length) { 5672 Thread.sleep(1.msecs); 5673 //waited ++; 5674 } 5675 //if(waited) {import std.stdio; writeln(waited);} 5676 synchronized(this) { 5677 queue[nextIndexBack] = sn; 5678 nextIndexBack++; 5679 atomicOp!"+="(queueLength, 1); 5680 } 5681 semaphore.notify(); 5682 } 5683 5684 5685 accept_new_connection(); 5686 if(sn !is null) 5687 existing_connection_new_data(); 5688 else if(sn is null && globalStopFlag) { 5689 foreach(thread; threads) { 5690 semaphore.notify(); 5691 } 5692 Thread.sleep(50.msecs); 5693 } 5694 5695 if(crash_check()) 5696 break; 5697 } 5698 } 5699 5700 // FIXME: i typically stop this with ctrl+c which never 5701 // actually gets here. i need to do a sigint handler. 5702 if(cleanup) 5703 cleanup(); 5704 } 5705 } 5706 5707 //version(linux) 5708 //int epoll_fd; 5709 5710 bool tcp; 5711 void delegate() cleanup; 5712 5713 private void function(Socket) fhandler; 5714 private void dg_handler(Socket s) { 5715 fhandler(s); 5716 } 5717 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5718 fhandler = handler; 5719 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5720 } 5721 5722 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5723 this.handler = handler; 5724 this.useFork = useFork; 5725 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5726 5727 listener = startListening(host, port, tcp, cleanup, 128, dropPrivs); 5728 5729 version(cgi_use_fiber) 5730 if(useFork) 5731 listener.blocking = false; 5732 5733 // this is the UI control thread and thus gets more priority 5734 Thread.getThis.priority = Thread.PRIORITY_MAX; 5735 } 5736 5737 Socket listener; 5738 void delegate(Socket) handler; 5739 5740 immutable bool useFork; 5741 int numberOfThreads; 5742 } 5743 5744 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 5745 Socket listener; 5746 if(host.startsWith("unix:")) { 5747 version(Posix) { 5748 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5749 cloexec(listener); 5750 string filename = host["unix:".length .. $].idup; 5751 listener.bind(new UnixAddress(filename)); 5752 cleanup = delegate() { 5753 listener.close(); 5754 import std.file; 5755 remove(filename); 5756 }; 5757 tcp = false; 5758 } else { 5759 throw new Exception("unix sockets not supported on this system"); 5760 } 5761 } else if(host.startsWith("abstract:")) { 5762 version(linux) { 5763 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5764 cloexec(listener); 5765 string filename = "\0" ~ host["abstract:".length .. $]; 5766 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 5767 listener.bind(new UnixAddress(filename)); 5768 tcp = false; 5769 } else { 5770 throw new Exception("abstract unix sockets not supported on this system"); 5771 } 5772 } else { 5773 version(cgi_use_fiber) { 5774 version(Windows) 5775 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 5776 else 5777 listener = new TcpSocket(); 5778 } else { 5779 listener = new TcpSocket(); 5780 } 5781 cloexec(listener); 5782 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 5783 listener.bind(host.length ? parseAddress(host, port) : new InternetAddress(port)); 5784 cleanup = delegate() { 5785 listener.close(); 5786 }; 5787 tcp = true; 5788 } 5789 5790 listener.listen(backQueue); 5791 5792 if (dropPrivs !is null) // can be null, backwards compatibility 5793 dropPrivs(); 5794 5795 return listener; 5796 } 5797 5798 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 5799 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 5800 if(data.length == 0) return; 5801 ptrdiff_t amount; 5802 //import std.stdio; writeln("***",cast(string) data,"///"); 5803 do { 5804 amount = s.send(data); 5805 if(amount == Socket.ERROR) { 5806 version(cgi_use_fiber) { 5807 if(wouldHaveBlocked()) { 5808 bool registered = true; 5809 registerEventWakeup(®istered, s, WakeupEvent.Write); 5810 continue; 5811 } 5812 } 5813 throw new ConnectionException(s, lastSocketError, file, line); 5814 } 5815 assert(amount > 0); 5816 5817 data = data[amount .. $]; 5818 } while(data.length); 5819 } 5820 5821 class ConnectionException : Exception { 5822 Socket socket; 5823 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 5824 this.socket = s; 5825 super(msg, file, line); 5826 } 5827 } 5828 5829 alias void delegate(Socket) CMT; 5830 5831 import core.thread; 5832 /+ 5833 cgi.d now uses a hybrid of event i/o and threads at the top level. 5834 5835 Top level thread is responsible for accepting sockets and selecting on them. 5836 5837 It then indicates to a child that a request is pending, and any random worker 5838 thread that is free handles it. It goes into blocking mode and handles that 5839 http request to completion. 5840 5841 At that point, it goes back into the waiting queue. 5842 5843 5844 This concept is only implemented on Linux. On all other systems, it still 5845 uses the worker threads and semaphores (which is perfectly fine for a lot of 5846 things! Just having a great number of keep-alive connections will break that.) 5847 5848 5849 So the algorithm is: 5850 5851 select(accept, event, pending) 5852 if accept -> send socket to free thread, if any. if not, add socket to queue 5853 if event -> send the signaling thread a socket from the queue, if not, mark it free 5854 - event might block until it can be *written* to. it is a fifo sending socket fds! 5855 5856 A worker only does one http request at a time, then signals its availability back to the boss. 5857 5858 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 5859 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 5860 actual FD will not be kept out here. 5861 5862 So: 5863 queue = sockets we know are ready to read now, but no worker thread is available 5864 idle list = worker threads not doing anything else. they signal back and forth 5865 5866 the workers all read off the event fd. This is the semaphore wait 5867 5868 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 5869 it puts it in the queue and writes to the event fd. 5870 5871 The child could put the socket back in the epoll thing itself. 5872 5873 The child needs to be able to gracefully handle being given a socket that just closed with no work. 5874 +/ 5875 class ConnectionThread : Thread { 5876 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 5877 this.lcm = lcm; 5878 this.dg = dg; 5879 this.myThreadNumber = myThreadNumber; 5880 super(&run); 5881 } 5882 5883 void run() { 5884 while(true) { 5885 // so if there's a bunch of idle keep-alive connections, it can 5886 // consume all the worker threads... just sitting there. 5887 lcm.semaphore.wait(); 5888 if(globalStopFlag) 5889 return; 5890 Socket socket; 5891 synchronized(lcm) { 5892 auto idx = lcm.nextIndexFront; 5893 socket = lcm.queue[idx]; 5894 lcm.queue[idx] = null; 5895 atomicOp!"+="(lcm.nextIndexFront, 1); 5896 atomicOp!"-="(lcm.queueLength, 1); 5897 } 5898 try { 5899 //import std.stdio; writeln(myThreadNumber, " taking it"); 5900 dg(socket); 5901 /+ 5902 if(socket.isAlive) { 5903 // process it more later 5904 version(linux) { 5905 import core.sys.linux.epoll; 5906 epoll_event ev; 5907 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 5908 ev.data.fd = socket.handle; 5909 import std.stdio; writeln("adding"); 5910 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 5911 if(errno == EEXIST) { 5912 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 5913 ev.data.fd = socket.handle; 5914 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 5915 throw new Exception("epoll_ctl " ~ to!string(errno)); 5916 } else 5917 throw new Exception("epoll_ctl " ~ to!string(errno)); 5918 } 5919 //import std.stdio; writeln("keep alive"); 5920 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 5921 __traits(getMember, socket, "sock") = cast(socket_t) -1; 5922 } else { 5923 continue; // hope it times out in a reasonable amount of time... 5924 } 5925 } 5926 +/ 5927 } catch(ConnectionClosedException e) { 5928 // can just ignore this, it is fairly normal 5929 socket.close(); 5930 } catch(Throwable e) { 5931 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 5932 socket.close(); 5933 } 5934 } 5935 } 5936 5937 ListeningConnectionManager lcm; 5938 CMT dg; 5939 int myThreadNumber; 5940 } 5941 5942 version(cgi_use_fiber) 5943 class WorkerThread : Thread { 5944 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 5945 this.lcm = lcm; 5946 this.dg = dg; 5947 this.myThreadNumber = myThreadNumber; 5948 super(&run); 5949 } 5950 5951 version(Windows) 5952 void run() { 5953 auto timeout = INFINITE; 5954 PseudoblockingOverlappedSocket key; 5955 OVERLAPPED* overlapped; 5956 DWORD bytes; 5957 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 5958 if(key is null) 5959 continue; 5960 key.lastAnswer = bytes; 5961 if(key.fiber) { 5962 key.fiber.proceed(); 5963 } else { 5964 // we have a new connection, issue the first receive on it and issue the next accept 5965 5966 auto sn = key.accepted; 5967 5968 key.accept(); 5969 5970 cloexec(sn); 5971 if(lcm.tcp) { 5972 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5973 // on the socket because we do some buffering internally. I think this helps, 5974 // certainly does for small requests, and I think it does for larger ones too 5975 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5976 5977 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5978 } 5979 5980 dg(sn); 5981 } 5982 } 5983 //SleepEx(INFINITE, TRUE); 5984 } 5985 5986 version(linux) 5987 void run() { 5988 5989 import core.sys.linux.epoll; 5990 epfd = epoll_create1(EPOLL_CLOEXEC); 5991 if(epfd == -1) 5992 throw new Exception("epoll_create1 " ~ to!string(errno)); 5993 scope(exit) { 5994 import core.sys.posix.unistd; 5995 close(epfd); 5996 } 5997 5998 { 5999 epoll_event ev; 6000 ev.events = EPOLLIN; 6001 ev.data.fd = cancelfd; 6002 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6003 } 6004 6005 epoll_event ev; 6006 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6007 ev.data.fd = lcm.listener.handle; 6008 if(epoll_ctl(epfd, EPOLL_CTL_ADD, lcm.listener.handle, &ev) == -1) 6009 throw new Exception("epoll_ctl " ~ to!string(errno)); 6010 6011 6012 6013 while(!globalStopFlag) { 6014 Socket sn; 6015 6016 epoll_event[64] events; 6017 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6018 if(nfds == -1) { 6019 if(errno == EINTR) 6020 continue; 6021 throw new Exception("epoll_wait " ~ to!string(errno)); 6022 } 6023 6024 foreach(idx; 0 .. nfds) { 6025 auto flags = events[idx].events; 6026 6027 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6028 globalStopFlag = true; 6029 //import std.stdio; writeln("exit heard"); 6030 break; 6031 } else if(cast(size_t) events[idx].data.ptr == cast(size_t) lcm.listener.handle) { 6032 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6033 // this try/catch is because it is set to non-blocking mode 6034 // and Phobos' stupid api throws an exception instead of returning 6035 // if it would block. Why would it block? because a forked process 6036 // might have beat us to it, but the wakeup event thundered our herds. 6037 try 6038 sn = lcm.listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6039 catch(SocketAcceptException e) { continue; } 6040 6041 cloexec(sn); 6042 if(lcm.tcp) { 6043 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6044 // on the socket because we do some buffering internally. I think this helps, 6045 // certainly does for small requests, and I think it does for larger ones too 6046 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6047 6048 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6049 } 6050 6051 dg(sn); 6052 } else { 6053 if(cast(size_t) events[idx].data.ptr < 1024) { 6054 throw new Exception("this doesn't look like a fiber pointer..."); 6055 } 6056 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6057 fiber.proceed(); 6058 } 6059 } 6060 } 6061 } 6062 6063 ListeningConnectionManager lcm; 6064 CMT dg; 6065 int myThreadNumber; 6066 } 6067 6068 6069 /* Done with network helper */ 6070 6071 /* Helpers for doing temporary files. Used both here and in web.d */ 6072 6073 version(Windows) { 6074 import core.sys.windows.windows; 6075 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6076 alias GetTempPathW GetTempPath; 6077 } 6078 6079 version(Posix) { 6080 static import linux = core.sys.posix.unistd; 6081 } 6082 6083 string getTempDirectory() { 6084 string path; 6085 version(Windows) { 6086 wchar[1024] buffer; 6087 auto len = GetTempPath(1024, buffer.ptr); 6088 if(len == 0) 6089 throw new Exception("couldn't find a temporary path"); 6090 6091 auto b = buffer[0 .. len]; 6092 6093 path = to!string(b); 6094 } else 6095 path = "/tmp/"; 6096 6097 return path; 6098 } 6099 6100 6101 // I like std.date. These functions help keep my old code and data working with phobos changing. 6102 6103 long sysTimeToDTime(in SysTime sysTime) { 6104 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6105 } 6106 6107 long dateTimeToDTime(in DateTime dt) { 6108 return sysTimeToDTime(cast(SysTime) dt); 6109 } 6110 6111 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6112 return sysTimeToDTime(Clock.currTime(UTC())); 6113 } 6114 6115 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6116 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6117 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6118 return SysTime(hnsecs, tz); 6119 } 6120 6121 6122 6123 // this is a helper to read HTTP transfer-encoding: chunked responses 6124 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6125 immutable(ubyte)[] ret; 6126 6127 another_chunk: 6128 // If here, we are at the beginning of a chunk. 6129 auto a = ir.front(); 6130 int chunkSize; 6131 int loc = locationOf(a, "\r\n"); 6132 while(loc == -1) { 6133 ir.popFront(); 6134 a = ir.front(); 6135 loc = locationOf(a, "\r\n"); 6136 } 6137 6138 string hex; 6139 hex = ""; 6140 for(int i = 0; i < loc; i++) { 6141 char c = a[i]; 6142 if(c >= 'A' && c <= 'Z') 6143 c += 0x20; 6144 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6145 hex ~= c; 6146 } else { 6147 break; 6148 } 6149 } 6150 6151 assert(hex.length); 6152 6153 int power = 1; 6154 int size = 0; 6155 foreach(cc1; retro(hex)) { 6156 dchar cc = cc1; 6157 if(cc >= 'a' && cc <= 'z') 6158 cc -= 0x20; 6159 int val = 0; 6160 if(cc >= '0' && cc <= '9') 6161 val = cc - '0'; 6162 else 6163 val = cc - 'A' + 10; 6164 6165 size += power * val; 6166 power *= 16; 6167 } 6168 6169 chunkSize = size; 6170 assert(size >= 0); 6171 6172 if(loc + 2 > a.length) { 6173 ir.popFront(0, a.length + loc + 2); 6174 a = ir.front(); 6175 } 6176 6177 a = ir.consume(loc + 2); 6178 6179 if(chunkSize == 0) { // we're done with the response 6180 // if we got here, will change must be true.... 6181 more_footers: 6182 loc = locationOf(a, "\r\n"); 6183 if(loc == -1) { 6184 ir.popFront(); 6185 a = ir.front; 6186 goto more_footers; 6187 } else { 6188 assert(loc == 0); 6189 ir.consume(loc + 2); 6190 goto finish; 6191 } 6192 } else { 6193 // if we got here, will change must be true.... 6194 if(a.length < chunkSize + 2) { 6195 ir.popFront(0, chunkSize + 2); 6196 a = ir.front(); 6197 } 6198 6199 ret ~= (a[0..chunkSize]); 6200 6201 if(!(a.length > chunkSize + 2)) { 6202 ir.popFront(0, chunkSize + 2); 6203 a = ir.front(); 6204 } 6205 assert(a[chunkSize] == 13); 6206 assert(a[chunkSize+1] == 10); 6207 a = ir.consume(chunkSize + 2); 6208 chunkSize = 0; 6209 goto another_chunk; 6210 } 6211 6212 finish: 6213 return ret; 6214 } 6215 6216 // I want to be able to get data from multiple sources the same way... 6217 interface ByChunkRange { 6218 bool empty(); 6219 void popFront(); 6220 const(ubyte)[] front(); 6221 } 6222 6223 ByChunkRange byChunk(const(ubyte)[] data) { 6224 return new class ByChunkRange { 6225 override bool empty() { 6226 return !data.length; 6227 } 6228 6229 override void popFront() { 6230 if(data.length > 4096) 6231 data = data[4096 .. $]; 6232 else 6233 data = null; 6234 } 6235 6236 override const(ubyte)[] front() { 6237 return data[0 .. $ > 4096 ? 4096 : $]; 6238 } 6239 }; 6240 } 6241 6242 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6243 const(ubyte)[] f; 6244 6245 f = ir.front; 6246 if(f.length > atMost) 6247 f = f[0 .. atMost]; 6248 6249 return new class ByChunkRange { 6250 override bool empty() { 6251 return atMost == 0; 6252 } 6253 6254 override const(ubyte)[] front() { 6255 return f; 6256 } 6257 6258 override void popFront() { 6259 ir.consume(f.length); 6260 atMost -= f.length; 6261 auto a = ir.front(); 6262 6263 if(a.length <= atMost) { 6264 f = a; 6265 atMost -= a.length; 6266 a = ir.consume(a.length); 6267 if(atMost != 0) 6268 ir.popFront(); 6269 if(f.length == 0) { 6270 f = ir.front(); 6271 } 6272 } else { 6273 // we actually have *more* here than we need.... 6274 f = a[0..atMost]; 6275 atMost = 0; 6276 ir.consume(atMost); 6277 } 6278 } 6279 }; 6280 } 6281 6282 version(cgi_with_websocket) { 6283 // http://tools.ietf.org/html/rfc6455 6284 6285 /** 6286 WEBSOCKET SUPPORT: 6287 6288 Full example: 6289 --- 6290 import arsd.cgi; 6291 6292 void websocketEcho(Cgi cgi) { 6293 if(cgi.websocketRequested()) { 6294 if(cgi.origin != "http://arsdnet.net") 6295 throw new Exception("bad origin"); 6296 auto websocket = cgi.acceptWebsocket(); 6297 6298 websocket.send("hello"); 6299 websocket.send(" world!"); 6300 6301 auto msg = websocket.recv(); 6302 while(msg.opcode != WebSocketOpcode.close) { 6303 if(msg.opcode == WebSocketOpcode.text) { 6304 websocket.send(msg.textData); 6305 } else if(msg.opcode == WebSocketOpcode.binary) { 6306 websocket.send(msg.data); 6307 } 6308 6309 msg = websocket.recv(); 6310 } 6311 6312 websocket.close(); 6313 } else assert(0, "i want a web socket!"); 6314 } 6315 6316 mixin GenericMain!websocketEcho; 6317 --- 6318 */ 6319 6320 class WebSocket { 6321 Cgi cgi; 6322 6323 private this(Cgi cgi) { 6324 this.cgi = cgi; 6325 6326 Socket socket = cgi.idlol.source; 6327 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6328 } 6329 6330 // returns true if data available, false if it timed out 6331 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6332 if(!waitForNextMessageWouldBlock()) 6333 return true; 6334 if(isDataPending(timeout)) 6335 return true; // this is kinda a lie. 6336 6337 return false; 6338 } 6339 6340 public bool lowLevelReceive() { 6341 auto bfr = cgi.idlol; 6342 top: 6343 auto got = bfr.front; 6344 if(got.length) { 6345 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6346 receiveBuffer.length += receiveBufferUsedLength + got.length; 6347 6348 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6349 receiveBufferUsedLength += got.length; 6350 bfr.consume(got.length); 6351 6352 return true; 6353 } 6354 6355 if(bfr.sourceClosed) 6356 return false; 6357 6358 bfr.popFront(0); 6359 if(bfr.sourceClosed) 6360 return false; 6361 goto top; 6362 } 6363 6364 6365 bool isDataPending(Duration timeout = 0.seconds) { 6366 Socket socket = cgi.idlol.source; 6367 6368 auto check = new SocketSet(); 6369 check.add(socket); 6370 6371 auto got = Socket.select(check, null, null, timeout); 6372 if(got > 0) 6373 return true; 6374 return false; 6375 } 6376 6377 // note: this blocks 6378 WebSocketFrame recv() { 6379 return waitForNextMessage(); 6380 } 6381 6382 6383 6384 6385 private void llclose() { 6386 cgi.close(); 6387 } 6388 6389 private void llsend(ubyte[] data) { 6390 cgi.write(data); 6391 cgi.flush(); 6392 } 6393 6394 void unregisterActiveSocket(WebSocket) {} 6395 6396 /* copy/paste section { */ 6397 6398 private int readyState_; 6399 private ubyte[] receiveBuffer; 6400 private size_t receiveBufferUsedLength; 6401 6402 private Config config; 6403 6404 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6405 enum OPEN = 1; /// The connection is open and ready to communicate. 6406 enum CLOSING = 2; /// The connection is in the process of closing. 6407 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6408 6409 /++ 6410 6411 +/ 6412 /// Group: foundational 6413 static struct Config { 6414 /++ 6415 These control the size of the receive buffer. 6416 6417 It starts at the initial size, will temporarily 6418 balloon up to the maximum size, and will reuse 6419 a buffer up to the likely size. 6420 6421 Anything larger than the maximum size will cause 6422 the connection to be aborted and an exception thrown. 6423 This is to protect you against a peer trying to 6424 exhaust your memory, while keeping the user-level 6425 processing simple. 6426 +/ 6427 size_t initialReceiveBufferSize = 4096; 6428 size_t likelyReceiveBufferSize = 4096; /// ditto 6429 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6430 6431 /++ 6432 Maximum combined size of a message. 6433 +/ 6434 size_t maximumMessageSize = 10 * 1024 * 1024; 6435 6436 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6437 string origin; /// Origin URL to send with the handshake, if desired. 6438 string protocol; /// the protocol header, if desired. 6439 6440 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6441 } 6442 6443 /++ 6444 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6445 +/ 6446 int readyState() { 6447 return readyState_; 6448 } 6449 6450 /++ 6451 Closes the connection, sending a graceful teardown message to the other side. 6452 +/ 6453 /// Group: foundational 6454 void close(int code = 0, string reason = null) 6455 //in (reason.length < 123) 6456 in { assert(reason.length < 123); } do 6457 { 6458 if(readyState_ != OPEN) 6459 return; // it cool, we done 6460 WebSocketFrame wss; 6461 wss.fin = true; 6462 wss.opcode = WebSocketOpcode.close; 6463 wss.data = cast(ubyte[]) reason.dup; 6464 wss.send(&llsend); 6465 6466 readyState_ = CLOSING; 6467 6468 llclose(); 6469 } 6470 6471 /++ 6472 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6473 +/ 6474 /// Group: foundational 6475 void ping() { 6476 WebSocketFrame wss; 6477 wss.fin = true; 6478 wss.opcode = WebSocketOpcode.ping; 6479 wss.send(&llsend); 6480 } 6481 6482 // automatically handled.... 6483 void pong() { 6484 WebSocketFrame wss; 6485 wss.fin = true; 6486 wss.opcode = WebSocketOpcode.pong; 6487 wss.send(&llsend); 6488 } 6489 6490 /++ 6491 Sends a text message through the websocket. 6492 +/ 6493 /// Group: foundational 6494 void send(in char[] textData) { 6495 WebSocketFrame wss; 6496 wss.fin = true; 6497 wss.opcode = WebSocketOpcode.text; 6498 wss.data = cast(ubyte[]) textData.dup; 6499 wss.send(&llsend); 6500 } 6501 6502 /++ 6503 Sends a binary message through the websocket. 6504 +/ 6505 /// Group: foundational 6506 void send(in ubyte[] binaryData) { 6507 WebSocketFrame wss; 6508 wss.fin = true; 6509 wss.opcode = WebSocketOpcode.binary; 6510 wss.data = cast(ubyte[]) binaryData.dup; 6511 wss.send(&llsend); 6512 } 6513 6514 /++ 6515 Waits for and returns the next complete message on the socket. 6516 6517 Note that the onmessage function is still called, right before 6518 this returns. 6519 +/ 6520 /// Group: blocking_api 6521 public WebSocketFrame waitForNextMessage() { 6522 do { 6523 auto m = processOnce(); 6524 if(m.populated) 6525 return m; 6526 } while(lowLevelReceive()); 6527 6528 throw new ConnectionClosedException("Websocket receive timed out"); 6529 //return WebSocketFrame.init; // FIXME? maybe. 6530 } 6531 6532 /++ 6533 Tells if [waitForNextMessage] would block. 6534 +/ 6535 /// Group: blocking_api 6536 public bool waitForNextMessageWouldBlock() { 6537 checkAgain: 6538 if(isMessageBuffered()) 6539 return false; 6540 if(!isDataPending()) 6541 return true; 6542 while(isDataPending()) 6543 lowLevelReceive(); 6544 goto checkAgain; 6545 } 6546 6547 /++ 6548 Is there a message in the buffer already? 6549 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6550 If `false`, check [isDataPending] as the next step. 6551 +/ 6552 /// Group: blocking_api 6553 public bool isMessageBuffered() { 6554 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6555 auto s = d; 6556 if(d.length) { 6557 auto orig = d; 6558 auto m = WebSocketFrame.read(d); 6559 // that's how it indicates that it needs more data 6560 if(d !is orig) 6561 return true; 6562 } 6563 6564 return false; 6565 } 6566 6567 private ubyte continuingType; 6568 private ubyte[] continuingData; 6569 //private size_t continuingDataLength; 6570 6571 private WebSocketFrame processOnce() { 6572 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6573 auto s = d; 6574 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6575 WebSocketFrame m; 6576 if(d.length) { 6577 auto orig = d; 6578 m = WebSocketFrame.read(d); 6579 // that's how it indicates that it needs more data 6580 if(d is orig) 6581 return WebSocketFrame.init; 6582 m.unmaskInPlace(); 6583 switch(m.opcode) { 6584 case WebSocketOpcode.continuation: 6585 if(continuingData.length + m.data.length > config.maximumMessageSize) 6586 throw new Exception("message size exceeded"); 6587 6588 continuingData ~= m.data; 6589 if(m.fin) { 6590 if(ontextmessage) 6591 ontextmessage(cast(char[]) continuingData); 6592 if(onbinarymessage) 6593 onbinarymessage(continuingData); 6594 6595 continuingData = null; 6596 } 6597 break; 6598 case WebSocketOpcode.text: 6599 if(m.fin) { 6600 if(ontextmessage) 6601 ontextmessage(m.textData); 6602 } else { 6603 continuingType = m.opcode; 6604 //continuingDataLength = 0; 6605 continuingData = null; 6606 continuingData ~= m.data; 6607 } 6608 break; 6609 case WebSocketOpcode.binary: 6610 if(m.fin) { 6611 if(onbinarymessage) 6612 onbinarymessage(m.data); 6613 } else { 6614 continuingType = m.opcode; 6615 //continuingDataLength = 0; 6616 continuingData = null; 6617 continuingData ~= m.data; 6618 } 6619 break; 6620 case WebSocketOpcode.close: 6621 readyState_ = CLOSED; 6622 if(onclose) 6623 onclose(); 6624 6625 unregisterActiveSocket(this); 6626 break; 6627 case WebSocketOpcode.ping: 6628 pong(); 6629 break; 6630 case WebSocketOpcode.pong: 6631 // just really references it is still alive, nbd. 6632 break; 6633 default: // ignore though i could and perhaps should throw too 6634 } 6635 } 6636 6637 // the recv thing can be invalidated so gotta copy it over ugh 6638 if(d.length) { 6639 m.data = m.data.dup(); 6640 } 6641 6642 import core.stdc.string; 6643 memmove(receiveBuffer.ptr, d.ptr, d.length); 6644 receiveBufferUsedLength = d.length; 6645 6646 return m; 6647 } 6648 6649 private void autoprocess() { 6650 // FIXME 6651 do { 6652 processOnce(); 6653 } while(lowLevelReceive()); 6654 } 6655 6656 6657 void delegate() onclose; /// 6658 void delegate() onerror; /// 6659 void delegate(in char[]) ontextmessage; /// 6660 void delegate(in ubyte[]) onbinarymessage; /// 6661 void delegate() onopen; /// 6662 6663 /++ 6664 6665 +/ 6666 /// Group: browser_api 6667 void onmessage(void delegate(in char[]) dg) { 6668 ontextmessage = dg; 6669 } 6670 6671 /// ditto 6672 void onmessage(void delegate(in ubyte[]) dg) { 6673 onbinarymessage = dg; 6674 } 6675 6676 /* } end copy/paste */ 6677 6678 6679 } 6680 6681 bool websocketRequested(Cgi cgi) { 6682 return 6683 "sec-websocket-key" in cgi.requestHeaders 6684 && 6685 "connection" in cgi.requestHeaders && 6686 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6687 && 6688 "upgrade" in cgi.requestHeaders && 6689 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6690 ; 6691 } 6692 6693 WebSocket acceptWebsocket(Cgi cgi) { 6694 assert(!cgi.closed); 6695 assert(!cgi.outputtedResponseData); 6696 cgi.setResponseStatus("101 Switching Protocols"); 6697 cgi.header("Upgrade: WebSocket"); 6698 cgi.header("Connection: upgrade"); 6699 6700 string key = cgi.requestHeaders["sec-websocket-key"]; 6701 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 6702 6703 import std.digest.sha; 6704 auto hash = sha1Of(key); 6705 auto accept = Base64.encode(hash); 6706 6707 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 6708 6709 cgi.websocketMode = true; 6710 cgi.write(""); 6711 6712 cgi.flush(); 6713 6714 return new WebSocket(cgi); 6715 } 6716 6717 // FIXME get websocket to work on other modes, not just embedded_httpd 6718 6719 /* copy/paste in http2.d { */ 6720 enum WebSocketOpcode : ubyte { 6721 continuation = 0, 6722 text = 1, 6723 binary = 2, 6724 // 3, 4, 5, 6, 7 RESERVED 6725 close = 8, 6726 ping = 9, 6727 pong = 10, 6728 // 11,12,13,14,15 RESERVED 6729 } 6730 6731 public struct WebSocketFrame { 6732 private bool populated; 6733 bool fin; 6734 bool rsv1; 6735 bool rsv2; 6736 bool rsv3; 6737 WebSocketOpcode opcode; // 4 bits 6738 bool masked; 6739 ubyte lengthIndicator; // don't set this when building one to send 6740 ulong realLength; // don't use when sending 6741 ubyte[4] maskingKey; // don't set this when sending 6742 ubyte[] data; 6743 6744 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 6745 WebSocketFrame msg; 6746 msg.fin = true; 6747 msg.opcode = opcode; 6748 msg.data = cast(ubyte[]) data.dup; 6749 6750 return msg; 6751 } 6752 6753 private void send(scope void delegate(ubyte[]) llsend) { 6754 ubyte[64] headerScratch; 6755 int headerScratchPos = 0; 6756 6757 realLength = data.length; 6758 6759 { 6760 ubyte b1; 6761 b1 |= cast(ubyte) opcode; 6762 b1 |= rsv3 ? (1 << 4) : 0; 6763 b1 |= rsv2 ? (1 << 5) : 0; 6764 b1 |= rsv1 ? (1 << 6) : 0; 6765 b1 |= fin ? (1 << 7) : 0; 6766 6767 headerScratch[0] = b1; 6768 headerScratchPos++; 6769 } 6770 6771 { 6772 headerScratchPos++; // we'll set header[1] at the end of this 6773 auto rlc = realLength; 6774 ubyte b2; 6775 b2 |= masked ? (1 << 7) : 0; 6776 6777 assert(headerScratchPos == 2); 6778 6779 if(realLength > 65535) { 6780 // use 64 bit length 6781 b2 |= 0x7f; 6782 6783 // FIXME: double check endinaness 6784 foreach(i; 0 .. 8) { 6785 headerScratch[2 + 7 - i] = rlc & 0x0ff; 6786 rlc >>>= 8; 6787 } 6788 6789 headerScratchPos += 8; 6790 } else if(realLength > 125) { 6791 // use 16 bit length 6792 b2 |= 0x7e; 6793 6794 // FIXME: double check endinaness 6795 foreach(i; 0 .. 2) { 6796 headerScratch[2 + 1 - i] = rlc & 0x0ff; 6797 rlc >>>= 8; 6798 } 6799 6800 headerScratchPos += 2; 6801 } else { 6802 // use 7 bit length 6803 b2 |= realLength & 0b_0111_1111; 6804 } 6805 6806 headerScratch[1] = b2; 6807 } 6808 6809 //assert(!masked, "masking key not properly implemented"); 6810 if(masked) { 6811 // FIXME: randomize this 6812 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 6813 headerScratchPos += 4; 6814 6815 // we'll just mask it in place... 6816 int keyIdx = 0; 6817 foreach(i; 0 .. data.length) { 6818 data[i] = data[i] ^ maskingKey[keyIdx]; 6819 if(keyIdx == 3) 6820 keyIdx = 0; 6821 else 6822 keyIdx++; 6823 } 6824 } 6825 6826 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 6827 llsend(headerScratch[0 .. headerScratchPos]); 6828 llsend(data); 6829 } 6830 6831 static WebSocketFrame read(ref ubyte[] d) { 6832 WebSocketFrame msg; 6833 6834 auto orig = d; 6835 6836 WebSocketFrame needsMoreData() { 6837 d = orig; 6838 return WebSocketFrame.init; 6839 } 6840 6841 if(d.length < 2) 6842 return needsMoreData(); 6843 6844 ubyte b = d[0]; 6845 6846 msg.populated = true; 6847 6848 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 6849 b >>= 4; 6850 msg.rsv3 = b & 0x01; 6851 b >>= 1; 6852 msg.rsv2 = b & 0x01; 6853 b >>= 1; 6854 msg.rsv1 = b & 0x01; 6855 b >>= 1; 6856 msg.fin = b & 0x01; 6857 6858 b = d[1]; 6859 msg.masked = (b & 0b1000_0000) ? true : false; 6860 msg.lengthIndicator = b & 0b0111_1111; 6861 6862 d = d[2 .. $]; 6863 6864 if(msg.lengthIndicator == 0x7e) { 6865 // 16 bit length 6866 msg.realLength = 0; 6867 6868 if(d.length < 2) return needsMoreData(); 6869 6870 foreach(i; 0 .. 2) { 6871 msg.realLength |= d[0] << ((1-i) * 8); 6872 d = d[1 .. $]; 6873 } 6874 } else if(msg.lengthIndicator == 0x7f) { 6875 // 64 bit length 6876 msg.realLength = 0; 6877 6878 if(d.length < 8) return needsMoreData(); 6879 6880 foreach(i; 0 .. 8) { 6881 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 6882 d = d[1 .. $]; 6883 } 6884 } else { 6885 // 7 bit length 6886 msg.realLength = msg.lengthIndicator; 6887 } 6888 6889 if(msg.masked) { 6890 6891 if(d.length < 4) return needsMoreData(); 6892 6893 msg.maskingKey = d[0 .. 4]; 6894 d = d[4 .. $]; 6895 } 6896 6897 if(msg.realLength > d.length) { 6898 return needsMoreData(); 6899 } 6900 6901 msg.data = d[0 .. cast(size_t) msg.realLength]; 6902 d = d[cast(size_t) msg.realLength .. $]; 6903 6904 return msg; 6905 } 6906 6907 void unmaskInPlace() { 6908 if(this.masked) { 6909 int keyIdx = 0; 6910 foreach(i; 0 .. this.data.length) { 6911 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 6912 if(keyIdx == 3) 6913 keyIdx = 0; 6914 else 6915 keyIdx++; 6916 } 6917 } 6918 } 6919 6920 char[] textData() { 6921 return cast(char[]) data; 6922 } 6923 } 6924 /* } */ 6925 } 6926 6927 6928 version(Windows) 6929 { 6930 version(CRuntime_DigitalMars) 6931 { 6932 extern(C) int setmode(int, int) nothrow @nogc; 6933 } 6934 else version(CRuntime_Microsoft) 6935 { 6936 extern(C) int _setmode(int, int) nothrow @nogc; 6937 alias setmode = _setmode; 6938 } 6939 else static assert(0); 6940 } 6941 6942 version(Posix) { 6943 import core.sys.posix.unistd; 6944 version(CRuntime_Musl) {} else { 6945 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 6946 } 6947 } 6948 6949 6950 // FIXME: these aren't quite public yet. 6951 //private: 6952 6953 // template for laziness 6954 void startAddonServer()(string arg) { 6955 version(OSX) { 6956 assert(0, "Not implemented"); 6957 } else version(linux) { 6958 import core.sys.posix.unistd; 6959 pid_t pid; 6960 const(char)*[16] args; 6961 args[0] = "ARSD_CGI_ADDON_SERVER"; 6962 args[1] = arg.ptr; 6963 posix_spawn(&pid, "/proc/self/exe", 6964 null, 6965 null, 6966 args.ptr, 6967 null // env 6968 ); 6969 } else version(Windows) { 6970 wchar[2048] filename; 6971 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 6972 if(len == 0 || len == filename.length) 6973 throw new Exception("could not get process name to start helper server"); 6974 6975 STARTUPINFOW startupInfo; 6976 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 6977 PROCESS_INFORMATION processInfo; 6978 6979 import std.utf; 6980 6981 // I *MIGHT* need to run it as a new job or a service... 6982 auto ret = CreateProcessW( 6983 filename.ptr, 6984 toUTF16z(arg), 6985 null, // process attributes 6986 null, // thread attributes 6987 false, // inherit handles 6988 0, // creation flags 6989 null, // environment 6990 null, // working directory 6991 &startupInfo, 6992 &processInfo 6993 ); 6994 6995 if(!ret) 6996 throw new Exception("create process failed"); 6997 6998 // when done with those, if we set them 6999 /* 7000 CloseHandle(hStdInput); 7001 CloseHandle(hStdOutput); 7002 CloseHandle(hStdError); 7003 */ 7004 7005 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7006 } 7007 7008 // template for laziness 7009 /* 7010 The websocket server is a single-process, single-thread, event 7011 I/O thing. It is passed websockets from other CGI processes 7012 and is then responsible for handling their messages and responses. 7013 Note that the CGI process is responsible for websocket setup, 7014 including authentication, etc. 7015 7016 It also gets data sent to it by other processes and is responsible 7017 for distributing that, as necessary. 7018 */ 7019 void runWebsocketServer()() { 7020 assert(0, "not implemented"); 7021 } 7022 7023 void sendToWebsocketServer(WebSocket ws, string group) { 7024 assert(0, "not implemented"); 7025 } 7026 7027 void sendToWebsocketServer(string content, string group) { 7028 assert(0, "not implemented"); 7029 } 7030 7031 7032 void runEventServer()() { 7033 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7034 } 7035 7036 void runTimerServer()() { 7037 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7038 } 7039 7040 version(Posix) { 7041 alias LocalServerConnectionHandle = int; 7042 alias CgiConnectionHandle = int; 7043 alias SocketConnectionHandle = int; 7044 7045 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7046 } else version(Windows) { 7047 alias LocalServerConnectionHandle = HANDLE; 7048 version(embedded_httpd_threads) { 7049 alias CgiConnectionHandle = SOCKET; 7050 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7051 } else version(fastcgi) { 7052 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7053 enum INVALID_CGI_CONNECTION_HANDLE = null; 7054 } else version(scgi) { 7055 alias CgiConnectionHandle = SOCKET; 7056 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7057 } else { /* version(plain_cgi) */ 7058 alias CgiConnectionHandle = HANDLE; 7059 enum INVALID_CGI_CONNECTION_HANDLE = null; 7060 } 7061 alias SocketConnectionHandle = SOCKET; 7062 } 7063 7064 version(with_addon_servers_connections) 7065 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7066 version(Posix) { 7067 import core.sys.posix.unistd; 7068 import core.sys.posix.sys.un; 7069 7070 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7071 if(sock == -1) 7072 throw new Exception("socket " ~ to!string(errno)); 7073 7074 scope(failure) 7075 close(sock); 7076 7077 cloexec(sock); 7078 7079 // add-on server processes are assumed to be local, and thus will 7080 // use unix domain sockets. Besides, I want to pass sockets to them, 7081 // so it basically must be local (except for the session server, but meh). 7082 sockaddr_un addr; 7083 addr.sun_family = AF_UNIX; 7084 version(linux) { 7085 // on linux, we will use the abstract namespace 7086 addr.sun_path[0] = 0; 7087 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7088 } else { 7089 // but otherwise, just use a file cuz we must. 7090 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7091 } 7092 7093 bool alreadyTried; 7094 7095 try_again: 7096 7097 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7098 if(!alreadyTried && errno == ECONNREFUSED) { 7099 // try auto-spawning the server, then attempt connection again 7100 startAddonServer(arg); 7101 import core.thread; 7102 Thread.sleep(50.msecs); 7103 alreadyTried = true; 7104 goto try_again; 7105 } else 7106 throw new Exception("connect " ~ to!string(errno)); 7107 } 7108 7109 return sock; 7110 } else version(Windows) { 7111 return null; // FIXME 7112 } 7113 } 7114 7115 version(with_addon_servers_connections) 7116 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7117 version(Posix) { 7118 import core.sys.posix.unistd; 7119 close(handle); 7120 } else version(Windows) 7121 CloseHandle(handle); 7122 } 7123 7124 void runSessionServer()() { 7125 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7126 } 7127 7128 import core.stdc.errno; 7129 7130 struct IoOp { 7131 @disable this(); 7132 @disable this(this); 7133 7134 /* 7135 So we want to be able to eventually handle generic sockets too. 7136 */ 7137 7138 enum Read = 1; 7139 enum Write = 2; 7140 enum Accept = 3; 7141 enum ReadSocketHandle = 4; 7142 7143 // Your handler may be called in a different thread than the one that initiated the IO request! 7144 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7145 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7146 private void delegate(IoOp*) closeHandler; 7147 private void delegate(IoOp*) completeHandler; 7148 private int internalFd; 7149 private int operation; 7150 private int bufferLengthAllocated; 7151 private int bufferLengthUsed; 7152 private ubyte[1] internalBuffer; // it can be overallocated! 7153 7154 ubyte[] allocatedBuffer() return { 7155 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7156 } 7157 7158 ubyte[] usedBuffer() return { 7159 return allocatedBuffer[0 .. bufferLengthUsed]; 7160 } 7161 7162 void reset() { 7163 bufferLengthUsed = 0; 7164 } 7165 7166 int fd() { 7167 return internalFd; 7168 } 7169 } 7170 7171 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7172 import core.stdc.stdlib; 7173 7174 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7175 if(ptr is null) 7176 assert(0); // out of memory! 7177 7178 auto op = cast(IoOp*) ptr; 7179 7180 op.handler = handler; 7181 op.internalFd = fd; 7182 op.operation = operation; 7183 op.bufferLengthAllocated = bufferSize; 7184 op.bufferLengthUsed = 0; 7185 7186 import core.memory; 7187 7188 GC.addRoot(ptr); 7189 7190 return op; 7191 } 7192 7193 void freeIoOp(ref IoOp* ptr) { 7194 7195 import core.memory; 7196 GC.removeRoot(ptr); 7197 7198 import core.stdc.stdlib; 7199 free(ptr); 7200 ptr = null; 7201 } 7202 7203 version(Posix) 7204 version(with_addon_servers_connections) 7205 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7206 7207 //import std.stdio : writeln; writeln(cast(string) data); 7208 7209 import core.sys.posix.unistd; 7210 7211 auto ret = write(connection, data.ptr, data.length); 7212 if(ret != data.length) { 7213 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7214 // the file is closed, remove it 7215 eis.fileClosed(connection); 7216 } else 7217 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7218 } 7219 } 7220 version(Windows) 7221 version(with_addon_servers_connections) 7222 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7223 // FIXME 7224 } 7225 7226 bool isInvalidHandle(CgiConnectionHandle h) { 7227 return h == INVALID_CGI_CONNECTION_HANDLE; 7228 } 7229 7230 /+ 7231 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7232 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7233 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7234 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7235 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7236 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7237 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7238 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7239 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7240 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7241 7242 +/ 7243 7244 /++ 7245 You can customize your server by subclassing the appropriate server. Then, register your 7246 subclass at compile time with the [registerEventIoServer] template, or implement your own 7247 main function and call it yourself. 7248 7249 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7250 +/ 7251 version(with_addon_servers_connections) 7252 interface EventIoServer { 7253 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7254 void handleLocalConnectionClose(IoOp* op); 7255 void handleLocalConnectionComplete(IoOp* op); 7256 void wait_timeout(); 7257 void fileClosed(int fd); 7258 7259 void epoll_fd(int fd); 7260 } 7261 7262 // the sink should buffer it 7263 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7264 static if(is(T == struct)) { 7265 foreach(member; __traits(allMembers, T)) 7266 serialize(sink, __traits(getMember, t, member)); 7267 } else static if(is(T : int)) { 7268 // no need to think of endianness just because this is only used 7269 // for local, same-machine stuff anyway. thanks private lol 7270 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7271 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7272 // these are common enough to optimize 7273 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7274 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7275 sink(cast(ubyte[]) t[]); 7276 } else static if(is(T : A[], A)) { 7277 // generic array is less optimal but still prolly ok 7278 int len = cast(int) t.length; 7279 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7280 foreach(item; t) 7281 serialize(sink, item); 7282 } else static assert(0, T.stringof); 7283 } 7284 7285 // all may be stack buffers, so use cautio 7286 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7287 static if(is(T == struct)) { 7288 T t; 7289 foreach(member; __traits(allMembers, T)) 7290 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7291 dg(t); 7292 } else static if(is(T : int)) { 7293 // no need to think of endianness just because this is only used 7294 // for local, same-machine stuff anyway. thanks private lol 7295 T t; 7296 auto data = get(t.sizeof); 7297 t = (cast(T[]) data)[0]; 7298 dg(t); 7299 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7300 // these are common enough to optimize 7301 int len; 7302 auto data = get(len.sizeof); 7303 len = (cast(int[]) data)[0]; 7304 7305 /* 7306 typeof(T[0])[2000] stackBuffer; 7307 T buffer; 7308 7309 if(len < stackBuffer.length) 7310 buffer = stackBuffer[0 .. len]; 7311 else 7312 buffer = new T(len); 7313 7314 data = get(len * typeof(T[0]).sizeof); 7315 */ 7316 7317 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7318 7319 dg(t); 7320 } else static if(is(T == E[], E)) { 7321 T t; 7322 int len; 7323 auto data = get(len.sizeof); 7324 len = (cast(int[]) data)[0]; 7325 t.length = len; 7326 foreach(ref e; t) { 7327 deserialize!E(get, (ele) { e = ele; }); 7328 } 7329 dg(t); 7330 } else static assert(0, T.stringof); 7331 } 7332 7333 unittest { 7334 serialize((ubyte[] b) { 7335 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7336 }, 1); 7337 serialize((ubyte[] b) { 7338 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7339 }, 56674); 7340 ubyte[1000] buffer; 7341 int bufferPoint; 7342 void add(ubyte[] b) { 7343 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7344 bufferPoint += b.length; 7345 } 7346 ubyte[] get(int sz) { 7347 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7348 bufferPoint += sz; 7349 return b; 7350 } 7351 serialize(&add, "test here"); 7352 bufferPoint = 0; 7353 deserialize!string(&get, (t) { assert(t == "test here"); }); 7354 bufferPoint = 0; 7355 7356 struct Foo { 7357 int a; 7358 ubyte c; 7359 string d; 7360 } 7361 serialize(&add, Foo(403, 37, "amazing")); 7362 bufferPoint = 0; 7363 deserialize!Foo(&get, (t) { 7364 assert(t.a == 403); 7365 assert(t.c == 37); 7366 assert(t.d == "amazing"); 7367 }); 7368 bufferPoint = 0; 7369 } 7370 7371 /* 7372 Here's the way the RPC interface works: 7373 7374 You define the interface that lists the functions you can call on the remote process. 7375 The interface may also have static methods for convenience. These forward to a singleton 7376 instance of an auto-generated class, which actually sends the args over the pipe. 7377 7378 An impl class actually implements it. A receiving server deserializes down the pipe and 7379 calls methods on the class. 7380 7381 I went with the interface to get some nice compiler checking and documentation stuff. 7382 7383 I could have skipped the interface and just implemented it all from the server class definition 7384 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7385 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7386 7387 7388 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7389 7390 Realistically though the bodies would just be 7391 connection.call(this.mangleof, args...) sooooo. 7392 7393 FIXME: overloads aren't supported 7394 */ 7395 7396 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7397 interface SessionObject {} 7398 7399 private immutable void delegate(string[])[string] scheduledJobHandlers; 7400 private immutable void delegate(string[])[string] websocketServers; 7401 7402 version(with_breaking_cgi_features) 7403 mixin(q{ 7404 7405 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7406 static import std.traits; 7407 7408 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7409 static foreach(idx, member; __traits(derivedMembers, T)) { 7410 static if(__traits(isVirtualFunction, __traits(getMember, T, member))) 7411 mixin( q{ 7412 std.traits.ReturnType!(__traits(getMember, T, member)) 7413 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7414 { 7415 SerializationBuffer buffer; 7416 auto i = cast(ushort) idx; 7417 serialize(&buffer.sink, i); 7418 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7419 foreach(param; params) 7420 serialize(&buffer.sink, param); 7421 7422 auto sendable = buffer.sendable; 7423 7424 version(Posix) {{ 7425 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7426 7427 if(ret == -1) { 7428 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7429 } else if(ret == 0) { 7430 throw new Exception("Connection to addon server lost"); 7431 } if(ret < sendable.length) 7432 throw new Exception("Send failed to send all"); 7433 assert(ret == sendable.length); 7434 }} // FIXME Windows impl 7435 7436 static if(!is(typeof(return) == void)) { 7437 // there is a return value; we need to wait for it too 7438 version(Posix) { 7439 ubyte[3000] revBuffer; 7440 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7441 auto got = revBuffer[0 .. ret]; 7442 7443 int dataLocation; 7444 ubyte[] grab(int sz) { 7445 auto dataLocation1 = dataLocation; 7446 dataLocation += sz; 7447 return got[dataLocation1 .. dataLocation]; 7448 } 7449 7450 typeof(return) retu; 7451 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7452 return retu; 7453 } else { 7454 // FIXME Windows impl 7455 return typeof(return).init; 7456 } 7457 7458 } 7459 }}); 7460 } 7461 7462 private static typeof(this) singletonInstance; 7463 private LocalServerConnectionHandle connectionHandle; 7464 7465 static typeof(this) connection() { 7466 if(singletonInstance is null) { 7467 singletonInstance = new typeof(this)(); 7468 singletonInstance.connect(); 7469 } 7470 return singletonInstance; 7471 } 7472 7473 void connect() { 7474 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7475 } 7476 7477 void disconnect() { 7478 closeLocalServerConnection(connectionHandle); 7479 } 7480 } 7481 7482 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7483 ushort calledIdx; 7484 string calledFunction; 7485 7486 int dataLocation; 7487 ubyte[] grab(int sz) { 7488 if(sz == 0) assert(0); 7489 auto d = data[dataLocation .. dataLocation + sz]; 7490 dataLocation += sz; 7491 return d; 7492 } 7493 7494 again: 7495 7496 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7497 deserialize!string(&grab, (a) { calledFunction = a; }); 7498 7499 import std.traits; 7500 7501 sw: switch(calledIdx) { 7502 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7503 static if(__traits(isVirtualFunction, __traits(getMember, Interface, memberName))) { 7504 case idx: 7505 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7506 7507 Parameters!(__traits(getMember, Interface, memberName)) params; 7508 foreach(ref param; params) 7509 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7510 7511 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7512 __traits(getMember, this_, memberName)(params); 7513 } else { 7514 auto ret = __traits(getMember, this_, memberName)(params); 7515 SerializationBuffer buffer; 7516 serialize(&buffer.sink, ret); 7517 7518 auto sendable = buffer.sendable; 7519 7520 version(Posix) { 7521 auto r = send(fd, sendable.ptr, sendable.length, 0); 7522 if(r == -1) { 7523 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7524 } else if(r == 0) { 7525 throw new Exception("Connection to addon client lost"); 7526 } if(r < sendable.length) 7527 throw new Exception("Send failed to send all"); 7528 7529 } // FIXME Windows impl 7530 } 7531 break sw; 7532 } 7533 default: assert(0); 7534 } 7535 7536 if(dataLocation != data.length) 7537 goto again; 7538 } 7539 7540 7541 private struct SerializationBuffer { 7542 ubyte[2048] bufferBacking; 7543 int bufferLocation; 7544 void sink(scope ubyte[] data) { 7545 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7546 bufferLocation += data.length; 7547 } 7548 7549 ubyte[] sendable() return { 7550 return bufferBacking[0 .. bufferLocation]; 7551 } 7552 } 7553 7554 /* 7555 FIXME: 7556 add a version command line arg 7557 version data in the library 7558 management gui as external program 7559 7560 at server with event_fd for each run 7561 use .mangleof in the at function name 7562 7563 i think the at server will have to: 7564 pipe args to the child 7565 collect child output for logging 7566 get child return value for logging 7567 7568 on windows timers work differently. idk how to best combine with the io stuff. 7569 7570 will have to have dump and restore too, so i can restart without losing stuff. 7571 */ 7572 7573 /++ 7574 A convenience object for talking to the [BasicDataServer] from a higher level. 7575 See: [Cgi.getSessionObject]. 7576 7577 You pass it a `Data` struct describing the data you want saved in the session. 7578 Then, this class will generate getter and setter properties that allow access 7579 to that data. 7580 7581 Note that each load and store will be done as-accessed; it doesn't front-load 7582 mutable data nor does it batch updates out of fear of read-modify-write race 7583 conditions. (In fact, right now it does this for everything, but in the future, 7584 I might batch load `immutable` members of the Data struct.) 7585 7586 At some point in the future, I might also let it do different backends, like 7587 a client-side cookie store too, but idk. 7588 7589 Note that the plain-old-data members of your `Data` struct are wrapped by this 7590 interface via a static foreach to make property functions. 7591 7592 See_Also: [MockSession] 7593 +/ 7594 interface Session(Data) : SessionObject { 7595 @property string sessionId() const; 7596 7597 /++ 7598 Starts a new session. Note that a session is also 7599 implicitly started as soon as you write data to it, 7600 so if you need to alter these parameters from their 7601 defaults, be sure to explicitly call this BEFORE doing 7602 any writes to session data. 7603 7604 Params: 7605 idleLifetime = How long, in seconds, the session 7606 should remain in memory when not being read from 7607 or written to. The default is one day. 7608 7609 NOT IMPLEMENTED 7610 7611 useExtendedLifetimeCookie = The session ID is always 7612 stored in a HTTP cookie, and by default, that cookie 7613 is discarded when the user closes their browser. 7614 7615 But if you set this to true, it will use a non-perishable 7616 cookie for the given idleLifetime. 7617 7618 NOT IMPLEMENTED 7619 +/ 7620 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7621 7622 /++ 7623 Regenerates the session ID and updates the associated 7624 cookie. 7625 7626 This is also your chance to change immutable data 7627 (not yet implemented). 7628 +/ 7629 void regenerateId(); 7630 7631 /++ 7632 Terminates this session, deleting all saved data. 7633 +/ 7634 void terminate(); 7635 7636 /++ 7637 Plain-old-data members of your `Data` struct are wrapped here via 7638 the property getters and setters. 7639 7640 If the member is a non-string array, it returns a magical array proxy 7641 object which allows for atomic appends and replaces via overloaded operators. 7642 You can slice this to get a range representing a $(B const) view of the array. 7643 This is to protect you against read-modify-write race conditions. 7644 +/ 7645 static foreach(memberName; __traits(allMembers, Data)) 7646 static if(is(typeof(__traits(getMember, Data, memberName)))) 7647 mixin(q{ 7648 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7649 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7650 }); 7651 7652 } 7653 7654 /++ 7655 An implementation of [Session] that works on real cgi connections utilizing the 7656 [BasicDataServer]. 7657 7658 As opposed to a [MockSession] which is made for testing purposes. 7659 7660 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7661 +/ 7662 class BasicDataServerSession(Data) : Session!Data { 7663 private Cgi cgi; 7664 private string sessionId_; 7665 7666 public @property string sessionId() const { 7667 return sessionId_; 7668 } 7669 7670 protected @property string sessionId(string s) { 7671 return this.sessionId_ = s; 7672 } 7673 7674 private this(Cgi cgi) { 7675 this.cgi = cgi; 7676 if(auto ptr = "sessionId" in cgi.cookies) 7677 sessionId = (*ptr).length ? *ptr : null; 7678 } 7679 7680 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7681 assert(sessionId is null); 7682 7683 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7684 7685 import std.random, std.conv; 7686 sessionId = to!string(uniform(1, long.max)); 7687 7688 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7689 setCookie(); 7690 } 7691 7692 protected void setCookie() { 7693 cgi.setCookie( 7694 "sessionId", sessionId, 7695 0 /* expiration */, 7696 "/" /* path */, 7697 null /* domain */, 7698 true /* http only */, 7699 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 7700 } 7701 7702 void regenerateId() { 7703 if(sessionId is null) { 7704 start(); 7705 return; 7706 } 7707 import std.random, std.conv; 7708 auto oldSessionId = sessionId; 7709 sessionId = to!string(uniform(1, long.max)); 7710 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 7711 setCookie(); 7712 } 7713 7714 void terminate() { 7715 BasicDataServer.connection.destroySession(sessionId); 7716 sessionId = null; 7717 setCookie(); 7718 } 7719 7720 static foreach(memberName; __traits(allMembers, Data)) 7721 static if(is(typeof(__traits(getMember, Data, memberName)))) 7722 mixin(q{ 7723 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7724 if(sessionId is null) 7725 return typeof(return).init; 7726 7727 import std.traits; 7728 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 7729 if(v.length == 0) 7730 return typeof(return).init; 7731 import std.conv; 7732 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 7733 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 7734 // basically. Assuming the session is POD this should be fine. 7735 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 7736 } 7737 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 7738 if(sessionId is null) 7739 start(); 7740 import std.conv; 7741 import std.traits; 7742 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 7743 return value; 7744 } 7745 }); 7746 } 7747 7748 /++ 7749 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 7750 Simply stores the data in its instance members. 7751 +/ 7752 class MockSession(Data) : Session!Data { 7753 pure { 7754 @property string sessionId() const { return "mock"; } 7755 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 7756 void regenerateId() {} 7757 void terminate() {} 7758 7759 private Data store_; 7760 7761 static foreach(memberName; __traits(allMembers, Data)) 7762 static if(is(typeof(__traits(getMember, Data, memberName)))) 7763 mixin(q{ 7764 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7765 return __traits(getMember, store_, memberName); 7766 } 7767 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 7768 return __traits(getMember, store_, memberName) = value; 7769 } 7770 }); 7771 } 7772 } 7773 7774 /++ 7775 Direct interface to the basic data add-on server. You can 7776 typically use [Cgi.getSessionObject] as a more convenient interface. 7777 +/ 7778 version(with_addon_servers_connections) 7779 interface BasicDataServer { 7780 /// 7781 void createSession(string sessionId, int lifetime); 7782 /// 7783 void renewSession(string sessionId, int lifetime); 7784 /// 7785 void destroySession(string sessionId); 7786 /// 7787 void renameSession(string oldSessionId, string newSessionId); 7788 7789 /// 7790 void setSessionData(string sessionId, string dataKey, string dataValue); 7791 /// 7792 string getSessionData(string sessionId, string dataKey); 7793 7794 /// 7795 static BasicDataServerConnection connection() { 7796 return BasicDataServerConnection.connection(); 7797 } 7798 } 7799 7800 version(with_addon_servers_connections) 7801 class BasicDataServerConnection : BasicDataServer { 7802 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 7803 } 7804 7805 version(with_addon_servers) 7806 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 7807 7808 void createSession(string sessionId, int lifetime) { 7809 sessions[sessionId.idup] = Session(lifetime); 7810 } 7811 void destroySession(string sessionId) { 7812 sessions.remove(sessionId); 7813 } 7814 void renewSession(string sessionId, int lifetime) { 7815 sessions[sessionId].lifetime = lifetime; 7816 } 7817 void renameSession(string oldSessionId, string newSessionId) { 7818 sessions[newSessionId.idup] = sessions[oldSessionId]; 7819 sessions.remove(oldSessionId); 7820 } 7821 void setSessionData(string sessionId, string dataKey, string dataValue) { 7822 if(sessionId !in sessions) 7823 createSession(sessionId, 3600); // FIXME? 7824 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 7825 } 7826 string getSessionData(string sessionId, string dataKey) { 7827 if(auto session = sessionId in sessions) { 7828 if(auto data = dataKey in (*session).values) 7829 return *data; 7830 else 7831 return null; // no such data 7832 7833 } else { 7834 return null; // no session 7835 } 7836 } 7837 7838 7839 protected: 7840 7841 struct Session { 7842 int lifetime; 7843 7844 string[string] values; 7845 } 7846 7847 Session[string] sessions; 7848 7849 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 7850 auto data = op.usedBuffer; 7851 dispatchRpcServer!BasicDataServer(this, data, op.fd); 7852 return false; 7853 } 7854 7855 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 7856 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 7857 void wait_timeout() {} 7858 void fileClosed(int fd) {} // stateless so irrelevant 7859 void epoll_fd(int fd) {} 7860 } 7861 7862 /++ 7863 See [schedule] to make one of these. You then call one of the methods here to set it up: 7864 7865 --- 7866 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 7867 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 7868 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 7869 --- 7870 +/ 7871 version(with_addon_servers_connections) 7872 struct ScheduledJobHelper { 7873 private string func; 7874 private string[] args; 7875 private bool consumed; 7876 7877 private this(string func, string[] args) { 7878 this.func = func; 7879 this.args = args; 7880 } 7881 7882 ~this() { 7883 assert(consumed); 7884 } 7885 7886 /++ 7887 Schedules the job to be run at the given time. 7888 +/ 7889 void at(DateTime when, immutable TimeZone timezone = UTC()) { 7890 consumed = true; 7891 7892 auto conn = ScheduledJobServerConnection.connection; 7893 import std.file; 7894 auto st = SysTime(when, timezone); 7895 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 7896 } 7897 7898 /++ 7899 Schedules the job to run at least after the specified delay. 7900 +/ 7901 void delay(Duration delay) { 7902 consumed = true; 7903 7904 auto conn = ScheduledJobServerConnection.connection; 7905 import std.file; 7906 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 7907 } 7908 7909 /++ 7910 Runs the job in the background ASAP. 7911 7912 $(NOTE It may run in a background thread. Don't segfault!) 7913 +/ 7914 void asap() { 7915 consumed = true; 7916 7917 auto conn = ScheduledJobServerConnection.connection; 7918 import std.file; 7919 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 7920 } 7921 7922 /+ 7923 /++ 7924 Schedules the job to recur on the given pattern. 7925 +/ 7926 void recur(string spec) { 7927 7928 } 7929 +/ 7930 } 7931 7932 /++ 7933 First step to schedule a job on the scheduled job server. 7934 7935 The scheduled job needs to be a top-level function that doesn't read any 7936 variables from outside its arguments because it may be run in a new process, 7937 without any context existing later. 7938 7939 You MUST set details on the returned object to actually do anything! 7940 +/ 7941 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 7942 /// 7943 ScheduledJobHelper schedule(T args) { 7944 // this isn't meant to ever be called, but instead just to 7945 // get the compiler to type check the arguments passed for us 7946 auto sample = delegate() { 7947 fn(args); 7948 }; 7949 string[] sargs; 7950 foreach(arg; args) 7951 sargs ~= to!string(arg); 7952 return ScheduledJobHelper(fn.mangleof, sargs); 7953 } 7954 7955 shared static this() { 7956 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 7957 import std.traits; 7958 Parameters!fn args; 7959 foreach(idx, ref arg; args) 7960 arg = to!(typeof(arg))(sargs[idx]); 7961 fn(args); 7962 }; 7963 } 7964 } 7965 7966 /// 7967 interface ScheduledJobServer { 7968 /// Use the [schedule] function for a higher-level interface. 7969 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 7970 /// 7971 void cancelJob(int jobId); 7972 } 7973 7974 version(with_addon_servers_connections) 7975 class ScheduledJobServerConnection : ScheduledJobServer { 7976 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 7977 } 7978 7979 version(with_addon_servers) 7980 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 7981 // FIXME: we need to handle SIGCHLD in this somehow 7982 // whenIs is 0 for relative, 1 for absolute 7983 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 7984 auto nj = nextJobId; 7985 nextJobId++; 7986 7987 version(linux) { 7988 import core.sys.linux.timerfd; 7989 import core.sys.linux.epoll; 7990 import core.sys.posix.unistd; 7991 7992 7993 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 7994 if(fd == -1) 7995 throw new Exception("fd timer create failed"); 7996 7997 foreach(ref arg; args) 7998 arg = arg.idup; 7999 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8000 8001 itimerspec value; 8002 value.it_value.tv_sec = when; 8003 value.it_value.tv_nsec = 0; 8004 8005 value.it_interval.tv_sec = 0; 8006 value.it_interval.tv_nsec = 0; 8007 8008 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8009 throw new Exception("couldn't set fd timer"); 8010 8011 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8012 jobs.remove(nj); 8013 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8014 close(fd); 8015 8016 8017 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8018 8019 return true; 8020 }); 8021 scope(failure) 8022 freeIoOp(op); 8023 8024 epoll_event ev; 8025 ev.events = EPOLLIN | EPOLLET; 8026 ev.data.ptr = op; 8027 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8028 throw new Exception("epoll_ctl " ~ to!string(errno)); 8029 8030 jobs[nj] = job; 8031 return nj; 8032 } else assert(0); 8033 } 8034 8035 protected void cancelJob(int jobId) { 8036 version(linux) { 8037 auto job = jobId in jobs; 8038 if(job is null) 8039 return; 8040 8041 jobs.remove(jobId); 8042 8043 version(linux) { 8044 import core.sys.linux.timerfd; 8045 import core.sys.linux.epoll; 8046 import core.sys.posix.unistd; 8047 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8048 close(job.timerfd); 8049 } 8050 } 8051 jobs.remove(jobId); 8052 } 8053 8054 int nextJobId = 1; 8055 static struct Job { 8056 string executable; 8057 string func; 8058 string[] args; 8059 int timerfd; 8060 int id; 8061 } 8062 Job[int] jobs; 8063 8064 8065 // event io server methods below 8066 8067 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8068 auto data = op.usedBuffer; 8069 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8070 return false; 8071 } 8072 8073 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8074 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8075 void wait_timeout() {} 8076 void fileClosed(int fd) {} // stateless so irrelevant 8077 8078 int epoll_fd_; 8079 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8080 int epoll_fd() { return epoll_fd_; } 8081 } 8082 8083 /// 8084 version(with_addon_servers_connections) 8085 interface EventSourceServer { 8086 /++ 8087 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8088 8089 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8090 8091 See_Also: 8092 [sendEvent] 8093 +/ 8094 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8095 /* 8096 If lastEventId is missing or empty, you just get new events as they come. 8097 8098 If it is set from something else, it sends all since then (that are still alive) 8099 down the pipe immediately. 8100 8101 The reason it can come from the header is that's what the standard defines for 8102 browser reconnects. The reason it can come from a query string is just convenience 8103 in catching up in a user-defined manner. 8104 8105 The reason the header overrides the query string is if the browser tries to reconnect, 8106 it will send the header AND the query (it reconnects to the same url), so we just 8107 want to do the restart thing. 8108 8109 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8110 */ 8111 string lastEventId = cgi.lastEventId; 8112 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8113 lastEventId = cgi.get["lastEventId"]; 8114 8115 cgi.setResponseContentType("text/event-stream"); 8116 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8117 cgi.flush(); 8118 8119 cgi.closed = true; 8120 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8121 scope(exit) 8122 closeLocalServerConnection(s); 8123 8124 version(fastcgi) 8125 throw new Exception("sending fcgi connections not supported"); 8126 else { 8127 auto fd = cgi.getOutputFileHandle(); 8128 if(isInvalidHandle(fd)) 8129 throw new Exception("bad fd from cgi!"); 8130 8131 EventSourceServerImplementation.SendableEventConnection sec; 8132 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8133 8134 version(Posix) { 8135 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8136 assert(res == sec.sizeof); 8137 } else version(Windows) { 8138 // FIXME 8139 } 8140 } 8141 } 8142 8143 /++ 8144 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8145 8146 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8147 8148 Params: 8149 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8150 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8151 data = the event data. Available in JS as `event.data`. 8152 lifetime = the amount of time to keep this event for replaying on the event server. 8153 8154 See_Also: 8155 [sendEventToEventServer] 8156 +/ 8157 public static void sendEvent(string url, string event, string data, int lifetime) { 8158 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8159 scope(exit) 8160 closeLocalServerConnection(s); 8161 8162 EventSourceServerImplementation.SendableEvent sev; 8163 sev.populate(url, event, data, lifetime); 8164 8165 version(Posix) { 8166 auto ret = send(s, &sev, sev.sizeof, 0); 8167 assert(ret == sev.sizeof); 8168 } else version(Windows) { 8169 // FIXME 8170 } 8171 } 8172 8173 /++ 8174 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8175 8176 See_Also: [disconnect] 8177 +/ 8178 void connect(string url, string forwardUrl); 8179 8180 /++ 8181 Disconnects `forwardUrl` from `url` 8182 8183 See_Also: [connect] 8184 +/ 8185 void disconnect(string url, string forwardUrl); 8186 } 8187 8188 /// 8189 version(with_addon_servers) 8190 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8191 8192 protected: 8193 8194 void connect(string url, string forwardUrl) { 8195 pipes[url] ~= forwardUrl; 8196 } 8197 void disconnect(string url, string forwardUrl) { 8198 auto t = url in pipes; 8199 if(t is null) 8200 return; 8201 foreach(idx, n; (*t)) 8202 if(n == forwardUrl) { 8203 (*t)[idx] = (*t)[$-1]; 8204 (*t) = (*t)[0 .. $-1]; 8205 break; 8206 } 8207 } 8208 8209 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8210 if(receivedFd != -1) { 8211 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8212 8213 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8214 8215 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8216 8217 auto url = got.url.idup; 8218 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8219 8220 // FIXME: catch up on past messages here 8221 } else { 8222 auto data = op.usedBuffer; 8223 auto event = cast(SendableEvent*) data.ptr; 8224 8225 if(event.magic == 0xdeadbeef) { 8226 handleInputEvent(event); 8227 8228 if(event.url in pipes) 8229 foreach(pipe; pipes[event.url]) { 8230 event.url = pipe; 8231 handleInputEvent(event); 8232 } 8233 } else { 8234 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8235 } 8236 } 8237 return false; 8238 } 8239 void handleLocalConnectionClose(IoOp* op) { 8240 fileClosed(op.fd); 8241 } 8242 void handleLocalConnectionComplete(IoOp* op) {} 8243 8244 void wait_timeout() { 8245 // just keeping alive 8246 foreach(url, connections; eventConnectionsByUrl) 8247 foreach(connection; connections) 8248 if(connection.needsChunking) 8249 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8250 else 8251 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8252 } 8253 8254 void fileClosed(int fd) { 8255 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8256 foreach(idx, conn; connections) { 8257 if(fd == conn.fd) { 8258 connections[idx] = connections[$-1]; 8259 connections = connections[0 .. $ - 1]; 8260 continue outer; 8261 } 8262 } 8263 } 8264 } 8265 8266 void epoll_fd(int fd) {} 8267 8268 8269 private: 8270 8271 8272 struct SendableEventConnection { 8273 ubyte responseChunked; 8274 8275 int urlLength; 8276 char[256] urlBuffer = 0; 8277 8278 int lastEventIdLength; 8279 char[32] lastEventIdBuffer = 0; 8280 8281 char[] url() return { 8282 return urlBuffer[0 .. urlLength]; 8283 } 8284 void url(in char[] u) { 8285 urlBuffer[0 .. u.length] = u[]; 8286 urlLength = cast(int) u.length; 8287 } 8288 char[] lastEventId() return { 8289 return lastEventIdBuffer[0 .. lastEventIdLength]; 8290 } 8291 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8292 in { 8293 assert(url.length < this.urlBuffer.length); 8294 assert(lastEventId.length < this.lastEventIdBuffer.length); 8295 } 8296 do { 8297 this.responseChunked = responseChunked ? 1 : 0; 8298 this.urlLength = cast(int) url.length; 8299 this.lastEventIdLength = cast(int) lastEventId.length; 8300 8301 this.urlBuffer[0 .. url.length] = url[]; 8302 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8303 } 8304 } 8305 8306 struct SendableEvent { 8307 int magic = 0xdeadbeef; 8308 int urlLength; 8309 char[256] urlBuffer = 0; 8310 int typeLength; 8311 char[32] typeBuffer = 0; 8312 int messageLength; 8313 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8314 int _lifetime; 8315 8316 char[] message() return { 8317 return messageBuffer[0 .. messageLength]; 8318 } 8319 char[] type() return { 8320 return typeBuffer[0 .. typeLength]; 8321 } 8322 char[] url() return { 8323 return urlBuffer[0 .. urlLength]; 8324 } 8325 void url(in char[] u) { 8326 urlBuffer[0 .. u.length] = u[]; 8327 urlLength = cast(int) u.length; 8328 } 8329 int lifetime() { 8330 return _lifetime; 8331 } 8332 8333 /// 8334 void populate(string url, string type, string message, int lifetime) 8335 in { 8336 assert(url.length < this.urlBuffer.length); 8337 assert(type.length < this.typeBuffer.length); 8338 assert(message.length < this.messageBuffer.length); 8339 } 8340 do { 8341 this.urlLength = cast(int) url.length; 8342 this.typeLength = cast(int) type.length; 8343 this.messageLength = cast(int) message.length; 8344 this._lifetime = lifetime; 8345 8346 this.urlBuffer[0 .. url.length] = url[]; 8347 this.typeBuffer[0 .. type.length] = type[]; 8348 this.messageBuffer[0 .. message.length] = message[]; 8349 } 8350 } 8351 8352 struct EventConnection { 8353 int fd; 8354 bool needsChunking; 8355 } 8356 8357 private EventConnection[][string] eventConnectionsByUrl; 8358 private string[][string] pipes; 8359 8360 private void handleInputEvent(scope SendableEvent* event) { 8361 static int eventId; 8362 8363 static struct StoredEvent { 8364 int id; 8365 string type; 8366 string message; 8367 int lifetimeRemaining; 8368 } 8369 8370 StoredEvent[][string] byUrl; 8371 8372 int thisId = ++eventId; 8373 8374 if(event.lifetime) 8375 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8376 8377 auto connectionsPtr = event.url in eventConnectionsByUrl; 8378 EventConnection[] connections; 8379 if(connectionsPtr is null) 8380 return; 8381 else 8382 connections = *connectionsPtr; 8383 8384 char[4096] buffer; 8385 char[] formattedMessage; 8386 8387 void append(const char[] a) { 8388 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8389 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8390 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8391 } 8392 8393 import std.algorithm.iteration; 8394 8395 if(connections.length) { 8396 append("id: "); 8397 append(to!string(thisId)); 8398 append("\n"); 8399 8400 append("event: "); 8401 append(event.type); 8402 append("\n"); 8403 8404 foreach(line; event.message.splitter("\n")) { 8405 append("data: "); 8406 append(line); 8407 append("\n"); 8408 } 8409 8410 append("\n"); 8411 } 8412 8413 // chunk it for HTTP! 8414 auto len = toHex(formattedMessage.length); 8415 buffer[4 .. 6] = "\r\n"[]; 8416 buffer[4 - len.length .. 4] = len[]; 8417 buffer[6 + formattedMessage.length] = '\r'; 8418 buffer[6 + formattedMessage.length + 1] = '\n'; 8419 8420 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8421 // done 8422 8423 // FIXME: send back requests when needed 8424 // FIXME: send a single ":\n" every 15 seconds to keep alive 8425 8426 foreach(connection; connections) { 8427 if(connection.needsChunking) { 8428 nonBlockingWrite(this, connection.fd, chunkedMessage); 8429 } else { 8430 nonBlockingWrite(this, connection.fd, formattedMessage); 8431 } 8432 } 8433 } 8434 } 8435 8436 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8437 version(Posix) { 8438 8439 import core.sys.posix.unistd; 8440 import core.sys.posix.fcntl; 8441 import core.sys.posix.sys.un; 8442 8443 import core.sys.posix.signal; 8444 signal(SIGPIPE, SIG_IGN); 8445 8446 static extern(C) void sigchldhandler(int) { 8447 int status; 8448 import w = core.sys.posix.sys.wait; 8449 w.wait(&status); 8450 } 8451 signal(SIGCHLD, &sigchldhandler); 8452 8453 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8454 if(sock == -1) 8455 throw new Exception("socket " ~ to!string(errno)); 8456 8457 scope(failure) 8458 close(sock); 8459 8460 cloexec(sock); 8461 8462 // add-on server processes are assumed to be local, and thus will 8463 // use unix domain sockets. Besides, I want to pass sockets to them, 8464 // so it basically must be local (except for the session server, but meh). 8465 sockaddr_un addr; 8466 addr.sun_family = AF_UNIX; 8467 version(linux) { 8468 // on linux, we will use the abstract namespace 8469 addr.sun_path[0] = 0; 8470 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8471 } else { 8472 // but otherwise, just use a file cuz we must. 8473 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8474 } 8475 8476 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8477 throw new Exception("bind " ~ to!string(errno)); 8478 8479 if(listen(sock, 128) == -1) 8480 throw new Exception("listen " ~ to!string(errno)); 8481 8482 makeNonBlocking(sock); 8483 8484 version(linux) { 8485 import core.sys.linux.epoll; 8486 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8487 if(epoll_fd == -1) 8488 throw new Exception("epoll_create1 " ~ to!string(errno)); 8489 scope(failure) 8490 close(epoll_fd); 8491 } else { 8492 import core.sys.posix.poll; 8493 } 8494 8495 version(linux) 8496 eis.epoll_fd = epoll_fd; 8497 8498 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8499 scope(exit) 8500 freeIoOp(acceptOp); 8501 8502 version(linux) { 8503 epoll_event ev; 8504 ev.events = EPOLLIN | EPOLLET; 8505 ev.data.ptr = acceptOp; 8506 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8507 throw new Exception("epoll_ctl " ~ to!string(errno)); 8508 8509 epoll_event[64] events; 8510 } else { 8511 pollfd[] pollfds; 8512 IoOp*[int] ioops; 8513 pollfds ~= pollfd(sock, POLLIN); 8514 ioops[sock] = acceptOp; 8515 } 8516 8517 import core.time : MonoTime, seconds; 8518 8519 MonoTime timeout = MonoTime.currTime + 15.seconds; 8520 8521 while(true) { 8522 8523 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8524 8525 int timeout_milliseconds = 0; // -1; // infinite 8526 8527 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8528 if(timeout_milliseconds < 0) 8529 timeout_milliseconds = 0; 8530 8531 //writeln("waiting for ", name); 8532 8533 version(linux) { 8534 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8535 if(nfds == -1) { 8536 if(errno == EINTR) 8537 continue; 8538 throw new Exception("epoll_wait " ~ to!string(errno)); 8539 } 8540 } else { 8541 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8542 size_t lastIdx = 0; 8543 } 8544 8545 if(nfds == 0) { 8546 eis.wait_timeout(); 8547 timeout += 15.seconds; 8548 } 8549 8550 foreach(idx; 0 .. nfds) { 8551 version(linux) { 8552 auto flags = events[idx].events; 8553 auto ioop = cast(IoOp*) events[idx].data.ptr; 8554 } else { 8555 IoOp* ioop; 8556 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8557 if(thing.revents) { 8558 ioop = ioops[thing.fd]; 8559 lastIdx += tidx + 1; 8560 break; 8561 } 8562 } 8563 } 8564 8565 //writeln(flags, " ", ioop.fd); 8566 8567 void newConnection() { 8568 // on edge triggering, it is important that we get it all 8569 while(true) { 8570 version(Android) { 8571 auto size = cast(int) addr.sizeof; 8572 } else { 8573 auto size = cast(uint) addr.sizeof; 8574 } 8575 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8576 if(ns == -1) { 8577 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8578 // all done, got it all 8579 break; 8580 } 8581 throw new Exception("accept " ~ to!string(errno)); 8582 } 8583 cloexec(ns); 8584 8585 makeNonBlocking(ns); 8586 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8587 niop.closeHandler = &eis.handleLocalConnectionClose; 8588 niop.completeHandler = &eis.handleLocalConnectionComplete; 8589 scope(failure) freeIoOp(niop); 8590 8591 version(linux) { 8592 epoll_event nev; 8593 nev.events = EPOLLIN | EPOLLET; 8594 nev.data.ptr = niop; 8595 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8596 throw new Exception("epoll_ctl " ~ to!string(errno)); 8597 } else { 8598 bool found = false; 8599 foreach(ref pfd; pollfds) { 8600 if(pfd.fd < 0) { 8601 pfd.fd = ns; 8602 found = true; 8603 } 8604 } 8605 if(!found) 8606 pollfds ~= pollfd(ns, POLLIN); 8607 ioops[ns] = niop; 8608 } 8609 } 8610 } 8611 8612 bool newConnectionCondition() { 8613 version(linux) 8614 return ioop.fd == sock && (flags & EPOLLIN); 8615 else 8616 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8617 } 8618 8619 if(newConnectionCondition()) { 8620 newConnection(); 8621 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8622 while(true) { 8623 int in_fd; 8624 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8625 if(got == -1) { 8626 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8627 // all done, got it all 8628 if(ioop.completeHandler) 8629 ioop.completeHandler(ioop); 8630 break; 8631 } 8632 throw new Exception("recv " ~ to!string(errno)); 8633 } 8634 8635 if(got == 0) { 8636 if(ioop.closeHandler) { 8637 ioop.closeHandler(ioop); 8638 version(linux) {} // nothing needed 8639 else { 8640 foreach(ref pfd; pollfds) { 8641 if(pfd.fd == ioop.fd) 8642 pfd.fd = -1; 8643 } 8644 } 8645 } 8646 close(ioop.fd); 8647 freeIoOp(ioop); 8648 break; 8649 } 8650 8651 ioop.bufferLengthUsed = cast(int) got; 8652 ioop.handler(ioop, in_fd); 8653 } 8654 } else if(ioop.operation == IoOp.Read) { 8655 while(true) { 8656 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8657 if(got == -1) { 8658 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8659 // all done, got it all 8660 if(ioop.completeHandler) 8661 ioop.completeHandler(ioop); 8662 break; 8663 } 8664 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8665 } 8666 8667 if(got == 0) { 8668 if(ioop.closeHandler) 8669 ioop.closeHandler(ioop); 8670 close(ioop.fd); 8671 freeIoOp(ioop); 8672 break; 8673 } 8674 8675 ioop.bufferLengthUsed = cast(int) got; 8676 if(ioop.handler(ioop, ioop.fd)) { 8677 close(ioop.fd); 8678 freeIoOp(ioop); 8679 break; 8680 } 8681 } 8682 } 8683 8684 // EPOLLHUP? 8685 } 8686 } 8687 } else version(Windows) { 8688 8689 // set up a named pipe 8690 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8691 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8692 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8693 8694 } else static assert(0); 8695 } 8696 8697 8698 version(with_sendfd) 8699 // copied from the web and ported from C 8700 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 8701 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 8702 msghdr msg; 8703 iovec[1] iov; 8704 8705 version(OSX) { 8706 //msg.msg_accrights = cast(cattr_t) &sendfd; 8707 //msg.msg_accrightslen = int.sizeof; 8708 } else version(Android) { 8709 } else { 8710 union ControlUnion { 8711 cmsghdr cm; 8712 char[CMSG_SPACE(int.sizeof)] control; 8713 } 8714 8715 ControlUnion control_un; 8716 cmsghdr* cmptr; 8717 8718 msg.msg_control = control_un.control.ptr; 8719 msg.msg_controllen = control_un.control.length; 8720 8721 cmptr = CMSG_FIRSTHDR(&msg); 8722 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 8723 cmptr.cmsg_level = SOL_SOCKET; 8724 cmptr.cmsg_type = SCM_RIGHTS; 8725 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 8726 } 8727 8728 msg.msg_name = null; 8729 msg.msg_namelen = 0; 8730 8731 iov[0].iov_base = ptr; 8732 iov[0].iov_len = nbytes; 8733 msg.msg_iov = iov.ptr; 8734 msg.msg_iovlen = 1; 8735 8736 return sendmsg(fd, &msg, 0); 8737 } 8738 8739 version(with_sendfd) 8740 // copied from the web and ported from C 8741 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 8742 msghdr msg; 8743 iovec[1] iov; 8744 ssize_t n; 8745 int newfd; 8746 8747 version(OSX) { 8748 //msg.msg_accrights = cast(cattr_t) recvfd; 8749 //msg.msg_accrightslen = int.sizeof; 8750 } else version(Android) { 8751 } else { 8752 union ControlUnion { 8753 cmsghdr cm; 8754 char[CMSG_SPACE(int.sizeof)] control; 8755 } 8756 ControlUnion control_un; 8757 cmsghdr* cmptr; 8758 8759 msg.msg_control = control_un.control.ptr; 8760 msg.msg_controllen = control_un.control.length; 8761 } 8762 8763 msg.msg_name = null; 8764 msg.msg_namelen = 0; 8765 8766 iov[0].iov_base = ptr; 8767 iov[0].iov_len = nbytes; 8768 msg.msg_iov = iov.ptr; 8769 msg.msg_iovlen = 1; 8770 8771 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 8772 return n; 8773 8774 version(OSX) { 8775 //if(msg.msg_accrightslen != int.sizeof) 8776 //*recvfd = -1; 8777 } else version(Android) { 8778 } else { 8779 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 8780 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 8781 if (cmptr.cmsg_level != SOL_SOCKET) 8782 throw new Exception("control level != SOL_SOCKET"); 8783 if (cmptr.cmsg_type != SCM_RIGHTS) 8784 throw new Exception("control type != SCM_RIGHTS"); 8785 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 8786 } else 8787 *recvfd = -1; /* descriptor was not passed */ 8788 } 8789 8790 return n; 8791 } 8792 /* end read_fd */ 8793 8794 8795 /* 8796 Event source stuff 8797 8798 The api is: 8799 8800 sendEvent(string url, string type, string data, int timeout = 60*10); 8801 8802 attachEventListener(string url, int fd, lastId) 8803 8804 8805 It just sends to all attached listeners, and stores it until the timeout 8806 for replaying via lastEventId. 8807 */ 8808 8809 /* 8810 Session process stuff 8811 8812 it stores it all. the cgi object has a session object that can grab it 8813 8814 session may be done in the same process if possible, there is a version 8815 switch to choose if you want to override. 8816 */ 8817 8818 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 8819 alias handler = dispatchHandler; 8820 string urlPrefix; 8821 bool rejectFurther; 8822 immutable(DispatcherDetails) details; 8823 } 8824 8825 private string urlify(string name) pure { 8826 return beautify(name, '-', true); 8827 } 8828 8829 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 8830 if(name == "id") 8831 return allLowerCase ? name : "ID"; 8832 8833 char[160] buffer; 8834 int bufferIndex = 0; 8835 bool shouldCap = true; 8836 bool shouldSpace; 8837 bool lastWasCap; 8838 foreach(idx, char ch; name) { 8839 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 8840 8841 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 8842 if(lastWasCap) { 8843 // two caps in a row, don't change. Prolly acronym. 8844 } else { 8845 if(idx) 8846 shouldSpace = true; // new word, add space 8847 } 8848 8849 lastWasCap = true; 8850 } else { 8851 lastWasCap = false; 8852 } 8853 8854 if(shouldSpace) { 8855 buffer[bufferIndex++] = space; 8856 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 8857 shouldSpace = false; 8858 } 8859 if(shouldCap) { 8860 if(ch >= 'a' && ch <= 'z') 8861 ch -= 32; 8862 shouldCap = false; 8863 } 8864 if(allLowerCase && ch >= 'A' && ch <= 'Z') 8865 ch += 32; 8866 buffer[bufferIndex++] = ch; 8867 } 8868 return buffer[0 .. bufferIndex].idup; 8869 } 8870 8871 /* 8872 string urlFor(alias func)() { 8873 return __traits(identifier, func); 8874 } 8875 */ 8876 8877 /++ 8878 UDA: The name displayed to the user in auto-generated HTML. 8879 8880 Default is `beautify(identifier)`. 8881 +/ 8882 struct DisplayName { 8883 string name; 8884 } 8885 8886 /++ 8887 UDA: The name used in the URL or web parameter. 8888 8889 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 8890 +/ 8891 struct UrlName { 8892 string name; 8893 } 8894 8895 /++ 8896 UDA: default format to respond for this method 8897 +/ 8898 struct DefaultFormat { string value; } 8899 8900 class MissingArgumentException : Exception { 8901 string functionName; 8902 string argumentName; 8903 string argumentType; 8904 8905 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 8906 this.functionName = functionName; 8907 this.argumentName = argumentName; 8908 this.argumentType = argumentType; 8909 8910 super("Missing Argument: " ~ this.argumentName, file, line, next); 8911 } 8912 } 8913 8914 /++ 8915 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 8916 8917 History: 8918 Added December 15, 2021 (dub v10.5) 8919 +/ 8920 class ResourceNotFoundException : Exception { 8921 string resourceType; 8922 string resourceId; 8923 8924 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 8925 this.resourceType = resourceType; 8926 this.resourceId = resourceId; 8927 8928 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 8929 } 8930 8931 } 8932 8933 /++ 8934 This can be attached to any constructor or function called from the cgi system. 8935 8936 If it is present, the function argument can NOT be set from web params, but instead 8937 is set to the return value of the given `func`. 8938 8939 If `func` can take a parameter of type [Cgi], it will be passed the one representing 8940 the current request. Otherwise, it must take zero arguments. 8941 8942 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 8943 for the connection. Any of type [Session] (with an argument) is also assumed to come from 8944 the cgi object. 8945 8946 const arguments are also supported. 8947 +/ 8948 struct ifCalledFromWeb(alias func) {} 8949 8950 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 8951 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 8952 8953 // FIXME: any array of structs should also be settable or gettable from csv as well. 8954 8955 // FIXME: think more about checkboxes and bools. 8956 8957 import std.traits; 8958 8959 Parameters!method params; 8960 alias idents = ParameterIdentifierTuple!method; 8961 alias defaults = ParameterDefaults!method; 8962 8963 const(string)[] names; 8964 const(string)[] values; 8965 8966 // first, check for missing arguments and initialize to defaults if necessary 8967 8968 static if(is(typeof(method) P == __parameters)) 8969 foreach(idx, param; P) {{ 8970 // see: mustNotBeSetFromWebParams 8971 static if(is(param : Cgi)) { 8972 static assert(!is(param == immutable)); 8973 cast() params[idx] = cgi; 8974 } else static if(is(param == Session!D, D)) { 8975 static assert(!is(param == immutable)); 8976 cast() params[idx] = cgi.getSessionObject!D(); 8977 } else { 8978 bool populated; 8979 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 8980 static if(is(uda == ifCalledFromWeb!func, alias func)) { 8981 static if(is(typeof(func(cgi)))) 8982 params[idx] = func(cgi); 8983 else 8984 params[idx] = func(); 8985 8986 populated = true; 8987 } 8988 } 8989 8990 if(!populated) { 8991 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 8992 params[idx] = param.getAutomaticallyForCgi(cgi); 8993 populated = true; 8994 } 8995 } 8996 8997 if(!populated) { 8998 auto ident = idents[idx]; 8999 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9000 if(ident !in cgi.get) { 9001 static if(is(defaults[idx] == void)) { 9002 static if(is(param == bool)) 9003 params[idx] = false; 9004 else 9005 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9006 } else 9007 params[idx] = defaults[idx]; 9008 } 9009 } else { 9010 if(ident !in cgi.post) { 9011 static if(is(defaults[idx] == void)) { 9012 static if(is(param == bool)) 9013 params[idx] = false; 9014 else 9015 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9016 } else 9017 params[idx] = defaults[idx]; 9018 } 9019 } 9020 } 9021 } 9022 }} 9023 9024 // second, parse the arguments in order to build up arrays, etc. 9025 9026 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9027 static if(is(T == struct)) { 9028 if(name == paramName) { 9029 *what = T.init; 9030 return true; 9031 } else { 9032 // could be a child. gonna allow either obj.field OR obj[field] 9033 9034 string afterName; 9035 9036 if(name[paramName.length] == '[') { 9037 int count = 1; 9038 auto idx = paramName.length + 1; 9039 while(idx < name.length && count > 0) { 9040 if(name[idx] == '[') 9041 count++; 9042 else if(name[idx] == ']') { 9043 count--; 9044 if(count == 0) break; 9045 } 9046 idx++; 9047 } 9048 9049 if(idx == name.length) 9050 return false; // malformed 9051 9052 auto insideBrackets = name[paramName.length + 1 .. idx]; 9053 afterName = name[idx + 1 .. $]; 9054 9055 name = name[0 .. paramName.length]; 9056 9057 paramName = insideBrackets; 9058 9059 } else if(name[paramName.length] == '.') { 9060 paramName = name[paramName.length + 1 .. $]; 9061 name = paramName; 9062 int p = 0; 9063 foreach(ch; paramName) { 9064 if(ch == '.' || ch == '[') 9065 break; 9066 p++; 9067 } 9068 9069 afterName = paramName[p .. $]; 9070 paramName = paramName[0 .. p]; 9071 } else { 9072 return false; 9073 } 9074 9075 if(paramName.length) 9076 // set the child member 9077 switch(paramName) { 9078 foreach(idx, memberName; __traits(allMembers, T)) 9079 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9080 // data member! 9081 case memberName: 9082 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9083 } 9084 default: 9085 // ok, not a member 9086 } 9087 } 9088 9089 return false; 9090 } else static if(is(T == enum)) { 9091 *what = to!T(value); 9092 return true; 9093 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9094 *what = to!T(value); 9095 return true; 9096 } else static if(is(T == bool)) { 9097 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9098 return true; 9099 } else static if(is(T == K[], K)) { 9100 K tmp; 9101 if(name == paramName) { 9102 // direct - set and append 9103 if(setVariable(name, paramName, &tmp, value)) { 9104 (*what) ~= tmp; 9105 return true; 9106 } else { 9107 return false; 9108 } 9109 } else { 9110 // child, append to last element 9111 // FIXME: what about range violations??? 9112 auto ptr = &(*what)[(*what).length - 1]; 9113 return setVariable(name, paramName, ptr, value); 9114 9115 } 9116 } else static if(is(T == V[K], K, V)) { 9117 // assoc array, name[key] is valid 9118 if(name == paramName) { 9119 // no action necessary 9120 return true; 9121 } else if(name[paramName.length] == '[') { 9122 int count = 1; 9123 auto idx = paramName.length + 1; 9124 while(idx < name.length && count > 0) { 9125 if(name[idx] == '[') 9126 count++; 9127 else if(name[idx] == ']') { 9128 count--; 9129 if(count == 0) break; 9130 } 9131 idx++; 9132 } 9133 if(idx == name.length) 9134 return false; // malformed 9135 9136 auto insideBrackets = name[paramName.length + 1 .. idx]; 9137 auto afterName = name[idx + 1 .. $]; 9138 9139 auto k = to!K(insideBrackets); 9140 V v; 9141 if(auto ptr = k in *what) 9142 v = *ptr; 9143 9144 name = name[0 .. paramName.length]; 9145 //writeln(name, afterName, " ", paramName); 9146 9147 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9148 if(ret) { 9149 (*what)[k] = v; 9150 return true; 9151 } 9152 } 9153 9154 return false; 9155 } else { 9156 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9157 } 9158 9159 //return false; 9160 } 9161 9162 void setArgument(string name, string value) { 9163 int p; 9164 foreach(ch; name) { 9165 if(ch == '.' || ch == '[') 9166 break; 9167 p++; 9168 } 9169 9170 auto paramName = name[0 .. p]; 9171 9172 sw: switch(paramName) { 9173 static if(is(typeof(method) P == __parameters)) 9174 foreach(idx, param; P) { 9175 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9176 // cannot be set from the outside 9177 } else { 9178 case idents[idx]: 9179 static if(is(param == Cgi.UploadedFile)) { 9180 params[idx] = cgi.files[name]; 9181 } else { 9182 setVariable(name, paramName, ¶ms[idx], value); 9183 } 9184 break sw; 9185 } 9186 } 9187 default: 9188 // ignore; not relevant argument 9189 } 9190 } 9191 9192 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9193 names = cgi.allGetNamesInOrder; 9194 values = cgi.allGetValuesInOrder; 9195 } else { 9196 names = cgi.allPostNamesInOrder; 9197 values = cgi.allPostValuesInOrder; 9198 } 9199 9200 foreach(idx, name; names) { 9201 setArgument(name, values[idx]); 9202 } 9203 9204 static if(is(ReturnType!method == void)) { 9205 typeof(null) ret; 9206 dg(params); 9207 } else { 9208 auto ret = dg(params); 9209 } 9210 9211 // FIXME: format return values 9212 // options are: json, html, csv. 9213 // also may need to wrap in envelope format: none, html, or json. 9214 return ret; 9215 } 9216 9217 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9218 static if(is(T : const(Cgi))) { 9219 return true; 9220 } else static if(is(T : const(Session!D), D)) { 9221 return true; 9222 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9223 return true; 9224 } else { 9225 foreach(uda; attrs) 9226 static if(is(uda == ifCalledFromWeb!func, alias func)) 9227 return true; 9228 return false; 9229 } 9230 } 9231 9232 private bool hasIfCalledFromWeb(attrs...)() { 9233 foreach(uda; attrs) 9234 static if(is(uda == ifCalledFromWeb!func, alias func)) 9235 return true; 9236 return false; 9237 } 9238 9239 /++ 9240 Implies POST path for the thing itself, then GET will get the automatic form. 9241 9242 The given customizer, if present, will be called as a filter on the Form object. 9243 9244 History: 9245 Added December 27, 2020 9246 +/ 9247 template AutomaticForm(alias customizer) { } 9248 9249 /++ 9250 This is meant to be returned by a function that takes a form POST submission. You 9251 want to set the url of the new resource it created, which is set as the http 9252 Location header for a "201 Created" result, and you can also set a separate 9253 destination for browser users, which it sets via a "Refresh" header. 9254 9255 The `resourceRepresentation` should generally be the thing you just created, and 9256 it will be the body of the http response when formatted through the presenter. 9257 The exact thing is up to you - it could just return an id, or the whole object, or 9258 perhaps a partial object. 9259 9260 Examples: 9261 --- 9262 class Test : WebObject { 9263 @(Cgi.RequestMethod.POST) 9264 CreatedResource!int makeThing(string value) { 9265 return CreatedResource!int(value.to!int, "/resources/id"); 9266 } 9267 } 9268 --- 9269 9270 History: 9271 Added December 18, 2021 9272 +/ 9273 struct CreatedResource(T) { 9274 static if(!is(T == void)) 9275 T resourceRepresentation; 9276 string resourceUrl; 9277 string refreshUrl; 9278 } 9279 9280 /+ 9281 /++ 9282 This can be attached as a UDA to a handler to add a http Refresh header on a 9283 successful run. (It will not be attached if the function throws an exception.) 9284 This will refresh the browser the given number of seconds after the page loads, 9285 to the url returned by `urlFunc`, which can be either a static function or a 9286 member method of the current handler object. 9287 9288 You might use this for a POST handler that is normally used from ajax, but you 9289 want it to degrade gracefully to a temporarily flashed message before reloading 9290 the main page. 9291 9292 History: 9293 Added December 18, 2021 9294 +/ 9295 struct Refresh(alias urlFunc) { 9296 int waitInSeconds; 9297 9298 string url() { 9299 static if(__traits(isStaticFunction, urlFunc)) 9300 return urlFunc(); 9301 else static if(is(urlFunc : string)) 9302 return urlFunc; 9303 } 9304 } 9305 +/ 9306 9307 /+ 9308 /++ 9309 Sets a filter to be run before 9310 9311 A before function can do validations of params and log and stop the function from running. 9312 +/ 9313 template Before(alias b) {} 9314 template After(alias b) {} 9315 +/ 9316 9317 /+ 9318 Argument conversions: for the most part, it is to!Thing(string). 9319 9320 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9321 they are passed 9322 9323 arr=foo&arr=bar <-- notice the same name. 9324 9325 Structs are first declared with an empty thing, then have their members set individually, 9326 with dot notation. The members are not required, just the initial declaration. 9327 9328 struct Foo { 9329 int a; 9330 string b; 9331 } 9332 void test(Foo foo){} 9333 9334 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9335 9336 Arrays of structs use this declaration. 9337 9338 void test(Foo[] foo) {} 9339 9340 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9341 9342 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9343 declaration is ignored. 9344 9345 Mind that order matters! The declaration MUST come first in the string. 9346 9347 Arrays of struct members follow this rule recursively. 9348 9349 struct Foo { 9350 int[] a; 9351 } 9352 9353 foo&foo.a=1&foo.a=2&foo&foo.a=1 9354 9355 9356 Associative arrays are formatted with brackets, after a declaration, like structs: 9357 9358 foo&foo[key]=value&foo[other_key]=value 9359 9360 9361 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9362 do not support the strict ordering requirements to work with these struct protocols. 9363 9364 FIXME: also perhaps accept application/json to better work with outside trash. 9365 9366 9367 Return values are also auto-formatted according to user-requested type: 9368 for json, it loops over and converts. 9369 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9370 +/ 9371 9372 /++ 9373 A web presenter is responsible for rendering things to HTML to be usable 9374 in a web browser. 9375 9376 They are passed as template arguments to the base classes of [WebObject] 9377 9378 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9379 and override it. Use forwarding and specialization to customize it. 9380 9381 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9382 class templates and your overridden templates work with each other. 9383 9384 --- 9385 class MyPresenter : WebPresenter!(MyPresenter) { 9386 @Override 9387 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9388 // present the CustomType 9389 } 9390 @Override 9391 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9392 // handle everything else via the super class, which will call 9393 // back to your class when appropriate 9394 super.presentSuccessfulReturnAsHtml(cgi, ret); 9395 } 9396 } 9397 --- 9398 9399 The meta argument in there can be overridden by your own facility. 9400 9401 +/ 9402 class WebPresenter(CRTP) { 9403 9404 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9405 /// If you override a plain method, use `override`. If a template, use `@Override`. 9406 enum Override; 9407 9408 string script() { 9409 return ` 9410 `; 9411 } 9412 9413 string style() { 9414 return ` 9415 :root { 9416 --mild-border: #ccc; 9417 --middle-border: #999; 9418 --accent-color: #f2f2f2; 9419 --sidebar-color: #fefefe; 9420 } 9421 ` ~ genericFormStyling() ~ genericSiteStyling(); 9422 } 9423 9424 string genericFormStyling() { 9425 return 9426 q"css 9427 table.automatic-data-display { 9428 border-collapse: collapse; 9429 border: solid 1px var(--mild-border); 9430 } 9431 9432 table.automatic-data-display td { 9433 vertical-align: top; 9434 border: solid 1px var(--mild-border); 9435 padding: 2px 4px; 9436 } 9437 9438 table.automatic-data-display th { 9439 border: solid 1px var(--mild-border); 9440 border-bottom: solid 1px var(--middle-border); 9441 padding: 2px 4px; 9442 } 9443 9444 ol.automatic-data-display { 9445 margin: 0px; 9446 list-style-position: inside; 9447 padding: 0px; 9448 } 9449 9450 dl.automatic-data-display { 9451 9452 } 9453 9454 .automatic-form { 9455 max-width: 600px; 9456 } 9457 9458 .form-field { 9459 margin: 0.5em; 9460 padding-left: 0.5em; 9461 } 9462 9463 .label-text { 9464 display: block; 9465 font-weight: bold; 9466 margin-left: -0.5em; 9467 } 9468 9469 .submit-button-holder { 9470 padding-left: 2em; 9471 } 9472 9473 .add-array-button { 9474 9475 } 9476 css"; 9477 } 9478 9479 string genericSiteStyling() { 9480 return 9481 q"css 9482 * { box-sizing: border-box; } 9483 html, body { margin: 0px; } 9484 body { 9485 font-family: sans-serif; 9486 } 9487 header { 9488 background: var(--accent-color); 9489 height: 64px; 9490 } 9491 footer { 9492 background: var(--accent-color); 9493 height: 64px; 9494 } 9495 #site-container { 9496 display: flex; 9497 } 9498 main { 9499 flex: 1 1 auto; 9500 order: 2; 9501 min-height: calc(100vh - 64px - 64px); 9502 padding: 4px; 9503 padding-left: 1em; 9504 } 9505 #sidebar { 9506 flex: 0 0 16em; 9507 order: 1; 9508 background: var(--sidebar-color); 9509 } 9510 css"; 9511 } 9512 9513 import arsd.dom; 9514 Element htmlContainer() { 9515 auto document = new Document(q"html 9516 <!DOCTYPE html> 9517 <html class="no-script"> 9518 <head> 9519 <script>document.documentElement.classList.remove("no-script");</script> 9520 <style>.no-script requires-script { display: none; }</style> 9521 <title>D Application</title> 9522 <link rel="stylesheet" href="style.css" /> 9523 </head> 9524 <body> 9525 <header></header> 9526 <div id="site-container"> 9527 <main></main> 9528 <div id="sidebar"></div> 9529 </div> 9530 <footer></footer> 9531 <script src="script.js"></script> 9532 </body> 9533 </html> 9534 html", true, true); 9535 9536 return document.requireSelector("main"); 9537 } 9538 9539 /// Renders a response as an HTTP error 9540 void renderBasicError(Cgi cgi, int httpErrorCode) { 9541 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9542 auto c = htmlContainer(); 9543 c.innerText = getHttpCodeText(httpErrorCode); 9544 cgi.setResponseContentType("text/html; charset=utf-8"); 9545 cgi.write(c.parentDocument.toString(), true); 9546 } 9547 9548 template methodMeta(alias method) { 9549 enum methodMeta = null; 9550 } 9551 9552 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9553 switch(format) { 9554 case "html": 9555 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9556 break; 9557 case "json": 9558 import arsd.jsvar; 9559 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9560 var json; 9561 foreach(index, type; Types) { 9562 if(ret.contains == index) 9563 json = ret.payload[index]; 9564 } 9565 } else { 9566 var json = ret; 9567 } 9568 var envelope = json; // var.emptyObject; 9569 /* 9570 envelope.success = true; 9571 envelope.result = json; 9572 envelope.error = null; 9573 */ 9574 cgi.setResponseContentType("application/json"); 9575 cgi.write(envelope.toJson(), true); 9576 break; 9577 default: 9578 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9579 } 9580 } 9581 9582 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9583 /// in the default presenter - allowing the function to have full low-level control over the 9584 /// response. 9585 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9586 // nothing intentionally! 9587 } 9588 9589 /// Redirections are forwarded to [Cgi.setResponseLocation] 9590 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9591 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9592 } 9593 9594 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9595 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9596 cgi.setResponseStatus(getHttpCodeText(201)); 9597 if(ret.resourceUrl.length) 9598 cgi.header("Location: " ~ ret.resourceUrl); 9599 if(ret.refreshUrl.length) 9600 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9601 static if(!is(R == void)) 9602 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9603 } 9604 9605 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9606 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9607 bool outputted = false; 9608 foreach(index, type; Types) { 9609 if(ret.contains == index) { 9610 assert(!outputted); 9611 outputted = true; 9612 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9613 } 9614 } 9615 if(!outputted) 9616 assert(0); 9617 } 9618 9619 /++ 9620 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9621 +/ 9622 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9623 cgi.setCache(true); // not necessarily true but meh 9624 if(auto fn = ret.filename()) { 9625 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9626 } 9627 cgi.setResponseContentType(ret.contentType); 9628 cgi.write(ret.getData(), true); 9629 } 9630 9631 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9632 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9633 auto container = this.htmlContainer(); 9634 container.appendChild(formatReturnValueAsHtml(ret)); 9635 cgi.write(container.parentDocument.toString(), true); 9636 } 9637 9638 /++ 9639 9640 History: 9641 Added January 23, 2023 (dub v11.0) 9642 +/ 9643 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9644 switch(format) { 9645 case "html": 9646 presentExceptionAsHtml(cgi, t, meta); 9647 break; 9648 default: 9649 } 9650 } 9651 9652 9653 /++ 9654 If you override this, you will need to cast the exception type `t` dynamically, 9655 but can then use the template arguments here to refer back to the function. 9656 9657 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9658 method on the live object. You could, in theory, change arguments and retry, but I 9659 provide that information mostly with the expectation that you will use them to make 9660 useful forms or richer error messages for the user. 9661 9662 History: 9663 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 9664 I removed this in favor of a `Meta` param. 9665 9666 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 9667 9668 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 9669 9670 If you used the func for something, move that something into your `methodMeta` template. 9671 9672 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 9673 enabling an easier implementation of [presentExceptionalReturn]. 9674 +/ 9675 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 9676 Form af; 9677 /+ 9678 foreach(attr; __traits(getAttributes, func)) { 9679 static if(__traits(isSame, attr, AutomaticForm)) { 9680 af = createAutomaticFormForFunction!(func)(dg); 9681 } 9682 } 9683 +/ 9684 presentExceptionAsHtmlImpl(cgi, t, af); 9685 } 9686 9687 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 9688 if(auto e = cast(ResourceNotFoundException) t) { 9689 auto container = this.htmlContainer(); 9690 9691 container.addChild("p", e.msg); 9692 9693 if(!cgi.outputtedResponseData) 9694 cgi.setResponseStatus("404 Not Found"); 9695 cgi.write(container.parentDocument.toString(), true); 9696 } else if(auto mae = cast(MissingArgumentException) t) { 9697 if(automaticForm is null) 9698 goto generic; 9699 auto container = this.htmlContainer(); 9700 if(cgi.requestMethod == Cgi.RequestMethod.POST) 9701 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 9702 container.appendChild(automaticForm); 9703 9704 cgi.write(container.parentDocument.toString(), true); 9705 } else { 9706 generic: 9707 auto container = this.htmlContainer(); 9708 9709 // import std.stdio; writeln(t.toString()); 9710 9711 container.appendChild(exceptionToElement(t)); 9712 9713 container.addChild("h4", "GET"); 9714 foreach(k, v; cgi.get) { 9715 auto deets = container.addChild("details"); 9716 deets.addChild("summary", k); 9717 deets.addChild("div", v); 9718 } 9719 9720 container.addChild("h4", "POST"); 9721 foreach(k, v; cgi.post) { 9722 auto deets = container.addChild("details"); 9723 deets.addChild("summary", k); 9724 deets.addChild("div", v); 9725 } 9726 9727 9728 if(!cgi.outputtedResponseData) 9729 cgi.setResponseStatus("500 Internal Server Error"); 9730 cgi.write(container.parentDocument.toString(), true); 9731 } 9732 } 9733 9734 Element exceptionToElement(Throwable t) { 9735 auto div = Element.make("div"); 9736 div.addClass("exception-display"); 9737 9738 div.addChild("p", t.msg); 9739 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 9740 9741 auto pre = div.addChild("pre"); 9742 string s; 9743 s = t.toString(); 9744 Element currentBox; 9745 bool on = false; 9746 foreach(line; s.splitLines) { 9747 if(!on && line.startsWith("-----")) 9748 on = true; 9749 if(!on) continue; 9750 if(line.indexOf("arsd/") != -1) { 9751 if(currentBox is null) { 9752 currentBox = pre.addChild("details"); 9753 currentBox.addChild("summary", "Framework code"); 9754 } 9755 currentBox.addChild("span", line ~ "\n"); 9756 } else { 9757 pre.addChild("span", line ~ "\n"); 9758 currentBox = null; 9759 } 9760 } 9761 9762 return div; 9763 } 9764 9765 /++ 9766 Returns an element for a particular type 9767 +/ 9768 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 9769 import std.traits; 9770 9771 auto div = Element.make("div"); 9772 div.addClass("form-field"); 9773 9774 static if(is(T == Cgi.UploadedFile)) { 9775 Element lbl; 9776 if(displayName !is null) { 9777 lbl = div.addChild("label"); 9778 lbl.addChild("span", displayName, "label-text"); 9779 lbl.appendText(" "); 9780 } else { 9781 lbl = div; 9782 } 9783 auto i = lbl.addChild("input", name); 9784 i.attrs.name = name; 9785 i.attrs.type = "file"; 9786 } else static if(is(T == enum)) { 9787 Element lbl; 9788 if(displayName !is null) { 9789 lbl = div.addChild("label"); 9790 lbl.addChild("span", displayName, "label-text"); 9791 lbl.appendText(" "); 9792 } else { 9793 lbl = div; 9794 } 9795 auto i = lbl.addChild("select", name); 9796 i.attrs.name = name; 9797 9798 foreach(memberName; __traits(allMembers, T)) 9799 i.addChild("option", memberName); 9800 9801 } else static if(is(T == struct)) { 9802 if(displayName !is null) 9803 div.addChild("span", displayName, "label-text"); 9804 auto fieldset = div.addChild("fieldset"); 9805 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 9806 fieldset.addChild("input", name); 9807 foreach(idx, memberName; __traits(allMembers, T)) 9808 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9809 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 9810 } 9811 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9812 Element lbl; 9813 if(displayName !is null) { 9814 lbl = div.addChild("label"); 9815 lbl.addChild("span", displayName, "label-text"); 9816 lbl.appendText(" "); 9817 } else { 9818 lbl = div; 9819 } 9820 Element i; 9821 if(udaSuggestion) { 9822 i = udaSuggestion(); 9823 lbl.appendChild(i); 9824 } else { 9825 i = lbl.addChild("input", name); 9826 } 9827 i.attrs.name = name; 9828 static if(isSomeString!T) 9829 i.attrs.type = "text"; 9830 else 9831 i.attrs.type = "number"; 9832 if(i.tagName == "textarea") 9833 i.textContent = to!string(T.init); 9834 else 9835 i.attrs.value = to!string(T.init); 9836 } else static if(is(T == bool)) { 9837 Element lbl; 9838 if(displayName !is null) { 9839 lbl = div.addChild("label"); 9840 lbl.addChild("span", displayName, "label-text"); 9841 lbl.appendText(" "); 9842 } else { 9843 lbl = div; 9844 } 9845 auto i = lbl.addChild("input", name); 9846 i.attrs.type = "checkbox"; 9847 i.attrs.value = "true"; 9848 i.attrs.name = name; 9849 } else static if(is(T == K[], K)) { 9850 auto templ = div.addChild("template"); 9851 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 9852 if(displayName !is null) 9853 div.addChild("span", displayName, "label-text"); 9854 auto btn = div.addChild("button"); 9855 btn.addClass("add-array-button"); 9856 btn.attrs.type = "button"; 9857 btn.innerText = "Add"; 9858 btn.attrs.onclick = q{ 9859 var a = document.importNode(this.parentNode.firstChild.content, true); 9860 this.parentNode.insertBefore(a, this); 9861 }; 9862 } else static if(is(T == V[K], K, V)) { 9863 div.innerText = "assoc array not implemented for automatic form at this time"; 9864 } else { 9865 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9866 } 9867 9868 9869 return div; 9870 } 9871 9872 /// creates a form for gathering the function's arguments 9873 Form createAutomaticFormForFunction(alias method, T)(T dg) { 9874 9875 auto form = cast(Form) Element.make("form"); 9876 9877 form.method = "POST"; // FIXME 9878 9879 form.addClass("automatic-form"); 9880 9881 string formDisplayName = beautify(__traits(identifier, method)); 9882 foreach(attr; __traits(getAttributes, method)) 9883 static if(is(typeof(attr) == DisplayName)) 9884 formDisplayName = attr.name; 9885 form.addChild("h3", formDisplayName); 9886 9887 import std.traits; 9888 9889 //Parameters!method params; 9890 //alias idents = ParameterIdentifierTuple!method; 9891 //alias defaults = ParameterDefaults!method; 9892 9893 static if(is(typeof(method) P == __parameters)) 9894 foreach(idx, _; P) {{ 9895 9896 alias param = P[idx .. idx + 1]; 9897 9898 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 9899 string displayName = beautify(__traits(identifier, param)); 9900 Element function() element; 9901 foreach(attr; __traits(getAttributes, param)) { 9902 static if(is(typeof(attr) == DisplayName)) 9903 displayName = attr.name; 9904 else static if(is(typeof(attr) : typeof(element))) { 9905 element = attr; 9906 } 9907 } 9908 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 9909 if(i.querySelector("input[type=file]") !is null) 9910 form.setAttribute("enctype", "multipart/form-data"); 9911 } 9912 }} 9913 9914 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 9915 9916 return form; 9917 } 9918 9919 /// creates a form for gathering object members (for the REST object thing right now) 9920 Form createAutomaticFormForObject(T)(T obj) { 9921 auto form = cast(Form) Element.make("form"); 9922 9923 form.addClass("automatic-form"); 9924 9925 form.addChild("h3", beautify(__traits(identifier, T))); 9926 9927 import std.traits; 9928 9929 //Parameters!method params; 9930 //alias idents = ParameterIdentifierTuple!method; 9931 //alias defaults = ParameterDefaults!method; 9932 9933 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 9934 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 9935 string displayName = beautify(memberName); 9936 Element function() element; 9937 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 9938 static if(is(typeof(attr) == DisplayName)) 9939 displayName = attr.name; 9940 else static if(is(typeof(attr) : typeof(element))) 9941 element = attr; 9942 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 9943 9944 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 9945 }}} 9946 9947 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 9948 9949 return form; 9950 } 9951 9952 /// 9953 Element formatReturnValueAsHtml(T)(T t) { 9954 import std.traits; 9955 9956 static if(is(T == typeof(null))) { 9957 return Element.make("span"); 9958 } else static if(is(T : Element)) { 9959 return t; 9960 } else static if(is(T == MultipleResponses!Types, Types...)) { 9961 foreach(index, type; Types) { 9962 if(t.contains == index) 9963 return formatReturnValueAsHtml(t.payload[index]); 9964 } 9965 assert(0); 9966 } else static if(is(T == Paginated!E, E)) { 9967 auto e = Element.make("div").addClass("paginated-result"); 9968 e.appendChild(formatReturnValueAsHtml(t.items)); 9969 if(t.nextPageUrl.length) 9970 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 9971 return e; 9972 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 9973 return Element.make("span", to!string(t), "automatic-data-display"); 9974 } else static if(is(T == V[K], K, V)) { 9975 auto dl = Element.make("dl"); 9976 dl.addClass("automatic-data-display associative-array"); 9977 foreach(k, v; t) { 9978 dl.addChild("dt", to!string(k)); 9979 dl.addChild("dd", formatReturnValueAsHtml(v)); 9980 } 9981 return dl; 9982 } else static if(is(T == struct)) { 9983 auto dl = Element.make("dl"); 9984 dl.addClass("automatic-data-display struct"); 9985 9986 foreach(idx, memberName; __traits(allMembers, T)) 9987 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9988 dl.addChild("dt", beautify(memberName)); 9989 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 9990 } 9991 9992 return dl; 9993 } else static if(is(T == bool)) { 9994 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 9995 } else static if(is(T == E[], E)) { 9996 static if(is(E : RestObject!Proxy, Proxy)) { 9997 // treat RestObject similar to struct 9998 auto table = cast(Table) Element.make("table"); 9999 table.addClass("automatic-data-display"); 10000 string[] names; 10001 foreach(idx, memberName; __traits(derivedMembers, E)) 10002 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10003 names ~= beautify(memberName); 10004 } 10005 table.appendHeaderRow(names); 10006 10007 foreach(l; t) { 10008 auto tr = table.appendRow(); 10009 foreach(idx, memberName; __traits(derivedMembers, E)) 10010 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10011 static if(memberName == "id") { 10012 string val = to!string(__traits(getMember, l, memberName)); 10013 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10014 } else { 10015 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10016 } 10017 } 10018 } 10019 10020 return table; 10021 } else static if(is(E == struct)) { 10022 // an array of structs is kinda special in that I like 10023 // having those formatted as tables. 10024 auto table = cast(Table) Element.make("table"); 10025 table.addClass("automatic-data-display"); 10026 string[] names; 10027 foreach(idx, memberName; __traits(allMembers, E)) 10028 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10029 names ~= beautify(memberName); 10030 } 10031 table.appendHeaderRow(names); 10032 10033 foreach(l; t) { 10034 auto tr = table.appendRow(); 10035 foreach(idx, memberName; __traits(allMembers, E)) 10036 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10037 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10038 } 10039 } 10040 10041 return table; 10042 } else { 10043 // otherwise, I will just make a list. 10044 auto ol = Element.make("ol"); 10045 ol.addClass("automatic-data-display"); 10046 foreach(e; t) 10047 ol.addChild("li", formatReturnValueAsHtml(e)); 10048 return ol; 10049 } 10050 } else static if(is(T : Object)) { 10051 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10052 return Element.make("div", t.toHtml()); 10053 else 10054 return Element.make("div", t.toString()); 10055 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10056 10057 assert(0); 10058 } 10059 10060 } 10061 10062 /++ 10063 The base class for the [dispatcher] function and object support. 10064 +/ 10065 class WebObject { 10066 //protected Cgi cgi; 10067 10068 protected void initialize(Cgi cgi) { 10069 //this.cgi = cgi; 10070 } 10071 } 10072 10073 /++ 10074 Can return one of the given types, decided at runtime. The syntax 10075 is to declare all the possible types in the return value, then you 10076 can `return typeof(return)(...value...)` to construct it. 10077 10078 It has an auto-generated constructor for each value it can hold. 10079 10080 --- 10081 MultipleResponses!(Redirection, string) getData(int how) { 10082 if(how & 1) 10083 return typeof(return)(Redirection("http://dpldocs.info/")); 10084 else 10085 return typeof(return)("hi there!"); 10086 } 10087 --- 10088 10089 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10090 +/ 10091 struct MultipleResponses(T...) { 10092 private size_t contains; 10093 private union { 10094 private T payload; 10095 } 10096 10097 static foreach(index, type; T) 10098 public this(type t) { 10099 contains = index; 10100 payload[index] = t; 10101 } 10102 10103 /++ 10104 This is primarily for testing. It is your way of getting to the response. 10105 10106 Let's say you wanted to test that one holding a Redirection and a string actually 10107 holds a string, by name of "test": 10108 10109 --- 10110 auto valueToTest = your_test_function(); 10111 10112 valueToTest.visit( 10113 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10114 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10115 ); 10116 --- 10117 10118 History: 10119 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10120 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10121 +/ 10122 void visit(Handlers...)(Handlers handlers) { 10123 template findHandler(type, int count, HandlersToCheck...) { 10124 static if(HandlersToCheck.length == 0) 10125 enum findHandler = -1; 10126 else { 10127 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10128 enum findHandler = count; 10129 else 10130 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10131 } 10132 } 10133 foreach(index, type; T) { 10134 enum handlerIndex = findHandler!(type, 0, Handlers); 10135 static if(handlerIndex == -1) 10136 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10137 else { 10138 if(index == this.contains) 10139 handlers[handlerIndex](this.payload[index]); 10140 } 10141 } 10142 } 10143 10144 /+ 10145 auto toArsdJsvar()() { 10146 import arsd.jsvar; 10147 return var(null); 10148 } 10149 +/ 10150 } 10151 10152 // FIXME: implement this somewhere maybe 10153 struct RawResponse { 10154 int code; 10155 string[] headers; 10156 const(ubyte)[] responseBody; 10157 } 10158 10159 /++ 10160 You can return this from [WebObject] subclasses for redirections. 10161 10162 (though note the static types means that class must ALWAYS redirect if 10163 you return this directly. You might want to return [MultipleResponses] if it 10164 can be conditional) 10165 +/ 10166 struct Redirection { 10167 string to; /// The URL to redirect to. 10168 int code = 303; /// The HTTP code to return. 10169 } 10170 10171 /++ 10172 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10173 10174 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10175 the presenter in the dispatcher. 10176 10177 FIXME: explain this better 10178 10179 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10180 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10181 the runtime result of that is undefined. 10182 10183 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10184 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10185 10186 $(WARNING 10187 --- 10188 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10189 // even if you put different URL udas on it, the current code ignores them. 10190 void foo(int a) {} 10191 void foo(string a) {} 10192 --- 10193 ) 10194 10195 See_Also: [serveRestObject], [serveStaticFile] 10196 +/ 10197 auto serveApi(T)(string urlPrefix) { 10198 assert(urlPrefix[$ - 1] == '/'); 10199 return serveApiInternal!T(urlPrefix); 10200 } 10201 10202 private string nextPieceFromSlash(ref string remainingUrl) { 10203 if(remainingUrl.length == 0) 10204 return remainingUrl; 10205 int slash = 0; 10206 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10207 slash++; 10208 10209 // I am specifically passing `null` to differentiate it vs empty string 10210 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10211 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10212 // so if it is the last item, the dot can be used to load an alternative view 10213 // otherwise tho the dot is considered part of the identifier 10214 // FIXME 10215 10216 // again notice "" vs null here! 10217 if(slash == remainingUrl.length) 10218 remainingUrl = null; 10219 else 10220 remainingUrl = remainingUrl[slash + 1 .. $]; 10221 10222 return ident; 10223 } 10224 10225 /++ 10226 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10227 +/ 10228 enum AddTrailingSlash; 10229 /// ditto 10230 enum RemoveTrailingSlash; 10231 10232 private auto serveApiInternal(T)(string urlPrefix) { 10233 10234 import arsd.dom; 10235 import arsd.jsvar; 10236 10237 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10238 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10239 10240 try { 10241 // see duplicated code below by searching subresource_ctor 10242 // also see mustNotBeSetFromWebParams 10243 10244 static if(is(typeof(T.__ctor) P == __parameters)) { 10245 P params; 10246 10247 foreach(pidx, param; P) { 10248 static if(is(param : Cgi)) { 10249 static assert(!is(param == immutable)); 10250 cast() params[pidx] = cgi; 10251 } else static if(is(param == Session!D, D)) { 10252 static assert(!is(param == immutable)); 10253 cast() params[pidx] = cgi.getSessionObject!D(); 10254 10255 } else { 10256 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10257 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10258 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10259 static if(is(typeof(func(cgi)))) 10260 params[pidx] = func(cgi); 10261 else 10262 params[pidx] = func(); 10263 } 10264 } 10265 } else { 10266 10267 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10268 params[pidx] = param.getAutomaticallyForCgi(cgi); 10269 } else static if(is(param == string)) { 10270 auto ident = nextPieceFromSlash(remainingUrl); 10271 params[pidx] = ident; 10272 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10273 } 10274 } 10275 } 10276 10277 auto obj = new T(params); 10278 } else { 10279 auto obj = new T(); 10280 } 10281 10282 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10283 } catch(Throwable t) { 10284 switch(cgi.request("format", "html")) { 10285 case "html": 10286 static void dummy() {} 10287 presenter.presentExceptionAsHtml(cgi, t, null); 10288 return true; 10289 case "json": 10290 var envelope = var.emptyObject; 10291 envelope.success = false; 10292 envelope.result = null; 10293 envelope.error = t.toString(); 10294 cgi.setResponseContentType("application/json"); 10295 cgi.write(envelope.toJson(), true); 10296 return true; 10297 default: 10298 throw t; 10299 // return true; 10300 } 10301 // return true; 10302 } 10303 10304 assert(0); 10305 } 10306 10307 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10308 10309 obj.initialize(cgi); 10310 10311 /+ 10312 Overload rules: 10313 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10314 statically. 10315 10316 Moreover, some args vs no args can be overloaded dynamically. 10317 +/ 10318 10319 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10320 /+ 10321 auto orig = remainingUrl; 10322 assert(0, 10323 (orig is null ? "__null" : orig) 10324 ~ " .. " ~ 10325 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10326 +/ 10327 10328 if(methodNameFromUrl is null) 10329 methodNameFromUrl = "__null"; 10330 10331 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10332 10333 if(remainingUrl.length) 10334 hack ~= "/"; 10335 10336 switch(hack) { 10337 foreach(methodName; __traits(derivedMembers, T)) 10338 static if(methodName != "__ctor") 10339 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10340 static if(is(typeof(overload) P == __parameters)) 10341 static if(is(typeof(overload) R == return)) 10342 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10343 { 10344 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10345 case urlNameForMethod: 10346 10347 static if(is(R : WebObject)) { 10348 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10349 10350 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10351 10352 // subresource_ctor 10353 // also see mustNotBeSetFromWebParams 10354 10355 P params; 10356 10357 string ident; 10358 10359 foreach(pidx, param; P) { 10360 static if(is(param : Cgi)) { 10361 static assert(!is(param == immutable)); 10362 cast() params[pidx] = cgi; 10363 } else static if(is(param == typeof(presenter))) { 10364 cast() param[pidx] = presenter; 10365 } else static if(is(param == Session!D, D)) { 10366 static assert(!is(param == immutable)); 10367 cast() params[pidx] = cgi.getSessionObject!D(); 10368 } else { 10369 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10370 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10371 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10372 static if(is(typeof(func(cgi)))) 10373 params[pidx] = func(cgi); 10374 else 10375 params[pidx] = func(); 10376 } 10377 } 10378 } else { 10379 10380 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10381 params[pidx] = param.getAutomaticallyForCgi(cgi); 10382 } else static if(is(param == string)) { 10383 ident = nextPieceFromSlash(remainingUrl); 10384 if(ident is null) { 10385 // trailing slash mandated on subresources 10386 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10387 return true; 10388 } else { 10389 params[pidx] = ident; 10390 } 10391 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10392 } 10393 } 10394 } 10395 10396 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10397 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10398 } else { 10399 // 404 it if any url left - not a subresource means we don't get to play with that! 10400 if(remainingUrl.length) 10401 return false; 10402 10403 bool automaticForm; 10404 10405 foreach(attr; __traits(getAttributes, overload)) 10406 static if(is(attr == AddTrailingSlash)) { 10407 if(remainingUrl is null) { 10408 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10409 return true; 10410 } 10411 } else static if(is(attr == RemoveTrailingSlash)) { 10412 if(remainingUrl !is null) { 10413 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10414 return true; 10415 } 10416 10417 } else static if(__traits(isSame, AutomaticForm, attr)) { 10418 automaticForm = true; 10419 } 10420 10421 /+ 10422 int zeroArgOverload = -1; 10423 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10424 bool calledWithZeroArgs = true; 10425 foreach(k, v; cgi.get) 10426 if(k != "format") { 10427 calledWithZeroArgs = false; 10428 break; 10429 } 10430 foreach(k, v; cgi.post) 10431 if(k != "format") { 10432 calledWithZeroArgs = false; 10433 break; 10434 } 10435 10436 // first, we need to go through and see if there is an empty one, since that 10437 // changes inside. But otherwise, all the stuff I care about can be done via 10438 // simple looping (other improper overloads might be flagged for runtime semantic check) 10439 // 10440 // an argument of type Cgi is ignored for these purposes 10441 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10442 static if(is(typeof(overload) P == __parameters)) 10443 static if(P.length == 0) 10444 zeroArgOverload = cast(int) idx; 10445 else static if(P.length == 1 && is(P[0] : Cgi)) 10446 zeroArgOverload = cast(int) idx; 10447 }} 10448 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10449 bool overloadHasBeenCalled = false; 10450 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10451 bool callFunction = true; 10452 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10453 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10454 callFunction = false; 10455 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10456 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10457 callFunction = false; 10458 10459 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10460 10461 bool hadAnyMethodRestrictions = false; 10462 bool foundAcceptableMethod = false; 10463 foreach(attr; __traits(getAttributes, overload)) { 10464 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10465 hadAnyMethodRestrictions = true; 10466 if(attr == cgi.requestMethod) 10467 foundAcceptableMethod = true; 10468 } 10469 } 10470 10471 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10472 callFunction = false; 10473 10474 /+ 10475 The overloads we really want to allow are the sane ones 10476 from the web perspective. Which is likely on HTTP verbs, 10477 for the most part, but might also be potentially based on 10478 some args vs zero args, or on argument names. Can't really 10479 do argument types very reliable through the web though; those 10480 should probably be different URLs. 10481 10482 Even names I feel is better done inside the function, so I'm not 10483 going to support that here. But the HTTP verbs and zero vs some 10484 args makes sense - it lets you define custom forms pretty easily. 10485 10486 Moreover, I'm of the opinion that empty overload really only makes 10487 sense on GET for this case. On a POST, it is just a missing argument 10488 exception and that should be handled by the presenter. But meh, I'll 10489 let the user define that, D only allows one empty arg thing anyway 10490 so the method UDAs are irrelevant. 10491 +/ 10492 if(callFunction) 10493 +/ 10494 10495 auto format = cgi.request("format", defaultFormat!overload()); 10496 auto wantsFormFormat = format.startsWith("form-"); 10497 10498 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10499 // Should I still show the form on a json thing? idk... 10500 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10501 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10502 return true; 10503 } 10504 10505 try { 10506 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10507 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10508 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10509 } catch(Throwable t) { 10510 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10511 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10512 } 10513 return true; 10514 //}} 10515 10516 //cgi.header("Accept: POST"); // FIXME list the real thing 10517 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10518 //return true; 10519 } 10520 } 10521 } 10522 case "GET script.js": 10523 cgi.setResponseContentType("text/javascript"); 10524 cgi.gzipResponse = true; 10525 cgi.write(presenter.script(), true); 10526 return true; 10527 case "GET style.css": 10528 cgi.setResponseContentType("text/css"); 10529 cgi.gzipResponse = true; 10530 cgi.write(presenter.style(), true); 10531 return true; 10532 default: 10533 return false; 10534 } 10535 10536 assert(0); 10537 } 10538 return DispatcherDefinition!internalHandler(urlPrefix, false); 10539 } 10540 10541 string defaultFormat(alias method)() { 10542 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10543 foreach(attr; __traits(getAttributes, method)) { 10544 static if(is(typeof(attr) == DefaultFormat)) { 10545 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10546 return attr.value; 10547 } 10548 } 10549 return "html"; 10550 } 10551 10552 struct Paginated(T) { 10553 T[] items; 10554 string nextPageUrl; 10555 } 10556 10557 template urlNamesForMethod(alias method, string default_) { 10558 string[] helper() { 10559 auto verb = Cgi.RequestMethod.GET; 10560 bool foundVerb = false; 10561 bool foundNoun = false; 10562 10563 string def = default_; 10564 10565 bool hasAutomaticForm = false; 10566 10567 foreach(attr; __traits(getAttributes, method)) { 10568 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10569 verb = attr; 10570 if(foundVerb) 10571 assert(0, "Multiple http verbs on one function is not currently supported"); 10572 foundVerb = true; 10573 } 10574 static if(is(typeof(attr) == UrlName)) { 10575 if(foundNoun) 10576 assert(0, "Multiple url names on one function is not currently supported"); 10577 foundNoun = true; 10578 def = attr.name; 10579 } 10580 static if(__traits(isSame, attr, AutomaticForm)) { 10581 hasAutomaticForm = true; 10582 } 10583 } 10584 10585 if(def is null) 10586 def = "__null"; 10587 10588 string[] ret; 10589 10590 static if(is(typeof(method) R == return)) { 10591 static if(is(R : WebObject)) { 10592 def ~= "/"; 10593 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10594 ret ~= v ~ " " ~ def; 10595 } else { 10596 if(hasAutomaticForm) { 10597 ret ~= "GET " ~ def; 10598 ret ~= "POST " ~ def; 10599 } else { 10600 ret ~= to!string(verb) ~ " " ~ def; 10601 } 10602 } 10603 } else static assert(0); 10604 10605 return ret; 10606 } 10607 enum urlNamesForMethod = helper(); 10608 } 10609 10610 10611 enum AccessCheck { 10612 allowed, 10613 denied, 10614 nonExistant, 10615 } 10616 10617 enum Operation { 10618 show, 10619 create, 10620 replace, 10621 remove, 10622 update 10623 } 10624 10625 enum UpdateResult { 10626 accessDenied, 10627 noSuchResource, 10628 success, 10629 failure, 10630 unnecessary 10631 } 10632 10633 enum ValidationResult { 10634 valid, 10635 invalid 10636 } 10637 10638 10639 /++ 10640 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10641 10642 WARNING: this is not stable. 10643 +/ 10644 class RestObject(CRTP) : WebObject { 10645 10646 import arsd.dom; 10647 import arsd.jsvar; 10648 10649 /// Prepare the object to be shown. 10650 void show() {} 10651 /// ditto 10652 void show(string urlId) { 10653 load(urlId); 10654 show(); 10655 } 10656 10657 /// Override this to provide access control to this object. 10658 AccessCheck accessCheck(string urlId, Operation operation) { 10659 return AccessCheck.allowed; 10660 } 10661 10662 ValidationResult validate() { 10663 // FIXME 10664 return ValidationResult.valid; 10665 } 10666 10667 string getUrlSlug() { 10668 import std.conv; 10669 static if(is(typeof(CRTP.id))) 10670 return to!string((cast(CRTP) this).id); 10671 else 10672 return null; 10673 } 10674 10675 // The functions with more arguments are the low-level ones, 10676 // they forward to the ones with fewer arguments by default. 10677 10678 // POST on a parent collection - this is called from a collection class after the members are updated 10679 /++ 10680 Given a populated object, this creates a new entry. Returns the url identifier 10681 of the new object. 10682 +/ 10683 string create(scope void delegate() applyChanges) { 10684 applyChanges(); 10685 save(); 10686 return getUrlSlug(); 10687 } 10688 10689 void replace() { 10690 save(); 10691 } 10692 void replace(string urlId, scope void delegate() applyChanges) { 10693 load(urlId); 10694 applyChanges(); 10695 replace(); 10696 } 10697 10698 void update(string[] fieldList) { 10699 save(); 10700 } 10701 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 10702 load(urlId); 10703 applyChanges(); 10704 update(fieldList); 10705 } 10706 10707 void remove() {} 10708 10709 void remove(string urlId) { 10710 load(urlId); 10711 remove(); 10712 } 10713 10714 abstract void load(string urlId); 10715 abstract void save(); 10716 10717 Element toHtml(Presenter)(Presenter presenter) { 10718 import arsd.dom; 10719 import std.conv; 10720 auto obj = cast(CRTP) this; 10721 auto div = Element.make("div"); 10722 div.addClass("Dclass_" ~ CRTP.stringof); 10723 div.dataset.url = getUrlSlug(); 10724 bool first = true; 10725 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 10726 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10727 if(!first) div.addChild("br"); else first = false; 10728 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 10729 } 10730 return div; 10731 } 10732 10733 var toJson() { 10734 import arsd.jsvar; 10735 var v = var.emptyObject(); 10736 auto obj = cast(CRTP) this; 10737 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 10738 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10739 v[memberName] = __traits(getMember, obj, memberName); 10740 } 10741 return v; 10742 } 10743 10744 /+ 10745 auto structOf(this This) { 10746 10747 } 10748 +/ 10749 } 10750 10751 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 10752 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 10753 10754 /++ 10755 Base class for REST collections. 10756 +/ 10757 class CollectionOf(Obj) : RestObject!(CollectionOf) { 10758 /// You might subclass this and use the cgi object's query params 10759 /// to implement a search filter, for example. 10760 /// 10761 /// FIXME: design a way to auto-generate that form 10762 /// (other than using the WebObject thing above lol 10763 // it'll prolly just be some searchParams UDA or maybe an enum. 10764 // 10765 // pagination too perhaps. 10766 // 10767 // and sorting too 10768 IndexResult index() { return IndexResult.init; } 10769 10770 string[] sortableFields() { return null; } 10771 string[] searchableFields() { return null; } 10772 10773 struct IndexResult { 10774 Obj[] results; 10775 10776 string[] sortableFields; 10777 10778 string previousPageIdentifier; 10779 string nextPageIdentifier; 10780 string firstPageIdentifier; 10781 string lastPageIdentifier; 10782 10783 int numberOfPages; 10784 } 10785 10786 override string create(scope void delegate() applyChanges) { assert(0); } 10787 override void load(string urlId) { assert(0); } 10788 override void save() { assert(0); } 10789 override void show() { 10790 index(); 10791 } 10792 override void show(string urlId) { 10793 show(); 10794 } 10795 10796 /// Proxy POST requests (create calls) to the child collection 10797 alias PostProxy = Obj; 10798 } 10799 10800 /++ 10801 Serves a REST object, similar to a Ruby on Rails resource. 10802 10803 You put data members in your class. cgi.d will automatically make something out of those. 10804 10805 It will call your constructor with the ID from the URL. This may be null. 10806 It will then populate the data members from the request. 10807 It will then call a method, if present, telling what happened. You don't need to write these! 10808 It finally returns a reply. 10809 10810 Your methods are passed a list of fields it actually set. 10811 10812 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 10813 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 10814 with relative linking. But meh.) 10815 10816 GET /items -> index. all values not set. 10817 GET /items/id -> get. only ID will be set, other params ignored. 10818 POST /items -> create. values set as given 10819 PUT /items/id -> replace. values set as given 10820 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 10821 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 10822 PATCH /items/id -> update. values set as given, list of changed fields passed 10823 or POST /items/id with cgi.post["_method"] == "PATCH" 10824 DELETE /items/id -> destroy. only ID guaranteed to be set 10825 or POST /items/id with cgi.post["_method"] == "DELETE" 10826 10827 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 10828 redirect you away from it. 10829 10830 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 10831 10832 I will also let you change the default, if you must. 10833 10834 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 10835 10836 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 10837 that follow the same thing. They may be individual resources or collections themselves. 10838 10839 Your class is expected to have at least the following methods: 10840 10841 FIXME: i kinda wanna add a routes object to the initialize call 10842 10843 create 10844 Create returns the new address on success, some code on failure. 10845 show 10846 index 10847 update 10848 remove 10849 10850 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 10851 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 10852 10853 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 10854 10855 NOT IMPLEMENTED 10856 10857 10858 Really, a collection is a resource with a bunch of subresources. 10859 10860 GET /items 10861 index because it is GET on the top resource 10862 10863 GET /items/foo 10864 item but different than items? 10865 10866 class Items { 10867 10868 } 10869 10870 ... but meh, a collection can be automated. not worth making it 10871 a separate thing, let's look at a real example. Users has many 10872 items and a virtual one, /users/current. 10873 10874 the individual users have properties and two sub-resources: 10875 session, which is just one, and comments, a collection. 10876 10877 class User : RestObject!() { // no parent 10878 int id; 10879 string name; 10880 10881 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 10882 // but you can override them to do it differently. 10883 10884 // any member which is of type RestObject can be linked automatically via href btw. 10885 10886 void show() {} 10887 void show(string urlId) {} // automated! GET of this specific thing 10888 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 10889 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 10890 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 10891 void remove(string urlId) {} // DELETE 10892 10893 void load(string urlId) {} // the default implementation of show() populates the id, then 10894 10895 this() {} 10896 10897 mixin Subresource!Session; 10898 mixin Subresource!Comment; 10899 } 10900 10901 class Session : RestObject!() { 10902 // the parent object may not be fully constructed/loaded 10903 this(User parent) {} 10904 10905 } 10906 10907 class Comment : CollectionOf!Comment { 10908 this(User parent) {} 10909 } 10910 10911 class Users : CollectionOf!User { 10912 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 10913 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 10914 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 10915 } 10916 10917 +/ 10918 auto serveRestObject(T)(string urlPrefix) { 10919 assert(urlPrefix[0] == '/'); 10920 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 10921 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10922 string url = cgi.pathInfo[urlPrefix.length .. $]; 10923 10924 if(url.length && url[$ - 1] == '/') { 10925 // remove the final slash... 10926 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 10927 return true; 10928 } 10929 10930 return restObjectServeHandler!T(cgi, presenter, url); 10931 } 10932 return DispatcherDefinition!internalHandler(urlPrefix, false); 10933 } 10934 10935 /+ 10936 /// Convenience method for serving a collection. It will be named the same 10937 /// as type T, just with an s at the end. If you need any further, just 10938 /// write the class yourself. 10939 auto serveRestCollectionOf(T)(string urlPrefix) { 10940 assert(urlPrefix[0] == '/'); 10941 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 10942 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 10943 } 10944 +/ 10945 10946 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 10947 string urlId = null; 10948 if(url.length && url[0] == '/') { 10949 // asking for a subobject 10950 urlId = url[1 .. $]; 10951 foreach(idx, ch; urlId) { 10952 if(ch == '/') { 10953 urlId = urlId[0 .. idx]; 10954 break; 10955 } 10956 } 10957 } 10958 10959 // FIXME handle other subresources 10960 10961 static if(is(T : CollectionOf!(C), C)) { 10962 if(urlId !is null) { 10963 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 10964 } 10965 } 10966 10967 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 10968 10969 auto obj = new T(); 10970 obj.initialize(cgi); 10971 // FIXME: populate reflection info delegates 10972 10973 10974 // FIXME: I am not happy with this. 10975 switch(urlId) { 10976 case "script.js": 10977 cgi.setResponseContentType("text/javascript"); 10978 cgi.gzipResponse = true; 10979 cgi.write(presenter.script(), true); 10980 return true; 10981 case "style.css": 10982 cgi.setResponseContentType("text/css"); 10983 cgi.gzipResponse = true; 10984 cgi.write(presenter.style(), true); 10985 return true; 10986 default: 10987 // intentionally blank 10988 } 10989 10990 10991 10992 10993 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 10994 foreach(idx, memberName; __traits(derivedMembers, Obj)) 10995 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10996 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 10997 } 10998 } 10999 void applyChanges() { 11000 applyChangesTemplate(cgi, obj); 11001 } 11002 11003 string[] modifiedList; 11004 11005 void writeObject(bool addFormLinks) { 11006 if(cgi.request("format") == "json") { 11007 cgi.setResponseContentType("application/json"); 11008 cgi.write(obj.toJson().toString, true); 11009 } else { 11010 auto container = presenter.htmlContainer(); 11011 if(addFormLinks) { 11012 static if(is(T : CollectionOf!(C), C)) 11013 container.appendHtml(` 11014 <form> 11015 <button type="submit" name="_method" value="POST">Create New</button> 11016 </form> 11017 `); 11018 else 11019 container.appendHtml(` 11020 <a href="..">Back</a> 11021 <form> 11022 <button type="submit" name="_method" value="PATCH">Edit</button> 11023 <button type="submit" name="_method" value="DELETE">Delete</button> 11024 </form> 11025 `); 11026 } 11027 container.appendChild(obj.toHtml(presenter)); 11028 cgi.write(container.parentDocument.toString, true); 11029 } 11030 } 11031 11032 // FIXME: I think I need a set type in here.... 11033 // it will be nice to pass sets of members. 11034 11035 try 11036 switch(cgi.requestMethod) { 11037 case Cgi.RequestMethod.GET: 11038 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11039 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11040 11041 // automatic forms here for usable basic auto site from browser. 11042 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11043 switch(cgi.request("_method", "GET")) { 11044 case "GET": 11045 static if(is(T : CollectionOf!(C), C)) { 11046 auto results = obj.index(); 11047 if(cgi.request("format", "html") == "html") { 11048 auto container = presenter.htmlContainer(); 11049 auto html = presenter.formatReturnValueAsHtml(results.results); 11050 container.appendHtml(` 11051 <form> 11052 <button type="submit" name="_method" value="POST">Create New</button> 11053 </form> 11054 `); 11055 11056 container.appendChild(html); 11057 cgi.write(container.parentDocument.toString, true); 11058 } else { 11059 cgi.setResponseContentType("application/json"); 11060 import arsd.jsvar; 11061 var json = var.emptyArray; 11062 foreach(r; results.results) { 11063 var o = var.emptyObject; 11064 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11065 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11066 o[memberName] = __traits(getMember, r, memberName); 11067 } 11068 11069 json ~= o; 11070 } 11071 cgi.write(json.toJson(), true); 11072 } 11073 } else { 11074 obj.show(urlId); 11075 writeObject(true); 11076 } 11077 break; 11078 case "PATCH": 11079 obj.load(urlId); 11080 goto case; 11081 case "PUT": 11082 case "POST": 11083 // an editing form for the object 11084 auto container = presenter.htmlContainer(); 11085 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11086 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11087 } else { 11088 auto form = presenter.createAutomaticFormForObject(obj); 11089 } 11090 form.attrs.method = "POST"; 11091 form.setValue("_method", cgi.request("_method", "GET")); 11092 container.appendChild(form); 11093 cgi.write(container.parentDocument.toString(), true); 11094 break; 11095 case "DELETE": 11096 // FIXME: a delete form for the object (can be phrased "are you sure?") 11097 auto container = presenter.htmlContainer(); 11098 container.appendHtml(` 11099 <form method="POST"> 11100 Are you sure you want to delete this item? 11101 <input type="hidden" name="_method" value="DELETE" /> 11102 <input type="submit" value="Yes, Delete It" /> 11103 </form> 11104 11105 `); 11106 cgi.write(container.parentDocument.toString(), true); 11107 break; 11108 default: 11109 cgi.write("bad method\n", true); 11110 } 11111 break; 11112 case Cgi.RequestMethod.POST: 11113 // this is to allow compatibility with HTML forms 11114 switch(cgi.request("_method", "POST")) { 11115 case "PUT": 11116 goto PUT; 11117 case "PATCH": 11118 goto PATCH; 11119 case "DELETE": 11120 goto DELETE; 11121 case "POST": 11122 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11123 auto p = new obj.PostProxy(); 11124 void specialApplyChanges() { 11125 applyChangesTemplate(cgi, p); 11126 } 11127 string n = p.create(&specialApplyChanges); 11128 } else { 11129 string n = obj.create(&applyChanges); 11130 } 11131 11132 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11133 cgi.setResponseLocation(newUrl); 11134 cgi.setResponseStatus("201 Created"); 11135 cgi.write(`The object has been created.`); 11136 break; 11137 default: 11138 cgi.write("bad method\n", true); 11139 } 11140 // FIXME this should be valid on the collection, but not the child.... 11141 // 303 See Other 11142 break; 11143 case Cgi.RequestMethod.PUT: 11144 PUT: 11145 obj.replace(urlId, &applyChanges); 11146 writeObject(false); 11147 break; 11148 case Cgi.RequestMethod.PATCH: 11149 PATCH: 11150 obj.update(urlId, &applyChanges, modifiedList); 11151 writeObject(false); 11152 break; 11153 case Cgi.RequestMethod.DELETE: 11154 DELETE: 11155 obj.remove(urlId); 11156 cgi.setResponseStatus("204 No Content"); 11157 break; 11158 default: 11159 // FIXME: OPTIONS, HEAD 11160 } 11161 catch(Throwable t) { 11162 presenter.presentExceptionAsHtml(cgi, t); 11163 } 11164 11165 return true; 11166 } 11167 11168 /+ 11169 struct SetOfFields(T) { 11170 private void[0][string] storage; 11171 void set(string what) { 11172 //storage[what] = 11173 } 11174 void unset(string what) {} 11175 void setAll() {} 11176 void unsetAll() {} 11177 bool isPresent(string what) { return false; } 11178 } 11179 +/ 11180 11181 /+ 11182 enum readonly; 11183 enum hideonindex; 11184 +/ 11185 11186 /++ 11187 Returns true if I recommend gzipping content of this type. You might 11188 want to call it from your Presenter classes before calling cgi.write. 11189 11190 --- 11191 cgi.setResponseContentType(yourContentType); 11192 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11193 cgi.write(yourData, true); 11194 --- 11195 11196 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11197 11198 11199 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11200 11201 History: 11202 Added January 28, 2023 (dub v11.0) 11203 +/ 11204 bool gzipRecommendedForContentType(string contentType) { 11205 if(contentType.startsWith("text/")) 11206 return true; 11207 if(contentType.startsWith("application/javascript")) 11208 return true; 11209 11210 return false; 11211 } 11212 11213 /++ 11214 Serves a static file. To be used with [dispatcher]. 11215 11216 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11217 +/ 11218 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11219 // https://baus.net/on-tcp_cork/ 11220 // man 2 sendfile 11221 assert(urlPrefix[0] == '/'); 11222 if(filename is null) 11223 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11224 if(contentType is null) { 11225 contentType = contentTypeFromFileExtension(filename); 11226 } 11227 11228 static struct DispatcherDetails { 11229 string filename; 11230 string contentType; 11231 } 11232 11233 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11234 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11235 cgi.setCache(true); 11236 cgi.setResponseContentType(details.contentType); 11237 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11238 cgi.write(std.file.read(details.filename), true); 11239 return true; 11240 } 11241 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11242 } 11243 11244 /++ 11245 Serves static data. To be used with [dispatcher]. 11246 11247 History: 11248 Added October 31, 2021 11249 +/ 11250 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11251 assert(urlPrefix[0] == '/'); 11252 if(contentType is null) { 11253 contentType = contentTypeFromFileExtension(urlPrefix); 11254 } 11255 11256 static struct DispatcherDetails { 11257 immutable(void)[] data; 11258 string contentType; 11259 } 11260 11261 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11262 cgi.setCache(true); 11263 cgi.setResponseContentType(details.contentType); 11264 cgi.write(details.data, true); 11265 return true; 11266 } 11267 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11268 } 11269 11270 string contentTypeFromFileExtension(string filename) { 11271 if(filename.endsWith(".png")) 11272 return "image/png"; 11273 if(filename.endsWith(".apng")) 11274 return "image/apng"; 11275 if(filename.endsWith(".svg")) 11276 return "image/svg+xml"; 11277 if(filename.endsWith(".jpg")) 11278 return "image/jpeg"; 11279 if(filename.endsWith(".html")) 11280 return "text/html"; 11281 if(filename.endsWith(".css")) 11282 return "text/css"; 11283 if(filename.endsWith(".js")) 11284 return "application/javascript"; 11285 if(filename.endsWith(".wasm")) 11286 return "application/wasm"; 11287 if(filename.endsWith(".mp3")) 11288 return "audio/mpeg"; 11289 if(filename.endsWith(".pdf")) 11290 return "application/pdf"; 11291 return null; 11292 } 11293 11294 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11295 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11296 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11297 assert(urlPrefix[0] == '/'); 11298 assert(urlPrefix[$-1] == '/'); 11299 11300 static struct DispatcherDetails { 11301 string directory; 11302 bool recursive; 11303 } 11304 11305 if(directory is null) 11306 directory = urlPrefix[1 .. $]; 11307 11308 if(directory.length == 0) 11309 directory = "./"; 11310 11311 assert(directory[$-1] == '/'); 11312 11313 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11314 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11315 11316 if(details.recursive) { 11317 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11318 if(file.indexOf("\\") != -1) 11319 return false; 11320 11321 import std.path; 11322 11323 file = std.path.buildNormalizedPath(file); 11324 enum upOneDir = ".." ~ std.path.dirSeparator; 11325 11326 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11327 if(file == ".." || file.startsWith(upOneDir)) 11328 return false; 11329 if(std.path.isAbsolute(file)) 11330 return false; 11331 11332 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11333 11334 // once it passes these filters it is probably ok. 11335 } else { 11336 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11337 return false; 11338 } 11339 11340 auto contentType = contentTypeFromFileExtension(file); 11341 11342 auto fn = details.directory ~ file; 11343 if(std.file.exists(fn)) { 11344 //if(contentType.indexOf("image/") == 0) 11345 //cgi.setCache(true); 11346 //else if(contentType.indexOf("audio/") == 0) 11347 cgi.setCache(true); 11348 cgi.setResponseContentType(contentType); 11349 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11350 cgi.write(std.file.read(fn), true); 11351 return true; 11352 } else { 11353 return false; 11354 } 11355 } 11356 11357 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11358 } 11359 11360 /++ 11361 Redirects one url to another 11362 11363 See_Also: [dispatcher], [serveStaticFile] 11364 +/ 11365 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11366 assert(urlPrefix[0] == '/'); 11367 static struct DispatcherDetails { 11368 string redirectTo; 11369 string code; 11370 } 11371 11372 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11373 cgi.setResponseLocation(details.redirectTo, true, details.code); 11374 return true; 11375 } 11376 11377 11378 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11379 } 11380 11381 /// Used exclusively with `dispatchTo` 11382 struct DispatcherData(Presenter) { 11383 Cgi cgi; /// You can use this cgi object. 11384 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11385 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11386 } 11387 11388 /++ 11389 Dispatches the URL to a specific function. 11390 +/ 11391 auto handleWith(alias handler)(string urlPrefix) { 11392 // cuz I'm too lazy to do it better right now 11393 static class Hack : WebObject { 11394 static import std.traits; 11395 @UrlName("") 11396 auto handle(std.traits.Parameters!handler args) { 11397 return handler(args); 11398 } 11399 } 11400 11401 return urlPrefix.serveApiInternal!Hack; 11402 } 11403 11404 /++ 11405 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11406 11407 --- 11408 bool other(DD)(DD dd) { 11409 return dd.dispatcher!( 11410 "/whatever".serveRedirect("/success"), 11411 "/api/".serveApi!MyClass 11412 ); 11413 } 11414 --- 11415 11416 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11417 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11418 Or, of course, you could just use the exact type in your own code. 11419 11420 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11421 good job. 11422 11423 11424 +/ 11425 auto dispatchTo(alias handler)(string urlPrefix) { 11426 assert(urlPrefix[0] == '/'); 11427 assert(urlPrefix[$-1] != '/'); 11428 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11429 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11430 } 11431 11432 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11433 } 11434 11435 /++ 11436 See [serveStaticFile] if you want to serve a file off disk. 11437 11438 History: 11439 Added January 28, 2023 (dub v11.0) 11440 +/ 11441 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11442 assert(urlPrefix[0] == '/'); 11443 11444 static struct DispatcherDetails { 11445 immutable(ubyte)[] data; 11446 string contentType; 11447 string filenameToSuggestAsDownload; 11448 } 11449 11450 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11451 cgi.setCache(true); 11452 cgi.setResponseContentType(details.contentType); 11453 if(details.filenameToSuggestAsDownload.length) 11454 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11455 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11456 cgi.write(details.data, true); 11457 return true; 11458 } 11459 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11460 } 11461 11462 /++ 11463 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11464 11465 History: 11466 Added January 28, 2023 (dub v11.0) 11467 +/ 11468 alias KeepExistingPresenter = typeof(null); 11469 11470 /++ 11471 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11472 this issues the given errorCode and stops processing. 11473 11474 --- 11475 bool hasAdminPermissions(Cgi cgi) { 11476 return true; 11477 } 11478 11479 mixin DispatcherMain!( 11480 "/admin".dispatchSubsection!( 11481 passFilterOrIssueError!(hasAdminPermissions, 403), 11482 KeepExistingPresenter, 11483 "/".serveApi!AdminFunctions 11484 ) 11485 ); 11486 --- 11487 11488 History: 11489 Added January 28, 2023 (dub v11.0) 11490 +/ 11491 template passFilterOrIssueError(alias filter, int errorCode) { 11492 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11493 if(filter(dd.cgi)) 11494 return true; 11495 dd.presenter.renderBasicError(dd.cgi, errorCode); 11496 return false; 11497 } 11498 } 11499 11500 /++ 11501 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11502 and then be dispatched to their own handlers. 11503 11504 --- 11505 /+ 11506 // a long-form filter function 11507 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11508 // you are permitted to call mutable methods on the Cgi object 11509 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11510 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11511 11512 if(checkPermissionOnRequest(dd.cgi)) { 11513 return true; // OK, allow processing to continue 11514 } else { 11515 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11516 return false; // and stop further processing into this subsection 11517 } 11518 } 11519 +/ 11520 11521 // but you can also do short-form filters: 11522 11523 bool permissionCheck(Cgi cgi) { 11524 return ("ok" in cgi.get) !is null; 11525 } 11526 11527 // handler for the subsection 11528 class AdminClass : WebObject { 11529 int foo() { return 5; } 11530 } 11531 11532 // handler for the main site 11533 class TheMainSite : WebObject {} 11534 11535 mixin DispatcherMain!( 11536 "/admin".dispatchSubsection!( 11537 // converts our short-form filter into a long-form filter 11538 passFilterOrIssueError!(permissionCheck, 403), 11539 // can use a new presenter if wanted for the subsection 11540 KeepExistingPresenter, 11541 // and then provide child route dispatchers 11542 "/".serveApi!AdminClass 11543 ), 11544 // and back to the top level 11545 "/".serveApi!TheMainSite 11546 ); 11547 --- 11548 11549 Note you can encapsulate sections in files like this: 11550 11551 --- 11552 auto adminDispatcher(string urlPrefix) { 11553 return urlPrefix.dispatchSubsection!( 11554 .... 11555 ); 11556 } 11557 11558 mixin DispatcherMain!( 11559 "/admin".adminDispatcher, 11560 // and so on 11561 ) 11562 --- 11563 11564 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11565 11566 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11567 11568 11569 History: 11570 Added January 28, 2023 (dub v11.0) 11571 +/ 11572 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11573 assert(urlPrefix[0] == '/'); 11574 assert(urlPrefix[$-1] != '/'); 11575 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11576 static if(!is(PreRequestFilter == typeof(null))) { 11577 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11578 return true; // we handled it by rejecting it 11579 } 11580 11581 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11582 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11583 } else { 11584 auto newPresenter = new NewPresenter(); 11585 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11586 } 11587 } 11588 11589 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11590 } 11591 11592 /++ 11593 A URL dispatcher. 11594 11595 --- 11596 if(cgi.dispatcher!( 11597 "/api/".serveApi!MyApiClass, 11598 "/objects/lol".serveRestObject!MyRestObject, 11599 "/file.js".serveStaticFile, 11600 "/admin/".dispatchTo!adminHandler 11601 )) return; 11602 --- 11603 11604 11605 You define a series of url prefixes followed by handlers. 11606 11607 You may want to do different pre- and post- processing there, for example, 11608 an authorization check and different page layout. You can use different 11609 presenters and different function chains. See [dispatchSubsection] for details. 11610 11611 [dispatchTo] will send the request to another function for handling. 11612 +/ 11613 template dispatcher(definitions...) { 11614 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11615 static if(is(Presenter == typeof(null))) { 11616 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11617 auto presenter = new GenericWebPresenter(); 11618 } else 11619 alias presenter = presenterArg; 11620 11621 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11622 } 11623 11624 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11625 // I can prolly make this more efficient later but meh. 11626 foreach(definition; definitions) { 11627 if(definition.rejectFurther) { 11628 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11629 auto ret = definition.handler( 11630 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11631 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11632 if(ret) 11633 return true; 11634 } 11635 } else if( 11636 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11637 // cgi.d dispatcher urls must be complete or have a /; 11638 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11639 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11640 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11641 ) { 11642 auto ret = definition.handler( 11643 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11644 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11645 if(ret) 11646 return true; 11647 } 11648 } 11649 return false; 11650 } 11651 } 11652 11653 }); 11654 11655 private struct StackBuffer { 11656 char[1024] initial = void; 11657 char[] buffer; 11658 size_t position; 11659 11660 this(int a) { 11661 buffer = initial[]; 11662 position = 0; 11663 } 11664 11665 void add(in char[] what) { 11666 if(position + what.length > buffer.length) 11667 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 11668 buffer[position .. position + what.length] = what[]; 11669 position += what.length; 11670 } 11671 11672 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 11673 add(w1); 11674 add(w2); 11675 add(w3); 11676 } 11677 11678 void add(long v) { 11679 char[16] buffer = void; 11680 auto pos = buffer.length; 11681 bool negative; 11682 if(v < 0) { 11683 negative = true; 11684 v = -v; 11685 } 11686 do { 11687 buffer[--pos] = cast(char) (v % 10 + '0'); 11688 v /= 10; 11689 } while(v); 11690 11691 if(negative) 11692 buffer[--pos] = '-'; 11693 11694 auto res = buffer[pos .. $]; 11695 11696 add(res[]); 11697 } 11698 11699 char[] get() @nogc { 11700 return buffer[0 .. position]; 11701 } 11702 } 11703 11704 // duplicated in http2.d 11705 private static string getHttpCodeText(int code) pure nothrow @nogc { 11706 switch(code) { 11707 case 200: return "200 OK"; 11708 case 201: return "201 Created"; 11709 case 202: return "202 Accepted"; 11710 case 203: return "203 Non-Authoritative Information"; 11711 case 204: return "204 No Content"; 11712 case 205: return "205 Reset Content"; 11713 case 206: return "206 Partial Content"; 11714 // 11715 case 300: return "300 Multiple Choices"; 11716 case 301: return "301 Moved Permanently"; 11717 case 302: return "302 Found"; 11718 case 303: return "303 See Other"; 11719 case 304: return "304 Not Modified"; 11720 case 305: return "305 Use Proxy"; 11721 case 307: return "307 Temporary Redirect"; 11722 case 308: return "308 Permanent Redirect"; 11723 11724 // 11725 case 400: return "400 Bad Request"; 11726 case 401: return "401 Unauthorized"; 11727 case 402: return "402 Payment Required"; 11728 case 403: return "403 Forbidden"; 11729 case 404: return "404 Not Found"; 11730 case 405: return "405 Method Not Allowed"; 11731 case 406: return "406 Not Acceptable"; 11732 case 407: return "407 Proxy Authentication Required"; 11733 case 408: return "408 Request Timeout"; 11734 case 409: return "409 Conflict"; 11735 case 410: return "410 Gone"; 11736 case 411: return "411 Length Required"; 11737 case 412: return "412 Precondition Failed"; 11738 case 413: return "413 Payload Too Large"; 11739 case 414: return "414 URI Too Long"; 11740 case 415: return "415 Unsupported Media Type"; 11741 case 416: return "416 Range Not Satisfiable"; 11742 case 417: return "417 Expectation Failed"; 11743 case 418: return "418 I'm a teapot"; 11744 case 421: return "421 Misdirected Request"; 11745 case 422: return "422 Unprocessable Entity (WebDAV)"; 11746 case 423: return "423 Locked (WebDAV)"; 11747 case 424: return "424 Failed Dependency (WebDAV)"; 11748 case 425: return "425 Too Early"; 11749 case 426: return "426 Upgrade Required"; 11750 case 428: return "428 Precondition Required"; 11751 case 431: return "431 Request Header Fields Too Large"; 11752 case 451: return "451 Unavailable For Legal Reasons"; 11753 11754 case 500: return "500 Internal Server Error"; 11755 case 501: return "501 Not Implemented"; 11756 case 502: return "502 Bad Gateway"; 11757 case 503: return "503 Service Unavailable"; 11758 case 504: return "504 Gateway Timeout"; 11759 case 505: return "505 HTTP Version Not Supported"; 11760 case 506: return "506 Variant Also Negotiates"; 11761 case 507: return "507 Insufficient Storage (WebDAV)"; 11762 case 508: return "508 Loop Detected (WebDAV)"; 11763 case 510: return "510 Not Extended"; 11764 case 511: return "511 Network Authentication Required"; 11765 // 11766 default: assert(0, "Unsupported http code"); 11767 } 11768 } 11769 11770 11771 /+ 11772 /++ 11773 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 11774 11775 It relies on jsvar.d and dom.d. 11776 11777 11778 You can get javascript out of it to call. The generated functions need to look 11779 like 11780 11781 function name(a,b,c,d,e) { 11782 return _call("name", {"realName":a,"sds":b}); 11783 } 11784 11785 And _call returns an object you can call or set up or whatever. 11786 +/ 11787 bool apiDispatcher()(Cgi cgi) { 11788 import arsd.jsvar; 11789 import arsd.dom; 11790 } 11791 +/ 11792 version(linux) 11793 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 11794 /* 11795 Copyright: Adam D. Ruppe, 2008 - 2023 11796 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 11797 Authors: Adam D. Ruppe 11798 11799 Copyright Adam D. Ruppe 2008 - 2023. 11800 Distributed under the Boost Software License, Version 1.0. 11801 (See accompanying file LICENSE_1_0.txt or copy at 11802 http://www.boost.org/LICENSE_1_0.txt) 11803 */