1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. Note: prior to version 11, this would be embedded_httpd_processes on Linux and embedded_httpd_threads everywhere else. It now means embedded_httpd_hybrid everywhere supported and embedded_httpd_threads everywhere else. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 490 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 491 492 /++ 493 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 494 +/ 495 version(Demo) 496 unittest { 497 import arsd.cgi; 498 499 mixin DispatcherMain!( 500 "/".serveStaticFileDirectory(null, true) 501 ); 502 } 503 504 /++ 505 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 506 +/ 507 version(Demo) 508 unittest { 509 import arsd.cgi; 510 511 void requestHandler(Cgi cgi) { 512 cgi.dispatcher!( 513 "/".serveStaticFileDirectory(null, true) 514 ); 515 } 516 517 // mixin GenericMain!requestHandler would add this function: 518 void main(string[] args) { 519 // this is all the content of [cgiMainImpl] which you can also call 520 521 // cgi.d embeds a few add on functions like real time event forwarders 522 // and session servers it can run in other processes. this spawns them, if needed. 523 if(tryAddonServers(args)) 524 return; 525 526 // cgi.d allows you to easily simulate http requests from the command line, 527 // without actually starting a server. this function will do that. 528 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 529 return; 530 531 RequestServer server; 532 // you can change the default port here if you like 533 // server.listeningPort = 9000; 534 535 // then call this to let the command line args override your default 536 server.configureFromCommandLine(args); 537 538 // here is where you could print out the listeningPort to the user if you wanted 539 540 // and serve the request(s) according to the compile configuration 541 server.serve!(requestHandler)(); 542 543 // or you could explicitly choose a serve mode like this: 544 // server.serveEmbeddedHttp!requestHandler(); 545 } 546 } 547 548 /++ 549 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 550 otherwise run through the rest of the internal mechanisms to call your functions without actually 551 spinning up a server. 552 +/ 553 version(Demo) 554 unittest { 555 import arsd.cgi; 556 557 void requestHandler(Cgi cgi) { 558 559 } 560 561 // D doesn't let me embed a unittest inside an example unittest 562 // so this is a function, but you can do it however in your real program 563 /* unittest */ void runTests() { 564 auto tester = new CgiTester(&requestHandler); 565 566 auto response = tester.GET("/"); 567 assert(response.code == 200); 568 } 569 } 570 571 static import std.file; 572 573 static import arsd.core; 574 version(Posix) 575 import arsd.core : makeNonBlocking; 576 577 578 // for a single thread, linear request thing, use: 579 // -version=embedded_httpd_threads -version=cgi_no_threads 580 581 version(Posix) { 582 version(CRuntime_Musl) { 583 584 } else version(minimal) { 585 586 } else { 587 version(FreeBSD) { 588 // I never implemented the fancy stuff there either 589 } else { 590 version=with_breaking_cgi_features; 591 version=with_sendfd; 592 version=with_addon_servers; 593 } 594 } 595 } 596 597 version(Windows) { 598 version(minimal) { 599 600 } else { 601 // not too concerned about gdc here since the mingw version is fairly new as well 602 version=with_breaking_cgi_features; 603 } 604 } 605 606 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 607 void cloexec(int fd) { 608 version(Posix) { 609 import core.sys.posix.fcntl; 610 fcntl(fd, F_SETFD, FD_CLOEXEC); 611 } 612 } 613 614 void cloexec(Socket s) { 615 version(Posix) { 616 import core.sys.posix.fcntl; 617 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 618 } 619 } 620 621 // the servers must know about the connections to talk to them; the interfaces are vital 622 version(with_addon_servers) 623 version=with_addon_servers_connections; 624 625 version(embedded_httpd) { 626 version=embedded_httpd_hybrid; 627 /* 628 version(with_openssl) { 629 pragma(lib, "crypto"); 630 pragma(lib, "ssl"); 631 } 632 */ 633 } 634 635 version(embedded_httpd_hybrid) { 636 version=embedded_httpd_threads; 637 version(cgi_no_fork) {} else version(Posix) 638 version=cgi_use_fork; 639 version=cgi_use_fiber; 640 } 641 642 version(cgi_use_fork) 643 enum cgi_use_fork_default = true; 644 else 645 enum cgi_use_fork_default = false; 646 647 version(embedded_httpd_processes) 648 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 649 650 version(embedded_httpd_threads) { 651 // unless the user overrides the default.. 652 version(cgi_session_server_process) 653 {} 654 else 655 version=cgi_embedded_sessions; 656 } 657 version(scgi) { 658 // unless the user overrides the default.. 659 version(cgi_session_server_process) 660 {} 661 else 662 version=cgi_embedded_sessions; 663 } 664 665 // fall back if the other is not defined so we can cleanly version it below 666 version(cgi_embedded_sessions) {} 667 else version=cgi_session_server_process; 668 669 670 version=cgi_with_websocket; 671 672 enum long defaultMaxContentLength = 5_000_000; 673 674 /* 675 676 To do a file download offer in the browser: 677 678 cgi.setResponseContentType("text/csv"); 679 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 680 */ 681 682 // FIXME: the location header is supposed to be an absolute url I guess. 683 684 // FIXME: would be cool to flush part of a dom document before complete 685 // somehow in here and dom.d. 686 687 688 // these are public so you can mixin GenericMain. 689 // FIXME: use a function level import instead! 690 public import std.string; 691 public import std.stdio; 692 public import std.conv; 693 import std.uri; 694 import std.uni; 695 import std.algorithm.comparison; 696 import std.algorithm.searching; 697 import std.exception; 698 import std.base64; 699 static import std.algorithm; 700 import std.datetime; 701 import std.range; 702 703 import std.process; 704 705 import std.zlib; 706 707 708 T[] consume(T)(T[] range, int count) { 709 if(count > range.length) 710 count = range.length; 711 return range[count..$]; 712 } 713 714 int locationOf(T)(T[] data, string item) { 715 const(ubyte[]) d = cast(const(ubyte[])) data; 716 const(ubyte[]) i = cast(const(ubyte[])) item; 717 718 // this is a vague sanity check to ensure we aren't getting insanely 719 // sized input that will infinite loop below. it should never happen; 720 // even huge file uploads ought to come in smaller individual pieces. 721 if(d.length > (int.max/2)) 722 throw new Exception("excessive block of input"); 723 724 for(int a = 0; a < d.length; a++) { 725 if(a + i.length > d.length) 726 return -1; 727 if(d[a..a+i.length] == i) 728 return a; 729 } 730 731 return -1; 732 } 733 734 /// If you are doing a custom cgi class, mixing this in can take care of 735 /// the required constructors for you 736 mixin template ForwardCgiConstructors() { 737 this(long maxContentLength = defaultMaxContentLength, 738 string[string] env = null, 739 const(ubyte)[] delegate() readdata = null, 740 void delegate(const(ubyte)[]) _rawDataOutput = null, 741 void delegate() _flush = null 742 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 743 744 this(string[] args) { super(args); } 745 746 this( 747 BufferedInputRange inputData, 748 string address, ushort _port, 749 int pathInfoStarts = 0, 750 bool _https = false, 751 void delegate(const(ubyte)[]) _rawDataOutput = null, 752 void delegate() _flush = null, 753 // this pointer tells if the connection is supposed to be closed after we handle this 754 bool* closeConnection = null) 755 { 756 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 757 } 758 759 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 760 } 761 762 /// thrown when a connection is closed remotely while we waiting on data from it 763 class ConnectionClosedException : Exception { 764 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 765 super(message, file, line, next); 766 } 767 } 768 769 770 version(Windows) { 771 // FIXME: ugly hack to solve stdin exception problems on Windows: 772 // reading stdin results in StdioException (Bad file descriptor) 773 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 774 private struct stdin { 775 struct ByChunk { // Replicates std.stdio.ByChunk 776 private: 777 ubyte[] chunk_; 778 public: 779 this(size_t size) 780 in { 781 assert(size, "size must be larger than 0"); 782 } 783 do { 784 chunk_ = new ubyte[](size); 785 popFront(); 786 } 787 788 @property bool empty() const { 789 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 790 } 791 @property nothrow ubyte[] front() { return chunk_; } 792 void popFront() { 793 enforce(!empty, "Cannot call popFront on empty range"); 794 chunk_ = stdin.rawRead(chunk_); 795 } 796 } 797 798 import core.sys.windows.windows; 799 static: 800 801 T[] rawRead(T)(T[] buf) { 802 uint bytesRead; 803 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 804 805 if (!result) { 806 auto err = GetLastError(); 807 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 808 return buf[0..0]; 809 // Some other error, throw it 810 811 char* buffer; 812 scope(exit) LocalFree(buffer); 813 814 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 815 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 816 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 817 throw new Exception(to!string(buffer)); 818 } 819 enforce(!(bytesRead % T.sizeof), "I/O error"); 820 return buf[0..bytesRead / T.sizeof]; 821 } 822 823 auto byChunk(size_t sz) { return ByChunk(sz); } 824 825 void close() { 826 std.stdio.stdin.close; 827 } 828 } 829 } 830 831 /// The main interface with the web request 832 class Cgi { 833 public: 834 /// the methods a request can be 835 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 836 // these are defined in the standard, but idk if they are useful for anything 837 OPTIONS, TRACE, CONNECT, 838 // These seem new, I have only recently seen them 839 PATCH, MERGE, 840 // this is an extension for when the method is not specified and you want to assume 841 CommandLine } 842 843 844 /+ 845 846 ubyte[] perRequestMemoryPool; 847 void[] perRequestMemoryPoolWithPointers; 848 // might want to just slice the buffer itself too when we happened to have gotten a full request inside it and don't need to decode 849 // then the buffer also can be recycled if it is set. 850 851 // we might also be able to set memory recyclable true by default, but then the property getters set it to false. but not all the things are property getters. but realistically anything except benchmarks are gonna get something lol so meh. 852 853 /+ 854 struct VariableCollection { 855 string[] opIndex(string name) { 856 857 } 858 } 859 860 /++ 861 Call this to indicate that you've not retained any reference to the request-local memory (including all strings returned from the Cgi object) outside the request (you can .idup anything you need to store) and it is thus free to be freed or reused by another request. 862 863 Most handlers should be able to call this; retaining memory is the exception in any cgi program, but since I can't prove it from inside the library, it plays it safe and lets the GC manage it unless you opt into this behavior. All cgi.d functions will duplicate strings if needed (e.g. session ids from cookies) so unless you're doing something yourself, this should be ok. 864 865 History: 866 Added 867 +/ 868 public void recycleMemory() { 869 870 } 871 +/ 872 873 874 /++ 875 Cgi provides a per-request memory pool 876 877 +/ 878 void[] allocateMemory(size_t nBytes) { 879 880 } 881 882 /// ditto 883 void[] reallocateMemory(void[] old, size_t nBytes) { 884 885 } 886 887 /// ditto 888 void freeMemory(void[] memory) { 889 890 } 891 +/ 892 893 894 /* 895 import core.runtime; 896 auto args = Runtime.args(); 897 898 we can call the app a few ways: 899 900 1) set up the environment variables and call the app (manually simulating CGI) 901 2) simulate a call automatically: 902 ./app method 'uri' 903 904 for example: 905 ./app get /path?arg arg2=something 906 907 Anything on the uri is treated as query string etc 908 909 on get method, further args are appended to the query string (encoded automatically) 910 on post method, further args are done as post 911 912 913 @name means import from file "name". if name == -, it uses stdin 914 (so info=@- means set info to the value of stdin) 915 916 917 Other arguments include: 918 --cookie name=value (these are all concated together) 919 --header 'X-Something: cool' 920 --referrer 'something' 921 --port 80 922 --remote-address some.ip.address.here 923 --https yes 924 --user-agent 'something' 925 --userpass 'user:pass' 926 --authorization 'Basic base64encoded_user:pass' 927 --accept 'content' // FIXME: better example 928 --last-event-id 'something' 929 --host 'something.com' 930 931 Non-simulation arguments: 932 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 933 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 934 935 */ 936 937 /** Initializes it with command line arguments (for easy testing) */ 938 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 939 rawDataOutput = _rawDataOutput; 940 // these are all set locally so the loop works 941 // without triggering errors in dmd 2.064 942 // we go ahead and set them at the end of it to the this version 943 int port; 944 string referrer; 945 string remoteAddress; 946 string userAgent; 947 string authorization; 948 string origin; 949 string accept; 950 string lastEventId; 951 bool https; 952 string host; 953 RequestMethod requestMethod; 954 string requestUri; 955 string pathInfo; 956 string queryString; 957 958 bool lookingForMethod; 959 bool lookingForUri; 960 string nextArgIs; 961 962 string _cookie; 963 string _queryString; 964 string[][string] _post; 965 string[string] _headers; 966 967 string[] breakUp(string s) { 968 string k, v; 969 auto idx = s.indexOf("="); 970 if(idx == -1) { 971 k = s; 972 } else { 973 k = s[0 .. idx]; 974 v = s[idx + 1 .. $]; 975 } 976 977 return [k, v]; 978 } 979 980 lookingForMethod = true; 981 982 scriptName = args[0]; 983 scriptFileName = args[0]; 984 985 environmentVariables = cast(const) environment.toAA; 986 987 foreach(arg; args[1 .. $]) { 988 if(arg.startsWith("--")) { 989 nextArgIs = arg[2 .. $]; 990 } else if(nextArgIs.length) { 991 if (nextArgIs == "cookie") { 992 auto info = breakUp(arg); 993 if(_cookie.length) 994 _cookie ~= "; "; 995 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 996 } 997 else if (nextArgIs == "port") { 998 port = to!int(arg); 999 } 1000 else if (nextArgIs == "referrer") { 1001 referrer = arg; 1002 } 1003 else if (nextArgIs == "remote-address") { 1004 remoteAddress = arg; 1005 } 1006 else if (nextArgIs == "user-agent") { 1007 userAgent = arg; 1008 } 1009 else if (nextArgIs == "authorization") { 1010 authorization = arg; 1011 } 1012 else if (nextArgIs == "userpass") { 1013 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 1014 } 1015 else if (nextArgIs == "origin") { 1016 origin = arg; 1017 } 1018 else if (nextArgIs == "accept") { 1019 accept = arg; 1020 } 1021 else if (nextArgIs == "last-event-id") { 1022 lastEventId = arg; 1023 } 1024 else if (nextArgIs == "https") { 1025 if(arg == "yes") 1026 https = true; 1027 } 1028 else if (nextArgIs == "header") { 1029 string thing, other; 1030 auto idx = arg.indexOf(":"); 1031 if(idx == -1) 1032 throw new Exception("need a colon in a http header"); 1033 thing = arg[0 .. idx]; 1034 other = arg[idx + 1.. $]; 1035 _headers[thing.strip.toLower()] = other.strip; 1036 } 1037 else if (nextArgIs == "host") { 1038 host = arg; 1039 } 1040 // else 1041 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1042 1043 nextArgIs = null; 1044 } else if(lookingForMethod) { 1045 lookingForMethod = false; 1046 lookingForUri = true; 1047 1048 if(arg.asLowerCase().equal("commandline")) 1049 requestMethod = RequestMethod.CommandLine; 1050 else 1051 requestMethod = to!RequestMethod(arg.toUpper()); 1052 } else if(lookingForUri) { 1053 lookingForUri = false; 1054 1055 requestUri = arg; 1056 1057 auto idx = arg.indexOf("?"); 1058 if(idx == -1) 1059 pathInfo = arg; 1060 else { 1061 pathInfo = arg[0 .. idx]; 1062 _queryString = arg[idx + 1 .. $]; 1063 } 1064 } else { 1065 // it is an argument of some sort 1066 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1067 auto parts = breakUp(arg); 1068 _post[parts[0]] ~= parts[1]; 1069 allPostNamesInOrder ~= parts[0]; 1070 allPostValuesInOrder ~= parts[1]; 1071 } else { 1072 if(_queryString.length) 1073 _queryString ~= "&"; 1074 auto parts = breakUp(arg); 1075 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1076 } 1077 } 1078 } 1079 1080 acceptsGzip = false; 1081 keepAliveRequested = false; 1082 requestHeaders = cast(immutable) _headers; 1083 1084 cookie = _cookie; 1085 cookiesArray = getCookieArray(); 1086 cookies = keepLastOf(cookiesArray); 1087 1088 queryString = _queryString; 1089 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1090 get = keepLastOf(getArray); 1091 1092 postArray = cast(immutable) _post; 1093 post = keepLastOf(_post); 1094 1095 // FIXME 1096 filesArray = null; 1097 files = null; 1098 1099 isCalledWithCommandLineArguments = true; 1100 1101 this.port = port; 1102 this.referrer = referrer; 1103 this.remoteAddress = remoteAddress; 1104 this.userAgent = userAgent; 1105 this.authorization = authorization; 1106 this.origin = origin; 1107 this.accept = accept; 1108 this.lastEventId = lastEventId; 1109 this.https = https; 1110 this.host = host; 1111 this.requestMethod = requestMethod; 1112 this.requestUri = requestUri; 1113 this.pathInfo = pathInfo; 1114 this.queryString = queryString; 1115 this.postBody = null; 1116 } 1117 1118 private { 1119 string[] allPostNamesInOrder; 1120 string[] allPostValuesInOrder; 1121 string[] allGetNamesInOrder; 1122 string[] allGetValuesInOrder; 1123 } 1124 1125 CgiConnectionHandle getOutputFileHandle() { 1126 return _outputFileHandle; 1127 } 1128 1129 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1130 1131 /** Initializes it using a CGI or CGI-like interface */ 1132 this(long maxContentLength = defaultMaxContentLength, 1133 // use this to override the environment variable listing 1134 in string[string] env = null, 1135 // and this should return a chunk of data. return empty when done 1136 const(ubyte)[] delegate() readdata = null, 1137 // finally, use this to do custom output if needed 1138 void delegate(const(ubyte)[]) _rawDataOutput = null, 1139 // to flush teh custom output 1140 void delegate() _flush = null 1141 ) 1142 { 1143 1144 // these are all set locally so the loop works 1145 // without triggering errors in dmd 2.064 1146 // we go ahead and set them at the end of it to the this version 1147 int port; 1148 string referrer; 1149 string remoteAddress; 1150 string userAgent; 1151 string authorization; 1152 string origin; 1153 string accept; 1154 string lastEventId; 1155 bool https; 1156 string host; 1157 RequestMethod requestMethod; 1158 string requestUri; 1159 string pathInfo; 1160 string queryString; 1161 1162 1163 1164 isCalledWithCommandLineArguments = false; 1165 rawDataOutput = _rawDataOutput; 1166 flushDelegate = _flush; 1167 auto getenv = delegate string(string var) { 1168 if(env is null) 1169 return std.process.environment.get(var); 1170 auto e = var in env; 1171 if(e is null) 1172 return null; 1173 return *e; 1174 }; 1175 1176 environmentVariables = env is null ? 1177 cast(const) environment.toAA : 1178 env; 1179 1180 // fetching all the request headers 1181 string[string] requestHeadersHere; 1182 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1183 if(k.startsWith("HTTP_")) { 1184 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1185 } 1186 } 1187 1188 this.requestHeaders = assumeUnique(requestHeadersHere); 1189 1190 requestUri = getenv("REQUEST_URI"); 1191 1192 cookie = getenv("HTTP_COOKIE"); 1193 cookiesArray = getCookieArray(); 1194 cookies = keepLastOf(cookiesArray); 1195 1196 referrer = getenv("HTTP_REFERER"); 1197 userAgent = getenv("HTTP_USER_AGENT"); 1198 remoteAddress = getenv("REMOTE_ADDR"); 1199 host = getenv("HTTP_HOST"); 1200 pathInfo = getenv("PATH_INFO"); 1201 1202 queryString = getenv("QUERY_STRING"); 1203 scriptName = getenv("SCRIPT_NAME"); 1204 { 1205 import core.runtime; 1206 auto sfn = getenv("SCRIPT_FILENAME"); 1207 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1208 } 1209 1210 bool iis = false; 1211 1212 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1213 if(requestUri.length == 0) { 1214 // IIS sometimes includes the script name as part of the path info - we don't want that 1215 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1216 pathInfo = pathInfo[scriptName.length .. $]; 1217 1218 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1219 1220 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1221 1222 // FIXME: this works for apache and iis... but what about others? 1223 } 1224 1225 1226 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1227 getArray = assumeUnique(ugh); 1228 get = keepLastOf(getArray); 1229 1230 1231 // NOTE: on shitpache, you need to specifically forward this 1232 authorization = getenv("HTTP_AUTHORIZATION"); 1233 // this is a hack because Apache is a shitload of fuck and 1234 // refuses to send the real header to us. Compatible 1235 // programs should send both the standard and X- versions 1236 1237 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1238 // unnecessary with mod_rewrite, so it is commented 1239 1240 //if(authorization.length == 0) // if the std is there, use it 1241 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1242 1243 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1244 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1245 port = to!int(getenv("SERVER_PORT")); 1246 else 1247 port = 0; // this was probably called from the command line 1248 1249 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1250 if(ae.length && ae.indexOf("gzip") != -1) 1251 acceptsGzip = true; 1252 1253 accept = getenv("HTTP_ACCEPT"); 1254 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1255 1256 auto ka = getenv("HTTP_CONNECTION"); 1257 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1258 keepAliveRequested = true; 1259 1260 auto or = getenv("HTTP_ORIGIN"); 1261 origin = or; 1262 1263 auto rm = getenv("REQUEST_METHOD"); 1264 if(rm.length) 1265 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1266 else 1267 requestMethod = RequestMethod.CommandLine; 1268 1269 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1270 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1271 1272 // FIXME: DOCUMENT_ROOT? 1273 1274 // FIXME: what about PUT? 1275 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1276 version(preserveData) // a hack to make forwarding simpler 1277 immutable(ubyte)[] data; 1278 size_t amountReceived = 0; 1279 auto contentType = getenv("CONTENT_TYPE"); 1280 1281 // FIXME: is this ever not going to be set? I guess it depends 1282 // on if the server de-chunks and buffers... seems like it has potential 1283 // to be slow if they did that. The spec says it is always there though. 1284 // And it has worked reliably for me all year in the live environment, 1285 // but some servers might be different. 1286 auto cls = getenv("CONTENT_LENGTH"); 1287 auto contentLength = to!size_t(cls.length ? cls : "0"); 1288 1289 immutable originalContentLength = contentLength; 1290 if(contentLength) { 1291 if(maxContentLength > 0 && contentLength > maxContentLength) { 1292 setResponseStatus("413 Request entity too large"); 1293 write("You tried to upload a file that is too large."); 1294 close(); 1295 throw new Exception("POST too large"); 1296 } 1297 prepareForIncomingDataChunks(contentType, contentLength); 1298 1299 1300 int processChunk(in ubyte[] chunk) { 1301 if(chunk.length > contentLength) { 1302 handleIncomingDataChunk(chunk[0..contentLength]); 1303 amountReceived += contentLength; 1304 contentLength = 0; 1305 return 1; 1306 } else { 1307 handleIncomingDataChunk(chunk); 1308 contentLength -= chunk.length; 1309 amountReceived += chunk.length; 1310 } 1311 if(contentLength == 0) 1312 return 1; 1313 1314 onRequestBodyDataReceived(amountReceived, originalContentLength); 1315 return 0; 1316 } 1317 1318 1319 if(readdata is null) { 1320 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1321 if(processChunk(chunk)) 1322 break; 1323 } else { 1324 // we have a custom data source.. 1325 auto chunk = readdata(); 1326 while(chunk.length) { 1327 if(processChunk(chunk)) 1328 break; 1329 chunk = readdata(); 1330 } 1331 } 1332 1333 onRequestBodyDataReceived(amountReceived, originalContentLength); 1334 postArray = assumeUnique(pps._post); 1335 filesArray = assumeUnique(pps._files); 1336 files = keepLastOf(filesArray); 1337 post = keepLastOf(postArray); 1338 this.postBody = pps.postBody; 1339 cleanUpPostDataState(); 1340 } 1341 1342 version(preserveData) 1343 originalPostData = data; 1344 } 1345 // fixme: remote_user script name 1346 1347 1348 this.port = port; 1349 this.referrer = referrer; 1350 this.remoteAddress = remoteAddress; 1351 this.userAgent = userAgent; 1352 this.authorization = authorization; 1353 this.origin = origin; 1354 this.accept = accept; 1355 this.lastEventId = lastEventId; 1356 this.https = https; 1357 this.host = host; 1358 this.requestMethod = requestMethod; 1359 this.requestUri = requestUri; 1360 this.pathInfo = pathInfo; 1361 this.queryString = queryString; 1362 } 1363 1364 /// Cleans up any temporary files. Do not use the object 1365 /// after calling this. 1366 /// 1367 /// NOTE: it is called automatically by GenericMain 1368 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1369 void dispose() { 1370 foreach(file; files) { 1371 if(!file.contentInMemory) 1372 if(std.file.exists(file.contentFilename)) 1373 std.file.remove(file.contentFilename); 1374 } 1375 } 1376 1377 private { 1378 struct PostParserState { 1379 string contentType; 1380 string boundary; 1381 string localBoundary; // the ones used at the end or something lol 1382 bool isMultipart; 1383 bool needsSavedBody; 1384 1385 ulong expectedLength; 1386 ulong contentConsumed; 1387 immutable(ubyte)[] buffer; 1388 1389 // multipart parsing state 1390 int whatDoWeWant; 1391 bool weHaveAPart; 1392 string[] thisOnesHeaders; 1393 immutable(ubyte)[] thisOnesData; 1394 1395 string postBody; 1396 1397 UploadedFile piece; 1398 bool isFile = false; 1399 1400 size_t memoryCommitted; 1401 1402 // do NOT keep mutable references to these anywhere! 1403 // I assume they are unique in the constructor once we're all done getting data. 1404 string[][string] _post; 1405 UploadedFile[][string] _files; 1406 } 1407 1408 PostParserState pps; 1409 } 1410 1411 /// This represents a file the user uploaded via a POST request. 1412 static struct UploadedFile { 1413 /// If you want to create one of these structs for yourself from some data, 1414 /// use this function. 1415 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1416 Cgi.UploadedFile f; 1417 f.filename = name; 1418 f.content = cast(immutable(ubyte)[]) data; 1419 f.contentInMemory = true; 1420 return f; 1421 } 1422 1423 string name; /// The name of the form element. 1424 string filename; /// The filename the user set. 1425 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1426 1427 /** 1428 For small files, cgi.d will buffer the uploaded file in memory, and make it 1429 directly accessible to you through the content member. I find this very convenient 1430 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1431 often want to inspect and modify the file anyway!) 1432 1433 I find the file is very large, it is undesirable to eat that much memory just 1434 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1435 to the constructor so they are accepted, cgi.d will write the content to a temporary 1436 file that you can re-read later. 1437 1438 You can override this behavior by subclassing Cgi and overriding the protected 1439 handlePostChunk method. Note that the object is not initialized when you 1440 write that method - the http headers are available, but the cgi.post method 1441 is not. You may parse the file as it streams in using this method. 1442 1443 1444 Anyway, if the file is small enough to be in memory, contentInMemory will be 1445 set to true, and the content is available in the content member. 1446 1447 If not, contentInMemory will be set to false, and the content saved in a file, 1448 whose name will be available in the contentFilename member. 1449 1450 1451 Tip: if you know you are always dealing with small files, and want the convenience 1452 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1453 a large file comes in, it simply throws an exception (and HTTP error response) 1454 instead of trying to handle it. 1455 1456 The default value of maxContentLength in the constructor is for small files. 1457 */ 1458 bool contentInMemory = true; // the default ought to always be true 1459 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1460 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1461 1462 /// 1463 ulong fileSize() const { 1464 if(contentInMemory) 1465 return content.length; 1466 import std.file; 1467 return std.file.getSize(contentFilename); 1468 1469 } 1470 1471 /// 1472 void writeToFile(string filenameToSaveTo) const { 1473 import std.file; 1474 if(contentInMemory) 1475 std.file.write(filenameToSaveTo, content); 1476 else 1477 std.file.rename(contentFilename, filenameToSaveTo); 1478 } 1479 } 1480 1481 // given a content type and length, decide what we're going to do with the data.. 1482 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1483 pps.expectedLength = contentLength; 1484 1485 auto terminator = contentType.indexOf(";"); 1486 if(terminator == -1) 1487 terminator = contentType.length; 1488 1489 pps.contentType = contentType[0 .. terminator]; 1490 auto b = contentType[terminator .. $]; 1491 if(b.length) { 1492 auto idx = b.indexOf("boundary="); 1493 if(idx != -1) { 1494 pps.boundary = b[idx + "boundary=".length .. $]; 1495 pps.localBoundary = "\r\n--" ~ pps.boundary; 1496 } 1497 } 1498 1499 // while a content type SHOULD be sent according to the RFC, it is 1500 // not required. We're told we SHOULD guess by looking at the content 1501 // but it seems to me that this only happens when it is urlencoded. 1502 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1503 pps.isMultipart = false; 1504 pps.needsSavedBody = false; 1505 } else if(pps.contentType == "multipart/form-data") { 1506 pps.isMultipart = true; 1507 enforce(pps.boundary.length, "no boundary"); 1508 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1509 // save the body so the application can handle it 1510 pps.isMultipart = false; 1511 pps.needsSavedBody = true; 1512 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1513 // save the body so the application can handle it 1514 pps.needsSavedBody = true; 1515 pps.isMultipart = false; 1516 } else { 1517 // the rest is 100% handled by the application. just save the body and send it to them 1518 pps.needsSavedBody = true; 1519 pps.isMultipart = false; 1520 } 1521 } 1522 1523 // handles streaming POST data. If you handle some other content type, you should 1524 // override this. If the data isn't the content type you want, you ought to call 1525 // super.handleIncomingDataChunk so regular forms and files still work. 1526 1527 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1528 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1529 // input anyway, so I'm not going to get too worked up about it right now. 1530 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1531 if(chunk.length == 0) 1532 return; 1533 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1534 // if we're passed big chunks, it might throw unnecessarily. 1535 // just pass it smaller chunks at a time. 1536 if(pps.isMultipart) { 1537 // multipart/form-data 1538 1539 1540 // FIXME: this might want to be factored out and factorized 1541 // need to make sure the stream hooks actually work. 1542 void pieceHasNewContent() { 1543 // we just grew the piece's buffer. Do we have to switch to file backing? 1544 if(pps.piece.contentInMemory) { 1545 if(pps.piece.content.length <= 10 * 1024 * 1024) 1546 // meh, I'm ok with it. 1547 return; 1548 else { 1549 // this is too big. 1550 if(!pps.isFile) 1551 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1552 else { 1553 // a file this large is probably acceptable though... let's use a backing file. 1554 pps.piece.contentInMemory = false; 1555 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1556 1557 int count = 0; 1558 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1559 // odds are this loop will never be entered, but we want it just in case. 1560 while(std.file.exists(pps.piece.contentFilename)) { 1561 count++; 1562 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1563 } 1564 // I hope this creates the file pretty quickly, or the loop might be useless... 1565 // FIXME: maybe I should write some kind of custom transaction here. 1566 std.file.write(pps.piece.contentFilename, pps.piece.content); 1567 1568 pps.piece.content = null; 1569 } 1570 } 1571 } else { 1572 // it's already in a file, so just append it to what we have 1573 if(pps.piece.content.length) { 1574 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1575 std.file.append(pps.piece.contentFilename, pps.piece.content); 1576 pps.piece.content = null; 1577 } 1578 } 1579 } 1580 1581 1582 void commitPart() { 1583 if(!pps.weHaveAPart) 1584 return; 1585 1586 pieceHasNewContent(); // be sure the new content is handled every time 1587 1588 if(pps.isFile) { 1589 // I'm not sure if other environments put files in post or not... 1590 // I used to not do it, but I think I should, since it is there... 1591 pps._post[pps.piece.name] ~= pps.piece.filename; 1592 pps._files[pps.piece.name] ~= pps.piece; 1593 1594 allPostNamesInOrder ~= pps.piece.name; 1595 allPostValuesInOrder ~= pps.piece.filename; 1596 } else { 1597 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1598 1599 allPostNamesInOrder ~= pps.piece.name; 1600 allPostValuesInOrder ~= cast(string) pps.piece.content; 1601 } 1602 1603 /* 1604 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1605 pps.piece.content.length < 1000 1606 ? 1607 to!string(pps.piece.content) 1608 : 1609 "too long"); 1610 */ 1611 1612 // FIXME: the limit here 1613 pps.memoryCommitted += pps.piece.content.length; 1614 1615 pps.weHaveAPart = false; 1616 pps.whatDoWeWant = 1; 1617 pps.thisOnesHeaders = null; 1618 pps.thisOnesData = null; 1619 1620 pps.piece = UploadedFile.init; 1621 pps.isFile = false; 1622 } 1623 1624 void acceptChunk() { 1625 pps.buffer ~= chunk; 1626 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1627 } 1628 1629 immutable(ubyte)[] consume(size_t howMuch) { 1630 pps.contentConsumed += howMuch; 1631 auto ret = pps.buffer[0 .. howMuch]; 1632 pps.buffer = pps.buffer[howMuch .. $]; 1633 return ret; 1634 } 1635 1636 dataConsumptionLoop: do { 1637 switch(pps.whatDoWeWant) { 1638 default: assert(0); 1639 case 0: 1640 acceptChunk(); 1641 // the format begins with two extra leading dashes, then we should be at the boundary 1642 if(pps.buffer.length < 2) 1643 return; 1644 assert(pps.buffer[0] == '-', "no leading dash"); 1645 consume(1); 1646 assert(pps.buffer[0] == '-', "no second leading dash"); 1647 consume(1); 1648 1649 pps.whatDoWeWant = 1; 1650 goto case 1; 1651 /* fallthrough */ 1652 case 1: // looking for headers 1653 // here, we should be lined up right at the boundary, which is followed by a \r\n 1654 1655 // want to keep the buffer under control in case we're under attack 1656 //stderr.writeln("here once"); 1657 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1658 // throw new Exception("wtf is up with the huge mime part headers"); 1659 1660 acceptChunk(); 1661 1662 if(pps.buffer.length < pps.boundary.length) 1663 return; // not enough data, since there should always be a boundary here at least 1664 1665 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1666 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1667 // we *should* be at the end here! 1668 assert(pps.buffer[0] == '-'); 1669 consume(1); 1670 assert(pps.buffer[0] == '-'); 1671 consume(1); 1672 1673 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1674 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1675 "not lined up on boundary " ~ pps.boundary); 1676 consume(pps.boundary.length); 1677 1678 assert(pps.buffer[0] == '-'); 1679 consume(1); 1680 assert(pps.buffer[0] == '-'); 1681 consume(1); 1682 1683 assert(pps.buffer[0] == '\r'); 1684 consume(1); 1685 assert(pps.buffer[0] == '\n'); 1686 consume(1); 1687 1688 assert(pps.buffer.length == 0); 1689 assert(pps.contentConsumed == pps.expectedLength); 1690 break dataConsumptionLoop; // we're done! 1691 } else { 1692 // we're not done yet. We should be lined up on a boundary. 1693 1694 // But, we want to ensure the headers are here before we consume anything! 1695 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1696 if(headerEndLocation == -1) 1697 return; // they *should* all be here, so we can handle them all at once. 1698 1699 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1700 "not lined up on boundary " ~ pps.boundary); 1701 1702 consume(pps.boundary.length); 1703 // the boundary is always followed by a \r\n 1704 assert(pps.buffer[0] == '\r'); 1705 consume(1); 1706 assert(pps.buffer[0] == '\n'); 1707 consume(1); 1708 } 1709 1710 // re-running since by consuming the boundary, we invalidate the old index. 1711 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1712 assert(headerEndLocation >= 0, "no header"); 1713 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1714 1715 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1716 1717 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1718 1719 // now we'll parse the headers 1720 foreach(h; pps.thisOnesHeaders) { 1721 auto p = h.indexOf(":"); 1722 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1723 string hn = h[0..p]; 1724 string hv = h[p+2..$]; 1725 1726 switch(hn.toLower) { 1727 default: assert(0); 1728 case "content-disposition": 1729 auto info = hv.split("; "); 1730 foreach(i; info[1..$]) { // skipping the form-data 1731 auto o = i.split("="); // FIXME 1732 string pn = o[0]; 1733 string pv = o[1][1..$-1]; 1734 1735 if(pn == "name") { 1736 pps.piece.name = pv; 1737 } else if (pn == "filename") { 1738 pps.piece.filename = pv; 1739 pps.isFile = true; 1740 } 1741 } 1742 break; 1743 case "content-type": 1744 pps.piece.contentType = hv; 1745 break; 1746 } 1747 } 1748 1749 pps.whatDoWeWant++; // move to the next step - the data 1750 break; 1751 case 2: 1752 // when we get here, pps.buffer should contain our first chunk of data 1753 1754 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1755 throw new Exception("wtf is up with the huge mime part buffer"); 1756 1757 acceptChunk(); 1758 1759 // so the trick is, we want to process all the data up to the boundary, 1760 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1761 // want to wait for the next chunk. We start by looking for the whole boundary 1762 // in the buffer somewhere. 1763 1764 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1765 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1766 if(boundaryLocation != -1) { 1767 // this is easy - we can see it in it's entirety! 1768 1769 pps.piece.content ~= consume(boundaryLocation); 1770 1771 assert(pps.buffer[0] == '\r'); 1772 consume(1); 1773 assert(pps.buffer[0] == '\n'); 1774 consume(1); 1775 assert(pps.buffer[0] == '-'); 1776 consume(1); 1777 assert(pps.buffer[0] == '-'); 1778 consume(1); 1779 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1780 pps.weHaveAPart = true; 1781 pps.whatDoWeWant = 1; // back to getting headers for the next part 1782 1783 commitPart(); // we're done here 1784 } else { 1785 // we can't see the whole thing, but what if there's a partial boundary? 1786 1787 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1788 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1789 bool potentialBoundaryFound = false; 1790 1791 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1792 // we grow the boundary a bit each time. If we think it looks the 1793 // same, better pull another chunk to be sure it's not the end. 1794 // Starting small because exiting the loop early is desirable, since 1795 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1796 // the best we can do. 1797 if(a > pps.buffer.length) 1798 break; // FIXME: is this right? 1799 assert(a <= pps.buffer.length); 1800 assert(a > 0); 1801 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1802 // ok, there *might* be a boundary here, so let's 1803 // not treat the end as data yet. The rest is good to 1804 // use though, since if there was a boundary there, we'd 1805 // have handled it up above after locationOf. 1806 1807 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1808 consume(pps.buffer.length - a); 1809 pieceHasNewContent(); 1810 potentialBoundaryFound = true; 1811 break boundaryCheck; 1812 } 1813 } 1814 1815 if(!potentialBoundaryFound) { 1816 // we can consume the whole thing 1817 pps.piece.content ~= pps.buffer; 1818 pieceHasNewContent(); 1819 consume(pps.buffer.length); 1820 } else { 1821 // we found a possible boundary, but there was 1822 // insufficient data to be sure. 1823 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1824 1825 return; // wait for the next chunk. 1826 } 1827 } 1828 } 1829 } while(pps.buffer.length); 1830 1831 // btw all boundaries except the first should have a \r\n before them 1832 } else { 1833 // application/x-www-form-urlencoded and application/json 1834 1835 // not using maxContentLength because that might be cranked up to allow 1836 // large file uploads. We can handle them, but a huge post[] isn't any good. 1837 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1838 throw new Exception("wtf is up with such a gigantic form submission????"); 1839 1840 pps.buffer ~= chunk; 1841 1842 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1843 if(pps.buffer.length == pps.expectedLength) { 1844 if(pps.needsSavedBody) 1845 pps.postBody = cast(string) pps.buffer; 1846 else 1847 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1848 version(preserveData) 1849 originalPostData = pps.buffer; 1850 } else { 1851 // just for debugging 1852 } 1853 } 1854 } 1855 1856 protected void cleanUpPostDataState() { 1857 pps = PostParserState.init; 1858 } 1859 1860 /// you can override this function to somehow react 1861 /// to an upload in progress. 1862 /// 1863 /// Take note that parts of the CGI object is not yet 1864 /// initialized! Stuff from HTTP headers, including get[], is usable. 1865 /// But, none of post[] is usable, and you cannot write here. That's 1866 /// why this method is const - mutating the object won't do much anyway. 1867 /// 1868 /// My idea here was so you can output a progress bar or 1869 /// something to a cooperative client (see arsd.rtud for a potential helper) 1870 /// 1871 /// The default is to do nothing. Subclass cgi and use the 1872 /// CustomCgiMain mixin to do something here. 1873 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1874 // This space intentionally left blank. 1875 } 1876 1877 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1878 /// *closeConnection will be set to true if you should close the connection after handling this request 1879 this(BufferedInputRange ir, bool* closeConnection) { 1880 isCalledWithCommandLineArguments = false; 1881 import al = std.algorithm; 1882 1883 immutable(ubyte)[] data; 1884 1885 void rdo(const(ubyte)[] d) { 1886 //import std.stdio; writeln(d); 1887 sendAll(ir.source, d); 1888 } 1889 1890 auto ira = ir.source.remoteAddress(); 1891 auto irLocalAddress = ir.source.localAddress(); 1892 1893 ushort port = 80; 1894 if(auto ia = cast(InternetAddress) irLocalAddress) { 1895 port = ia.port; 1896 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1897 port = ia.port; 1898 } 1899 1900 // that check for UnixAddress is to work around a Phobos bug 1901 // see: https://github.com/dlang/phobos/pull/7383 1902 // but this might be more useful anyway tbh for this case 1903 version(Posix) 1904 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1905 else 1906 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1907 } 1908 1909 /** 1910 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1911 1912 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1913 1914 Params: 1915 inputData = the incoming data, including headers and other raw http data. 1916 When the constructor exits, it will leave this range exactly at the start of 1917 the next request on the connection (if there is one). 1918 1919 address = the IP address of the remote user 1920 _port = the port number of the connection 1921 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1922 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1923 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1924 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1925 closeConnection = if the request asks to close the connection, *closeConnection == true. 1926 */ 1927 this( 1928 BufferedInputRange inputData, 1929 // string[] headers, immutable(ubyte)[] data, 1930 string address, ushort _port, 1931 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1932 bool _https = false, 1933 void delegate(const(ubyte)[]) _rawDataOutput = null, 1934 void delegate() _flush = null, 1935 // this pointer tells if the connection is supposed to be closed after we handle this 1936 bool* closeConnection = null) 1937 { 1938 // these are all set locally so the loop works 1939 // without triggering errors in dmd 2.064 1940 // we go ahead and set them at the end of it to the this version 1941 int port; 1942 string referrer; 1943 string remoteAddress; 1944 string userAgent; 1945 string authorization; 1946 string origin; 1947 string accept; 1948 string lastEventId; 1949 bool https; 1950 string host; 1951 RequestMethod requestMethod; 1952 string requestUri; 1953 string pathInfo; 1954 string queryString; 1955 string scriptName; 1956 string[string] get; 1957 string[][string] getArray; 1958 bool keepAliveRequested; 1959 bool acceptsGzip; 1960 string cookie; 1961 1962 1963 1964 environmentVariables = cast(const) environment.toAA; 1965 1966 idlol = inputData; 1967 1968 isCalledWithCommandLineArguments = false; 1969 1970 https = _https; 1971 port = _port; 1972 1973 rawDataOutput = _rawDataOutput; 1974 flushDelegate = _flush; 1975 nph = true; 1976 1977 remoteAddress = address; 1978 1979 // streaming parser 1980 import al = std.algorithm; 1981 1982 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1983 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1984 while(idx == -1) { 1985 inputData.popFront(0); 1986 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1987 } 1988 1989 assert(idx != -1); 1990 1991 1992 string contentType = ""; 1993 string[string] requestHeadersHere; 1994 1995 size_t contentLength; 1996 1997 bool isChunked; 1998 1999 { 2000 import core.runtime; 2001 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 2002 } 2003 2004 2005 int headerNumber = 0; 2006 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 2007 if(line.length) { 2008 headerNumber++; 2009 auto header = cast(string) line.idup; 2010 if(headerNumber == 1) { 2011 // request line 2012 auto parts = al.splitter(header, " "); 2013 if(parts.front == "PRI") { 2014 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 2015 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 2016 // to bring me benefit) 2017 throw new HttpVersionNotSupportedException(); 2018 } 2019 requestMethod = to!RequestMethod(parts.front); 2020 parts.popFront(); 2021 requestUri = parts.front; 2022 2023 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 2024 scriptName = requestUri[0 .. pathInfoStarts]; 2025 2026 auto question = requestUri.indexOf("?"); 2027 if(question == -1) { 2028 queryString = ""; 2029 // FIXME: double check, this might be wrong since it could be url encoded 2030 pathInfo = requestUri[pathInfoStarts..$]; 2031 } else { 2032 queryString = requestUri[question+1..$]; 2033 pathInfo = requestUri[pathInfoStarts..question]; 2034 } 2035 2036 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2037 getArray = cast(string[][string]) assumeUnique(ugh); 2038 2039 if(header.indexOf("HTTP/1.0") != -1) { 2040 http10 = true; 2041 autoBuffer = true; 2042 if(closeConnection) { 2043 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2044 *closeConnection = true; 2045 } 2046 } 2047 } else { 2048 // other header 2049 auto colon = header.indexOf(":"); 2050 if(colon == -1) 2051 throw new Exception("HTTP headers should have a colon!"); 2052 string name = header[0..colon].toLower; 2053 string value = header[colon+2..$]; // skip the colon and the space 2054 2055 requestHeadersHere[name] = value; 2056 2057 if (name == "accept") { 2058 accept = value; 2059 } 2060 else if (name == "origin") { 2061 origin = value; 2062 } 2063 else if (name == "connection") { 2064 if(value == "close" && closeConnection) 2065 *closeConnection = true; 2066 if(value.asLowerCase().canFind("keep-alive")) { 2067 keepAliveRequested = true; 2068 2069 // on http 1.0, the connection is closed by default, 2070 // but not if they request keep-alive. then we don't close 2071 // anymore - undoing the set above 2072 if(http10 && closeConnection) { 2073 *closeConnection = false; 2074 } 2075 } 2076 } 2077 else if (name == "transfer-encoding") { 2078 if(value == "chunked") 2079 isChunked = true; 2080 } 2081 else if (name == "last-event-id") { 2082 lastEventId = value; 2083 } 2084 else if (name == "authorization") { 2085 authorization = value; 2086 } 2087 else if (name == "content-type") { 2088 contentType = value; 2089 } 2090 else if (name == "content-length") { 2091 contentLength = to!size_t(value); 2092 } 2093 else if (name == "x-forwarded-for") { 2094 remoteAddress = value; 2095 } 2096 else if (name == "x-forwarded-host" || name == "host") { 2097 if(name != "host" || host is null) 2098 host = value; 2099 } 2100 // FIXME: https://tools.ietf.org/html/rfc7239 2101 else if (name == "accept-encoding") { 2102 if(value.indexOf("gzip") != -1) 2103 acceptsGzip = true; 2104 } 2105 else if (name == "user-agent") { 2106 userAgent = value; 2107 } 2108 else if (name == "referer") { 2109 referrer = value; 2110 } 2111 else if (name == "cookie") { 2112 cookie ~= value; 2113 } else if(name == "expect") { 2114 if(value == "100-continue") { 2115 // FIXME we should probably give user code a chance 2116 // to process and reject but that needs to be virtual, 2117 // perhaps part of the CGI redesign. 2118 2119 // FIXME: if size is > max content length it should 2120 // also fail at this point. 2121 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2122 2123 // FIXME: let the user write out 103 early hints too 2124 } 2125 } 2126 // else 2127 // ignore it 2128 2129 } 2130 } 2131 2132 inputData.consume(idx + 4); 2133 // done 2134 2135 requestHeaders = assumeUnique(requestHeadersHere); 2136 2137 ByChunkRange dataByChunk; 2138 2139 // reading Content-Length type data 2140 // We need to read up the data we have, and write it out as a chunk. 2141 if(!isChunked) { 2142 dataByChunk = byChunk(inputData, contentLength); 2143 } else { 2144 // chunked requests happen, but not every day. Since we need to know 2145 // the content length (for now, maybe that should change), we'll buffer 2146 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2147 auto data = dechunk(inputData); 2148 2149 // set the range here 2150 dataByChunk = byChunk(data); 2151 contentLength = data.length; 2152 } 2153 2154 assert(dataByChunk !is null); 2155 2156 if(contentLength) { 2157 prepareForIncomingDataChunks(contentType, contentLength); 2158 foreach(dataChunk; dataByChunk) { 2159 handleIncomingDataChunk(dataChunk); 2160 } 2161 postArray = assumeUnique(pps._post); 2162 filesArray = assumeUnique(pps._files); 2163 files = keepLastOf(filesArray); 2164 post = keepLastOf(postArray); 2165 postBody = pps.postBody; 2166 cleanUpPostDataState(); 2167 } 2168 2169 this.port = port; 2170 this.referrer = referrer; 2171 this.remoteAddress = remoteAddress; 2172 this.userAgent = userAgent; 2173 this.authorization = authorization; 2174 this.origin = origin; 2175 this.accept = accept; 2176 this.lastEventId = lastEventId; 2177 this.https = https; 2178 this.host = host; 2179 this.requestMethod = requestMethod; 2180 this.requestUri = requestUri; 2181 this.pathInfo = pathInfo; 2182 this.queryString = queryString; 2183 2184 this.scriptName = scriptName; 2185 this.get = keepLastOf(getArray); 2186 this.getArray = cast(immutable) getArray; 2187 this.keepAliveRequested = keepAliveRequested; 2188 this.acceptsGzip = acceptsGzip; 2189 this.cookie = cookie; 2190 2191 cookiesArray = getCookieArray(); 2192 cookies = keepLastOf(cookiesArray); 2193 2194 } 2195 BufferedInputRange idlol; 2196 2197 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2198 string[string] ca; 2199 foreach(k, v; arr) 2200 ca[k] = v[$-1]; 2201 2202 return assumeUnique(ca); 2203 } 2204 2205 // FIXME duplication 2206 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2207 UploadedFile[string] ca; 2208 foreach(k, v; arr) 2209 ca[k] = v[$-1]; 2210 2211 return assumeUnique(ca); 2212 } 2213 2214 2215 private immutable(string[][string]) getCookieArray() { 2216 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2217 return assumeUnique(forTheLoveOfGod); 2218 } 2219 2220 /// Very simple method to require a basic auth username and password. 2221 /// If the http request doesn't include the required credentials, it throws a 2222 /// HTTP 401 error, and an exception. 2223 /// 2224 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 2225 /// the user's credentials are sent in plain text on every request. 2226 /// 2227 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2228 /// application. Either use Apache's built in methods for basic authentication, or add 2229 /// something along these lines to your server configuration: 2230 /// 2231 /// RewriteEngine On 2232 /// RewriteCond %{HTTP:Authorization} ^(.*) 2233 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2234 /// 2235 /// To ensure the necessary data is available to cgi.d. 2236 void requireBasicAuth(string user, string pass, string message = null) { 2237 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2238 setResponseStatus("401 Authorization Required"); 2239 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 2240 close(); 2241 throw new Exception("Not authorized; got " ~ authorization); 2242 } 2243 } 2244 2245 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2246 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2247 /// Use setResponseExpires and updateResponseExpires for more control 2248 void setCache(bool allowCaching) { 2249 noCache = !allowCaching; 2250 } 2251 2252 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2253 /// who can accept it 2254 bool gzipResponse; 2255 2256 immutable bool acceptsGzip; 2257 immutable bool keepAliveRequested; 2258 2259 /// Set to true if and only if this was initialized with command line arguments 2260 immutable bool isCalledWithCommandLineArguments; 2261 2262 /// This gets a full url for the current request, including port, protocol, host, path, and query 2263 string getCurrentCompleteUri() const { 2264 ushort defaultPort = https ? 443 : 80; 2265 2266 string uri = "http"; 2267 if(https) 2268 uri ~= "s"; 2269 uri ~= "://"; 2270 uri ~= host; 2271 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2272 version(none) 2273 if(!(!port || port == defaultPort)) { 2274 uri ~= ":"; 2275 uri ~= to!string(port); 2276 } 2277 +/ 2278 uri ~= requestUri; 2279 return uri; 2280 } 2281 2282 /// You can override this if your site base url isn't the same as the script name 2283 string logicalScriptName() const { 2284 return scriptName; 2285 } 2286 2287 /++ 2288 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2289 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2290 Note setResponseStatus() must be called *before* you write() any data to the output. 2291 2292 History: 2293 The `int` overload was added on January 11, 2021. 2294 +/ 2295 void setResponseStatus(string status) { 2296 assert(!outputtedResponseData); 2297 responseStatus = status; 2298 } 2299 /// ditto 2300 void setResponseStatus(int statusCode) { 2301 setResponseStatus(getHttpCodeText(statusCode)); 2302 } 2303 private string responseStatus = null; 2304 2305 /// Returns true if it is still possible to output headers 2306 bool canOutputHeaders() { 2307 return !isClosed && !outputtedResponseData; 2308 } 2309 2310 /// Sets the location header, which the browser will redirect the user to automatically. 2311 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2312 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2313 void setResponseLocation(string uri, bool important = true, string status = null) { 2314 if(!important && isCurrentResponseLocationImportant) 2315 return; // important redirects always override unimportant ones 2316 2317 if(uri is null) { 2318 responseStatus = "200 OK"; 2319 responseLocation = null; 2320 isCurrentResponseLocationImportant = important; 2321 return; // this just cancels the redirect 2322 } 2323 2324 assert(!outputtedResponseData); 2325 if(status is null) 2326 responseStatus = "302 Found"; 2327 else 2328 responseStatus = status; 2329 2330 responseLocation = uri.strip; 2331 isCurrentResponseLocationImportant = important; 2332 } 2333 protected string responseLocation = null; 2334 private bool isCurrentResponseLocationImportant = false; 2335 2336 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2337 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2338 /// Note: the when parameter is different than setCookie's expire parameter. 2339 void setResponseExpires(long when, bool isPublic = false) { 2340 responseExpires = when; 2341 setCache(true); // need to enable caching so the date has meaning 2342 2343 responseIsPublic = isPublic; 2344 responseExpiresRelative = false; 2345 } 2346 2347 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2348 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2349 responseExpires = whenFromNow; 2350 setCache(true); // need to enable caching so the date has meaning 2351 2352 responseIsPublic = isPublic; 2353 responseExpiresRelative = true; 2354 } 2355 private long responseExpires = long.min; 2356 private bool responseIsPublic = false; 2357 private bool responseExpiresRelative = false; 2358 2359 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2360 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2361 /// output as a whole is as cacheable as the least cachable part in the chain. 2362 2363 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2364 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2365 void updateResponseExpires(long when, bool isPublic) { 2366 if(responseExpires == long.min) 2367 setResponseExpires(when, isPublic); 2368 else if(when < responseExpires) 2369 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2370 } 2371 2372 /* 2373 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2374 /// Should generally be false if the user is logged in. It assumes private cache only. 2375 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2376 void setPublicCaching(bool allowPublicCaches) { 2377 publicCaching = allowPublicCaches; 2378 } 2379 private bool publicCaching = false; 2380 */ 2381 2382 /++ 2383 History: 2384 Added January 11, 2021 2385 +/ 2386 enum SameSitePolicy { 2387 Lax, 2388 Strict, 2389 None 2390 } 2391 2392 /++ 2393 Sets an HTTP cookie, automatically encoding the data to the correct string. 2394 expiresIn is how many milliseconds in the future the cookie will expire. 2395 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2396 Note setCookie() must be called *before* you write() any data to the output. 2397 2398 History: 2399 Parameter `sameSitePolicy` was added on January 11, 2021. 2400 +/ 2401 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2402 assert(!outputtedResponseData); 2403 string cookie = std.uri.encodeComponent(name) ~ "="; 2404 cookie ~= std.uri.encodeComponent(data); 2405 if(path !is null) 2406 cookie ~= "; path=" ~ path; 2407 // FIXME: should I just be using max-age here? (also in cache below) 2408 if(expiresIn != 0) 2409 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2410 if(domain !is null) 2411 cookie ~= "; domain=" ~ domain; 2412 if(secure == true) 2413 cookie ~= "; Secure"; 2414 if(httpOnly == true ) 2415 cookie ~= "; HttpOnly"; 2416 final switch(sameSitePolicy) { 2417 case SameSitePolicy.Lax: 2418 cookie ~= "; SameSite=Lax"; 2419 break; 2420 case SameSitePolicy.Strict: 2421 cookie ~= "; SameSite=Strict"; 2422 break; 2423 case SameSitePolicy.None: 2424 cookie ~= "; SameSite=None"; 2425 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2426 break; 2427 } 2428 2429 if(auto idx = name in cookieIndexes) { 2430 responseCookies[*idx] = cookie; 2431 } else { 2432 cookieIndexes[name] = responseCookies.length; 2433 responseCookies ~= cookie; 2434 } 2435 } 2436 private string[] responseCookies; 2437 private size_t[string] cookieIndexes; 2438 2439 /// Clears a previously set cookie with the given name, path, and domain. 2440 void clearCookie(string name, string path = null, string domain = null) { 2441 assert(!outputtedResponseData); 2442 setCookie(name, "", 1, path, domain); 2443 } 2444 2445 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2446 void setResponseContentType(string ct) { 2447 assert(!outputtedResponseData); 2448 responseContentType = ct; 2449 } 2450 private string responseContentType = null; 2451 2452 /// Adds a custom header. It should be the name: value, but without any line terminator. 2453 /// For example: header("X-My-Header: Some value"); 2454 /// Note you should use the specialized functions in this object if possible to avoid 2455 /// duplicates in the output. 2456 void header(string h) { 2457 customHeaders ~= h; 2458 } 2459 2460 /++ 2461 I named the original function `header` after PHP, but this pattern more fits 2462 the rest of the Cgi object. 2463 2464 Either name are allowed. 2465 2466 History: 2467 Alias added June 17, 2022. 2468 +/ 2469 alias setResponseHeader = header; 2470 2471 private string[] customHeaders; 2472 private bool websocketMode; 2473 2474 void flushHeaders(const(void)[] t, bool isAll = false) { 2475 StackBuffer buffer = StackBuffer(0); 2476 2477 prepHeaders(t, isAll, &buffer); 2478 2479 if(rawDataOutput !is null) 2480 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2481 else { 2482 stdout.rawWrite(buffer.get()); 2483 } 2484 } 2485 2486 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2487 string terminator = "\n"; 2488 if(rawDataOutput !is null) 2489 terminator = "\r\n"; 2490 2491 if(responseStatus !is null) { 2492 if(nph) { 2493 if(http10) 2494 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2495 else 2496 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2497 } else 2498 buffer.add("Status: ", responseStatus, terminator); 2499 } else if (nph) { 2500 if(http10) 2501 buffer.add("HTTP/1.0 200 OK", terminator); 2502 else 2503 buffer.add("HTTP/1.1 200 OK", terminator); 2504 } 2505 2506 if(websocketMode) 2507 goto websocket; 2508 2509 if(nph) { // we're responsible for setting the date too according to http 1.1 2510 char[29] db = void; 2511 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2512 buffer.add("Date: ", db[], terminator); 2513 } 2514 2515 // FIXME: what if the user wants to set his own content-length? 2516 // The custom header function can do it, so maybe that's best. 2517 // Or we could reuse the isAll param. 2518 if(responseLocation !is null) { 2519 buffer.add("Location: ", responseLocation, terminator); 2520 } 2521 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2522 if(responseExpiresRelative) { 2523 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2524 buffer.add(responseExpires); 2525 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2526 } else { 2527 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2528 char[29] db = void; 2529 printDateToBuffer(cast(DateTime) expires, db[]); 2530 buffer.add("Expires: ", db[], terminator); 2531 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2532 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2533 buffer.add(terminator); 2534 } 2535 } 2536 if(responseCookies !is null && responseCookies.length > 0) { 2537 foreach(c; responseCookies) 2538 buffer.add("Set-Cookie: ", c, terminator); 2539 } 2540 if(noCache) { // we specifically do not want caching (this is actually the default) 2541 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2542 buffer.add("Expires: 0", terminator); 2543 buffer.add("Pragma: no-cache", terminator); 2544 } else { 2545 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2546 buffer.add("Cache-Control: public", terminator); 2547 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2548 } 2549 } 2550 if(responseContentType !is null) { 2551 buffer.add("Content-Type: ", responseContentType, terminator); 2552 } else 2553 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2554 2555 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2556 buffer.add("Content-Encoding: gzip", terminator); 2557 } 2558 2559 2560 if(!isAll) { 2561 if(nph && !http10) { 2562 buffer.add("Transfer-Encoding: chunked", terminator); 2563 responseChunked = true; 2564 } 2565 } else { 2566 buffer.add("Content-Length: "); 2567 buffer.add(t.length); 2568 buffer.add(terminator); 2569 if(nph && keepAliveRequested) { 2570 buffer.add("Connection: Keep-Alive", terminator); 2571 } 2572 } 2573 2574 websocket: 2575 2576 foreach(hd; customHeaders) 2577 buffer.add(hd, terminator); 2578 2579 // FIXME: what about duplicated headers? 2580 2581 // end of header indicator 2582 buffer.add(terminator); 2583 2584 outputtedResponseData = true; 2585 } 2586 2587 /// Writes the data to the output, flushing headers if they have not yet been sent. 2588 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2589 assert(!closed, "Output has already been closed"); 2590 2591 StackBuffer buffer = StackBuffer(0); 2592 2593 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2594 // actually gzip the data here 2595 2596 auto c = new Compress(HeaderFormat.gzip); // want gzip 2597 2598 auto data = c.compress(t); 2599 data ~= c.flush(); 2600 2601 // std.file.write("/tmp/last-item", data); 2602 2603 t = data; 2604 } 2605 2606 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2607 prepHeaders(t, isAll, &buffer); 2608 } 2609 2610 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2611 if (autoBuffer && !isAll) { 2612 outputBuffer ~= cast(ubyte[]) t; 2613 } 2614 if(!autoBuffer || isAll) { 2615 if(rawDataOutput !is null) 2616 if(nph && responseChunked) { 2617 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2618 // we're making the chunk here instead of in a function 2619 // to avoid unneeded gc pressure 2620 buffer.add(toHex(t.length)); 2621 buffer.add("\r\n"); 2622 buffer.add(cast(char[]) t, "\r\n"); 2623 } else { 2624 buffer.add(cast(char[]) t); 2625 } 2626 else 2627 buffer.add(cast(char[]) t); 2628 } 2629 } 2630 2631 if(rawDataOutput !is null) 2632 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2633 else 2634 stdout.rawWrite(buffer.get()); 2635 2636 if(maybeAutoClose && isAll) 2637 close(); // if you say it is all, that means we're definitely done 2638 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2639 } 2640 2641 /++ 2642 Convenience method to set content type to json and write the string as the complete response. 2643 2644 History: 2645 Added January 16, 2020 2646 +/ 2647 void writeJson(string json) { 2648 this.setResponseContentType("application/json"); 2649 this.write(json, true); 2650 } 2651 2652 /// Flushes the pending buffer, leaving the connection open so you can send more. 2653 void flush() { 2654 if(rawDataOutput is null) 2655 stdout.flush(); 2656 else if(flushDelegate !is null) 2657 flushDelegate(); 2658 } 2659 2660 version(autoBuffer) 2661 bool autoBuffer = true; 2662 else 2663 bool autoBuffer = false; 2664 ubyte[] outputBuffer; 2665 2666 /// Flushes the buffers to the network, signifying that you are done. 2667 /// You should always call this explicitly when you are done outputting data. 2668 void close() { 2669 if(closed) 2670 return; // don't double close 2671 2672 if(!outputtedResponseData) 2673 write("", true, false); 2674 2675 // writing auto buffered data 2676 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2677 if(!nph) 2678 stdout.rawWrite(outputBuffer); 2679 else 2680 write(outputBuffer, true, false); // tell it this is everything 2681 } 2682 2683 // closing the last chunk... 2684 if(nph && rawDataOutput !is null && responseChunked) 2685 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2686 2687 if(flushDelegate) 2688 flushDelegate(); 2689 2690 closed = true; 2691 } 2692 2693 // Closes without doing anything, shouldn't be used often 2694 void rawClose() { 2695 closed = true; 2696 } 2697 2698 /++ 2699 Gets a request variable as a specific type, or the default value of it isn't there 2700 or isn't convertible to the request type. 2701 2702 Checks both GET and POST variables, preferring the POST variable, if available. 2703 2704 A nice trick is using the default value to choose the type: 2705 2706 --- 2707 /* 2708 The return value will match the type of the default. 2709 Here, I gave 10 as a default, so the return value will 2710 be an int. 2711 2712 If the user-supplied value cannot be converted to the 2713 requested type, you will get the default value back. 2714 */ 2715 int a = cgi.request("number", 10); 2716 2717 if(cgi.get["number"] == "11") 2718 assert(a == 11); // conversion succeeds 2719 2720 if("number" !in cgi.get) 2721 assert(a == 10); // no value means you can't convert - give the default 2722 2723 if(cgi.get["number"] == "twelve") 2724 assert(a == 10); // conversion from string to int would fail, so we get the default 2725 --- 2726 2727 You can use an enum as an easy whitelist, too: 2728 2729 --- 2730 enum Operations { 2731 add, remove, query 2732 } 2733 2734 auto op = cgi.request("op", Operations.query); 2735 2736 if(cgi.get["op"] == "add") 2737 assert(op == Operations.add); 2738 if(cgi.get["op"] == "remove") 2739 assert(op == Operations.remove); 2740 if(cgi.get["op"] == "query") 2741 assert(op == Operations.query); 2742 2743 if(cgi.get["op"] == "random string") 2744 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2745 --- 2746 +/ 2747 T request(T = string)(in string name, in T def = T.init) const nothrow { 2748 try { 2749 return 2750 (name in post) ? to!T(post[name]) : 2751 (name in get) ? to!T(get[name]) : 2752 def; 2753 } catch(Exception e) { return def; } 2754 } 2755 2756 /// Is the output already closed? 2757 bool isClosed() const { 2758 return closed; 2759 } 2760 2761 /++ 2762 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2763 +/ 2764 Session!Data getSessionObject(Data)() { 2765 if(testInProcess !is null) { 2766 // test mode 2767 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2768 if(obj !is null) 2769 return cast(typeof(return)) obj; 2770 else { 2771 auto o = new MockSession!Data(); 2772 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2773 return o; 2774 } 2775 } else { 2776 // normal operation 2777 return new BasicDataServerSession!Data(this); 2778 } 2779 } 2780 2781 // if it is in test mode; triggers mock sessions. Used by CgiTester 2782 version(with_breaking_cgi_features) 2783 private CgiTester testInProcess; 2784 2785 /* Hooks for redirecting input and output */ 2786 private void delegate(const(ubyte)[]) rawDataOutput = null; 2787 private void delegate() flushDelegate = null; 2788 2789 /* This info is used when handling a more raw HTTP protocol */ 2790 private bool nph; 2791 private bool http10; 2792 private bool closed; 2793 private bool responseChunked = false; 2794 2795 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2796 immutable(ubyte)[] originalPostData; 2797 2798 /++ 2799 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2800 2801 It is intended to be used for JSON and XML request content types, but also may be used 2802 for other content types your application can handle. But it will NOT be populated 2803 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2804 parsed into the post and postArray members. 2805 2806 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2807 will be discarded to the client with an error. This helps keep this array from being exploded in size 2808 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2809 client in certain build modes.) 2810 2811 History: 2812 Added January 5, 2021 2813 Documented February 21, 2023 (dub v11.0) 2814 +/ 2815 public immutable string postBody; 2816 alias postJson = postBody; // old name 2817 2818 /* Internal state flags */ 2819 private bool outputtedResponseData; 2820 private bool noCache = true; 2821 2822 const(string[string]) environmentVariables; 2823 2824 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2825 partially because it logically is (your code doesn't change what the user requested...) 2826 and partially because I hate how bad programs in PHP change those superglobals to do 2827 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2828 2829 For some of these, you'll want to refer to the http or cgi specs for more details. 2830 */ 2831 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2832 2833 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2834 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2835 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2836 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2837 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2838 immutable(char[]) scriptFileName; /// The physical filename of your script 2839 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2840 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2841 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2842 2843 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2844 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2845 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2846 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2847 2848 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2849 */ 2850 immutable(char[]) referrer; 2851 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2852 2853 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2854 2855 immutable bool https; /// Was the request encrypted via https? 2856 immutable int port; /// On what TCP port number did the server receive the request? 2857 2858 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2859 2860 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2861 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2862 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2863 2864 /** 2865 Represents user uploaded files. 2866 2867 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2868 */ 2869 immutable(UploadedFile[][string]) filesArray; 2870 immutable(UploadedFile[string]) files; 2871 2872 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2873 /// the order of the arrays is the order the data arrives 2874 immutable(string[][string]) getArray; /// like get, but an array of values per name 2875 immutable(string[][string]) postArray; /// ditto for post 2876 immutable(string[][string]) cookiesArray; /// ditto for cookies 2877 2878 // convenience function for appending to a uri without extra ? 2879 // matches the name and effect of javascript's location.search property 2880 string search() const { 2881 if(queryString.length) 2882 return "?" ~ queryString; 2883 return ""; 2884 } 2885 2886 // FIXME: what about multiple files with the same name? 2887 private: 2888 //RequestMethod _requestMethod; 2889 } 2890 2891 /// use this for testing or other isolated things when you want it to be no-ops 2892 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2893 // we want to ignore, not use stdout 2894 if(outputSink is null) 2895 outputSink = delegate void(const(ubyte)[]) { }; 2896 2897 string[string] env; 2898 env["REQUEST_METHOD"] = to!string(method); 2899 env["CONTENT_LENGTH"] = to!string(data.length); 2900 2901 auto cgi = new Cgi( 2902 0, 2903 env, 2904 { return data; }, 2905 outputSink, 2906 null); 2907 2908 return cgi; 2909 } 2910 2911 /++ 2912 A helper test class for request handler unittests. 2913 +/ 2914 version(with_breaking_cgi_features) 2915 class CgiTester { 2916 private { 2917 SessionObject[TypeInfo] mockSessions; 2918 SessionObject getSessionOverride(TypeInfo ti) { 2919 if(auto o = ti in mockSessions) 2920 return *o; 2921 else 2922 return null; 2923 } 2924 void setSessionOverride(TypeInfo ti, SessionObject so) { 2925 mockSessions[ti] = so; 2926 } 2927 } 2928 2929 /++ 2930 Gets (and creates if necessary) a mock session object for this test. Note 2931 it will be the same one used for any test operations through this CgiTester instance. 2932 +/ 2933 Session!Data getSessionObject(Data)() { 2934 auto obj = getSessionOverride(typeid(typeof(return))); 2935 if(obj !is null) 2936 return cast(typeof(return)) obj; 2937 else { 2938 auto o = new MockSession!Data(); 2939 setSessionOverride(typeid(typeof(return)), o); 2940 return o; 2941 } 2942 } 2943 2944 /++ 2945 Pass a reference to your request handler when creating the tester. 2946 +/ 2947 this(void function(Cgi) requestHandler) { 2948 this.requestHandler = requestHandler; 2949 } 2950 2951 /++ 2952 You can check response information with these methods after you call the request handler. 2953 +/ 2954 struct Response { 2955 int code; 2956 string[string] headers; 2957 string responseText; 2958 ubyte[] responseBody; 2959 } 2960 2961 /++ 2962 Executes a test request on your request handler, and returns the response. 2963 2964 Params: 2965 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2966 args = additional arguments. Same format as cgi's command line handler. 2967 +/ 2968 Response GET(string url, string[] args = null) { 2969 return executeTest("GET", url, args); 2970 } 2971 /// ditto 2972 Response POST(string url, string[] args = null) { 2973 return executeTest("POST", url, args); 2974 } 2975 2976 /// ditto 2977 Response executeTest(string method, string url, string[] args) { 2978 ubyte[] outputtedRawData; 2979 void outputSink(const(ubyte)[] data) { 2980 outputtedRawData ~= data; 2981 } 2982 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2983 cgi.testInProcess = this; 2984 scope(exit) cgi.dispose(); 2985 2986 requestHandler(cgi); 2987 2988 cgi.close(); 2989 2990 Response response; 2991 2992 if(outputtedRawData.length) { 2993 enum LINE = "\r\n"; 2994 2995 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2996 assert(idx != -1, to!string(outputtedRawData)); 2997 auto headers = cast(string) outputtedRawData[0 .. idx]; 2998 response.code = 200; 2999 while(headers.length) { 3000 auto i = headers.locationOf(LINE); 3001 if(i == -1) i = cast(int) headers.length; 3002 3003 auto header = headers[0 .. i]; 3004 3005 auto c = header.locationOf(":"); 3006 if(c != -1) { 3007 auto name = header[0 .. c]; 3008 auto value = header[c + 2 ..$]; 3009 3010 if(name == "Status") 3011 response.code = value[0 .. value.locationOf(" ")].to!int; 3012 3013 response.headers[name] = value; 3014 } else { 3015 assert(0); 3016 } 3017 3018 if(i != headers.length) 3019 i += 2; 3020 headers = headers[i .. $]; 3021 } 3022 response.responseBody = outputtedRawData[idx + 4 .. $]; 3023 response.responseText = cast(string) response.responseBody; 3024 } 3025 3026 return response; 3027 } 3028 3029 private void function(Cgi) requestHandler; 3030 } 3031 3032 3033 // should this be a separate module? Probably, but that's a hassle. 3034 3035 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3036 string makeDataUrl(string mimeType, in void[] data) { 3037 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3038 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3039 } 3040 3041 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3042 /// Represents a url that can be broken down or built up through properties 3043 struct Uri { 3044 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3045 3046 // scheme//userinfo@host:port/path?query#fragment 3047 3048 string scheme; /// e.g. "http" in "http://example.com/" 3049 string userinfo; /// the username (and possibly a password) in the uri 3050 string host; /// the domain name 3051 int port; /// port number, if given. Will be zero if a port was not explicitly given 3052 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3053 string query; /// the stuff after the ? in a uri 3054 string fragment; /// the stuff after the # in a uri. 3055 3056 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3057 // the decode ones need to keep different names anyway because we can't overload on return values... 3058 static string encode(string s) { return std.uri.encodeComponent(s); } 3059 static string encode(string[string] s) { return encodeVariables(s); } 3060 static string encode(string[][string] s) { return encodeVariables(s); } 3061 3062 /// Breaks down a uri string to its components 3063 this(string uri) { 3064 reparse(uri); 3065 } 3066 3067 private void reparse(string uri) { 3068 // from RFC 3986 3069 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3070 // it was a nice experiment but just not worth it. 3071 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3072 /* 3073 Captures: 3074 0 = whole url 3075 1 = scheme, with : 3076 2 = scheme, no : 3077 3 = authority, with // 3078 4 = authority, no // 3079 5 = path 3080 6 = query string, with ? 3081 7 = query string, no ? 3082 8 = anchor, with # 3083 9 = anchor, no # 3084 */ 3085 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3086 // instead, I will DIY and cut that down to 0.6s on the same computer. 3087 /* 3088 3089 Note that authority is 3090 user:password@domain:port 3091 where the user:password@ part is optional, and the :port is optional. 3092 3093 Regex translation: 3094 3095 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3096 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3097 Path cannot have any ? or # in it. It is optional. 3098 Query must start with ? and must not have # in it. It is optional. 3099 Anchor must start with # and can have anything else in it to end of string. It is optional. 3100 */ 3101 3102 this = Uri.init; // reset all state 3103 3104 // empty uri = nothing special 3105 if(uri.length == 0) { 3106 return; 3107 } 3108 3109 size_t idx; 3110 3111 scheme_loop: foreach(char c; uri[idx .. $]) { 3112 switch(c) { 3113 case ':': 3114 case '/': 3115 case '?': 3116 case '#': 3117 break scheme_loop; 3118 default: 3119 } 3120 idx++; 3121 } 3122 3123 if(idx == 0 && uri[idx] == ':') { 3124 // this is actually a path! we skip way ahead 3125 goto path_loop; 3126 } 3127 3128 if(idx == uri.length) { 3129 // the whole thing is a path, apparently 3130 path = uri; 3131 return; 3132 } 3133 3134 if(idx > 0 && uri[idx] == ':') { 3135 scheme = uri[0 .. idx]; 3136 idx++; 3137 } else { 3138 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3139 idx = 0; 3140 } 3141 3142 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3143 // we have an authority.... 3144 idx += 2; 3145 3146 auto authority_start = idx; 3147 authority_loop: foreach(char c; uri[idx .. $]) { 3148 switch(c) { 3149 case '/': 3150 case '?': 3151 case '#': 3152 break authority_loop; 3153 default: 3154 } 3155 idx++; 3156 } 3157 3158 auto authority = uri[authority_start .. idx]; 3159 3160 auto idx2 = authority.indexOf("@"); 3161 if(idx2 != -1) { 3162 userinfo = authority[0 .. idx2]; 3163 authority = authority[idx2 + 1 .. $]; 3164 } 3165 3166 if(authority.length && authority[0] == '[') { 3167 // ipv6 address special casing 3168 idx2 = authority.indexOf(']'); 3169 if(idx2 != -1) { 3170 auto end = authority[idx2 + 1 .. $]; 3171 if(end.length && end[0] == ':') 3172 idx2 = idx2 + 1; 3173 else 3174 idx2 = -1; 3175 } 3176 } else { 3177 idx2 = authority.indexOf(":"); 3178 } 3179 3180 if(idx2 == -1) { 3181 port = 0; // 0 means not specified; we should use the default for the scheme 3182 host = authority; 3183 } else { 3184 host = authority[0 .. idx2]; 3185 if(idx2 + 1 < authority.length) 3186 port = to!int(authority[idx2 + 1 .. $]); 3187 else 3188 port = 0; 3189 } 3190 } 3191 3192 path_loop: 3193 auto path_start = idx; 3194 3195 foreach(char c; uri[idx .. $]) { 3196 if(c == '?' || c == '#') 3197 break; 3198 idx++; 3199 } 3200 3201 path = uri[path_start .. idx]; 3202 3203 if(idx == uri.length) 3204 return; // nothing more to examine... 3205 3206 if(uri[idx] == '?') { 3207 idx++; 3208 auto query_start = idx; 3209 foreach(char c; uri[idx .. $]) { 3210 if(c == '#') 3211 break; 3212 idx++; 3213 } 3214 query = uri[query_start .. idx]; 3215 } 3216 3217 if(idx < uri.length && uri[idx] == '#') { 3218 idx++; 3219 fragment = uri[idx .. $]; 3220 } 3221 3222 // uriInvalidated = false; 3223 } 3224 3225 private string rebuildUri() const { 3226 string ret; 3227 if(scheme.length) 3228 ret ~= scheme ~ ":"; 3229 if(userinfo.length || host.length) 3230 ret ~= "//"; 3231 if(userinfo.length) 3232 ret ~= userinfo ~ "@"; 3233 if(host.length) 3234 ret ~= host; 3235 if(port) 3236 ret ~= ":" ~ to!string(port); 3237 3238 ret ~= path; 3239 3240 if(query.length) 3241 ret ~= "?" ~ query; 3242 3243 if(fragment.length) 3244 ret ~= "#" ~ fragment; 3245 3246 // uri = ret; 3247 // uriInvalidated = false; 3248 return ret; 3249 } 3250 3251 /// Converts the broken down parts back into a complete string 3252 string toString() const { 3253 // if(uriInvalidated) 3254 return rebuildUri(); 3255 } 3256 3257 /// Returns a new absolute Uri given a base. It treats this one as 3258 /// relative where possible, but absolute if not. (If protocol, domain, or 3259 /// other info is not set, the new one inherits it from the base.) 3260 /// 3261 /// Browsers use a function like this to figure out links in html. 3262 Uri basedOn(in Uri baseUrl) const { 3263 Uri n = this; // copies 3264 if(n.scheme == "data") 3265 return n; 3266 // n.uriInvalidated = true; // make sure we regenerate... 3267 3268 // userinfo is not inherited... is this wrong? 3269 3270 // if anything is given in the existing url, we don't use the base anymore. 3271 if(n.scheme.empty) { 3272 n.scheme = baseUrl.scheme; 3273 if(n.host.empty) { 3274 n.host = baseUrl.host; 3275 if(n.port == 0) { 3276 n.port = baseUrl.port; 3277 if(n.path.length > 0 && n.path[0] != '/') { 3278 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3279 if(b.length == 0) 3280 b = "/"; 3281 n.path = b ~ n.path; 3282 } else if(n.path.length == 0) { 3283 n.path = baseUrl.path; 3284 } 3285 } 3286 } 3287 } 3288 3289 n.removeDots(); 3290 3291 return n; 3292 } 3293 3294 void removeDots() { 3295 auto parts = this.path.split("/"); 3296 string[] toKeep; 3297 foreach(part; parts) { 3298 if(part == ".") { 3299 continue; 3300 } else if(part == "..") { 3301 //if(toKeep.length > 1) 3302 toKeep = toKeep[0 .. $-1]; 3303 //else 3304 //toKeep = [""]; 3305 continue; 3306 } else { 3307 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3308 //continue; // skip a `//` situation 3309 toKeep ~= part; 3310 } 3311 } 3312 3313 auto path = toKeep.join("/"); 3314 if(path.length && path[0] != '/') 3315 path = "/" ~ path; 3316 3317 this.path = path; 3318 } 3319 3320 unittest { 3321 auto uri = Uri("test.html"); 3322 assert(uri.path == "test.html"); 3323 uri = Uri("path/1/lol"); 3324 assert(uri.path == "path/1/lol"); 3325 uri = Uri("http://me@example.com"); 3326 assert(uri.scheme == "http"); 3327 assert(uri.userinfo == "me"); 3328 assert(uri.host == "example.com"); 3329 uri = Uri("http://example.com/#a"); 3330 assert(uri.scheme == "http"); 3331 assert(uri.host == "example.com"); 3332 assert(uri.fragment == "a"); 3333 uri = Uri("#foo"); 3334 assert(uri.fragment == "foo"); 3335 uri = Uri("?lol"); 3336 assert(uri.query == "lol"); 3337 uri = Uri("#foo?lol"); 3338 assert(uri.fragment == "foo?lol"); 3339 uri = Uri("?lol#foo"); 3340 assert(uri.fragment == "foo"); 3341 assert(uri.query == "lol"); 3342 3343 uri = Uri("http://127.0.0.1/"); 3344 assert(uri.host == "127.0.0.1"); 3345 assert(uri.port == 0); 3346 3347 uri = Uri("http://127.0.0.1:123/"); 3348 assert(uri.host == "127.0.0.1"); 3349 assert(uri.port == 123); 3350 3351 uri = Uri("http://[ff:ff::0]/"); 3352 assert(uri.host == "[ff:ff::0]"); 3353 3354 uri = Uri("http://[ff:ff::0]:123/"); 3355 assert(uri.host == "[ff:ff::0]"); 3356 assert(uri.port == 123); 3357 } 3358 3359 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3360 // the possibilities. 3361 unittest { 3362 auto url = Uri("cool.html"); // checking relative links 3363 3364 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3365 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3366 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3367 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3368 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3369 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3370 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3371 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3372 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3373 3374 url = Uri("/something/cool.html"); // same server, different path 3375 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3376 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3377 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3378 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3379 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3380 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3381 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3382 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3383 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3384 3385 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3386 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3387 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3388 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3389 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3390 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3391 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3392 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3393 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3394 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3395 3396 url = Uri("/test/bar"); 3397 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3398 assert(Uri("../").basedOn(url) == "/"); 3399 3400 url = Uri("http://example.com/"); 3401 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3402 3403 //auto uriBefore = url; 3404 url = Uri("#anchor"); // everything should remain the same except the anchor 3405 //uriBefore.anchor = "anchor"); 3406 //assert(url == uriBefore); 3407 3408 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3409 3410 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3411 3412 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3413 3414 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3415 3416 // FIXME: add something for port too 3417 } 3418 3419 // these are like javascript's location.search and location.hash 3420 string search() const { 3421 return query.length ? ("?" ~ query) : ""; 3422 } 3423 string hash() const { 3424 return fragment.length ? ("#" ~ fragment) : ""; 3425 } 3426 } 3427 3428 3429 /* 3430 for session, see web.d 3431 */ 3432 3433 /// breaks down a url encoded string 3434 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3435 auto vars = data.split(separator); 3436 string[][string] _get; 3437 foreach(var; vars) { 3438 auto equal = var.indexOf("="); 3439 string name; 3440 string value; 3441 if(equal == -1) { 3442 name = decodeComponent(var); 3443 value = ""; 3444 } else { 3445 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3446 // stupid + -> space conversion. 3447 name = decodeComponent(var[0..equal].replace("+", " ")); 3448 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3449 } 3450 3451 _get[name] ~= value; 3452 if(namesInOrder) 3453 (*namesInOrder) ~= name; 3454 if(valuesInOrder) 3455 (*valuesInOrder) ~= value; 3456 } 3457 return _get; 3458 } 3459 3460 /// breaks down a url encoded string, but only returns the last value of any array 3461 string[string] decodeVariablesSingle(string data) { 3462 string[string] va; 3463 auto varArray = decodeVariables(data); 3464 foreach(k, v; varArray) 3465 va[k] = v[$-1]; 3466 3467 return va; 3468 } 3469 3470 /// url encodes the whole string 3471 string encodeVariables(in string[string] data) { 3472 string ret; 3473 3474 bool outputted = false; 3475 foreach(k, v; data) { 3476 if(outputted) 3477 ret ~= "&"; 3478 else 3479 outputted = true; 3480 3481 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3482 } 3483 3484 return ret; 3485 } 3486 3487 /// url encodes a whole string 3488 string encodeVariables(in string[][string] data) { 3489 string ret; 3490 3491 bool outputted = false; 3492 foreach(k, arr; data) { 3493 foreach(v; arr) { 3494 if(outputted) 3495 ret ~= "&"; 3496 else 3497 outputted = true; 3498 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3499 } 3500 } 3501 3502 return ret; 3503 } 3504 3505 /// Encodes all but the explicitly unreserved characters per rfc 3986 3506 /// Alphanumeric and -_.~ are the only ones left unencoded 3507 /// name is borrowed from php 3508 string rawurlencode(in char[] data) { 3509 string ret; 3510 ret.reserve(data.length * 2); 3511 foreach(char c; data) { 3512 if( 3513 (c >= 'a' && c <= 'z') || 3514 (c >= 'A' && c <= 'Z') || 3515 (c >= '0' && c <= '9') || 3516 c == '-' || c == '_' || c == '.' || c == '~') 3517 { 3518 ret ~= c; 3519 } else { 3520 ret ~= '%'; 3521 // since we iterate on char, this should give us the octets of the full utf8 string 3522 ret ~= toHexUpper(c); 3523 } 3524 } 3525 3526 return ret; 3527 } 3528 3529 3530 // http helper functions 3531 3532 // for chunked responses (which embedded http does whenever possible) 3533 version(none) // this is moved up above to avoid making a copy of the data 3534 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3535 const(ubyte)[] ret; 3536 3537 ret = cast(const(ubyte)[]) toHex(data.length); 3538 ret ~= cast(const(ubyte)[]) "\r\n"; 3539 ret ~= data; 3540 ret ~= cast(const(ubyte)[]) "\r\n"; 3541 3542 return ret; 3543 } 3544 3545 string toHex(long num) { 3546 string ret; 3547 while(num) { 3548 int v = num % 16; 3549 num /= 16; 3550 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3551 ret ~= d; 3552 } 3553 3554 return to!string(array(ret.retro)); 3555 } 3556 3557 string toHexUpper(long num) { 3558 string ret; 3559 while(num) { 3560 int v = num % 16; 3561 num /= 16; 3562 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3563 ret ~= d; 3564 } 3565 3566 if(ret.length == 1) 3567 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3568 3569 return to!string(array(ret.retro)); 3570 } 3571 3572 3573 // the generic mixins 3574 3575 /++ 3576 Use this instead of writing your own main 3577 3578 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3579 +/ 3580 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3581 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3582 } 3583 3584 /++ 3585 Boilerplate mixin for a main function that uses the [dispatcher] function. 3586 3587 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3588 3589 History: 3590 Added July 9, 2021 3591 +/ 3592 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3593 /// forwards to [CustomCgiDispatcherMain] with default args 3594 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3595 } 3596 3597 /// ditto 3598 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3599 class GenericPresenter : WebPresenter!GenericPresenter {} 3600 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3601 } 3602 3603 /++ 3604 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3605 3606 History: 3607 Added May 13, 2023 (dub v11.0) 3608 +/ 3609 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3610 /++ 3611 Handler to the generated presenter you can use from your objects, etc. 3612 +/ 3613 Presenter activePresenter; 3614 3615 /++ 3616 Request handler that creates the presenter then forwards to the [dispatcher] function. 3617 Renders 404 if the dispatcher did not handle the request. 3618 3619 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3620 +/ 3621 void handler(Cgi cgi) { 3622 auto presenter = new Presenter; 3623 activePresenter = presenter; 3624 scope(exit) activePresenter = null; 3625 3626 if(cgi.dispatcher!DispatcherArgs(presenter)) 3627 return; 3628 3629 switch(cgi.pathInfo) { 3630 case "/style.css": 3631 cgi.setCache(true); 3632 cgi.setResponseContentType("text/css"); 3633 cgi.write(presenter.style(), true); 3634 break; 3635 case "/script.js": 3636 cgi.setCache(true); 3637 cgi.setResponseContentType("application/javascript"); 3638 cgi.write(presenter.script(), true); 3639 break; 3640 default: 3641 presenter.renderBasicError(cgi, 404); 3642 } 3643 } 3644 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3645 } 3646 3647 /// ditto 3648 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3649 class GenericPresenter : WebPresenter!GenericPresenter {} 3650 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3651 3652 } 3653 3654 private string simpleHtmlEncode(string s) { 3655 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3656 } 3657 3658 string messageFromException(Throwable t) { 3659 string message; 3660 if(t !is null) { 3661 debug message = t.toString(); 3662 else message = "An unexpected error has occurred."; 3663 } else { 3664 message = "Unknown error"; 3665 } 3666 return message; 3667 } 3668 3669 string plainHttpError(bool isCgi, string type, Throwable t) { 3670 auto message = messageFromException(t); 3671 message = simpleHtmlEncode(message); 3672 3673 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3674 isCgi ? "Status:" : "HTTP/1.1", 3675 type, message.length, message); 3676 } 3677 3678 // returns true if we were able to recover reasonably 3679 bool handleException(Cgi cgi, Throwable t) { 3680 if(cgi.isClosed) { 3681 // if the channel has been explicitly closed, we can't handle it here 3682 return true; 3683 } 3684 3685 if(cgi.outputtedResponseData) { 3686 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3687 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3688 } else { 3689 // no headers are sent, we can send a full blown error and recover 3690 cgi.setCache(false); 3691 cgi.setResponseContentType("text/html"); 3692 cgi.setResponseLocation(null); // cancel the redirect 3693 cgi.setResponseStatus("500 Internal Server Error"); 3694 cgi.write(simpleHtmlEncode(messageFromException(t))); 3695 cgi.close(); 3696 return true; 3697 } 3698 } 3699 3700 bool isCgiRequestMethod(string s) { 3701 s = s.toUpper(); 3702 if(s == "COMMANDLINE") 3703 return true; 3704 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3705 if(s == member) 3706 return true; 3707 return false; 3708 } 3709 3710 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3711 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3712 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3713 void main(string[] args) { 3714 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3715 } 3716 } 3717 3718 version(embedded_httpd_processes) 3719 __gshared int processPoolSize = 8; 3720 3721 // Returns true if run. You should exit the program after that. 3722 bool tryAddonServers(string[] args) { 3723 if(args.length > 1) { 3724 // run the special separate processes if needed 3725 switch(args[1]) { 3726 case "--websocket-server": 3727 version(with_addon_servers) 3728 websocketServers[args[2]](args[3 .. $]); 3729 else 3730 printf("Add-on servers not compiled in.\n"); 3731 return true; 3732 case "--websocket-servers": 3733 import core.demangle; 3734 version(with_addon_servers_connections) 3735 foreach(k, v; websocketServers) 3736 writeln(k, "\t", demangle(k)); 3737 return true; 3738 case "--session-server": 3739 version(with_addon_servers) 3740 runSessionServer(); 3741 else 3742 printf("Add-on servers not compiled in.\n"); 3743 return true; 3744 case "--event-server": 3745 version(with_addon_servers) 3746 runEventServer(); 3747 else 3748 printf("Add-on servers not compiled in.\n"); 3749 return true; 3750 case "--timer-server": 3751 version(with_addon_servers) 3752 runTimerServer(); 3753 else 3754 printf("Add-on servers not compiled in.\n"); 3755 return true; 3756 case "--timed-jobs": 3757 import core.demangle; 3758 version(with_addon_servers_connections) 3759 foreach(k, v; scheduledJobHandlers) 3760 writeln(k, "\t", demangle(k)); 3761 return true; 3762 case "--timed-job": 3763 scheduledJobHandlers[args[2]](args[3 .. $]); 3764 return true; 3765 default: 3766 // intentionally blank - do nothing and carry on to run normally 3767 } 3768 } 3769 return false; 3770 } 3771 3772 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3773 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3774 // we support command line thing for easy testing everywhere 3775 // it needs to be called ./app method uri [other args...] 3776 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3777 Cgi cgi = new CustomCgi(args); 3778 scope(exit) cgi.dispose(); 3779 fun(cgi); 3780 cgi.close(); 3781 return true; 3782 } 3783 return false; 3784 } 3785 3786 /++ 3787 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3788 3789 As of version 11 (released August 2023), you can also make things like this: 3790 3791 --- 3792 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3793 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3794 3795 // can also: 3796 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3797 3798 // NOT IMPLEMENTED YET 3799 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3800 3801 foreach(listenSpec; server.listenSpecs) { 3802 // you can check what it actually bound to here and see your assigned ports 3803 } 3804 3805 // NOT IMPLEMENTED YET 3806 // server.start!handler(); // starts and runs in the arsd.core event loop 3807 3808 server.serve!handler(); // blocks the thread until the server exits 3809 --- 3810 3811 History: 3812 Added Sept 26, 2020 (release version 8.5). 3813 3814 The `listenSpec` member was added July 31, 2023. 3815 +/ 3816 struct RequestServer { 3817 /++ 3818 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3819 +/ 3820 string listeningHost = defaultListeningHost(); 3821 /// ditto 3822 ushort listeningPort = defaultListeningPort(); 3823 3824 static struct ListenSpec { 3825 enum Protocol { 3826 http, 3827 https, 3828 scgi 3829 } 3830 Protocol protocol; 3831 3832 enum AddressType { 3833 ip, 3834 unix, 3835 abstract_ 3836 } 3837 AddressType addressType; 3838 3839 string address; 3840 ushort port; 3841 } 3842 3843 /++ 3844 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3845 3846 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3847 3848 `http://localhost:8080` 3849 3850 `http://unix:filename/here` 3851 3852 `scgi://abstract:/name/here` 3853 3854 `http://[::1]:4444` 3855 3856 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3857 3858 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3859 3860 `localhost:8080` serves the default protocol. 3861 3862 `8080` or `:8080` assumes default protocol on localhost. 3863 3864 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3865 3866 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3867 3868 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3869 3870 $(PITFALL 3871 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3872 ) 3873 3874 Bugs: 3875 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3876 3877 History: 3878 Added July 31, 2023 (dub v11.0) 3879 +/ 3880 string[] listenSpec; 3881 3882 /++ 3883 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3884 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3885 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3886 [stop] may not work as well. 3887 3888 History: 3889 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3890 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3891 compatibility. 3892 +/ 3893 bool useFork = cgi_use_fork_default; 3894 3895 /++ 3896 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3897 default based on the number of cpus modified by the server mode. 3898 3899 History: 3900 Added August 12, 2022 (dub v10.9) 3901 +/ 3902 int numberOfThreads = 0; 3903 3904 /++ 3905 Creates a server configured to listen to multiple URLs. 3906 3907 History: 3908 Added July 31, 2023 (dub v11.0) 3909 +/ 3910 this(string[] listenTo) { 3911 this.listenSpec = listenTo; 3912 } 3913 3914 /// Creates a server object configured to listen on a single host and port. 3915 this(string defaultHost, ushort defaultPort) { 3916 this.listeningHost = defaultHost; 3917 this.listeningPort = defaultPort; 3918 } 3919 3920 /// ditto 3921 this(ushort defaultPort) { 3922 listeningPort = defaultPort; 3923 } 3924 3925 /++ 3926 Reads the command line arguments into the values here. 3927 3928 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3929 3930 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 3931 +/ 3932 void configureFromCommandLine(string[] args) { 3933 bool portOrHostFound = false; 3934 3935 bool foundPort = false; 3936 bool foundHost = false; 3937 bool foundUid = false; 3938 bool foundGid = false; 3939 bool foundListen = false; 3940 foreach(arg; args) { 3941 if(foundPort) { 3942 listeningPort = to!ushort(arg); 3943 portOrHostFound = true; 3944 foundPort = false; 3945 continue; 3946 } 3947 if(foundHost) { 3948 listeningHost = arg; 3949 portOrHostFound = true; 3950 foundHost = false; 3951 continue; 3952 } 3953 if(foundUid) { 3954 privilegesDropToUid = to!uid_t(arg); 3955 foundUid = false; 3956 continue; 3957 } 3958 if(foundGid) { 3959 privilegesDropToGid = to!gid_t(arg); 3960 foundGid = false; 3961 continue; 3962 } 3963 if(foundListen) { 3964 this.listenSpec ~= arg; 3965 foundListen = false; 3966 continue; 3967 } 3968 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3969 foundHost = true; 3970 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3971 foundPort = true; 3972 else if(arg == "--uid") 3973 foundUid = true; 3974 else if(arg == "--gid") 3975 foundGid = true; 3976 else if(arg == "--listen") 3977 foundListen = true; 3978 } 3979 3980 if(portOrHostFound && listenSpec.length) { 3981 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 3982 } 3983 } 3984 3985 version(Windows) { 3986 private alias uid_t = int; 3987 private alias gid_t = int; 3988 } 3989 3990 /// user (uid) to drop privileges to 3991 /// 0 … do nothing 3992 uid_t privilegesDropToUid = 0; 3993 /// group (gid) to drop privileges to 3994 /// 0 … do nothing 3995 gid_t privilegesDropToGid = 0; 3996 3997 private void dropPrivileges() { 3998 version(Posix) { 3999 import core.sys.posix.unistd; 4000 4001 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 4002 throw new Exception("Dropping privileges via setgid() failed."); 4003 4004 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 4005 throw new Exception("Dropping privileges via setuid() failed."); 4006 } 4007 else { 4008 // FIXME: Windows? 4009 //pragma(msg, "Dropping privileges is not implemented for this platform"); 4010 } 4011 4012 // done, set zero 4013 privilegesDropToGid = 0; 4014 privilegesDropToUid = 0; 4015 } 4016 4017 /++ 4018 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 4019 4020 History: 4021 Added Oct 10, 2020. 4022 Example: 4023 4024 --- 4025 import arsd.cgi; 4026 void main() { 4027 RequestServer server = RequestServer("127.0.0.1", 6789); 4028 string oauthCode; 4029 string oauthScope; 4030 server.serveHttpOnce!((cgi) { 4031 oauthCode = cgi.request("code"); 4032 oauthScope = cgi.request("scope"); 4033 cgi.write("Thank you, please return to the application."); 4034 }); 4035 // use the code and scope given 4036 } 4037 --- 4038 +/ 4039 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4040 import std.socket; 4041 4042 bool tcp; 4043 void delegate() cleanup; 4044 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4045 auto connection = socket.accept(); 4046 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4047 4048 if(cleanup) 4049 cleanup(); 4050 } 4051 4052 /++ 4053 Starts serving requests according to the current configuration. 4054 +/ 4055 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4056 version(netman_httpd) { 4057 // Obsolete! 4058 4059 import arsd.httpd; 4060 // what about forwarding the other constructor args? 4061 // this probably needs a whole redoing... 4062 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4063 return; 4064 } else 4065 version(embedded_httpd_processes) { 4066 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4067 } else 4068 version(embedded_httpd_threads) { 4069 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4070 } else 4071 version(scgi) { 4072 serveScgi!(fun, CustomCgi, maxContentLength)(); 4073 } else 4074 version(fastcgi) { 4075 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4076 } else 4077 version(stdio_http) { 4078 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4079 } else { 4080 //version=plain_cgi; 4081 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4082 } 4083 } 4084 4085 /++ 4086 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4087 4088 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4089 +/ 4090 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4091 globalStopFlag = false; 4092 static if(__traits(isStaticFunction, fun)) 4093 alias funToUse = fun; 4094 else 4095 void funToUse(CustomCgi cgi) { 4096 static if(__VERSION__ > 2097) 4097 __traits(child, _this, fun)(cgi); 4098 else static assert(0, "Not implemented in your compiler version!"); 4099 } 4100 auto manager = this.listenSpec is null ? 4101 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4102 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4103 manager.listen(); 4104 } 4105 4106 /++ 4107 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4108 +/ 4109 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4110 globalStopFlag = false; 4111 auto manager = this.listenSpec is null ? 4112 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4113 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4114 manager.listen(); 4115 } 4116 4117 /++ 4118 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4119 4120 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4121 4122 History: 4123 Added May 29, 2021 4124 +/ 4125 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4126 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4127 } 4128 4129 /++ 4130 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4131 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4132 +/ 4133 enum ForceStop { 4134 /++ 4135 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4136 +/ 4137 afterQueuedRequestsComplete, 4138 /++ 4139 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4140 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4141 +/ 4142 afterCurrentRequestsComplete, 4143 /++ 4144 Partial response writes will throw an exception, cancelling any streaming response, but complete 4145 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4146 +/ 4147 cancelStreamingRequestsEarly, 4148 /++ 4149 All writes will throw. 4150 +/ 4151 cancelAllRequestsEarly, 4152 /++ 4153 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4154 +/ 4155 forciblyTerminate, 4156 } 4157 4158 version(embedded_httpd_processes) {} else 4159 /++ 4160 Stops serving after the current requests are completed. 4161 4162 Bugs: 4163 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4164 on Windows however). Only partially implemented on non-Linux posix systems. 4165 4166 You might also try SIGINT perhaps. 4167 4168 The stopPriority is not yet fully implemented. 4169 +/ 4170 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4171 globalStopFlag = true; 4172 4173 version(Posix) { 4174 if(cancelfd > 0) { 4175 ulong a = 1; 4176 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4177 } 4178 } 4179 version(Windows) { 4180 if(iocp) { 4181 foreach(i; 0 .. 16) // FIXME 4182 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4183 } 4184 } 4185 } 4186 } 4187 4188 private alias AliasSeq(T...) = T; 4189 4190 version(with_breaking_cgi_features) 4191 mixin(q{ 4192 template ThisFor(alias t) { 4193 static if(__traits(isStaticFunction, t)) { 4194 alias ThisFor = AliasSeq!(); 4195 } else { 4196 alias ThisFor = __traits(parent, t); 4197 } 4198 } 4199 }); 4200 else 4201 alias ThisFor(alias t) = AliasSeq!(); 4202 4203 private __gshared bool globalStopFlag = false; 4204 4205 version(embedded_httpd_processes) 4206 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4207 import core.sys.posix.unistd; 4208 import core.sys.posix.sys.socket; 4209 import core.sys.posix.netinet.in_; 4210 //import std.c.linux.socket; 4211 4212 int sock = socket(AF_INET, SOCK_STREAM, 0); 4213 if(sock == -1) 4214 throw new Exception("socket"); 4215 4216 cloexec(sock); 4217 4218 { 4219 4220 sockaddr_in addr; 4221 addr.sin_family = AF_INET; 4222 addr.sin_port = htons(params.listeningPort); 4223 auto lh = params.listeningHost; 4224 if(lh.length) { 4225 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4226 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4227 } else 4228 addr.sin_addr.s_addr = INADDR_ANY; 4229 4230 // HACKISH 4231 int on = 1; 4232 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4233 // end hack 4234 4235 4236 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4237 close(sock); 4238 throw new Exception("bind"); 4239 } 4240 4241 // FIXME: if this queue is full, it will just ignore it 4242 // and wait for the client to retransmit it. This is an 4243 // obnoxious timeout condition there. 4244 if(sock.listen(128) == -1) { 4245 close(sock); 4246 throw new Exception("listen"); 4247 } 4248 params.dropPrivileges(); 4249 } 4250 4251 version(embedded_httpd_processes_accept_after_fork) {} else { 4252 int pipeReadFd; 4253 int pipeWriteFd; 4254 4255 { 4256 int[2] pipeFd; 4257 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4258 import core.stdc.errno; 4259 throw new Exception("pipe failed " ~ to!string(errno)); 4260 } 4261 4262 pipeReadFd = pipeFd[0]; 4263 pipeWriteFd = pipeFd[1]; 4264 } 4265 } 4266 4267 4268 int processCount; 4269 pid_t newPid; 4270 reopen: 4271 while(processCount < processPoolSize) { 4272 newPid = fork(); 4273 if(newPid == 0) { 4274 // start serving on the socket 4275 //ubyte[4096] backingBuffer; 4276 for(;;) { 4277 bool closeConnection; 4278 uint i; 4279 sockaddr addr; 4280 i = addr.sizeof; 4281 version(embedded_httpd_processes_accept_after_fork) { 4282 int s = accept(sock, &addr, &i); 4283 int opt = 1; 4284 import core.sys.posix.netinet.tcp; 4285 // the Cgi class does internal buffering, so disabling this 4286 // helps with latency in many cases... 4287 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4288 cloexec(s); 4289 } else { 4290 int s; 4291 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4292 if(readret != s.sizeof) { 4293 import core.stdc.errno; 4294 throw new Exception("pipe read failed " ~ to!string(errno)); 4295 } 4296 4297 //writeln("process ", getpid(), " got socket ", s); 4298 } 4299 4300 try { 4301 4302 if(s == -1) 4303 throw new Exception("accept"); 4304 4305 scope(failure) close(s); 4306 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4307 auto ir = new BufferedInputRange(s); 4308 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4309 4310 while(!ir.empty) { 4311 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4312 4313 Cgi cgi; 4314 try { 4315 cgi = new CustomCgi(ir, &closeConnection); 4316 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4317 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4318 if(processPoolSize <= 1) 4319 closeConnection = true; 4320 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4321 } catch(HttpVersionNotSupportedException he) { 4322 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4323 closeConnection = true; 4324 break; 4325 } catch(Throwable t) { 4326 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4327 // anyway let's kill the connection 4328 version(CRuntime_Musl) { 4329 // LockingTextWriter fails here 4330 // so working around it 4331 auto estr = t.toString(); 4332 stderr.rawWrite(estr); 4333 stderr.rawWrite("\n"); 4334 } else 4335 stderr.writeln(t.toString()); 4336 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4337 closeConnection = true; 4338 break; 4339 } 4340 assert(cgi !is null); 4341 scope(exit) 4342 cgi.dispose(); 4343 4344 try { 4345 fun(cgi); 4346 cgi.close(); 4347 if(cgi.websocketMode) 4348 closeConnection = true; 4349 } catch(ConnectionException ce) { 4350 closeConnection = true; 4351 } catch(Throwable t) { 4352 // a processing error can be recovered from 4353 version(CRuntime_Musl) { 4354 // LockingTextWriter fails here 4355 // so working around it 4356 auto estr = t.toString(); 4357 stderr.rawWrite(estr); 4358 } else { 4359 stderr.writeln(t.toString); 4360 } 4361 if(!handleException(cgi, t)) 4362 closeConnection = true; 4363 } 4364 4365 if(closeConnection) { 4366 ir.source.close(); 4367 break; 4368 } else { 4369 if(!ir.empty) 4370 ir.popFront(); // get the next 4371 else if(ir.sourceClosed) { 4372 ir.source.close(); 4373 } 4374 } 4375 } 4376 4377 ir.source.close(); 4378 } catch(Throwable t) { 4379 version(CRuntime_Musl) {} else 4380 debug writeln(t); 4381 // most likely cause is a timeout 4382 } 4383 } 4384 } else if(newPid < 0) { 4385 throw new Exception("fork failed"); 4386 } else { 4387 processCount++; 4388 } 4389 } 4390 4391 // the parent should wait for its children... 4392 if(newPid) { 4393 import core.sys.posix.sys.wait; 4394 4395 version(embedded_httpd_processes_accept_after_fork) {} else { 4396 import core.sys.posix.sys.select; 4397 int[] fdQueue; 4398 while(true) { 4399 // writeln("select call"); 4400 int nfds = pipeWriteFd; 4401 if(sock > pipeWriteFd) 4402 nfds = sock; 4403 nfds += 1; 4404 fd_set read_fds; 4405 fd_set write_fds; 4406 FD_ZERO(&read_fds); 4407 FD_ZERO(&write_fds); 4408 FD_SET(sock, &read_fds); 4409 if(fdQueue.length) 4410 FD_SET(pipeWriteFd, &write_fds); 4411 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4412 if(ret == -1) { 4413 import core.stdc.errno; 4414 if(errno == EINTR) 4415 goto try_wait; 4416 else 4417 throw new Exception("wtf select"); 4418 } 4419 4420 int s = -1; 4421 if(FD_ISSET(sock, &read_fds)) { 4422 uint i; 4423 sockaddr addr; 4424 i = addr.sizeof; 4425 s = accept(sock, &addr, &i); 4426 cloexec(s); 4427 import core.sys.posix.netinet.tcp; 4428 int opt = 1; 4429 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4430 } 4431 4432 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4433 if(s == -1 && fdQueue.length) { 4434 s = fdQueue[0]; 4435 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4436 } 4437 write_fd(pipeWriteFd, &s, s.sizeof, s); 4438 close(s); // we are done with it, let the other process take ownership 4439 } else 4440 fdQueue ~= s; 4441 } 4442 } 4443 4444 try_wait: 4445 4446 int status; 4447 while(-1 != wait(&status)) { 4448 version(CRuntime_Musl) {} else { 4449 import std.stdio; writeln("Process died ", status); 4450 } 4451 processCount--; 4452 goto reopen; 4453 } 4454 close(sock); 4455 } 4456 } 4457 4458 version(fastcgi) 4459 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4460 // SetHandler fcgid-script 4461 FCGX_Stream* input, output, error; 4462 FCGX_ParamArray env; 4463 4464 4465 4466 const(ubyte)[] getFcgiChunk() { 4467 const(ubyte)[] ret; 4468 while(FCGX_HasSeenEOF(input) != -1) 4469 ret ~= cast(ubyte) FCGX_GetChar(input); 4470 return ret; 4471 } 4472 4473 void writeFcgi(const(ubyte)[] data) { 4474 FCGX_PutStr(data.ptr, data.length, output); 4475 } 4476 4477 void doARequest() { 4478 string[string] fcgienv; 4479 4480 for(auto e = env; e !is null && *e !is null; e++) { 4481 string cur = to!string(*e); 4482 auto idx = cur.indexOf("="); 4483 string name, value; 4484 if(idx == -1) 4485 name = cur; 4486 else { 4487 name = cur[0 .. idx]; 4488 value = cur[idx + 1 .. $]; 4489 } 4490 4491 fcgienv[name] = value; 4492 } 4493 4494 void flushFcgi() { 4495 FCGX_FFlush(output); 4496 } 4497 4498 Cgi cgi; 4499 try { 4500 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4501 } catch(Throwable t) { 4502 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4503 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4504 return; //continue; 4505 } 4506 assert(cgi !is null); 4507 scope(exit) cgi.dispose(); 4508 try { 4509 fun(cgi); 4510 cgi.close(); 4511 } catch(Throwable t) { 4512 // log it to the error stream 4513 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4514 // handle it for the user, if we can 4515 if(!handleException(cgi, t)) 4516 return; // continue; 4517 } 4518 } 4519 4520 auto lp = params.listeningPort; 4521 auto host = params.listeningHost; 4522 4523 FCGX_Request request; 4524 if(lp || !host.empty) { 4525 // if a listening port was specified on the command line, we want to spawn ourself 4526 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4527 FCGX_Init(); 4528 4529 int sock; 4530 4531 if(host.startsWith("unix:")) { 4532 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4533 } else if(host.startsWith("abstract:")) { 4534 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4535 } else { 4536 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4537 } 4538 4539 if(sock < 0) 4540 throw new Exception("Couldn't listen on the port"); 4541 FCGX_InitRequest(&request, sock, 0); 4542 while(FCGX_Accept_r(&request) >= 0) { 4543 input = request.inStream; 4544 output = request.outStream; 4545 error = request.errStream; 4546 env = request.envp; 4547 doARequest(); 4548 } 4549 } else { 4550 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4551 // using the version with a global variable since we are separate processes anyway 4552 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4553 doARequest(); 4554 } 4555 } 4556 } 4557 4558 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4559 ushort defaultListeningPort() { 4560 version(netman_httpd) 4561 return 8080; 4562 else version(embedded_httpd_processes) 4563 return 8085; 4564 else version(embedded_httpd_threads) 4565 return 8085; 4566 else version(scgi) 4567 return 4000; 4568 else 4569 return 0; 4570 } 4571 4572 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4573 string defaultListeningHost() { 4574 version(netman_httpd) 4575 return null; 4576 else version(embedded_httpd_processes) 4577 return null; 4578 else version(embedded_httpd_threads) 4579 return null; 4580 else version(scgi) 4581 return "127.0.0.1"; 4582 else 4583 return null; 4584 4585 } 4586 4587 /++ 4588 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4589 4590 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4591 4592 Params: 4593 fun = Your request handler 4594 CustomCgi = a subclass of Cgi, if you wise to customize it further 4595 maxContentLength = max POST size you want to allow 4596 args = command-line arguments 4597 4598 History: 4599 Documented Sept 26, 2020. 4600 +/ 4601 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4602 if(tryAddonServers(args)) 4603 return; 4604 4605 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4606 return; 4607 4608 RequestServer server; 4609 // you can change the port here if you like 4610 // server.listeningPort = 9000; 4611 4612 // then call this to let the command line args override your default 4613 server.configureFromCommandLine(args); 4614 4615 // and serve the request(s). 4616 server.serve!(fun, CustomCgi, maxContentLength)(); 4617 } 4618 4619 //version(plain_cgi) 4620 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4621 // standard CGI is the default version 4622 4623 4624 // Set stdin to binary mode if necessary to avoid mangled newlines 4625 // the fact that stdin is global means this could be trouble but standard cgi request 4626 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4627 version(Windows) { 4628 version(Win64) 4629 _setmode(std.stdio.stdin.fileno(), 0x8000); 4630 else 4631 setmode(std.stdio.stdin.fileno(), 0x8000); 4632 } 4633 4634 Cgi cgi; 4635 try { 4636 cgi = new CustomCgi(maxContentLength); 4637 version(Posix) 4638 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4639 else version(Windows) 4640 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4641 else static assert(0); 4642 } catch(Throwable t) { 4643 version(CRuntime_Musl) { 4644 // LockingTextWriter fails here 4645 // so working around it 4646 auto s = t.toString(); 4647 stderr.rawWrite(s); 4648 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4649 } else { 4650 stderr.writeln(t.msg); 4651 // the real http server will probably handle this; 4652 // most likely, this is a bug in Cgi. But, oh well. 4653 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4654 } 4655 return; 4656 } 4657 assert(cgi !is null); 4658 scope(exit) cgi.dispose(); 4659 4660 try { 4661 fun(cgi); 4662 cgi.close(); 4663 } catch (Throwable t) { 4664 version(CRuntime_Musl) { 4665 // LockingTextWriter fails here 4666 // so working around it 4667 auto s = t.msg; 4668 stderr.rawWrite(s); 4669 } else { 4670 stderr.writeln(t.msg); 4671 } 4672 if(!handleException(cgi, t)) 4673 return; 4674 } 4675 } 4676 4677 private __gshared int cancelfd = -1; 4678 4679 /+ 4680 The event loop for embedded_httpd_threads will prolly fiber dispatch 4681 cgi constructors too, so slow posts will not monopolize a worker thread. 4682 4683 May want to provide the worker task system just need to ensure all the fibers 4684 has a big enough stack for real work... would also ideally like to reuse them. 4685 4686 4687 So prolly bir would switch it to nonblocking. If it would block, it epoll 4688 registers one shot with this existing fiber to take it over. 4689 4690 new connection comes in. it picks a fiber off the free list, 4691 or if there is none, it creates a new one. this fiber handles 4692 this connection the whole time. 4693 4694 epoll triggers the fiber when something comes in. it is called by 4695 a random worker thread, it might change at any time. at least during 4696 the constructor. maybe into the main body it will stay tied to a thread 4697 just so TLS stuff doesn't randomly change in the middle. but I could 4698 specify if you yield all bets are off. 4699 4700 when the request is finished, if there's more data buffered, it just 4701 keeps going. if there is no more data buffered, it epoll ctls to 4702 get triggered when more data comes in. all one shot. 4703 4704 when a connection is closed, the fiber returns and is then reset 4705 and added to the free list. if the free list is full, the fiber is 4706 just freed, this means it will balloon to a certain size but not generally 4707 grow beyond that unless the activity keeps going. 4708 4709 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4710 4711 So the fiber has its own magic methods to read and write. if they would block, it registers 4712 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4713 4714 basically you issue the command and it tells you when it is done 4715 4716 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4717 4718 +/ 4719 4720 /++ 4721 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4722 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4723 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4724 4725 History: 4726 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4727 +/ 4728 version(cgi_use_fiber) 4729 __gshared size_t fiberStackSize = 4096 * 100; 4730 4731 version(cgi_use_fiber) 4732 class CgiFiber : Fiber { 4733 private void function(Socket) f_handler; 4734 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4735 f_handler(s); 4736 } 4737 this(void function(Socket) handler) { 4738 this.f_handler = handler; 4739 this(&f_handler_dg); 4740 } 4741 4742 this(void delegate(Socket) handler) { 4743 this.handler = handler; 4744 super(&run, fiberStackSize); 4745 } 4746 4747 Socket connection; 4748 void delegate(Socket) handler; 4749 4750 void run() { 4751 handler(connection); 4752 } 4753 4754 void delegate() postYield; 4755 4756 private void setPostYield(scope void delegate() py) @nogc { 4757 postYield = cast(void delegate()) py; 4758 } 4759 4760 void proceed() { 4761 try { 4762 call(); 4763 auto py = postYield; 4764 postYield = null; 4765 if(py !is null) 4766 py(); 4767 } catch(Exception e) { 4768 if(connection) 4769 connection.close(); 4770 goto terminate; 4771 } 4772 4773 if(state == State.TERM) { 4774 terminate: 4775 import core.memory; 4776 GC.removeRoot(cast(void*) this); 4777 } 4778 } 4779 } 4780 4781 version(cgi_use_fiber) 4782 version(Windows) { 4783 4784 extern(Windows) private { 4785 4786 import core.sys.windows.mswsock; 4787 4788 alias GROUP=uint; 4789 alias LPWSAPROTOCOL_INFOW = void*; 4790 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4791 alias WSASend = arsd.core.WSASend; 4792 alias WSARecv = arsd.core.WSARecv; 4793 alias WSABUF = arsd.core.WSABUF; 4794 4795 /+ 4796 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4797 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4798 4799 struct WSABUF { 4800 ULONG len; 4801 CHAR *buf; 4802 } 4803 +/ 4804 alias LPWSABUF = WSABUF*; 4805 4806 alias WSAOVERLAPPED = OVERLAPPED; 4807 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4808 /+ 4809 4810 alias LPFN_ACCEPTEX = 4811 BOOL 4812 function( 4813 SOCKET sListenSocket, 4814 SOCKET sAcceptSocket, 4815 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4816 void* lpOutputBuffer, 4817 WORD dwReceiveDataLength, 4818 WORD dwLocalAddressLength, 4819 WORD dwRemoteAddressLength, 4820 LPDWORD lpdwBytesReceived, 4821 LPOVERLAPPED lpOverlapped 4822 ); 4823 4824 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4825 +/ 4826 4827 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4828 } 4829 4830 private class PseudoblockingOverlappedSocket : Socket { 4831 SOCKET handle; 4832 4833 CgiFiber fiber; 4834 4835 this(AddressFamily af, SocketType st) { 4836 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4837 if(!handle) 4838 throw new Exception("WSASocketW"); 4839 this.handle = handle; 4840 4841 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4842 4843 if(iocp is null) { 4844 writeln(GetLastError()); 4845 throw new Exception("CreateIoCompletionPort"); 4846 } 4847 4848 super(cast(socket_t) handle, af); 4849 } 4850 this() pure nothrow @trusted { assert(0); } 4851 4852 override void blocking(bool) {} // meaningless to us, just ignore it. 4853 4854 protected override Socket accepting() pure nothrow { 4855 assert(0); 4856 } 4857 4858 bool addressesParsed; 4859 Address la; 4860 Address ra; 4861 4862 private void populateAddresses() { 4863 if(addressesParsed) 4864 return; 4865 addressesParsed = true; 4866 4867 int lalen, ralen; 4868 4869 sockaddr_in* la; 4870 sockaddr_in* ra; 4871 4872 lpfnGetAcceptExSockaddrs( 4873 scratchBuffer.ptr, 4874 0, // same as in the AcceptEx call! 4875 sockaddr_in.sizeof + 16, 4876 sockaddr_in.sizeof + 16, 4877 cast(sockaddr**) &la, 4878 &lalen, 4879 cast(sockaddr**) &ra, 4880 &ralen 4881 ); 4882 4883 if(la) 4884 this.la = new InternetAddress(*la); 4885 if(ra) 4886 this.ra = new InternetAddress(*ra); 4887 4888 } 4889 4890 override @property @trusted Address localAddress() { 4891 populateAddresses(); 4892 return la; 4893 } 4894 override @property @trusted Address remoteAddress() { 4895 populateAddresses(); 4896 return ra; 4897 } 4898 4899 PseudoblockingOverlappedSocket accepted; 4900 4901 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4902 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4903 4904 override Socket accept() @trusted { 4905 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4906 4907 if(lpfnAcceptEx is null) { 4908 DWORD dwBytes; 4909 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4910 4911 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4912 &GuidAcceptEx, GuidAcceptEx.sizeof, 4913 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4914 &dwBytes, null, null); 4915 4916 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4917 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4918 &GuidAcceptEx, GuidAcceptEx.sizeof, 4919 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4920 &dwBytes, null, null); 4921 4922 } 4923 4924 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4925 accepted = pfa; 4926 4927 SOCKET pendingForAccept = pfa.handle; 4928 DWORD ignored; 4929 4930 auto ret = lpfnAcceptEx(handle, 4931 pendingForAccept, 4932 // buffer to receive up front 4933 pfa.scratchBuffer.ptr, 4934 0, 4935 // size of local and remote addresses. normally + 16. 4936 sockaddr_in.sizeof + 16, 4937 sockaddr_in.sizeof + 16, 4938 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4939 &overlapped 4940 ); 4941 4942 return pfa; 4943 } 4944 4945 override void connect(Address to) { assert(0); } 4946 4947 DWORD lastAnswer; 4948 ubyte[1024] scratchBuffer; 4949 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4950 4951 WSABUF[1] buffer; 4952 OVERLAPPED overlapped; 4953 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4954 overlapped = overlapped.init; 4955 buffer[0].len = cast(DWORD) buf.length; 4956 buffer[0].buf = cast(ubyte*) buf.ptr; 4957 fiber.setPostYield( () { 4958 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4959 if(GetLastError() != 997) { 4960 //throw new Exception("WSASend fail"); 4961 } 4962 } 4963 }); 4964 4965 Fiber.yield(); 4966 return lastAnswer; 4967 } 4968 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4969 overlapped = overlapped.init; 4970 buffer[0].len = cast(DWORD) buf.length; 4971 buffer[0].buf = cast(ubyte*) buf.ptr; 4972 4973 DWORD flags2 = 0; 4974 4975 fiber.setPostYield(() { 4976 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4977 if(GetLastError() != 997) { 4978 //writeln("WSARecv ", WSAGetLastError()); 4979 //throw new Exception("WSARecv fail"); 4980 } 4981 } 4982 }); 4983 4984 Fiber.yield(); 4985 return lastAnswer; 4986 } 4987 4988 // I might go back and implement these for udp things. 4989 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4990 assert(0); 4991 } 4992 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4993 assert(0); 4994 } 4995 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4996 assert(0); 4997 } 4998 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4999 assert(0); 5000 } 5001 5002 // lol overload sets 5003 alias send = typeof(super).send; 5004 alias receive = typeof(super).receive; 5005 alias sendTo = typeof(super).sendTo; 5006 alias receiveFrom = typeof(super).receiveFrom; 5007 5008 } 5009 } 5010 5011 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 5012 assert(connection !is null); 5013 version(cgi_use_fiber) { 5014 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 5015 5016 version(Windows) { 5017 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 5018 } 5019 5020 import core.memory; 5021 GC.addRoot(cast(void*) fiber); 5022 fiber.connection = connection; 5023 fiber.proceed(); 5024 } else { 5025 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 5026 } 5027 } 5028 5029 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 5030 scope(failure) { 5031 // catch all for other errors 5032 try { 5033 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5034 connection.close(); 5035 } catch(Exception e) {} // swallow it, we're aborting anyway. 5036 } 5037 5038 bool closeConnection = alwaysCloseConnection; 5039 5040 /+ 5041 ubyte[4096] inputBuffer = void; 5042 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5043 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5044 5045 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5046 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5047 ir.__ctor(connection, inputBuffer[], true); 5048 +/ 5049 5050 auto ir = new BufferedInputRange(connection); 5051 5052 while(!ir.empty) { 5053 5054 if(ir.view.length == 0) { 5055 ir.popFront(); 5056 if(ir.sourceClosed) { 5057 connection.close(); 5058 closeConnection = true; 5059 break; 5060 } 5061 } 5062 5063 Cgi cgi; 5064 try { 5065 cgi = new CustomCgi(ir, &closeConnection); 5066 // There's a bunch of these casts around because the type matches up with 5067 // the -version=.... specifiers, just you can also create a RequestServer 5068 // and instantiate the things where the types don't match up. It isn't exactly 5069 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5070 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5071 } catch(ConnectionClosedException ce) { 5072 closeConnection = true; 5073 break; 5074 } catch(ConnectionException ce) { 5075 // broken pipe or something, just abort the connection 5076 closeConnection = true; 5077 break; 5078 } catch(HttpVersionNotSupportedException ve) { 5079 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5080 closeConnection = true; 5081 break; 5082 } catch(Throwable t) { 5083 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5084 // anyway let's kill the connection 5085 version(CRuntime_Musl) { 5086 stderr.rawWrite(t.toString()); 5087 stderr.rawWrite("\n"); 5088 } else { 5089 stderr.writeln(t.toString()); 5090 } 5091 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5092 closeConnection = true; 5093 break; 5094 } 5095 assert(cgi !is null); 5096 scope(exit) 5097 cgi.dispose(); 5098 5099 try { 5100 fun(cgi); 5101 cgi.close(); 5102 if(cgi.websocketMode) 5103 closeConnection = true; 5104 } catch(ConnectionException ce) { 5105 // broken pipe or something, just abort the connection 5106 closeConnection = true; 5107 } catch(ConnectionClosedException ce) { 5108 // broken pipe or something, just abort the connection 5109 closeConnection = true; 5110 } catch(Throwable t) { 5111 // a processing error can be recovered from 5112 version(CRuntime_Musl) {} else 5113 stderr.writeln(t.toString); 5114 if(!handleException(cgi, t)) 5115 closeConnection = true; 5116 } 5117 5118 if(globalStopFlag) 5119 closeConnection = true; 5120 5121 if(closeConnection || alwaysCloseConnection) { 5122 connection.shutdown(SocketShutdown.BOTH); 5123 connection.close(); 5124 ir.dispose(); 5125 closeConnection = false; // don't reclose after loop 5126 break; 5127 } else { 5128 if(ir.front.length) { 5129 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5130 } else if(ir.sourceClosed) { 5131 ir.source.shutdown(SocketShutdown.BOTH); 5132 ir.source.close(); 5133 ir.dispose(); 5134 closeConnection = false; 5135 } else { 5136 continue; 5137 // break; // this was for a keepalive experiment 5138 } 5139 } 5140 } 5141 5142 if(closeConnection) { 5143 connection.shutdown(SocketShutdown.BOTH); 5144 connection.close(); 5145 ir.dispose(); 5146 } 5147 5148 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5149 } 5150 5151 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5152 // and now we can buffer 5153 scope(failure) 5154 connection.close(); 5155 5156 import al = std.algorithm; 5157 5158 size_t size; 5159 5160 string[string] headers; 5161 5162 auto range = new BufferedInputRange(connection); 5163 more_data: 5164 auto chunk = range.front(); 5165 // waiting for colon for header length 5166 auto idx = indexOf(cast(string) chunk, ':'); 5167 if(idx == -1) { 5168 try { 5169 range.popFront(); 5170 } catch(Exception e) { 5171 // it is just closed, no big deal 5172 connection.close(); 5173 return; 5174 } 5175 goto more_data; 5176 } 5177 5178 size = to!size_t(cast(string) chunk[0 .. idx]); 5179 chunk = range.consume(idx + 1); 5180 // reading headers 5181 if(chunk.length < size) 5182 range.popFront(0, size + 1); 5183 // we are now guaranteed to have enough 5184 chunk = range.front(); 5185 assert(chunk.length > size); 5186 5187 idx = 0; 5188 string key; 5189 string value; 5190 foreach(part; al.splitter(chunk, '\0')) { 5191 if(idx & 1) { // odd is value 5192 value = cast(string)(part.idup); 5193 headers[key] = value; // commit 5194 } else 5195 key = cast(string)(part.idup); 5196 idx++; 5197 } 5198 5199 enforce(chunk[size] == ','); // the terminator 5200 5201 range.consume(size + 1); 5202 // reading data 5203 // this will be done by Cgi 5204 5205 const(ubyte)[] getScgiChunk() { 5206 // we are already primed 5207 auto data = range.front(); 5208 if(data.length == 0 && !range.sourceClosed) { 5209 range.popFront(0); 5210 data = range.front(); 5211 } else if (range.sourceClosed) 5212 range.source.close(); 5213 5214 return data; 5215 } 5216 5217 void writeScgi(const(ubyte)[] data) { 5218 sendAll(connection, data); 5219 } 5220 5221 void flushScgi() { 5222 // I don't *think* I have to do anything.... 5223 } 5224 5225 Cgi cgi; 5226 try { 5227 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5228 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5229 } catch(Throwable t) { 5230 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5231 connection.close(); 5232 return; // this connection is dead 5233 } 5234 assert(cgi !is null); 5235 scope(exit) cgi.dispose(); 5236 try { 5237 fun(cgi); 5238 cgi.close(); 5239 connection.close(); 5240 } catch(Throwable t) { 5241 // no std err 5242 if(!handleException(cgi, t)) { 5243 connection.close(); 5244 return; 5245 } else { 5246 connection.close(); 5247 return; 5248 } 5249 } 5250 } 5251 5252 string printDate(DateTime date) { 5253 char[29] buffer = void; 5254 printDateToBuffer(date, buffer[]); 5255 return buffer.idup; 5256 } 5257 5258 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5259 assert(buffer.length >= 29); 5260 // 29 static length ? 5261 5262 static immutable daysOfWeek = [ 5263 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5264 ]; 5265 5266 static immutable months = [ 5267 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5268 ]; 5269 5270 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5271 buffer[3 .. 5] = ", "; 5272 buffer[5] = date.day / 10 + '0'; 5273 buffer[6] = date.day % 10 + '0'; 5274 buffer[7] = ' '; 5275 buffer[8 .. 11] = months[date.month]; 5276 buffer[11] = ' '; 5277 auto y = date.year; 5278 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5279 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5280 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5281 buffer[15] = cast(char) (y + '0'); 5282 buffer[16] = ' '; 5283 buffer[17] = date.hour / 10 + '0'; 5284 buffer[18] = date.hour % 10 + '0'; 5285 buffer[19] = ':'; 5286 buffer[20] = date.minute / 10 + '0'; 5287 buffer[21] = date.minute % 10 + '0'; 5288 buffer[22] = ':'; 5289 buffer[23] = date.second / 10 + '0'; 5290 buffer[24] = date.second % 10 + '0'; 5291 buffer[25 .. $] = " GMT"; 5292 5293 return 29; 5294 } 5295 5296 5297 // Referencing this gigantic typeid seems to remind the compiler 5298 // to actually put the symbol in the object file. I guess the immutable 5299 // assoc array array isn't actually included in druntime 5300 void hackAroundLinkerError() { 5301 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5302 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5303 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5304 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5305 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5306 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5307 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5308 // this is getting kinda ridiculous btw. Moving assoc arrays 5309 // to the library is the pain that keeps on coming. 5310 5311 // eh this broke the build on the work server 5312 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5313 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5314 } 5315 5316 5317 5318 5319 5320 version(fastcgi) { 5321 pragma(lib, "fcgi"); 5322 5323 static if(size_t.sizeof == 8) // 64 bit 5324 alias long c_int; 5325 else 5326 alias int c_int; 5327 5328 extern(C) { 5329 struct FCGX_Stream { 5330 ubyte* rdNext; 5331 ubyte* wrNext; 5332 ubyte* stop; 5333 ubyte* stopUnget; 5334 c_int isReader; 5335 c_int isClosed; 5336 c_int wasFCloseCalled; 5337 c_int FCGI_errno; 5338 void* function(FCGX_Stream* stream) fillBuffProc; 5339 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5340 void* data; 5341 } 5342 5343 // note: this is meant to be opaque, so don't access it directly 5344 struct FCGX_Request { 5345 int requestId; 5346 int role; 5347 FCGX_Stream* inStream; 5348 FCGX_Stream* outStream; 5349 FCGX_Stream* errStream; 5350 char** envp; 5351 void* paramsPtr; 5352 int ipcFd; 5353 int isBeginProcessed; 5354 int keepConnection; 5355 int appStatus; 5356 int nWriters; 5357 int flags; 5358 int listen_sock; 5359 } 5360 5361 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5362 void FCGX_Init(); 5363 5364 int FCGX_Accept_r(FCGX_Request *request); 5365 5366 5367 alias char** FCGX_ParamArray; 5368 5369 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5370 c_int FCGX_GetChar(FCGX_Stream* stream); 5371 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5372 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5373 c_int FCGX_FFlush(FCGX_Stream *stream); 5374 5375 int FCGX_OpenSocket(in char*, int); 5376 } 5377 } 5378 5379 5380 /* This might go int a separate module eventually. It is a network input helper class. */ 5381 5382 import std.socket; 5383 5384 version(cgi_use_fiber) { 5385 import core.thread; 5386 5387 version(linux) { 5388 import core.sys.linux.epoll; 5389 5390 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5391 } else version(Windows) { 5392 // declaring the iocp thing below... 5393 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5394 } 5395 5396 version(Windows) 5397 __gshared HANDLE iocp; 5398 5399 version(cgi_use_fiber) { 5400 version(linux) 5401 private enum WakeupEvent { 5402 Read = EPOLLIN, 5403 Write = EPOLLOUT 5404 } 5405 else version(Windows) 5406 private enum WakeupEvent { 5407 Read, Write 5408 } 5409 else static assert(0); 5410 } 5411 5412 version(cgi_use_fiber) 5413 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5414 5415 // static cast since I know what i have in here and don't want to pay for dynamic cast 5416 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5417 5418 version(linux) { 5419 f.setPostYield = () { 5420 if(*registered) { 5421 // rearm 5422 epoll_event evt; 5423 evt.events = e | EPOLLONESHOT; 5424 evt.data.ptr = cast(void*) f; 5425 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5426 throw new Exception("epoll_ctl"); 5427 } else { 5428 // initial registration 5429 *registered = true ; 5430 int fd = source.handle; 5431 epoll_event evt; 5432 evt.events = e | EPOLLONESHOT; 5433 evt.data.ptr = cast(void*) f; 5434 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5435 throw new Exception("epoll_ctl"); 5436 } 5437 }; 5438 5439 Fiber.yield(); 5440 5441 f.setPostYield(null); 5442 } else version(Windows) { 5443 Fiber.yield(); 5444 } 5445 else static assert(0); 5446 } 5447 5448 version(cgi_use_fiber) 5449 void unregisterSource(Socket s) { 5450 version(linux) { 5451 epoll_event evt; 5452 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5453 } else version(Windows) { 5454 // intentionally blank 5455 } 5456 else static assert(0); 5457 } 5458 5459 // it is a class primarily for reference semantics 5460 // I might change this interface 5461 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5462 class BufferedInputRange { 5463 version(Posix) 5464 this(int source, ubyte[] buffer = null) { 5465 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5466 } 5467 5468 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5469 // if they connect but never send stuff to us, we don't want it wasting the process 5470 // so setting a time out 5471 version(cgi_use_fiber) 5472 source.blocking = false; 5473 else 5474 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5475 5476 this.source = source; 5477 if(buffer is null) { 5478 underlyingBuffer = new ubyte[4096]; 5479 this.allowGrowth = true; 5480 } else { 5481 underlyingBuffer = buffer; 5482 this.allowGrowth = allowGrowth; 5483 } 5484 5485 assert(underlyingBuffer.length); 5486 5487 // we assume view.ptr is always inside underlyingBuffer 5488 view = underlyingBuffer[0 .. 0]; 5489 5490 popFront(); // prime 5491 } 5492 5493 version(cgi_use_fiber) { 5494 bool registered; 5495 } 5496 5497 void dispose() { 5498 version(cgi_use_fiber) { 5499 if(registered) 5500 unregisterSource(source); 5501 } 5502 } 5503 5504 /** 5505 A slight difference from regular ranges is you can give it the maximum 5506 number of bytes to consume. 5507 5508 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5509 consume() yourself and use a regular foreach, it will infinitely loop! 5510 5511 The default is to do what a normal range does, and consume the whole buffer 5512 and wait for additional input. 5513 5514 You can also specify 0, to append to the buffer, or any other number 5515 to remove the front n bytes and wait for more. 5516 */ 5517 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5518 if(sourceClosed) 5519 throw new ConnectionClosedException("can't get any more data from a closed source"); 5520 if(!skipConsume) 5521 consume(maxBytesToConsume); 5522 5523 // we might have to grow the buffer 5524 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5525 if(allowGrowth) { 5526 //import std.stdio; writeln("growth"); 5527 auto viewStart = view.ptr - underlyingBuffer.ptr; 5528 size_t growth = 4096; 5529 // make sure we have enough for what we're being asked for 5530 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5531 growth = minBytesToSettleFor - underlyingBuffer.length; 5532 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5533 underlyingBuffer.length += growth; 5534 view = underlyingBuffer[viewStart .. view.length]; 5535 } else 5536 throw new Exception("No room left in the buffer"); 5537 } 5538 5539 do { 5540 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5541 try_again: 5542 auto ret = source.receive(freeSpace); 5543 if(ret == Socket.ERROR) { 5544 if(wouldHaveBlocked()) { 5545 version(cgi_use_fiber) { 5546 registerEventWakeup(®istered, source, WakeupEvent.Read); 5547 goto try_again; 5548 } else { 5549 // gonna treat a timeout here as a close 5550 sourceClosed = true; 5551 return; 5552 } 5553 } 5554 version(Posix) { 5555 import core.stdc.errno; 5556 if(errno == EINTR || errno == EAGAIN) { 5557 goto try_again; 5558 } 5559 if(errno == ECONNRESET) { 5560 sourceClosed = true; 5561 return; 5562 } 5563 } 5564 throw new Exception(lastSocketError); // FIXME 5565 } 5566 if(ret == 0) { 5567 sourceClosed = true; 5568 return; 5569 } 5570 5571 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5572 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5573 //import std.stdio; writeln(cast(string) view); 5574 } while(view.length < minBytesToSettleFor); 5575 } 5576 5577 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5578 /// You might want to idup the data you are consuming if you store it, since it may 5579 /// be overwritten on the new popFront. 5580 /// 5581 /// You do not need to call this if you always want to wait for more data when you 5582 /// consume some. 5583 ubyte[] consume(size_t bytes) { 5584 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5585 view = view[bytes > $ ? $ : bytes .. $]; 5586 if(view.length == 0) { 5587 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5588 /* 5589 writeln("HERE"); 5590 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5591 writeln(cast(string)front); 5592 writeln("DONE"); 5593 */ 5594 } 5595 return front; 5596 } 5597 5598 bool empty() { 5599 return sourceClosed && view.length == 0; 5600 } 5601 5602 ubyte[] front() { 5603 return view; 5604 } 5605 5606 invariant() { 5607 assert(view.ptr >= underlyingBuffer.ptr); 5608 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5609 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5610 } 5611 5612 ubyte[] underlyingBuffer; 5613 bool allowGrowth; 5614 ubyte[] view; 5615 Socket source; 5616 bool sourceClosed; 5617 } 5618 5619 private class FakeSocketForStdin : Socket { 5620 import std.stdio; 5621 5622 this() { 5623 5624 } 5625 5626 private bool closed; 5627 5628 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5629 if(closed) 5630 throw new Exception("Closed"); 5631 return stdin.rawRead(buffer).length; 5632 } 5633 5634 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5635 if(closed) 5636 throw new Exception("Closed"); 5637 stdout.rawWrite(buffer); 5638 return buffer.length; 5639 } 5640 5641 override void close() @trusted scope { 5642 (cast(void delegate() @nogc nothrow) &realClose)(); 5643 } 5644 5645 override void shutdown(SocketShutdown s) { 5646 // FIXME 5647 } 5648 5649 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5650 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5651 5652 override @property @trusted Address remoteAddress() { return null; } 5653 override @property @trusted Address localAddress() { return null; } 5654 5655 void realClose() { 5656 closed = true; 5657 try { 5658 stdin.close(); 5659 stdout.close(); 5660 } catch(Exception e) { 5661 5662 } 5663 } 5664 } 5665 5666 import core.sync.semaphore; 5667 import core.atomic; 5668 5669 /** 5670 To use this thing: 5671 5672 --- 5673 void handler(Socket s) { do something... } 5674 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5675 manager.listen(); 5676 --- 5677 5678 The 4th parameter is optional. 5679 5680 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5681 comes in, you will get control. You can just continue; though to fetch more. 5682 5683 5684 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5685 */ 5686 class ListeningConnectionManager { 5687 Semaphore semaphore; 5688 Socket[256] queue; 5689 shared(ubyte) nextIndexFront; 5690 ubyte nextIndexBack; 5691 shared(int) queueLength; 5692 5693 Socket acceptCancelable() { 5694 version(Posix) { 5695 import core.sys.posix.sys.select; 5696 fd_set read_fds; 5697 FD_ZERO(&read_fds); 5698 int max = 0; 5699 foreach(listener; listeners) { 5700 FD_SET(listener.handle, &read_fds); 5701 if(listener.handle > max) 5702 max = listener.handle; 5703 } 5704 if(cancelfd != -1) { 5705 FD_SET(cancelfd, &read_fds); 5706 if(cancelfd > max) 5707 max = cancelfd; 5708 } 5709 auto ret = select(max + 1, &read_fds, null, null, null); 5710 if(ret == -1) { 5711 import core.stdc.errno; 5712 if(errno == EINTR) 5713 return null; 5714 else 5715 throw new Exception("wtf select"); 5716 } 5717 5718 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5719 return null; 5720 } 5721 5722 foreach(listener; listeners) { 5723 if(FD_ISSET(listener.handle, &read_fds)) 5724 return listener.accept(); 5725 } 5726 5727 return null; 5728 } else { 5729 5730 auto check = new SocketSet(); 5731 5732 keep_looping: 5733 check.reset(); 5734 foreach(listener; listeners) 5735 check.add(listener); 5736 5737 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5738 auto got = Socket.select(check, null, null, 3.seconds); 5739 if(got > 0) 5740 foreach(listener; listeners) 5741 if(check.isSet(listener)) 5742 return listener.accept(); 5743 if(globalStopFlag) 5744 return null; 5745 else 5746 goto keep_looping; 5747 } 5748 } 5749 5750 int defaultNumberOfThreads() { 5751 import std.parallelism; 5752 version(cgi_use_fiber) { 5753 return totalCPUs * 2 + 1; // still chance some will be pointlessly blocked anyway 5754 } else { 5755 // I times 4 here because there's a good chance some will be blocked on i/o. 5756 return totalCPUs * 4; 5757 } 5758 5759 } 5760 5761 void listen() { 5762 shared(int) loopBroken; 5763 5764 version(Posix) { 5765 import core.sys.posix.signal; 5766 signal(SIGPIPE, SIG_IGN); 5767 } 5768 5769 version(linux) { 5770 if(cancelfd == -1) 5771 cancelfd = eventfd(0, 0); 5772 } 5773 5774 version(cgi_no_threads) { 5775 // NEVER USE THIS 5776 // it exists only for debugging and other special occasions 5777 5778 // the thread mode is faster and less likely to stall the whole 5779 // thing when a request is slow 5780 while(!loopBroken && !globalStopFlag) { 5781 auto sn = acceptCancelable(); 5782 if(sn is null) continue; 5783 cloexec(sn); 5784 try { 5785 handler(sn); 5786 } catch(Exception e) { 5787 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5788 sn.close(); 5789 } 5790 } 5791 } else { 5792 5793 if(useFork) { 5794 version(linux) { 5795 //asm { int 3; } 5796 fork(); 5797 } 5798 } 5799 5800 version(cgi_use_fiber) { 5801 5802 version(Windows) { 5803 // please note these are overlapped sockets! so the accept just kicks things off 5804 foreach(listener; listeners) 5805 listener.accept(); 5806 } 5807 5808 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5809 foreach(i, ref thread; threads) { 5810 thread = new WorkerThread(this, handler, cast(int) i); 5811 thread.start(); 5812 } 5813 5814 bool fiber_crash_check() { 5815 bool hasAnyRunning; 5816 foreach(thread; threads) { 5817 if(!thread.isRunning) { 5818 thread.join(); 5819 } else hasAnyRunning = true; 5820 } 5821 5822 return (!hasAnyRunning); 5823 } 5824 5825 5826 while(!globalStopFlag) { 5827 Thread.sleep(1.seconds); 5828 if(fiber_crash_check()) 5829 break; 5830 } 5831 5832 } else { 5833 semaphore = new Semaphore(); 5834 5835 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5836 foreach(i, ref thread; threads) { 5837 thread = new ConnectionThread(this, handler, cast(int) i); 5838 thread.start(); 5839 } 5840 5841 while(!loopBroken && !globalStopFlag) { 5842 Socket sn; 5843 5844 bool crash_check() { 5845 bool hasAnyRunning; 5846 foreach(thread; threads) { 5847 if(!thread.isRunning) { 5848 thread.join(); 5849 } else hasAnyRunning = true; 5850 } 5851 5852 return (!hasAnyRunning); 5853 } 5854 5855 5856 void accept_new_connection() { 5857 sn = acceptCancelable(); 5858 if(sn is null) return; 5859 cloexec(sn); 5860 if(tcp) { 5861 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5862 // on the socket because we do some buffering internally. I think this helps, 5863 // certainly does for small requests, and I think it does for larger ones too 5864 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5865 5866 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5867 } 5868 } 5869 5870 void existing_connection_new_data() { 5871 // wait until a slot opens up 5872 // int waited = 0; 5873 while(queueLength >= queue.length) { 5874 Thread.sleep(1.msecs); 5875 // waited ++; 5876 } 5877 // if(waited) {import std.stdio; writeln(waited);} 5878 synchronized(this) { 5879 queue[nextIndexBack] = sn; 5880 nextIndexBack++; 5881 atomicOp!"+="(queueLength, 1); 5882 } 5883 semaphore.notify(); 5884 } 5885 5886 5887 accept_new_connection(); 5888 if(sn !is null) 5889 existing_connection_new_data(); 5890 else if(sn is null && globalStopFlag) { 5891 foreach(thread; threads) { 5892 semaphore.notify(); 5893 } 5894 Thread.sleep(50.msecs); 5895 } 5896 5897 if(crash_check()) 5898 break; 5899 } 5900 } 5901 5902 // FIXME: i typically stop this with ctrl+c which never 5903 // actually gets here. i need to do a sigint handler. 5904 if(cleanup) 5905 cleanup(); 5906 } 5907 } 5908 5909 //version(linux) 5910 //int epoll_fd; 5911 5912 bool tcp; 5913 void delegate() cleanup; 5914 5915 private void function(Socket) fhandler; 5916 private void dg_handler(Socket s) { 5917 fhandler(s); 5918 } 5919 5920 5921 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5922 fhandler = handler; 5923 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 5924 } 5925 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5926 string[] host; 5927 ushort[] port; 5928 5929 foreach(spec; listenSpec) { 5930 /+ 5931 The format: 5932 5933 protocol:// 5934 address_spec 5935 5936 Protocol is optional. Must be http, https, scgi, or fastcgi. 5937 5938 address_spec is either: 5939 ipv4 address : port 5940 [ipv6 address] : port 5941 unix:filename 5942 abstract:name 5943 port <which is tcp but on any interface> 5944 +/ 5945 5946 string protocol; 5947 string address_spec; 5948 5949 auto protocolIdx = spec.indexOf("://"); 5950 if(protocolIdx != -1) { 5951 protocol = spec[0 .. protocolIdx]; 5952 address_spec = spec[protocolIdx + "://".length .. $]; 5953 } else { 5954 address_spec = spec; 5955 } 5956 5957 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 5958 host ~= address_spec; 5959 port ~= 0; 5960 } else { 5961 auto idx = address_spec.lastIndexOf(":"); 5962 if(idx == -1) { 5963 host ~= null; 5964 } else { 5965 auto as = address_spec[0 .. idx]; 5966 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 5967 as = as[1 .. $-1]; 5968 host ~= as; 5969 } 5970 port ~= address_spec[idx + 1 .. $].to!ushort; 5971 } 5972 5973 } 5974 5975 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 5976 } 5977 5978 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5979 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5980 } 5981 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5982 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5983 } 5984 5985 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5986 fhandler = handler; 5987 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5988 } 5989 5990 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5991 assert(host.length == port.length); 5992 5993 this.handler = handler; 5994 this.useFork = useFork; 5995 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5996 5997 listeners.reserve(host.length); 5998 5999 foreach(i; 0 .. host.length) 6000 if(host[i] == "localhost") { 6001 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 6002 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 6003 } else { 6004 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 6005 } 6006 6007 version(cgi_use_fiber) 6008 if(useFork) { 6009 foreach(listener; listeners) 6010 listener.blocking = false; 6011 } 6012 6013 // this is the UI control thread and thus gets more priority 6014 Thread.getThis.priority = Thread.PRIORITY_MAX; 6015 } 6016 6017 Socket[] listeners; 6018 void delegate(Socket) handler; 6019 6020 immutable bool useFork; 6021 int numberOfThreads; 6022 } 6023 6024 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 6025 Socket listener; 6026 if(host.startsWith("unix:")) { 6027 version(Posix) { 6028 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6029 cloexec(listener); 6030 string filename = host["unix:".length .. $].idup; 6031 listener.bind(new UnixAddress(filename)); 6032 cleanup = delegate() { 6033 listener.close(); 6034 import std.file; 6035 remove(filename); 6036 }; 6037 tcp = false; 6038 } else { 6039 throw new Exception("unix sockets not supported on this system"); 6040 } 6041 } else if(host.startsWith("abstract:")) { 6042 version(linux) { 6043 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6044 cloexec(listener); 6045 string filename = "\0" ~ host["abstract:".length .. $]; 6046 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6047 listener.bind(new UnixAddress(filename)); 6048 tcp = false; 6049 } else { 6050 throw new Exception("abstract unix sockets not supported on this system"); 6051 } 6052 } else { 6053 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6054 version(cgi_use_fiber) { 6055 version(Windows) 6056 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6057 else 6058 listener = new Socket(address.addressFamily, SocketType.STREAM); 6059 } else { 6060 listener = new Socket(address.addressFamily, SocketType.STREAM); 6061 } 6062 cloexec(listener); 6063 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6064 if(address.addressFamily == AddressFamily.INET6) 6065 listener.setOption(SocketOptionLevel.IPV6, SocketOption.IPV6_V6ONLY, true); 6066 listener.bind(address); 6067 cleanup = delegate() { 6068 listener.close(); 6069 }; 6070 tcp = true; 6071 } 6072 6073 listener.listen(backQueue); 6074 6075 if (dropPrivs !is null) // can be null, backwards compatibility 6076 dropPrivs(); 6077 6078 return listener; 6079 } 6080 6081 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6082 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6083 if(data.length == 0) return; 6084 ptrdiff_t amount; 6085 //import std.stdio; writeln("***",cast(string) data,"///"); 6086 do { 6087 amount = s.send(data); 6088 if(amount == Socket.ERROR) { 6089 version(cgi_use_fiber) { 6090 if(wouldHaveBlocked()) { 6091 bool registered = true; 6092 registerEventWakeup(®istered, s, WakeupEvent.Write); 6093 continue; 6094 } 6095 } 6096 throw new ConnectionException(s, lastSocketError, file, line); 6097 } 6098 assert(amount > 0); 6099 6100 data = data[amount .. $]; 6101 } while(data.length); 6102 } 6103 6104 class ConnectionException : Exception { 6105 Socket socket; 6106 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6107 this.socket = s; 6108 super(msg, file, line); 6109 } 6110 } 6111 6112 class HttpVersionNotSupportedException : Exception { 6113 this(string file = __FILE__, size_t line = __LINE__) { 6114 super("HTTP Version Not Supported", file, line); 6115 } 6116 } 6117 6118 alias void delegate(Socket) CMT; 6119 6120 import core.thread; 6121 /+ 6122 cgi.d now uses a hybrid of event i/o and threads at the top level. 6123 6124 Top level thread is responsible for accepting sockets and selecting on them. 6125 6126 It then indicates to a child that a request is pending, and any random worker 6127 thread that is free handles it. It goes into blocking mode and handles that 6128 http request to completion. 6129 6130 At that point, it goes back into the waiting queue. 6131 6132 6133 This concept is only implemented on Linux. On all other systems, it still 6134 uses the worker threads and semaphores (which is perfectly fine for a lot of 6135 things! Just having a great number of keep-alive connections will break that.) 6136 6137 6138 So the algorithm is: 6139 6140 select(accept, event, pending) 6141 if accept -> send socket to free thread, if any. if not, add socket to queue 6142 if event -> send the signaling thread a socket from the queue, if not, mark it free 6143 - event might block until it can be *written* to. it is a fifo sending socket fds! 6144 6145 A worker only does one http request at a time, then signals its availability back to the boss. 6146 6147 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6148 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6149 actual FD will not be kept out here. 6150 6151 So: 6152 queue = sockets we know are ready to read now, but no worker thread is available 6153 idle list = worker threads not doing anything else. they signal back and forth 6154 6155 the workers all read off the event fd. This is the semaphore wait 6156 6157 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6158 it puts it in the queue and writes to the event fd. 6159 6160 The child could put the socket back in the epoll thing itself. 6161 6162 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6163 +/ 6164 class ConnectionThread : Thread { 6165 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6166 this.lcm = lcm; 6167 this.dg = dg; 6168 this.myThreadNumber = myThreadNumber; 6169 super(&run); 6170 } 6171 6172 void run() { 6173 while(true) { 6174 // so if there's a bunch of idle keep-alive connections, it can 6175 // consume all the worker threads... just sitting there. 6176 lcm.semaphore.wait(); 6177 if(globalStopFlag) 6178 return; 6179 Socket socket; 6180 synchronized(lcm) { 6181 auto idx = lcm.nextIndexFront; 6182 socket = lcm.queue[idx]; 6183 lcm.queue[idx] = null; 6184 atomicOp!"+="(lcm.nextIndexFront, 1); 6185 atomicOp!"-="(lcm.queueLength, 1); 6186 } 6187 try { 6188 //import std.stdio; writeln(myThreadNumber, " taking it"); 6189 dg(socket); 6190 /+ 6191 if(socket.isAlive) { 6192 // process it more later 6193 version(linux) { 6194 import core.sys.linux.epoll; 6195 epoll_event ev; 6196 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6197 ev.data.fd = socket.handle; 6198 import std.stdio; writeln("adding"); 6199 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6200 if(errno == EEXIST) { 6201 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6202 ev.data.fd = socket.handle; 6203 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6204 throw new Exception("epoll_ctl " ~ to!string(errno)); 6205 } else 6206 throw new Exception("epoll_ctl " ~ to!string(errno)); 6207 } 6208 //import std.stdio; writeln("keep alive"); 6209 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6210 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6211 } else { 6212 continue; // hope it times out in a reasonable amount of time... 6213 } 6214 } 6215 +/ 6216 } catch(ConnectionClosedException e) { 6217 // can just ignore this, it is fairly normal 6218 socket.close(); 6219 } catch(Throwable e) { 6220 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6221 socket.close(); 6222 } 6223 } 6224 } 6225 6226 ListeningConnectionManager lcm; 6227 CMT dg; 6228 int myThreadNumber; 6229 } 6230 6231 version(cgi_use_fiber) 6232 class WorkerThread : Thread { 6233 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6234 this.lcm = lcm; 6235 this.dg = dg; 6236 this.myThreadNumber = myThreadNumber; 6237 super(&run); 6238 } 6239 6240 version(Windows) 6241 void run() { 6242 auto timeout = INFINITE; 6243 PseudoblockingOverlappedSocket key; 6244 OVERLAPPED* overlapped; 6245 DWORD bytes; 6246 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6247 if(key is null) 6248 continue; 6249 key.lastAnswer = bytes; 6250 if(key.fiber) { 6251 key.fiber.proceed(); 6252 } else { 6253 // we have a new connection, issue the first receive on it and issue the next accept 6254 6255 auto sn = key.accepted; 6256 6257 key.accept(); 6258 6259 cloexec(sn); 6260 if(lcm.tcp) { 6261 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6262 // on the socket because we do some buffering internally. I think this helps, 6263 // certainly does for small requests, and I think it does for larger ones too 6264 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6265 6266 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6267 } 6268 6269 dg(sn); 6270 } 6271 } 6272 //SleepEx(INFINITE, TRUE); 6273 } 6274 6275 version(linux) 6276 void run() { 6277 6278 import core.sys.linux.epoll; 6279 epfd = epoll_create1(EPOLL_CLOEXEC); 6280 if(epfd == -1) 6281 throw new Exception("epoll_create1 " ~ to!string(errno)); 6282 scope(exit) { 6283 import core.sys.posix.unistd; 6284 close(epfd); 6285 } 6286 6287 { 6288 epoll_event ev; 6289 ev.events = EPOLLIN; 6290 ev.data.fd = cancelfd; 6291 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6292 } 6293 6294 foreach(listener; lcm.listeners) { 6295 epoll_event ev; 6296 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6297 ev.data.fd = listener.handle; 6298 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6299 throw new Exception("epoll_ctl " ~ to!string(errno)); 6300 } 6301 6302 6303 6304 while(!globalStopFlag) { 6305 Socket sn; 6306 6307 epoll_event[64] events; 6308 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6309 if(nfds == -1) { 6310 if(errno == EINTR) 6311 continue; 6312 throw new Exception("epoll_wait " ~ to!string(errno)); 6313 } 6314 6315 outer: foreach(idx; 0 .. nfds) { 6316 auto flags = events[idx].events; 6317 6318 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6319 globalStopFlag = true; 6320 //import std.stdio; writeln("exit heard"); 6321 break; 6322 } else { 6323 foreach(listener; lcm.listeners) { 6324 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6325 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6326 // this try/catch is because it is set to non-blocking mode 6327 // and Phobos' stupid api throws an exception instead of returning 6328 // if it would block. Why would it block? because a forked process 6329 // might have beat us to it, but the wakeup event thundered our herds. 6330 try 6331 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6332 catch(SocketAcceptException e) { continue outer; } 6333 6334 cloexec(sn); 6335 if(lcm.tcp) { 6336 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6337 // on the socket because we do some buffering internally. I think this helps, 6338 // certainly does for small requests, and I think it does for larger ones too 6339 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6340 6341 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6342 } 6343 6344 dg(sn); 6345 continue outer; 6346 } else { 6347 // writeln(events[idx].data.ptr); 6348 } 6349 } 6350 6351 if(cast(size_t) events[idx].data.ptr < 1024) { 6352 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6353 } 6354 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6355 fiber.proceed(); 6356 } 6357 } 6358 } 6359 } 6360 6361 ListeningConnectionManager lcm; 6362 CMT dg; 6363 int myThreadNumber; 6364 } 6365 6366 6367 /* Done with network helper */ 6368 6369 /* Helpers for doing temporary files. Used both here and in web.d */ 6370 6371 version(Windows) { 6372 import core.sys.windows.windows; 6373 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6374 alias GetTempPathW GetTempPath; 6375 } 6376 6377 version(Posix) { 6378 static import linux = core.sys.posix.unistd; 6379 } 6380 6381 string getTempDirectory() { 6382 string path; 6383 version(Windows) { 6384 wchar[1024] buffer; 6385 auto len = GetTempPath(1024, buffer.ptr); 6386 if(len == 0) 6387 throw new Exception("couldn't find a temporary path"); 6388 6389 auto b = buffer[0 .. len]; 6390 6391 path = to!string(b); 6392 } else 6393 path = "/tmp/"; 6394 6395 return path; 6396 } 6397 6398 6399 // I like std.date. These functions help keep my old code and data working with phobos changing. 6400 6401 long sysTimeToDTime(in SysTime sysTime) { 6402 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6403 } 6404 6405 long dateTimeToDTime(in DateTime dt) { 6406 return sysTimeToDTime(cast(SysTime) dt); 6407 } 6408 6409 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6410 return sysTimeToDTime(Clock.currTime(UTC())); 6411 } 6412 6413 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6414 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6415 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6416 return SysTime(hnsecs, tz); 6417 } 6418 6419 6420 6421 // this is a helper to read HTTP transfer-encoding: chunked responses 6422 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6423 immutable(ubyte)[] ret; 6424 6425 another_chunk: 6426 // If here, we are at the beginning of a chunk. 6427 auto a = ir.front(); 6428 int chunkSize; 6429 int loc = locationOf(a, "\r\n"); 6430 while(loc == -1) { 6431 ir.popFront(); 6432 a = ir.front(); 6433 loc = locationOf(a, "\r\n"); 6434 } 6435 6436 string hex; 6437 hex = ""; 6438 for(int i = 0; i < loc; i++) { 6439 char c = a[i]; 6440 if(c >= 'A' && c <= 'Z') 6441 c += 0x20; 6442 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6443 hex ~= c; 6444 } else { 6445 break; 6446 } 6447 } 6448 6449 assert(hex.length); 6450 6451 int power = 1; 6452 int size = 0; 6453 foreach(cc1; retro(hex)) { 6454 dchar cc = cc1; 6455 if(cc >= 'a' && cc <= 'z') 6456 cc -= 0x20; 6457 int val = 0; 6458 if(cc >= '0' && cc <= '9') 6459 val = cc - '0'; 6460 else 6461 val = cc - 'A' + 10; 6462 6463 size += power * val; 6464 power *= 16; 6465 } 6466 6467 chunkSize = size; 6468 assert(size >= 0); 6469 6470 if(loc + 2 > a.length) { 6471 ir.popFront(0, a.length + loc + 2); 6472 a = ir.front(); 6473 } 6474 6475 a = ir.consume(loc + 2); 6476 6477 if(chunkSize == 0) { // we're done with the response 6478 // if we got here, will change must be true.... 6479 more_footers: 6480 loc = locationOf(a, "\r\n"); 6481 if(loc == -1) { 6482 ir.popFront(); 6483 a = ir.front; 6484 goto more_footers; 6485 } else { 6486 assert(loc == 0); 6487 ir.consume(loc + 2); 6488 goto finish; 6489 } 6490 } else { 6491 // if we got here, will change must be true.... 6492 if(a.length < chunkSize + 2) { 6493 ir.popFront(0, chunkSize + 2); 6494 a = ir.front(); 6495 } 6496 6497 ret ~= (a[0..chunkSize]); 6498 6499 if(!(a.length > chunkSize + 2)) { 6500 ir.popFront(0, chunkSize + 2); 6501 a = ir.front(); 6502 } 6503 assert(a[chunkSize] == 13); 6504 assert(a[chunkSize+1] == 10); 6505 a = ir.consume(chunkSize + 2); 6506 chunkSize = 0; 6507 goto another_chunk; 6508 } 6509 6510 finish: 6511 return ret; 6512 } 6513 6514 // I want to be able to get data from multiple sources the same way... 6515 interface ByChunkRange { 6516 bool empty(); 6517 void popFront(); 6518 const(ubyte)[] front(); 6519 } 6520 6521 ByChunkRange byChunk(const(ubyte)[] data) { 6522 return new class ByChunkRange { 6523 override bool empty() { 6524 return !data.length; 6525 } 6526 6527 override void popFront() { 6528 if(data.length > 4096) 6529 data = data[4096 .. $]; 6530 else 6531 data = null; 6532 } 6533 6534 override const(ubyte)[] front() { 6535 return data[0 .. $ > 4096 ? 4096 : $]; 6536 } 6537 }; 6538 } 6539 6540 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6541 const(ubyte)[] f; 6542 6543 f = ir.front; 6544 if(f.length > atMost) 6545 f = f[0 .. atMost]; 6546 6547 return new class ByChunkRange { 6548 override bool empty() { 6549 return atMost == 0; 6550 } 6551 6552 override const(ubyte)[] front() { 6553 return f; 6554 } 6555 6556 override void popFront() { 6557 ir.consume(f.length); 6558 atMost -= f.length; 6559 auto a = ir.front(); 6560 6561 if(a.length <= atMost) { 6562 f = a; 6563 atMost -= a.length; 6564 a = ir.consume(a.length); 6565 if(atMost != 0) 6566 ir.popFront(); 6567 if(f.length == 0) { 6568 f = ir.front(); 6569 } 6570 } else { 6571 // we actually have *more* here than we need.... 6572 f = a[0..atMost]; 6573 atMost = 0; 6574 ir.consume(atMost); 6575 } 6576 } 6577 }; 6578 } 6579 6580 version(cgi_with_websocket) { 6581 // http://tools.ietf.org/html/rfc6455 6582 6583 /++ 6584 WEBSOCKET SUPPORT: 6585 6586 Full example: 6587 --- 6588 import arsd.cgi; 6589 6590 void websocketEcho(Cgi cgi) { 6591 if(cgi.websocketRequested()) { 6592 if(cgi.origin != "http://arsdnet.net") 6593 throw new Exception("bad origin"); 6594 auto websocket = cgi.acceptWebsocket(); 6595 6596 websocket.send("hello"); 6597 websocket.send(" world!"); 6598 6599 auto msg = websocket.recv(); 6600 while(msg.opcode != WebSocketOpcode.close) { 6601 if(msg.opcode == WebSocketOpcode.text) { 6602 websocket.send(msg.textData); 6603 } else if(msg.opcode == WebSocketOpcode.binary) { 6604 websocket.send(msg.data); 6605 } 6606 6607 msg = websocket.recv(); 6608 } 6609 6610 websocket.close(); 6611 } else { 6612 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6613 } 6614 } 6615 6616 mixin GenericMain!websocketEcho; 6617 --- 6618 +/ 6619 6620 class WebSocket { 6621 Cgi cgi; 6622 6623 private this(Cgi cgi) { 6624 this.cgi = cgi; 6625 6626 Socket socket = cgi.idlol.source; 6627 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6628 } 6629 6630 // returns true if data available, false if it timed out 6631 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6632 if(!waitForNextMessageWouldBlock()) 6633 return true; 6634 if(isDataPending(timeout)) 6635 return true; // this is kinda a lie. 6636 6637 return false; 6638 } 6639 6640 public bool lowLevelReceive() { 6641 auto bfr = cgi.idlol; 6642 top: 6643 auto got = bfr.front; 6644 if(got.length) { 6645 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6646 receiveBuffer.length += receiveBufferUsedLength + got.length; 6647 6648 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6649 receiveBufferUsedLength += got.length; 6650 bfr.consume(got.length); 6651 6652 return true; 6653 } 6654 6655 if(bfr.sourceClosed) 6656 return false; 6657 6658 bfr.popFront(0); 6659 if(bfr.sourceClosed) 6660 return false; 6661 goto top; 6662 } 6663 6664 6665 bool isDataPending(Duration timeout = 0.seconds) { 6666 Socket socket = cgi.idlol.source; 6667 6668 auto check = new SocketSet(); 6669 check.add(socket); 6670 6671 auto got = Socket.select(check, null, null, timeout); 6672 if(got > 0) 6673 return true; 6674 return false; 6675 } 6676 6677 // note: this blocks 6678 WebSocketFrame recv() { 6679 return waitForNextMessage(); 6680 } 6681 6682 6683 6684 6685 private void llclose() { 6686 cgi.close(); 6687 } 6688 6689 private void llsend(ubyte[] data) { 6690 cgi.write(data); 6691 cgi.flush(); 6692 } 6693 6694 void unregisterActiveSocket(WebSocket) {} 6695 6696 /* copy/paste section { */ 6697 6698 private int readyState_; 6699 private ubyte[] receiveBuffer; 6700 private size_t receiveBufferUsedLength; 6701 6702 private Config config; 6703 6704 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6705 enum OPEN = 1; /// The connection is open and ready to communicate. 6706 enum CLOSING = 2; /// The connection is in the process of closing. 6707 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6708 6709 /++ 6710 6711 +/ 6712 /// Group: foundational 6713 static struct Config { 6714 /++ 6715 These control the size of the receive buffer. 6716 6717 It starts at the initial size, will temporarily 6718 balloon up to the maximum size, and will reuse 6719 a buffer up to the likely size. 6720 6721 Anything larger than the maximum size will cause 6722 the connection to be aborted and an exception thrown. 6723 This is to protect you against a peer trying to 6724 exhaust your memory, while keeping the user-level 6725 processing simple. 6726 +/ 6727 size_t initialReceiveBufferSize = 4096; 6728 size_t likelyReceiveBufferSize = 4096; /// ditto 6729 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6730 6731 /++ 6732 Maximum combined size of a message. 6733 +/ 6734 size_t maximumMessageSize = 10 * 1024 * 1024; 6735 6736 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6737 string origin; /// Origin URL to send with the handshake, if desired. 6738 string protocol; /// the protocol header, if desired. 6739 6740 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6741 } 6742 6743 /++ 6744 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6745 +/ 6746 int readyState() { 6747 return readyState_; 6748 } 6749 6750 /++ 6751 Closes the connection, sending a graceful teardown message to the other side. 6752 +/ 6753 /// Group: foundational 6754 void close(int code = 0, string reason = null) 6755 //in (reason.length < 123) 6756 in { assert(reason.length < 123); } do 6757 { 6758 if(readyState_ != OPEN) 6759 return; // it cool, we done 6760 WebSocketFrame wss; 6761 wss.fin = true; 6762 wss.opcode = WebSocketOpcode.close; 6763 wss.data = cast(ubyte[]) reason.dup; 6764 wss.send(&llsend); 6765 6766 readyState_ = CLOSING; 6767 6768 llclose(); 6769 } 6770 6771 /++ 6772 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6773 +/ 6774 /// Group: foundational 6775 void ping() { 6776 WebSocketFrame wss; 6777 wss.fin = true; 6778 wss.opcode = WebSocketOpcode.ping; 6779 wss.send(&llsend); 6780 } 6781 6782 // automatically handled.... 6783 void pong() { 6784 WebSocketFrame wss; 6785 wss.fin = true; 6786 wss.opcode = WebSocketOpcode.pong; 6787 wss.send(&llsend); 6788 } 6789 6790 /++ 6791 Sends a text message through the websocket. 6792 +/ 6793 /// Group: foundational 6794 void send(in char[] textData) { 6795 WebSocketFrame wss; 6796 wss.fin = true; 6797 wss.opcode = WebSocketOpcode.text; 6798 wss.data = cast(ubyte[]) textData.dup; 6799 wss.send(&llsend); 6800 } 6801 6802 /++ 6803 Sends a binary message through the websocket. 6804 +/ 6805 /// Group: foundational 6806 void send(in ubyte[] binaryData) { 6807 WebSocketFrame wss; 6808 wss.fin = true; 6809 wss.opcode = WebSocketOpcode.binary; 6810 wss.data = cast(ubyte[]) binaryData.dup; 6811 wss.send(&llsend); 6812 } 6813 6814 /++ 6815 Waits for and returns the next complete message on the socket. 6816 6817 Note that the onmessage function is still called, right before 6818 this returns. 6819 +/ 6820 /// Group: blocking_api 6821 public WebSocketFrame waitForNextMessage() { 6822 do { 6823 auto m = processOnce(); 6824 if(m.populated) 6825 return m; 6826 } while(lowLevelReceive()); 6827 6828 throw new ConnectionClosedException("Websocket receive timed out"); 6829 //return WebSocketFrame.init; // FIXME? maybe. 6830 } 6831 6832 /++ 6833 Tells if [waitForNextMessage] would block. 6834 +/ 6835 /// Group: blocking_api 6836 public bool waitForNextMessageWouldBlock() { 6837 checkAgain: 6838 if(isMessageBuffered()) 6839 return false; 6840 if(!isDataPending()) 6841 return true; 6842 while(isDataPending()) 6843 lowLevelReceive(); 6844 goto checkAgain; 6845 } 6846 6847 /++ 6848 Is there a message in the buffer already? 6849 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6850 If `false`, check [isDataPending] as the next step. 6851 +/ 6852 /// Group: blocking_api 6853 public bool isMessageBuffered() { 6854 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6855 auto s = d; 6856 if(d.length) { 6857 auto orig = d; 6858 auto m = WebSocketFrame.read(d); 6859 // that's how it indicates that it needs more data 6860 if(d !is orig) 6861 return true; 6862 } 6863 6864 return false; 6865 } 6866 6867 private ubyte continuingType; 6868 private ubyte[] continuingData; 6869 //private size_t continuingDataLength; 6870 6871 private WebSocketFrame processOnce() { 6872 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6873 auto s = d; 6874 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6875 WebSocketFrame m; 6876 if(d.length) { 6877 auto orig = d; 6878 m = WebSocketFrame.read(d); 6879 // that's how it indicates that it needs more data 6880 if(d is orig) 6881 return WebSocketFrame.init; 6882 m.unmaskInPlace(); 6883 switch(m.opcode) { 6884 case WebSocketOpcode.continuation: 6885 if(continuingData.length + m.data.length > config.maximumMessageSize) 6886 throw new Exception("message size exceeded"); 6887 6888 continuingData ~= m.data; 6889 if(m.fin) { 6890 if(ontextmessage) 6891 ontextmessage(cast(char[]) continuingData); 6892 if(onbinarymessage) 6893 onbinarymessage(continuingData); 6894 6895 continuingData = null; 6896 } 6897 break; 6898 case WebSocketOpcode.text: 6899 if(m.fin) { 6900 if(ontextmessage) 6901 ontextmessage(m.textData); 6902 } else { 6903 continuingType = m.opcode; 6904 //continuingDataLength = 0; 6905 continuingData = null; 6906 continuingData ~= m.data; 6907 } 6908 break; 6909 case WebSocketOpcode.binary: 6910 if(m.fin) { 6911 if(onbinarymessage) 6912 onbinarymessage(m.data); 6913 } else { 6914 continuingType = m.opcode; 6915 //continuingDataLength = 0; 6916 continuingData = null; 6917 continuingData ~= m.data; 6918 } 6919 break; 6920 case WebSocketOpcode.close: 6921 readyState_ = CLOSED; 6922 if(onclose) 6923 onclose(); 6924 6925 unregisterActiveSocket(this); 6926 break; 6927 case WebSocketOpcode.ping: 6928 pong(); 6929 break; 6930 case WebSocketOpcode.pong: 6931 // just really references it is still alive, nbd. 6932 break; 6933 default: // ignore though i could and perhaps should throw too 6934 } 6935 } 6936 6937 // the recv thing can be invalidated so gotta copy it over ugh 6938 if(d.length) { 6939 m.data = m.data.dup(); 6940 } 6941 6942 import core.stdc.string; 6943 memmove(receiveBuffer.ptr, d.ptr, d.length); 6944 receiveBufferUsedLength = d.length; 6945 6946 return m; 6947 } 6948 6949 private void autoprocess() { 6950 // FIXME 6951 do { 6952 processOnce(); 6953 } while(lowLevelReceive()); 6954 } 6955 6956 6957 void delegate() onclose; /// 6958 void delegate() onerror; /// 6959 void delegate(in char[]) ontextmessage; /// 6960 void delegate(in ubyte[]) onbinarymessage; /// 6961 void delegate() onopen; /// 6962 6963 /++ 6964 6965 +/ 6966 /// Group: browser_api 6967 void onmessage(void delegate(in char[]) dg) { 6968 ontextmessage = dg; 6969 } 6970 6971 /// ditto 6972 void onmessage(void delegate(in ubyte[]) dg) { 6973 onbinarymessage = dg; 6974 } 6975 6976 /* } end copy/paste */ 6977 6978 6979 } 6980 6981 /++ 6982 Returns true if the request headers are asking for a websocket upgrade. 6983 6984 If this returns true, and you want to accept it, call [acceptWebsocket]. 6985 +/ 6986 bool websocketRequested(Cgi cgi) { 6987 return 6988 "sec-websocket-key" in cgi.requestHeaders 6989 && 6990 "connection" in cgi.requestHeaders && 6991 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6992 && 6993 "upgrade" in cgi.requestHeaders && 6994 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6995 ; 6996 } 6997 6998 /++ 6999 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 7000 +/ 7001 WebSocket acceptWebsocket(Cgi cgi) { 7002 assert(!cgi.closed); 7003 assert(!cgi.outputtedResponseData); 7004 cgi.setResponseStatus("101 Switching Protocols"); 7005 cgi.header("Upgrade: WebSocket"); 7006 cgi.header("Connection: upgrade"); 7007 7008 string key = cgi.requestHeaders["sec-websocket-key"]; 7009 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 7010 7011 import std.digest.sha; 7012 auto hash = sha1Of(key); 7013 auto accept = Base64.encode(hash); 7014 7015 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 7016 7017 cgi.websocketMode = true; 7018 cgi.write(""); 7019 7020 cgi.flush(); 7021 7022 return new WebSocket(cgi); 7023 } 7024 7025 // FIXME get websocket to work on other modes, not just embedded_httpd 7026 7027 /* copy/paste in http2.d { */ 7028 enum WebSocketOpcode : ubyte { 7029 continuation = 0, 7030 text = 1, 7031 binary = 2, 7032 // 3, 4, 5, 6, 7 RESERVED 7033 close = 8, 7034 ping = 9, 7035 pong = 10, 7036 // 11,12,13,14,15 RESERVED 7037 } 7038 7039 public struct WebSocketFrame { 7040 private bool populated; 7041 bool fin; 7042 bool rsv1; 7043 bool rsv2; 7044 bool rsv3; 7045 WebSocketOpcode opcode; // 4 bits 7046 bool masked; 7047 ubyte lengthIndicator; // don't set this when building one to send 7048 ulong realLength; // don't use when sending 7049 ubyte[4] maskingKey; // don't set this when sending 7050 ubyte[] data; 7051 7052 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7053 WebSocketFrame msg; 7054 msg.fin = true; 7055 msg.opcode = opcode; 7056 msg.data = cast(ubyte[]) data.dup; 7057 7058 return msg; 7059 } 7060 7061 private void send(scope void delegate(ubyte[]) llsend) { 7062 ubyte[64] headerScratch; 7063 int headerScratchPos = 0; 7064 7065 realLength = data.length; 7066 7067 { 7068 ubyte b1; 7069 b1 |= cast(ubyte) opcode; 7070 b1 |= rsv3 ? (1 << 4) : 0; 7071 b1 |= rsv2 ? (1 << 5) : 0; 7072 b1 |= rsv1 ? (1 << 6) : 0; 7073 b1 |= fin ? (1 << 7) : 0; 7074 7075 headerScratch[0] = b1; 7076 headerScratchPos++; 7077 } 7078 7079 { 7080 headerScratchPos++; // we'll set header[1] at the end of this 7081 auto rlc = realLength; 7082 ubyte b2; 7083 b2 |= masked ? (1 << 7) : 0; 7084 7085 assert(headerScratchPos == 2); 7086 7087 if(realLength > 65535) { 7088 // use 64 bit length 7089 b2 |= 0x7f; 7090 7091 // FIXME: double check endinaness 7092 foreach(i; 0 .. 8) { 7093 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7094 rlc >>>= 8; 7095 } 7096 7097 headerScratchPos += 8; 7098 } else if(realLength > 125) { 7099 // use 16 bit length 7100 b2 |= 0x7e; 7101 7102 // FIXME: double check endinaness 7103 foreach(i; 0 .. 2) { 7104 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7105 rlc >>>= 8; 7106 } 7107 7108 headerScratchPos += 2; 7109 } else { 7110 // use 7 bit length 7111 b2 |= realLength & 0b_0111_1111; 7112 } 7113 7114 headerScratch[1] = b2; 7115 } 7116 7117 //assert(!masked, "masking key not properly implemented"); 7118 if(masked) { 7119 // FIXME: randomize this 7120 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7121 headerScratchPos += 4; 7122 7123 // we'll just mask it in place... 7124 int keyIdx = 0; 7125 foreach(i; 0 .. data.length) { 7126 data[i] = data[i] ^ maskingKey[keyIdx]; 7127 if(keyIdx == 3) 7128 keyIdx = 0; 7129 else 7130 keyIdx++; 7131 } 7132 } 7133 7134 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7135 llsend(headerScratch[0 .. headerScratchPos]); 7136 llsend(data); 7137 } 7138 7139 static WebSocketFrame read(ref ubyte[] d) { 7140 WebSocketFrame msg; 7141 7142 auto orig = d; 7143 7144 WebSocketFrame needsMoreData() { 7145 d = orig; 7146 return WebSocketFrame.init; 7147 } 7148 7149 if(d.length < 2) 7150 return needsMoreData(); 7151 7152 ubyte b = d[0]; 7153 7154 msg.populated = true; 7155 7156 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7157 b >>= 4; 7158 msg.rsv3 = b & 0x01; 7159 b >>= 1; 7160 msg.rsv2 = b & 0x01; 7161 b >>= 1; 7162 msg.rsv1 = b & 0x01; 7163 b >>= 1; 7164 msg.fin = b & 0x01; 7165 7166 b = d[1]; 7167 msg.masked = (b & 0b1000_0000) ? true : false; 7168 msg.lengthIndicator = b & 0b0111_1111; 7169 7170 d = d[2 .. $]; 7171 7172 if(msg.lengthIndicator == 0x7e) { 7173 // 16 bit length 7174 msg.realLength = 0; 7175 7176 if(d.length < 2) return needsMoreData(); 7177 7178 foreach(i; 0 .. 2) { 7179 msg.realLength |= d[0] << ((1-i) * 8); 7180 d = d[1 .. $]; 7181 } 7182 } else if(msg.lengthIndicator == 0x7f) { 7183 // 64 bit length 7184 msg.realLength = 0; 7185 7186 if(d.length < 8) return needsMoreData(); 7187 7188 foreach(i; 0 .. 8) { 7189 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7190 d = d[1 .. $]; 7191 } 7192 } else { 7193 // 7 bit length 7194 msg.realLength = msg.lengthIndicator; 7195 } 7196 7197 if(msg.masked) { 7198 7199 if(d.length < 4) return needsMoreData(); 7200 7201 msg.maskingKey = d[0 .. 4]; 7202 d = d[4 .. $]; 7203 } 7204 7205 if(msg.realLength > d.length) { 7206 return needsMoreData(); 7207 } 7208 7209 msg.data = d[0 .. cast(size_t) msg.realLength]; 7210 d = d[cast(size_t) msg.realLength .. $]; 7211 7212 return msg; 7213 } 7214 7215 void unmaskInPlace() { 7216 if(this.masked) { 7217 int keyIdx = 0; 7218 foreach(i; 0 .. this.data.length) { 7219 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7220 if(keyIdx == 3) 7221 keyIdx = 0; 7222 else 7223 keyIdx++; 7224 } 7225 } 7226 } 7227 7228 char[] textData() { 7229 return cast(char[]) data; 7230 } 7231 } 7232 /* } */ 7233 } 7234 7235 7236 version(Windows) 7237 { 7238 version(CRuntime_DigitalMars) 7239 { 7240 extern(C) int setmode(int, int) nothrow @nogc; 7241 } 7242 else version(CRuntime_Microsoft) 7243 { 7244 extern(C) int _setmode(int, int) nothrow @nogc; 7245 alias setmode = _setmode; 7246 } 7247 else static assert(0); 7248 } 7249 7250 version(Posix) { 7251 import core.sys.posix.unistd; 7252 version(CRuntime_Musl) {} else { 7253 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7254 } 7255 } 7256 7257 7258 // FIXME: these aren't quite public yet. 7259 //private: 7260 7261 // template for laziness 7262 void startAddonServer()(string arg) { 7263 version(OSX) { 7264 assert(0, "Not implemented"); 7265 } else version(linux) { 7266 import core.sys.posix.unistd; 7267 pid_t pid; 7268 const(char)*[16] args; 7269 args[0] = "ARSD_CGI_ADDON_SERVER"; 7270 args[1] = arg.ptr; 7271 posix_spawn(&pid, "/proc/self/exe", 7272 null, 7273 null, 7274 args.ptr, 7275 null // env 7276 ); 7277 } else version(Windows) { 7278 wchar[2048] filename; 7279 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7280 if(len == 0 || len == filename.length) 7281 throw new Exception("could not get process name to start helper server"); 7282 7283 STARTUPINFOW startupInfo; 7284 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7285 PROCESS_INFORMATION processInfo; 7286 7287 import std.utf; 7288 7289 // I *MIGHT* need to run it as a new job or a service... 7290 auto ret = CreateProcessW( 7291 filename.ptr, 7292 toUTF16z(arg), 7293 null, // process attributes 7294 null, // thread attributes 7295 false, // inherit handles 7296 0, // creation flags 7297 null, // environment 7298 null, // working directory 7299 &startupInfo, 7300 &processInfo 7301 ); 7302 7303 if(!ret) 7304 throw new Exception("create process failed"); 7305 7306 // when done with those, if we set them 7307 /* 7308 CloseHandle(hStdInput); 7309 CloseHandle(hStdOutput); 7310 CloseHandle(hStdError); 7311 */ 7312 7313 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7314 } 7315 7316 // template for laziness 7317 /* 7318 The websocket server is a single-process, single-thread, event 7319 I/O thing. It is passed websockets from other CGI processes 7320 and is then responsible for handling their messages and responses. 7321 Note that the CGI process is responsible for websocket setup, 7322 including authentication, etc. 7323 7324 It also gets data sent to it by other processes and is responsible 7325 for distributing that, as necessary. 7326 */ 7327 void runWebsocketServer()() { 7328 assert(0, "not implemented"); 7329 } 7330 7331 void sendToWebsocketServer(WebSocket ws, string group) { 7332 assert(0, "not implemented"); 7333 } 7334 7335 void sendToWebsocketServer(string content, string group) { 7336 assert(0, "not implemented"); 7337 } 7338 7339 7340 void runEventServer()() { 7341 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7342 } 7343 7344 void runTimerServer()() { 7345 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7346 } 7347 7348 version(Posix) { 7349 alias LocalServerConnectionHandle = int; 7350 alias CgiConnectionHandle = int; 7351 alias SocketConnectionHandle = int; 7352 7353 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7354 } else version(Windows) { 7355 alias LocalServerConnectionHandle = HANDLE; 7356 version(embedded_httpd_threads) { 7357 alias CgiConnectionHandle = SOCKET; 7358 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7359 } else version(fastcgi) { 7360 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7361 enum INVALID_CGI_CONNECTION_HANDLE = null; 7362 } else version(scgi) { 7363 alias CgiConnectionHandle = SOCKET; 7364 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7365 } else { /* version(plain_cgi) */ 7366 alias CgiConnectionHandle = HANDLE; 7367 enum INVALID_CGI_CONNECTION_HANDLE = null; 7368 } 7369 alias SocketConnectionHandle = SOCKET; 7370 } 7371 7372 version(with_addon_servers_connections) 7373 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7374 version(Posix) { 7375 import core.sys.posix.unistd; 7376 import core.sys.posix.sys.un; 7377 7378 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7379 if(sock == -1) 7380 throw new Exception("socket " ~ to!string(errno)); 7381 7382 scope(failure) 7383 close(sock); 7384 7385 cloexec(sock); 7386 7387 // add-on server processes are assumed to be local, and thus will 7388 // use unix domain sockets. Besides, I want to pass sockets to them, 7389 // so it basically must be local (except for the session server, but meh). 7390 sockaddr_un addr; 7391 addr.sun_family = AF_UNIX; 7392 version(linux) { 7393 // on linux, we will use the abstract namespace 7394 addr.sun_path[0] = 0; 7395 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7396 } else { 7397 // but otherwise, just use a file cuz we must. 7398 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7399 } 7400 7401 bool alreadyTried; 7402 7403 try_again: 7404 7405 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7406 if(!alreadyTried && errno == ECONNREFUSED) { 7407 // try auto-spawning the server, then attempt connection again 7408 startAddonServer(arg); 7409 import core.thread; 7410 Thread.sleep(50.msecs); 7411 alreadyTried = true; 7412 goto try_again; 7413 } else 7414 throw new Exception("connect " ~ to!string(errno)); 7415 } 7416 7417 return sock; 7418 } else version(Windows) { 7419 return null; // FIXME 7420 } 7421 } 7422 7423 version(with_addon_servers_connections) 7424 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7425 version(Posix) { 7426 import core.sys.posix.unistd; 7427 close(handle); 7428 } else version(Windows) 7429 CloseHandle(handle); 7430 } 7431 7432 void runSessionServer()() { 7433 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7434 } 7435 7436 import core.stdc.errno; 7437 7438 struct IoOp { 7439 @disable this(); 7440 @disable this(this); 7441 7442 /* 7443 So we want to be able to eventually handle generic sockets too. 7444 */ 7445 7446 enum Read = 1; 7447 enum Write = 2; 7448 enum Accept = 3; 7449 enum ReadSocketHandle = 4; 7450 7451 // Your handler may be called in a different thread than the one that initiated the IO request! 7452 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7453 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7454 private void delegate(IoOp*) closeHandler; 7455 private void delegate(IoOp*) completeHandler; 7456 private int internalFd; 7457 private int operation; 7458 private int bufferLengthAllocated; 7459 private int bufferLengthUsed; 7460 private ubyte[1] internalBuffer; // it can be overallocated! 7461 7462 ubyte[] allocatedBuffer() return { 7463 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7464 } 7465 7466 ubyte[] usedBuffer() return { 7467 return allocatedBuffer[0 .. bufferLengthUsed]; 7468 } 7469 7470 void reset() { 7471 bufferLengthUsed = 0; 7472 } 7473 7474 int fd() { 7475 return internalFd; 7476 } 7477 } 7478 7479 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7480 import core.stdc.stdlib; 7481 7482 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7483 if(ptr is null) 7484 assert(0); // out of memory! 7485 7486 auto op = cast(IoOp*) ptr; 7487 7488 op.handler = handler; 7489 op.internalFd = fd; 7490 op.operation = operation; 7491 op.bufferLengthAllocated = bufferSize; 7492 op.bufferLengthUsed = 0; 7493 7494 import core.memory; 7495 7496 GC.addRoot(ptr); 7497 7498 return op; 7499 } 7500 7501 void freeIoOp(ref IoOp* ptr) { 7502 7503 import core.memory; 7504 GC.removeRoot(ptr); 7505 7506 import core.stdc.stdlib; 7507 free(ptr); 7508 ptr = null; 7509 } 7510 7511 version(Posix) 7512 version(with_addon_servers_connections) 7513 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7514 7515 //import std.stdio : writeln; writeln(cast(string) data); 7516 7517 import core.sys.posix.unistd; 7518 7519 auto ret = write(connection, data.ptr, data.length); 7520 if(ret != data.length) { 7521 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7522 // the file is closed, remove it 7523 eis.fileClosed(connection); 7524 } else 7525 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7526 } 7527 } 7528 version(Windows) 7529 version(with_addon_servers_connections) 7530 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7531 // FIXME 7532 } 7533 7534 bool isInvalidHandle(CgiConnectionHandle h) { 7535 return h == INVALID_CGI_CONNECTION_HANDLE; 7536 } 7537 7538 /+ 7539 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7540 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7541 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7542 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7543 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7544 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7545 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7546 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7547 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7548 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7549 7550 +/ 7551 7552 /++ 7553 You can customize your server by subclassing the appropriate server. Then, register your 7554 subclass at compile time with the [registerEventIoServer] template, or implement your own 7555 main function and call it yourself. 7556 7557 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7558 +/ 7559 version(with_addon_servers_connections) 7560 interface EventIoServer { 7561 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7562 void handleLocalConnectionClose(IoOp* op); 7563 void handleLocalConnectionComplete(IoOp* op); 7564 void wait_timeout(); 7565 void fileClosed(int fd); 7566 7567 void epoll_fd(int fd); 7568 } 7569 7570 // the sink should buffer it 7571 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7572 static if(is(T == struct)) { 7573 foreach(member; __traits(allMembers, T)) 7574 serialize(sink, __traits(getMember, t, member)); 7575 } else static if(is(T : int)) { 7576 // no need to think of endianness just because this is only used 7577 // for local, same-machine stuff anyway. thanks private lol 7578 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7579 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7580 // these are common enough to optimize 7581 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7582 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7583 sink(cast(ubyte[]) t[]); 7584 } else static if(is(T : A[], A)) { 7585 // generic array is less optimal but still prolly ok 7586 int len = cast(int) t.length; 7587 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7588 foreach(item; t) 7589 serialize(sink, item); 7590 } else static assert(0, T.stringof); 7591 } 7592 7593 // all may be stack buffers, so use cautio 7594 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7595 static if(is(T == struct)) { 7596 T t; 7597 foreach(member; __traits(allMembers, T)) 7598 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7599 dg(t); 7600 } else static if(is(T : int)) { 7601 // no need to think of endianness just because this is only used 7602 // for local, same-machine stuff anyway. thanks private lol 7603 T t; 7604 auto data = get(t.sizeof); 7605 t = (cast(T[]) data)[0]; 7606 dg(t); 7607 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7608 // these are common enough to optimize 7609 int len; 7610 auto data = get(len.sizeof); 7611 len = (cast(int[]) data)[0]; 7612 7613 /* 7614 typeof(T[0])[2000] stackBuffer; 7615 T buffer; 7616 7617 if(len < stackBuffer.length) 7618 buffer = stackBuffer[0 .. len]; 7619 else 7620 buffer = new T(len); 7621 7622 data = get(len * typeof(T[0]).sizeof); 7623 */ 7624 7625 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7626 7627 dg(t); 7628 } else static if(is(T == E[], E)) { 7629 T t; 7630 int len; 7631 auto data = get(len.sizeof); 7632 len = (cast(int[]) data)[0]; 7633 t.length = len; 7634 foreach(ref e; t) { 7635 deserialize!E(get, (ele) { e = ele; }); 7636 } 7637 dg(t); 7638 } else static assert(0, T.stringof); 7639 } 7640 7641 unittest { 7642 serialize((ubyte[] b) { 7643 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7644 }, 1); 7645 serialize((ubyte[] b) { 7646 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7647 }, 56674); 7648 ubyte[1000] buffer; 7649 int bufferPoint; 7650 void add(scope ubyte[] b) { 7651 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7652 bufferPoint += b.length; 7653 } 7654 ubyte[] get(int sz) { 7655 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7656 bufferPoint += sz; 7657 return b; 7658 } 7659 serialize(&add, "test here"); 7660 bufferPoint = 0; 7661 deserialize!string(&get, (t) { assert(t == "test here"); }); 7662 bufferPoint = 0; 7663 7664 struct Foo { 7665 int a; 7666 ubyte c; 7667 string d; 7668 } 7669 serialize(&add, Foo(403, 37, "amazing")); 7670 bufferPoint = 0; 7671 deserialize!Foo(&get, (t) { 7672 assert(t.a == 403); 7673 assert(t.c == 37); 7674 assert(t.d == "amazing"); 7675 }); 7676 bufferPoint = 0; 7677 } 7678 7679 /* 7680 Here's the way the RPC interface works: 7681 7682 You define the interface that lists the functions you can call on the remote process. 7683 The interface may also have static methods for convenience. These forward to a singleton 7684 instance of an auto-generated class, which actually sends the args over the pipe. 7685 7686 An impl class actually implements it. A receiving server deserializes down the pipe and 7687 calls methods on the class. 7688 7689 I went with the interface to get some nice compiler checking and documentation stuff. 7690 7691 I could have skipped the interface and just implemented it all from the server class definition 7692 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7693 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7694 7695 7696 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7697 7698 Realistically though the bodies would just be 7699 connection.call(this.mangleof, args...) sooooo. 7700 7701 FIXME: overloads aren't supported 7702 */ 7703 7704 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7705 interface SessionObject {} 7706 7707 private immutable void delegate(string[])[string] scheduledJobHandlers; 7708 private immutable void delegate(string[])[string] websocketServers; 7709 7710 version(with_breaking_cgi_features) 7711 mixin(q{ 7712 7713 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7714 static import std.traits; 7715 7716 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7717 static foreach(idx, member; __traits(derivedMembers, T)) { 7718 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7719 mixin( q{ 7720 std.traits.ReturnType!(__traits(getMember, T, member)) 7721 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7722 { 7723 SerializationBuffer buffer; 7724 auto i = cast(ushort) idx; 7725 serialize(&buffer.sink, i); 7726 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7727 foreach(param; params) 7728 serialize(&buffer.sink, param); 7729 7730 auto sendable = buffer.sendable; 7731 7732 version(Posix) {{ 7733 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7734 7735 if(ret == -1) { 7736 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7737 } else if(ret == 0) { 7738 throw new Exception("Connection to addon server lost"); 7739 } if(ret < sendable.length) 7740 throw new Exception("Send failed to send all"); 7741 assert(ret == sendable.length); 7742 }} // FIXME Windows impl 7743 7744 static if(!is(typeof(return) == void)) { 7745 // there is a return value; we need to wait for it too 7746 version(Posix) { 7747 ubyte[3000] revBuffer; 7748 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7749 auto got = revBuffer[0 .. ret]; 7750 7751 int dataLocation; 7752 ubyte[] grab(int sz) { 7753 auto dataLocation1 = dataLocation; 7754 dataLocation += sz; 7755 return got[dataLocation1 .. dataLocation]; 7756 } 7757 7758 typeof(return) retu; 7759 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7760 return retu; 7761 } else { 7762 // FIXME Windows impl 7763 return typeof(return).init; 7764 } 7765 7766 } 7767 }}); 7768 } 7769 7770 private static typeof(this) singletonInstance; 7771 private LocalServerConnectionHandle connectionHandle; 7772 7773 static typeof(this) connection() { 7774 if(singletonInstance is null) { 7775 singletonInstance = new typeof(this)(); 7776 singletonInstance.connect(); 7777 } 7778 return singletonInstance; 7779 } 7780 7781 void connect() { 7782 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7783 } 7784 7785 void disconnect() { 7786 closeLocalServerConnection(connectionHandle); 7787 } 7788 } 7789 7790 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7791 ushort calledIdx; 7792 string calledFunction; 7793 7794 int dataLocation; 7795 ubyte[] grab(int sz) { 7796 if(sz == 0) assert(0); 7797 auto d = data[dataLocation .. dataLocation + sz]; 7798 dataLocation += sz; 7799 return d; 7800 } 7801 7802 again: 7803 7804 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7805 deserialize!string(&grab, (a) { calledFunction = a; }); 7806 7807 import std.traits; 7808 7809 sw: switch(calledIdx) { 7810 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7811 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 7812 case idx: 7813 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7814 7815 Parameters!(__traits(getMember, Interface, memberName)) params; 7816 foreach(ref param; params) 7817 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7818 7819 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7820 __traits(getMember, this_, memberName)(params); 7821 } else { 7822 auto ret = __traits(getMember, this_, memberName)(params); 7823 SerializationBuffer buffer; 7824 serialize(&buffer.sink, ret); 7825 7826 auto sendable = buffer.sendable; 7827 7828 version(Posix) { 7829 auto r = send(fd, sendable.ptr, sendable.length, 0); 7830 if(r == -1) { 7831 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7832 } else if(r == 0) { 7833 throw new Exception("Connection to addon client lost"); 7834 } if(r < sendable.length) 7835 throw new Exception("Send failed to send all"); 7836 7837 } // FIXME Windows impl 7838 } 7839 break sw; 7840 } 7841 default: assert(0); 7842 } 7843 7844 if(dataLocation != data.length) 7845 goto again; 7846 } 7847 7848 7849 private struct SerializationBuffer { 7850 ubyte[2048] bufferBacking; 7851 int bufferLocation; 7852 void sink(scope ubyte[] data) { 7853 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7854 bufferLocation += data.length; 7855 } 7856 7857 ubyte[] sendable() return { 7858 return bufferBacking[0 .. bufferLocation]; 7859 } 7860 } 7861 7862 /* 7863 FIXME: 7864 add a version command line arg 7865 version data in the library 7866 management gui as external program 7867 7868 at server with event_fd for each run 7869 use .mangleof in the at function name 7870 7871 i think the at server will have to: 7872 pipe args to the child 7873 collect child output for logging 7874 get child return value for logging 7875 7876 on windows timers work differently. idk how to best combine with the io stuff. 7877 7878 will have to have dump and restore too, so i can restart without losing stuff. 7879 */ 7880 7881 /++ 7882 A convenience object for talking to the [BasicDataServer] from a higher level. 7883 See: [Cgi.getSessionObject]. 7884 7885 You pass it a `Data` struct describing the data you want saved in the session. 7886 Then, this class will generate getter and setter properties that allow access 7887 to that data. 7888 7889 Note that each load and store will be done as-accessed; it doesn't front-load 7890 mutable data nor does it batch updates out of fear of read-modify-write race 7891 conditions. (In fact, right now it does this for everything, but in the future, 7892 I might batch load `immutable` members of the Data struct.) 7893 7894 At some point in the future, I might also let it do different backends, like 7895 a client-side cookie store too, but idk. 7896 7897 Note that the plain-old-data members of your `Data` struct are wrapped by this 7898 interface via a static foreach to make property functions. 7899 7900 See_Also: [MockSession] 7901 +/ 7902 interface Session(Data) : SessionObject { 7903 @property string sessionId() const; 7904 7905 /++ 7906 Starts a new session. Note that a session is also 7907 implicitly started as soon as you write data to it, 7908 so if you need to alter these parameters from their 7909 defaults, be sure to explicitly call this BEFORE doing 7910 any writes to session data. 7911 7912 Params: 7913 idleLifetime = How long, in seconds, the session 7914 should remain in memory when not being read from 7915 or written to. The default is one day. 7916 7917 NOT IMPLEMENTED 7918 7919 useExtendedLifetimeCookie = The session ID is always 7920 stored in a HTTP cookie, and by default, that cookie 7921 is discarded when the user closes their browser. 7922 7923 But if you set this to true, it will use a non-perishable 7924 cookie for the given idleLifetime. 7925 7926 NOT IMPLEMENTED 7927 +/ 7928 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7929 7930 /++ 7931 Regenerates the session ID and updates the associated 7932 cookie. 7933 7934 This is also your chance to change immutable data 7935 (not yet implemented). 7936 +/ 7937 void regenerateId(); 7938 7939 /++ 7940 Terminates this session, deleting all saved data. 7941 +/ 7942 void terminate(); 7943 7944 /++ 7945 Plain-old-data members of your `Data` struct are wrapped here via 7946 the property getters and setters. 7947 7948 If the member is a non-string array, it returns a magical array proxy 7949 object which allows for atomic appends and replaces via overloaded operators. 7950 You can slice this to get a range representing a $(B const) view of the array. 7951 This is to protect you against read-modify-write race conditions. 7952 +/ 7953 static foreach(memberName; __traits(allMembers, Data)) 7954 static if(is(typeof(__traits(getMember, Data, memberName)))) 7955 mixin(q{ 7956 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7957 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7958 }); 7959 7960 } 7961 7962 /++ 7963 An implementation of [Session] that works on real cgi connections utilizing the 7964 [BasicDataServer]. 7965 7966 As opposed to a [MockSession] which is made for testing purposes. 7967 7968 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7969 +/ 7970 class BasicDataServerSession(Data) : Session!Data { 7971 private Cgi cgi; 7972 private string sessionId_; 7973 7974 public @property string sessionId() const { 7975 return sessionId_; 7976 } 7977 7978 protected @property string sessionId(string s) { 7979 return this.sessionId_ = s; 7980 } 7981 7982 private this(Cgi cgi) { 7983 this.cgi = cgi; 7984 if(auto ptr = "sessionId" in cgi.cookies) 7985 sessionId = (*ptr).length ? *ptr : null; 7986 } 7987 7988 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7989 assert(sessionId is null); 7990 7991 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7992 7993 import std.random, std.conv; 7994 sessionId = to!string(uniform(1, long.max)); 7995 7996 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7997 setCookie(); 7998 } 7999 8000 protected void setCookie() { 8001 cgi.setCookie( 8002 "sessionId", sessionId, 8003 0 /* expiration */, 8004 "/" /* path */, 8005 null /* domain */, 8006 true /* http only */, 8007 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 8008 } 8009 8010 void regenerateId() { 8011 if(sessionId is null) { 8012 start(); 8013 return; 8014 } 8015 import std.random, std.conv; 8016 auto oldSessionId = sessionId; 8017 sessionId = to!string(uniform(1, long.max)); 8018 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 8019 setCookie(); 8020 } 8021 8022 void terminate() { 8023 BasicDataServer.connection.destroySession(sessionId); 8024 sessionId = null; 8025 setCookie(); 8026 } 8027 8028 static foreach(memberName; __traits(allMembers, Data)) 8029 static if(is(typeof(__traits(getMember, Data, memberName)))) 8030 mixin(q{ 8031 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8032 if(sessionId is null) 8033 return typeof(return).init; 8034 8035 import std.traits; 8036 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8037 if(v.length == 0) 8038 return typeof(return).init; 8039 import std.conv; 8040 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8041 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8042 // basically. Assuming the session is POD this should be fine. 8043 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8044 } 8045 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8046 if(sessionId is null) 8047 start(); 8048 import std.conv; 8049 import std.traits; 8050 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8051 return value; 8052 } 8053 }); 8054 } 8055 8056 /++ 8057 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8058 Simply stores the data in its instance members. 8059 +/ 8060 class MockSession(Data) : Session!Data { 8061 pure { 8062 @property string sessionId() const { return "mock"; } 8063 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8064 void regenerateId() {} 8065 void terminate() {} 8066 8067 private Data store_; 8068 8069 static foreach(memberName; __traits(allMembers, Data)) 8070 static if(is(typeof(__traits(getMember, Data, memberName)))) 8071 mixin(q{ 8072 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8073 return __traits(getMember, store_, memberName); 8074 } 8075 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8076 return __traits(getMember, store_, memberName) = value; 8077 } 8078 }); 8079 } 8080 } 8081 8082 /++ 8083 Direct interface to the basic data add-on server. You can 8084 typically use [Cgi.getSessionObject] as a more convenient interface. 8085 +/ 8086 version(with_addon_servers_connections) 8087 interface BasicDataServer { 8088 /// 8089 void createSession(string sessionId, int lifetime); 8090 /// 8091 void renewSession(string sessionId, int lifetime); 8092 /// 8093 void destroySession(string sessionId); 8094 /// 8095 void renameSession(string oldSessionId, string newSessionId); 8096 8097 /// 8098 void setSessionData(string sessionId, string dataKey, string dataValue); 8099 /// 8100 string getSessionData(string sessionId, string dataKey); 8101 8102 /// 8103 static BasicDataServerConnection connection() { 8104 return BasicDataServerConnection.connection(); 8105 } 8106 } 8107 8108 version(with_addon_servers_connections) 8109 class BasicDataServerConnection : BasicDataServer { 8110 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8111 } 8112 8113 version(with_addon_servers) 8114 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8115 8116 void createSession(string sessionId, int lifetime) { 8117 sessions[sessionId.idup] = Session(lifetime); 8118 } 8119 void destroySession(string sessionId) { 8120 sessions.remove(sessionId); 8121 } 8122 void renewSession(string sessionId, int lifetime) { 8123 sessions[sessionId].lifetime = lifetime; 8124 } 8125 void renameSession(string oldSessionId, string newSessionId) { 8126 sessions[newSessionId.idup] = sessions[oldSessionId]; 8127 sessions.remove(oldSessionId); 8128 } 8129 void setSessionData(string sessionId, string dataKey, string dataValue) { 8130 if(sessionId !in sessions) 8131 createSession(sessionId, 3600); // FIXME? 8132 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8133 } 8134 string getSessionData(string sessionId, string dataKey) { 8135 if(auto session = sessionId in sessions) { 8136 if(auto data = dataKey in (*session).values) 8137 return *data; 8138 else 8139 return null; // no such data 8140 8141 } else { 8142 return null; // no session 8143 } 8144 } 8145 8146 8147 protected: 8148 8149 struct Session { 8150 int lifetime; 8151 8152 string[string] values; 8153 } 8154 8155 Session[string] sessions; 8156 8157 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8158 auto data = op.usedBuffer; 8159 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8160 return false; 8161 } 8162 8163 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8164 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8165 void wait_timeout() {} 8166 void fileClosed(int fd) {} // stateless so irrelevant 8167 void epoll_fd(int fd) {} 8168 } 8169 8170 /++ 8171 See [schedule] to make one of these. You then call one of the methods here to set it up: 8172 8173 --- 8174 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8175 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8176 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8177 --- 8178 +/ 8179 version(with_addon_servers_connections) 8180 struct ScheduledJobHelper { 8181 private string func; 8182 private string[] args; 8183 private bool consumed; 8184 8185 private this(string func, string[] args) { 8186 this.func = func; 8187 this.args = args; 8188 } 8189 8190 ~this() { 8191 assert(consumed); 8192 } 8193 8194 /++ 8195 Schedules the job to be run at the given time. 8196 +/ 8197 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8198 consumed = true; 8199 8200 auto conn = ScheduledJobServerConnection.connection; 8201 import std.file; 8202 auto st = SysTime(when, timezone); 8203 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8204 } 8205 8206 /++ 8207 Schedules the job to run at least after the specified delay. 8208 +/ 8209 void delay(Duration delay) { 8210 consumed = true; 8211 8212 auto conn = ScheduledJobServerConnection.connection; 8213 import std.file; 8214 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8215 } 8216 8217 /++ 8218 Runs the job in the background ASAP. 8219 8220 $(NOTE It may run in a background thread. Don't segfault!) 8221 +/ 8222 void asap() { 8223 consumed = true; 8224 8225 auto conn = ScheduledJobServerConnection.connection; 8226 import std.file; 8227 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8228 } 8229 8230 /+ 8231 /++ 8232 Schedules the job to recur on the given pattern. 8233 +/ 8234 void recur(string spec) { 8235 8236 } 8237 +/ 8238 } 8239 8240 /++ 8241 First step to schedule a job on the scheduled job server. 8242 8243 The scheduled job needs to be a top-level function that doesn't read any 8244 variables from outside its arguments because it may be run in a new process, 8245 without any context existing later. 8246 8247 You MUST set details on the returned object to actually do anything! 8248 +/ 8249 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8250 /// 8251 ScheduledJobHelper schedule(T args) { 8252 // this isn't meant to ever be called, but instead just to 8253 // get the compiler to type check the arguments passed for us 8254 auto sample = delegate() { 8255 fn(args); 8256 }; 8257 string[] sargs; 8258 foreach(arg; args) 8259 sargs ~= to!string(arg); 8260 return ScheduledJobHelper(fn.mangleof, sargs); 8261 } 8262 8263 shared static this() { 8264 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8265 import std.traits; 8266 Parameters!fn args; 8267 foreach(idx, ref arg; args) 8268 arg = to!(typeof(arg))(sargs[idx]); 8269 fn(args); 8270 }; 8271 } 8272 } 8273 8274 /// 8275 interface ScheduledJobServer { 8276 /// Use the [schedule] function for a higher-level interface. 8277 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8278 /// 8279 void cancelJob(int jobId); 8280 } 8281 8282 version(with_addon_servers_connections) 8283 class ScheduledJobServerConnection : ScheduledJobServer { 8284 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8285 } 8286 8287 version(with_addon_servers) 8288 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8289 // FIXME: we need to handle SIGCHLD in this somehow 8290 // whenIs is 0 for relative, 1 for absolute 8291 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8292 auto nj = nextJobId; 8293 nextJobId++; 8294 8295 version(linux) { 8296 import core.sys.linux.timerfd; 8297 import core.sys.linux.epoll; 8298 import core.sys.posix.unistd; 8299 8300 8301 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8302 if(fd == -1) 8303 throw new Exception("fd timer create failed"); 8304 8305 foreach(ref arg; args) 8306 arg = arg.idup; 8307 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8308 8309 itimerspec value; 8310 value.it_value.tv_sec = when; 8311 value.it_value.tv_nsec = 0; 8312 8313 value.it_interval.tv_sec = 0; 8314 value.it_interval.tv_nsec = 0; 8315 8316 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8317 throw new Exception("couldn't set fd timer"); 8318 8319 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8320 jobs.remove(nj); 8321 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8322 close(fd); 8323 8324 8325 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8326 8327 return true; 8328 }); 8329 scope(failure) 8330 freeIoOp(op); 8331 8332 epoll_event ev; 8333 ev.events = EPOLLIN | EPOLLET; 8334 ev.data.ptr = op; 8335 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8336 throw new Exception("epoll_ctl " ~ to!string(errno)); 8337 8338 jobs[nj] = job; 8339 return nj; 8340 } else assert(0); 8341 } 8342 8343 protected void cancelJob(int jobId) { 8344 version(linux) { 8345 auto job = jobId in jobs; 8346 if(job is null) 8347 return; 8348 8349 jobs.remove(jobId); 8350 8351 version(linux) { 8352 import core.sys.linux.timerfd; 8353 import core.sys.linux.epoll; 8354 import core.sys.posix.unistd; 8355 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8356 close(job.timerfd); 8357 } 8358 } 8359 jobs.remove(jobId); 8360 } 8361 8362 int nextJobId = 1; 8363 static struct Job { 8364 string executable; 8365 string func; 8366 string[] args; 8367 int timerfd; 8368 int id; 8369 } 8370 Job[int] jobs; 8371 8372 8373 // event io server methods below 8374 8375 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8376 auto data = op.usedBuffer; 8377 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8378 return false; 8379 } 8380 8381 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8382 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8383 void wait_timeout() {} 8384 void fileClosed(int fd) {} // stateless so irrelevant 8385 8386 int epoll_fd_; 8387 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8388 int epoll_fd() { return epoll_fd_; } 8389 } 8390 8391 /// 8392 version(with_addon_servers_connections) 8393 interface EventSourceServer { 8394 /++ 8395 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8396 8397 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8398 8399 See_Also: 8400 [sendEvent] 8401 +/ 8402 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8403 /* 8404 If lastEventId is missing or empty, you just get new events as they come. 8405 8406 If it is set from something else, it sends all since then (that are still alive) 8407 down the pipe immediately. 8408 8409 The reason it can come from the header is that's what the standard defines for 8410 browser reconnects. The reason it can come from a query string is just convenience 8411 in catching up in a user-defined manner. 8412 8413 The reason the header overrides the query string is if the browser tries to reconnect, 8414 it will send the header AND the query (it reconnects to the same url), so we just 8415 want to do the restart thing. 8416 8417 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8418 */ 8419 string lastEventId = cgi.lastEventId; 8420 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8421 lastEventId = cgi.get["lastEventId"]; 8422 8423 cgi.setResponseContentType("text/event-stream"); 8424 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8425 cgi.flush(); 8426 8427 cgi.closed = true; 8428 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8429 scope(exit) 8430 closeLocalServerConnection(s); 8431 8432 version(fastcgi) 8433 throw new Exception("sending fcgi connections not supported"); 8434 else { 8435 auto fd = cgi.getOutputFileHandle(); 8436 if(isInvalidHandle(fd)) 8437 throw new Exception("bad fd from cgi!"); 8438 8439 EventSourceServerImplementation.SendableEventConnection sec; 8440 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8441 8442 version(Posix) { 8443 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8444 assert(res == sec.sizeof); 8445 } else version(Windows) { 8446 // FIXME 8447 } 8448 } 8449 } 8450 8451 /++ 8452 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8453 8454 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8455 8456 Params: 8457 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8458 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8459 data = the event data. Available in JS as `event.data`. 8460 lifetime = the amount of time to keep this event for replaying on the event server. 8461 8462 See_Also: 8463 [sendEventToEventServer] 8464 +/ 8465 public static void sendEvent(string url, string event, string data, int lifetime) { 8466 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8467 scope(exit) 8468 closeLocalServerConnection(s); 8469 8470 EventSourceServerImplementation.SendableEvent sev; 8471 sev.populate(url, event, data, lifetime); 8472 8473 version(Posix) { 8474 auto ret = send(s, &sev, sev.sizeof, 0); 8475 assert(ret == sev.sizeof); 8476 } else version(Windows) { 8477 // FIXME 8478 } 8479 } 8480 8481 /++ 8482 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8483 8484 See_Also: [disconnect] 8485 +/ 8486 void connect(string url, string forwardUrl); 8487 8488 /++ 8489 Disconnects `forwardUrl` from `url` 8490 8491 See_Also: [connect] 8492 +/ 8493 void disconnect(string url, string forwardUrl); 8494 } 8495 8496 /// 8497 version(with_addon_servers) 8498 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8499 8500 protected: 8501 8502 void connect(string url, string forwardUrl) { 8503 pipes[url] ~= forwardUrl; 8504 } 8505 void disconnect(string url, string forwardUrl) { 8506 auto t = url in pipes; 8507 if(t is null) 8508 return; 8509 foreach(idx, n; (*t)) 8510 if(n == forwardUrl) { 8511 (*t)[idx] = (*t)[$-1]; 8512 (*t) = (*t)[0 .. $-1]; 8513 break; 8514 } 8515 } 8516 8517 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8518 if(receivedFd != -1) { 8519 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8520 8521 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8522 8523 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8524 8525 auto url = got.url.idup; 8526 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8527 8528 // FIXME: catch up on past messages here 8529 } else { 8530 auto data = op.usedBuffer; 8531 auto event = cast(SendableEvent*) data.ptr; 8532 8533 if(event.magic == 0xdeadbeef) { 8534 handleInputEvent(event); 8535 8536 if(event.url in pipes) 8537 foreach(pipe; pipes[event.url]) { 8538 event.url = pipe; 8539 handleInputEvent(event); 8540 } 8541 } else { 8542 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8543 } 8544 } 8545 return false; 8546 } 8547 void handleLocalConnectionClose(IoOp* op) { 8548 fileClosed(op.fd); 8549 } 8550 void handleLocalConnectionComplete(IoOp* op) {} 8551 8552 void wait_timeout() { 8553 // just keeping alive 8554 foreach(url, connections; eventConnectionsByUrl) 8555 foreach(connection; connections) 8556 if(connection.needsChunking) 8557 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8558 else 8559 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8560 } 8561 8562 void fileClosed(int fd) { 8563 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8564 foreach(idx, conn; connections) { 8565 if(fd == conn.fd) { 8566 connections[idx] = connections[$-1]; 8567 connections = connections[0 .. $ - 1]; 8568 continue outer; 8569 } 8570 } 8571 } 8572 } 8573 8574 void epoll_fd(int fd) {} 8575 8576 8577 private: 8578 8579 8580 struct SendableEventConnection { 8581 ubyte responseChunked; 8582 8583 int urlLength; 8584 char[256] urlBuffer = 0; 8585 8586 int lastEventIdLength; 8587 char[32] lastEventIdBuffer = 0; 8588 8589 char[] url() return { 8590 return urlBuffer[0 .. urlLength]; 8591 } 8592 void url(in char[] u) { 8593 urlBuffer[0 .. u.length] = u[]; 8594 urlLength = cast(int) u.length; 8595 } 8596 char[] lastEventId() return { 8597 return lastEventIdBuffer[0 .. lastEventIdLength]; 8598 } 8599 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8600 in { 8601 assert(url.length < this.urlBuffer.length); 8602 assert(lastEventId.length < this.lastEventIdBuffer.length); 8603 } 8604 do { 8605 this.responseChunked = responseChunked ? 1 : 0; 8606 this.urlLength = cast(int) url.length; 8607 this.lastEventIdLength = cast(int) lastEventId.length; 8608 8609 this.urlBuffer[0 .. url.length] = url[]; 8610 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8611 } 8612 } 8613 8614 struct SendableEvent { 8615 int magic = 0xdeadbeef; 8616 int urlLength; 8617 char[256] urlBuffer = 0; 8618 int typeLength; 8619 char[32] typeBuffer = 0; 8620 int messageLength; 8621 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8622 int _lifetime; 8623 8624 char[] message() return { 8625 return messageBuffer[0 .. messageLength]; 8626 } 8627 char[] type() return { 8628 return typeBuffer[0 .. typeLength]; 8629 } 8630 char[] url() return { 8631 return urlBuffer[0 .. urlLength]; 8632 } 8633 void url(in char[] u) { 8634 urlBuffer[0 .. u.length] = u[]; 8635 urlLength = cast(int) u.length; 8636 } 8637 int lifetime() { 8638 return _lifetime; 8639 } 8640 8641 /// 8642 void populate(string url, string type, string message, int lifetime) 8643 in { 8644 assert(url.length < this.urlBuffer.length); 8645 assert(type.length < this.typeBuffer.length); 8646 assert(message.length < this.messageBuffer.length); 8647 } 8648 do { 8649 this.urlLength = cast(int) url.length; 8650 this.typeLength = cast(int) type.length; 8651 this.messageLength = cast(int) message.length; 8652 this._lifetime = lifetime; 8653 8654 this.urlBuffer[0 .. url.length] = url[]; 8655 this.typeBuffer[0 .. type.length] = type[]; 8656 this.messageBuffer[0 .. message.length] = message[]; 8657 } 8658 } 8659 8660 struct EventConnection { 8661 int fd; 8662 bool needsChunking; 8663 } 8664 8665 private EventConnection[][string] eventConnectionsByUrl; 8666 private string[][string] pipes; 8667 8668 private void handleInputEvent(scope SendableEvent* event) { 8669 static int eventId; 8670 8671 static struct StoredEvent { 8672 int id; 8673 string type; 8674 string message; 8675 int lifetimeRemaining; 8676 } 8677 8678 StoredEvent[][string] byUrl; 8679 8680 int thisId = ++eventId; 8681 8682 if(event.lifetime) 8683 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8684 8685 auto connectionsPtr = event.url in eventConnectionsByUrl; 8686 EventConnection[] connections; 8687 if(connectionsPtr is null) 8688 return; 8689 else 8690 connections = *connectionsPtr; 8691 8692 char[4096] buffer; 8693 char[] formattedMessage; 8694 8695 void append(const char[] a) { 8696 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8697 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8698 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8699 } 8700 8701 import std.algorithm.iteration; 8702 8703 if(connections.length) { 8704 append("id: "); 8705 append(to!string(thisId)); 8706 append("\n"); 8707 8708 append("event: "); 8709 append(event.type); 8710 append("\n"); 8711 8712 foreach(line; event.message.splitter("\n")) { 8713 append("data: "); 8714 append(line); 8715 append("\n"); 8716 } 8717 8718 append("\n"); 8719 } 8720 8721 // chunk it for HTTP! 8722 auto len = toHex(formattedMessage.length); 8723 buffer[4 .. 6] = "\r\n"[]; 8724 buffer[4 - len.length .. 4] = len[]; 8725 buffer[6 + formattedMessage.length] = '\r'; 8726 buffer[6 + formattedMessage.length + 1] = '\n'; 8727 8728 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8729 // done 8730 8731 // FIXME: send back requests when needed 8732 // FIXME: send a single ":\n" every 15 seconds to keep alive 8733 8734 foreach(connection; connections) { 8735 if(connection.needsChunking) { 8736 nonBlockingWrite(this, connection.fd, chunkedMessage); 8737 } else { 8738 nonBlockingWrite(this, connection.fd, formattedMessage); 8739 } 8740 } 8741 } 8742 } 8743 8744 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8745 version(Posix) { 8746 8747 import core.sys.posix.unistd; 8748 import core.sys.posix.fcntl; 8749 import core.sys.posix.sys.un; 8750 8751 import core.sys.posix.signal; 8752 signal(SIGPIPE, SIG_IGN); 8753 8754 static extern(C) void sigchldhandler(int) { 8755 int status; 8756 import w = core.sys.posix.sys.wait; 8757 w.wait(&status); 8758 } 8759 signal(SIGCHLD, &sigchldhandler); 8760 8761 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8762 if(sock == -1) 8763 throw new Exception("socket " ~ to!string(errno)); 8764 8765 scope(failure) 8766 close(sock); 8767 8768 cloexec(sock); 8769 8770 // add-on server processes are assumed to be local, and thus will 8771 // use unix domain sockets. Besides, I want to pass sockets to them, 8772 // so it basically must be local (except for the session server, but meh). 8773 sockaddr_un addr; 8774 addr.sun_family = AF_UNIX; 8775 version(linux) { 8776 // on linux, we will use the abstract namespace 8777 addr.sun_path[0] = 0; 8778 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8779 } else { 8780 // but otherwise, just use a file cuz we must. 8781 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8782 } 8783 8784 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8785 throw new Exception("bind " ~ to!string(errno)); 8786 8787 if(listen(sock, 128) == -1) 8788 throw new Exception("listen " ~ to!string(errno)); 8789 8790 makeNonBlocking(sock); 8791 8792 version(linux) { 8793 import core.sys.linux.epoll; 8794 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8795 if(epoll_fd == -1) 8796 throw new Exception("epoll_create1 " ~ to!string(errno)); 8797 scope(failure) 8798 close(epoll_fd); 8799 } else { 8800 import core.sys.posix.poll; 8801 } 8802 8803 version(linux) 8804 eis.epoll_fd = epoll_fd; 8805 8806 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8807 scope(exit) 8808 freeIoOp(acceptOp); 8809 8810 version(linux) { 8811 epoll_event ev; 8812 ev.events = EPOLLIN | EPOLLET; 8813 ev.data.ptr = acceptOp; 8814 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8815 throw new Exception("epoll_ctl " ~ to!string(errno)); 8816 8817 epoll_event[64] events; 8818 } else { 8819 pollfd[] pollfds; 8820 IoOp*[int] ioops; 8821 pollfds ~= pollfd(sock, POLLIN); 8822 ioops[sock] = acceptOp; 8823 } 8824 8825 import core.time : MonoTime, seconds; 8826 8827 MonoTime timeout = MonoTime.currTime + 15.seconds; 8828 8829 while(true) { 8830 8831 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8832 8833 int timeout_milliseconds = 0; // -1; // infinite 8834 8835 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8836 if(timeout_milliseconds < 0) 8837 timeout_milliseconds = 0; 8838 8839 //writeln("waiting for ", name); 8840 8841 version(linux) { 8842 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8843 if(nfds == -1) { 8844 if(errno == EINTR) 8845 continue; 8846 throw new Exception("epoll_wait " ~ to!string(errno)); 8847 } 8848 } else { 8849 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8850 size_t lastIdx = 0; 8851 } 8852 8853 if(nfds == 0) { 8854 eis.wait_timeout(); 8855 timeout += 15.seconds; 8856 } 8857 8858 foreach(idx; 0 .. nfds) { 8859 version(linux) { 8860 auto flags = events[idx].events; 8861 auto ioop = cast(IoOp*) events[idx].data.ptr; 8862 } else { 8863 IoOp* ioop; 8864 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8865 if(thing.revents) { 8866 ioop = ioops[thing.fd]; 8867 lastIdx += tidx + 1; 8868 break; 8869 } 8870 } 8871 } 8872 8873 //writeln(flags, " ", ioop.fd); 8874 8875 void newConnection() { 8876 // on edge triggering, it is important that we get it all 8877 while(true) { 8878 auto size = cast(socklen_t) addr.sizeof; 8879 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8880 if(ns == -1) { 8881 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8882 // all done, got it all 8883 break; 8884 } 8885 throw new Exception("accept " ~ to!string(errno)); 8886 } 8887 cloexec(ns); 8888 8889 makeNonBlocking(ns); 8890 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8891 niop.closeHandler = &eis.handleLocalConnectionClose; 8892 niop.completeHandler = &eis.handleLocalConnectionComplete; 8893 scope(failure) freeIoOp(niop); 8894 8895 version(linux) { 8896 epoll_event nev; 8897 nev.events = EPOLLIN | EPOLLET; 8898 nev.data.ptr = niop; 8899 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8900 throw new Exception("epoll_ctl " ~ to!string(errno)); 8901 } else { 8902 bool found = false; 8903 foreach(ref pfd; pollfds) { 8904 if(pfd.fd < 0) { 8905 pfd.fd = ns; 8906 found = true; 8907 } 8908 } 8909 if(!found) 8910 pollfds ~= pollfd(ns, POLLIN); 8911 ioops[ns] = niop; 8912 } 8913 } 8914 } 8915 8916 bool newConnectionCondition() { 8917 version(linux) 8918 return ioop.fd == sock && (flags & EPOLLIN); 8919 else 8920 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8921 } 8922 8923 if(newConnectionCondition()) { 8924 newConnection(); 8925 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8926 while(true) { 8927 int in_fd; 8928 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8929 if(got == -1) { 8930 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8931 // all done, got it all 8932 if(ioop.completeHandler) 8933 ioop.completeHandler(ioop); 8934 break; 8935 } 8936 throw new Exception("recv " ~ to!string(errno)); 8937 } 8938 8939 if(got == 0) { 8940 if(ioop.closeHandler) { 8941 ioop.closeHandler(ioop); 8942 version(linux) {} // nothing needed 8943 else { 8944 foreach(ref pfd; pollfds) { 8945 if(pfd.fd == ioop.fd) 8946 pfd.fd = -1; 8947 } 8948 } 8949 } 8950 close(ioop.fd); 8951 freeIoOp(ioop); 8952 break; 8953 } 8954 8955 ioop.bufferLengthUsed = cast(int) got; 8956 ioop.handler(ioop, in_fd); 8957 } 8958 } else if(ioop.operation == IoOp.Read) { 8959 while(true) { 8960 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8961 if(got == -1) { 8962 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8963 // all done, got it all 8964 if(ioop.completeHandler) 8965 ioop.completeHandler(ioop); 8966 break; 8967 } 8968 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8969 } 8970 8971 if(got == 0) { 8972 if(ioop.closeHandler) 8973 ioop.closeHandler(ioop); 8974 close(ioop.fd); 8975 freeIoOp(ioop); 8976 break; 8977 } 8978 8979 ioop.bufferLengthUsed = cast(int) got; 8980 if(ioop.handler(ioop, ioop.fd)) { 8981 close(ioop.fd); 8982 freeIoOp(ioop); 8983 break; 8984 } 8985 } 8986 } 8987 8988 // EPOLLHUP? 8989 } 8990 } 8991 } else version(Windows) { 8992 8993 // set up a named pipe 8994 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8995 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8996 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8997 8998 } else static assert(0); 8999 } 9000 9001 9002 version(with_sendfd) 9003 // copied from the web and ported from C 9004 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 9005 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 9006 msghdr msg; 9007 iovec[1] iov; 9008 9009 version(OSX) { 9010 //msg.msg_accrights = cast(cattr_t) &sendfd; 9011 //msg.msg_accrightslen = int.sizeof; 9012 } else version(Android) { 9013 } else { 9014 union ControlUnion { 9015 cmsghdr cm; 9016 char[CMSG_SPACE(int.sizeof)] control; 9017 } 9018 9019 ControlUnion control_un; 9020 cmsghdr* cmptr; 9021 9022 msg.msg_control = control_un.control.ptr; 9023 msg.msg_controllen = control_un.control.length; 9024 9025 cmptr = CMSG_FIRSTHDR(&msg); 9026 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 9027 cmptr.cmsg_level = SOL_SOCKET; 9028 cmptr.cmsg_type = SCM_RIGHTS; 9029 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 9030 } 9031 9032 msg.msg_name = null; 9033 msg.msg_namelen = 0; 9034 9035 iov[0].iov_base = ptr; 9036 iov[0].iov_len = nbytes; 9037 msg.msg_iov = iov.ptr; 9038 msg.msg_iovlen = 1; 9039 9040 return sendmsg(fd, &msg, 0); 9041 } 9042 9043 version(with_sendfd) 9044 // copied from the web and ported from C 9045 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9046 msghdr msg; 9047 iovec[1] iov; 9048 ssize_t n; 9049 int newfd; 9050 9051 version(OSX) { 9052 //msg.msg_accrights = cast(cattr_t) recvfd; 9053 //msg.msg_accrightslen = int.sizeof; 9054 } else version(Android) { 9055 } else { 9056 union ControlUnion { 9057 cmsghdr cm; 9058 char[CMSG_SPACE(int.sizeof)] control; 9059 } 9060 ControlUnion control_un; 9061 cmsghdr* cmptr; 9062 9063 msg.msg_control = control_un.control.ptr; 9064 msg.msg_controllen = control_un.control.length; 9065 } 9066 9067 msg.msg_name = null; 9068 msg.msg_namelen = 0; 9069 9070 iov[0].iov_base = ptr; 9071 iov[0].iov_len = nbytes; 9072 msg.msg_iov = iov.ptr; 9073 msg.msg_iovlen = 1; 9074 9075 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9076 return n; 9077 9078 version(OSX) { 9079 //if(msg.msg_accrightslen != int.sizeof) 9080 //*recvfd = -1; 9081 } else version(Android) { 9082 } else { 9083 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9084 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9085 if (cmptr.cmsg_level != SOL_SOCKET) 9086 throw new Exception("control level != SOL_SOCKET"); 9087 if (cmptr.cmsg_type != SCM_RIGHTS) 9088 throw new Exception("control type != SCM_RIGHTS"); 9089 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9090 } else 9091 *recvfd = -1; /* descriptor was not passed */ 9092 } 9093 9094 return n; 9095 } 9096 /* end read_fd */ 9097 9098 9099 /* 9100 Event source stuff 9101 9102 The api is: 9103 9104 sendEvent(string url, string type, string data, int timeout = 60*10); 9105 9106 attachEventListener(string url, int fd, lastId) 9107 9108 9109 It just sends to all attached listeners, and stores it until the timeout 9110 for replaying via lastEventId. 9111 */ 9112 9113 /* 9114 Session process stuff 9115 9116 it stores it all. the cgi object has a session object that can grab it 9117 9118 session may be done in the same process if possible, there is a version 9119 switch to choose if you want to override. 9120 */ 9121 9122 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9123 alias handler = dispatchHandler; 9124 string urlPrefix; 9125 bool rejectFurther; 9126 immutable(DispatcherDetails) details; 9127 } 9128 9129 private string urlify(string name) pure { 9130 return beautify(name, '-', true); 9131 } 9132 9133 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9134 if(name == "id") 9135 return allLowerCase ? name : "ID"; 9136 9137 char[160] buffer; 9138 int bufferIndex = 0; 9139 bool shouldCap = true; 9140 bool shouldSpace; 9141 bool lastWasCap; 9142 foreach(idx, char ch; name) { 9143 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9144 9145 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9146 if(lastWasCap) { 9147 // two caps in a row, don't change. Prolly acronym. 9148 } else { 9149 if(idx) 9150 shouldSpace = true; // new word, add space 9151 } 9152 9153 lastWasCap = true; 9154 } else { 9155 lastWasCap = false; 9156 } 9157 9158 if(shouldSpace) { 9159 buffer[bufferIndex++] = space; 9160 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9161 shouldSpace = false; 9162 } 9163 if(shouldCap) { 9164 if(ch >= 'a' && ch <= 'z') 9165 ch -= 32; 9166 shouldCap = false; 9167 } 9168 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9169 ch += 32; 9170 buffer[bufferIndex++] = ch; 9171 } 9172 return buffer[0 .. bufferIndex].idup; 9173 } 9174 9175 /* 9176 string urlFor(alias func)() { 9177 return __traits(identifier, func); 9178 } 9179 */ 9180 9181 /++ 9182 UDA: The name displayed to the user in auto-generated HTML. 9183 9184 Default is `beautify(identifier)`. 9185 +/ 9186 struct DisplayName { 9187 string name; 9188 } 9189 9190 /++ 9191 UDA: The name used in the URL or web parameter. 9192 9193 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9194 +/ 9195 struct UrlName { 9196 string name; 9197 } 9198 9199 /++ 9200 UDA: default format to respond for this method 9201 +/ 9202 struct DefaultFormat { string value; } 9203 9204 class MissingArgumentException : Exception { 9205 string functionName; 9206 string argumentName; 9207 string argumentType; 9208 9209 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9210 this.functionName = functionName; 9211 this.argumentName = argumentName; 9212 this.argumentType = argumentType; 9213 9214 super("Missing Argument: " ~ this.argumentName, file, line, next); 9215 } 9216 } 9217 9218 /++ 9219 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9220 9221 History: 9222 Added December 15, 2021 (dub v10.5) 9223 +/ 9224 class ResourceNotFoundException : Exception { 9225 string resourceType; 9226 string resourceId; 9227 9228 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9229 this.resourceType = resourceType; 9230 this.resourceId = resourceId; 9231 9232 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9233 } 9234 9235 } 9236 9237 /++ 9238 This can be attached to any constructor or function called from the cgi system. 9239 9240 If it is present, the function argument can NOT be set from web params, but instead 9241 is set to the return value of the given `func`. 9242 9243 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9244 the current request. Otherwise, it must take zero arguments. 9245 9246 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9247 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9248 the cgi object. 9249 9250 const arguments are also supported. 9251 +/ 9252 struct ifCalledFromWeb(alias func) {} 9253 9254 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9255 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9256 9257 // FIXME: any array of structs should also be settable or gettable from csv as well. 9258 9259 // FIXME: think more about checkboxes and bools. 9260 9261 import std.traits; 9262 9263 Parameters!method params; 9264 alias idents = ParameterIdentifierTuple!method; 9265 alias defaults = ParameterDefaults!method; 9266 9267 const(string)[] names; 9268 const(string)[] values; 9269 9270 // first, check for missing arguments and initialize to defaults if necessary 9271 9272 static if(is(typeof(method) P == __parameters)) 9273 foreach(idx, param; P) {{ 9274 // see: mustNotBeSetFromWebParams 9275 static if(is(param : Cgi)) { 9276 static assert(!is(param == immutable)); 9277 cast() params[idx] = cgi; 9278 } else static if(is(param == Session!D, D)) { 9279 static assert(!is(param == immutable)); 9280 cast() params[idx] = cgi.getSessionObject!D(); 9281 } else { 9282 bool populated; 9283 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9284 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9285 static if(is(typeof(func(cgi)))) 9286 params[idx] = func(cgi); 9287 else 9288 params[idx] = func(); 9289 9290 populated = true; 9291 } 9292 } 9293 9294 if(!populated) { 9295 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9296 params[idx] = param.getAutomaticallyForCgi(cgi); 9297 populated = true; 9298 } 9299 } 9300 9301 if(!populated) { 9302 auto ident = idents[idx]; 9303 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9304 if(ident !in cgi.get) { 9305 static if(is(defaults[idx] == void)) { 9306 static if(is(param == bool)) 9307 params[idx] = false; 9308 else 9309 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9310 } else 9311 params[idx] = defaults[idx]; 9312 } 9313 } else { 9314 if(ident !in cgi.post) { 9315 static if(is(defaults[idx] == void)) { 9316 static if(is(param == bool)) 9317 params[idx] = false; 9318 else 9319 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9320 } else 9321 params[idx] = defaults[idx]; 9322 } 9323 } 9324 } 9325 } 9326 }} 9327 9328 // second, parse the arguments in order to build up arrays, etc. 9329 9330 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9331 static if(is(T == struct)) { 9332 if(name == paramName) { 9333 *what = T.init; 9334 return true; 9335 } else { 9336 // could be a child. gonna allow either obj.field OR obj[field] 9337 9338 string afterName; 9339 9340 if(name[paramName.length] == '[') { 9341 int count = 1; 9342 auto idx = paramName.length + 1; 9343 while(idx < name.length && count > 0) { 9344 if(name[idx] == '[') 9345 count++; 9346 else if(name[idx] == ']') { 9347 count--; 9348 if(count == 0) break; 9349 } 9350 idx++; 9351 } 9352 9353 if(idx == name.length) 9354 return false; // malformed 9355 9356 auto insideBrackets = name[paramName.length + 1 .. idx]; 9357 afterName = name[idx + 1 .. $]; 9358 9359 name = name[0 .. paramName.length]; 9360 9361 paramName = insideBrackets; 9362 9363 } else if(name[paramName.length] == '.') { 9364 paramName = name[paramName.length + 1 .. $]; 9365 name = paramName; 9366 int p = 0; 9367 foreach(ch; paramName) { 9368 if(ch == '.' || ch == '[') 9369 break; 9370 p++; 9371 } 9372 9373 afterName = paramName[p .. $]; 9374 paramName = paramName[0 .. p]; 9375 } else { 9376 return false; 9377 } 9378 9379 if(paramName.length) 9380 // set the child member 9381 switch(paramName) { 9382 foreach(idx, memberName; __traits(allMembers, T)) 9383 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9384 // data member! 9385 case memberName: 9386 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9387 } 9388 default: 9389 // ok, not a member 9390 } 9391 } 9392 9393 return false; 9394 } else static if(is(T == enum)) { 9395 *what = to!T(value); 9396 return true; 9397 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9398 *what = to!T(value); 9399 return true; 9400 } else static if(is(T == bool)) { 9401 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9402 return true; 9403 } else static if(is(T == K[], K)) { 9404 K tmp; 9405 if(name == paramName) { 9406 // direct - set and append 9407 if(setVariable(name, paramName, &tmp, value)) { 9408 (*what) ~= tmp; 9409 return true; 9410 } else { 9411 return false; 9412 } 9413 } else { 9414 // child, append to last element 9415 // FIXME: what about range violations??? 9416 auto ptr = &(*what)[(*what).length - 1]; 9417 return setVariable(name, paramName, ptr, value); 9418 9419 } 9420 } else static if(is(T == V[K], K, V)) { 9421 // assoc array, name[key] is valid 9422 if(name == paramName) { 9423 // no action necessary 9424 return true; 9425 } else if(name[paramName.length] == '[') { 9426 int count = 1; 9427 auto idx = paramName.length + 1; 9428 while(idx < name.length && count > 0) { 9429 if(name[idx] == '[') 9430 count++; 9431 else if(name[idx] == ']') { 9432 count--; 9433 if(count == 0) break; 9434 } 9435 idx++; 9436 } 9437 if(idx == name.length) 9438 return false; // malformed 9439 9440 auto insideBrackets = name[paramName.length + 1 .. idx]; 9441 auto afterName = name[idx + 1 .. $]; 9442 9443 auto k = to!K(insideBrackets); 9444 V v; 9445 if(auto ptr = k in *what) 9446 v = *ptr; 9447 9448 name = name[0 .. paramName.length]; 9449 //writeln(name, afterName, " ", paramName); 9450 9451 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9452 if(ret) { 9453 (*what)[k] = v; 9454 return true; 9455 } 9456 } 9457 9458 return false; 9459 } else { 9460 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9461 } 9462 9463 //return false; 9464 } 9465 9466 void setArgument(string name, string value) { 9467 int p; 9468 foreach(ch; name) { 9469 if(ch == '.' || ch == '[') 9470 break; 9471 p++; 9472 } 9473 9474 auto paramName = name[0 .. p]; 9475 9476 sw: switch(paramName) { 9477 static if(is(typeof(method) P == __parameters)) 9478 foreach(idx, param; P) { 9479 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9480 // cannot be set from the outside 9481 } else { 9482 case idents[idx]: 9483 static if(is(param == Cgi.UploadedFile)) { 9484 params[idx] = cgi.files[name]; 9485 } else static if(is(param : const Cgi.UploadedFile[])) { 9486 (cast() params[idx]) = cgi.filesArray[name]; 9487 } else { 9488 setVariable(name, paramName, ¶ms[idx], value); 9489 } 9490 break sw; 9491 } 9492 } 9493 default: 9494 // ignore; not relevant argument 9495 } 9496 } 9497 9498 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9499 names = cgi.allGetNamesInOrder; 9500 values = cgi.allGetValuesInOrder; 9501 } else { 9502 names = cgi.allPostNamesInOrder; 9503 values = cgi.allPostValuesInOrder; 9504 } 9505 9506 foreach(idx, name; names) { 9507 setArgument(name, values[idx]); 9508 } 9509 9510 static if(is(ReturnType!method == void)) { 9511 typeof(null) ret; 9512 dg(params); 9513 } else { 9514 auto ret = dg(params); 9515 } 9516 9517 // FIXME: format return values 9518 // options are: json, html, csv. 9519 // also may need to wrap in envelope format: none, html, or json. 9520 return ret; 9521 } 9522 9523 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9524 static if(is(T : const(Cgi))) { 9525 return true; 9526 } else static if(is(T : const(Session!D), D)) { 9527 return true; 9528 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9529 return true; 9530 } else { 9531 foreach(uda; attrs) 9532 static if(is(uda == ifCalledFromWeb!func, alias func)) 9533 return true; 9534 return false; 9535 } 9536 } 9537 9538 private bool hasIfCalledFromWeb(attrs...)() { 9539 foreach(uda; attrs) 9540 static if(is(uda == ifCalledFromWeb!func, alias func)) 9541 return true; 9542 return false; 9543 } 9544 9545 /++ 9546 Implies POST path for the thing itself, then GET will get the automatic form. 9547 9548 The given customizer, if present, will be called as a filter on the Form object. 9549 9550 History: 9551 Added December 27, 2020 9552 +/ 9553 template AutomaticForm(alias customizer) { } 9554 9555 /++ 9556 This is meant to be returned by a function that takes a form POST submission. You 9557 want to set the url of the new resource it created, which is set as the http 9558 Location header for a "201 Created" result, and you can also set a separate 9559 destination for browser users, which it sets via a "Refresh" header. 9560 9561 The `resourceRepresentation` should generally be the thing you just created, and 9562 it will be the body of the http response when formatted through the presenter. 9563 The exact thing is up to you - it could just return an id, or the whole object, or 9564 perhaps a partial object. 9565 9566 Examples: 9567 --- 9568 class Test : WebObject { 9569 @(Cgi.RequestMethod.POST) 9570 CreatedResource!int makeThing(string value) { 9571 return CreatedResource!int(value.to!int, "/resources/id"); 9572 } 9573 } 9574 --- 9575 9576 History: 9577 Added December 18, 2021 9578 +/ 9579 struct CreatedResource(T) { 9580 static if(!is(T == void)) 9581 T resourceRepresentation; 9582 string resourceUrl; 9583 string refreshUrl; 9584 } 9585 9586 /+ 9587 /++ 9588 This can be attached as a UDA to a handler to add a http Refresh header on a 9589 successful run. (It will not be attached if the function throws an exception.) 9590 This will refresh the browser the given number of seconds after the page loads, 9591 to the url returned by `urlFunc`, which can be either a static function or a 9592 member method of the current handler object. 9593 9594 You might use this for a POST handler that is normally used from ajax, but you 9595 want it to degrade gracefully to a temporarily flashed message before reloading 9596 the main page. 9597 9598 History: 9599 Added December 18, 2021 9600 +/ 9601 struct Refresh(alias urlFunc) { 9602 int waitInSeconds; 9603 9604 string url() { 9605 static if(__traits(isStaticFunction, urlFunc)) 9606 return urlFunc(); 9607 else static if(is(urlFunc : string)) 9608 return urlFunc; 9609 } 9610 } 9611 +/ 9612 9613 /+ 9614 /++ 9615 Sets a filter to be run before 9616 9617 A before function can do validations of params and log and stop the function from running. 9618 +/ 9619 template Before(alias b) {} 9620 template After(alias b) {} 9621 +/ 9622 9623 /+ 9624 Argument conversions: for the most part, it is to!Thing(string). 9625 9626 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9627 they are passed 9628 9629 arr=foo&arr=bar <-- notice the same name. 9630 9631 Structs are first declared with an empty thing, then have their members set individually, 9632 with dot notation. The members are not required, just the initial declaration. 9633 9634 struct Foo { 9635 int a; 9636 string b; 9637 } 9638 void test(Foo foo){} 9639 9640 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9641 9642 Arrays of structs use this declaration. 9643 9644 void test(Foo[] foo) {} 9645 9646 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9647 9648 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9649 declaration is ignored. 9650 9651 Mind that order matters! The declaration MUST come first in the string. 9652 9653 Arrays of struct members follow this rule recursively. 9654 9655 struct Foo { 9656 int[] a; 9657 } 9658 9659 foo&foo.a=1&foo.a=2&foo&foo.a=1 9660 9661 9662 Associative arrays are formatted with brackets, after a declaration, like structs: 9663 9664 foo&foo[key]=value&foo[other_key]=value 9665 9666 9667 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9668 do not support the strict ordering requirements to work with these struct protocols. 9669 9670 FIXME: also perhaps accept application/json to better work with outside trash. 9671 9672 9673 Return values are also auto-formatted according to user-requested type: 9674 for json, it loops over and converts. 9675 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9676 +/ 9677 9678 /++ 9679 A web presenter is responsible for rendering things to HTML to be usable 9680 in a web browser. 9681 9682 They are passed as template arguments to the base classes of [WebObject] 9683 9684 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9685 and override it. Use forwarding and specialization to customize it. 9686 9687 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9688 class templates and your overridden templates work with each other. 9689 9690 --- 9691 class MyPresenter : WebPresenter!(MyPresenter) { 9692 @Override 9693 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9694 // present the CustomType 9695 } 9696 @Override 9697 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9698 // handle everything else via the super class, which will call 9699 // back to your class when appropriate 9700 super.presentSuccessfulReturnAsHtml(cgi, ret); 9701 } 9702 } 9703 --- 9704 9705 The meta argument in there can be overridden by your own facility. 9706 9707 +/ 9708 class WebPresenter(CRTP) { 9709 9710 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9711 /// If you override a plain method, use `override`. If a template, use `@Override`. 9712 enum Override; 9713 9714 string script() { 9715 return ` 9716 `; 9717 } 9718 9719 string style() { 9720 return ` 9721 :root { 9722 --mild-border: #ccc; 9723 --middle-border: #999; 9724 --accent-color: #f2f2f2; 9725 --sidebar-color: #fefefe; 9726 } 9727 ` ~ genericFormStyling() ~ genericSiteStyling(); 9728 } 9729 9730 string genericFormStyling() { 9731 return 9732 q"css 9733 table.automatic-data-display { 9734 border-collapse: collapse; 9735 border: solid 1px var(--mild-border); 9736 } 9737 9738 table.automatic-data-display td { 9739 vertical-align: top; 9740 border: solid 1px var(--mild-border); 9741 padding: 2px 4px; 9742 } 9743 9744 table.automatic-data-display th { 9745 border: solid 1px var(--mild-border); 9746 border-bottom: solid 1px var(--middle-border); 9747 padding: 2px 4px; 9748 } 9749 9750 ol.automatic-data-display { 9751 margin: 0px; 9752 list-style-position: inside; 9753 padding: 0px; 9754 } 9755 9756 dl.automatic-data-display { 9757 9758 } 9759 9760 .automatic-form { 9761 max-width: 600px; 9762 } 9763 9764 .form-field { 9765 margin: 0.5em; 9766 padding-left: 0.5em; 9767 } 9768 9769 .label-text { 9770 display: block; 9771 font-weight: bold; 9772 margin-left: -0.5em; 9773 } 9774 9775 .submit-button-holder { 9776 padding-left: 2em; 9777 } 9778 9779 .add-array-button { 9780 9781 } 9782 css"; 9783 } 9784 9785 string genericSiteStyling() { 9786 return 9787 q"css 9788 * { box-sizing: border-box; } 9789 html, body { margin: 0px; } 9790 body { 9791 font-family: sans-serif; 9792 } 9793 header { 9794 background: var(--accent-color); 9795 height: 64px; 9796 } 9797 footer { 9798 background: var(--accent-color); 9799 height: 64px; 9800 } 9801 #site-container { 9802 display: flex; 9803 } 9804 main { 9805 flex: 1 1 auto; 9806 order: 2; 9807 min-height: calc(100vh - 64px - 64px); 9808 padding: 4px; 9809 padding-left: 1em; 9810 } 9811 #sidebar { 9812 flex: 0 0 16em; 9813 order: 1; 9814 background: var(--sidebar-color); 9815 } 9816 css"; 9817 } 9818 9819 import arsd.dom; 9820 Element htmlContainer() { 9821 auto document = new Document(q"html 9822 <!DOCTYPE html> 9823 <html class="no-script"> 9824 <head> 9825 <script>document.documentElement.classList.remove("no-script");</script> 9826 <style>.no-script requires-script { display: none; }</style> 9827 <title>D Application</title> 9828 <meta name="viewport" content="initial-scale=1, width=device-width" /> 9829 <link rel="stylesheet" href="style.css" /> 9830 </head> 9831 <body> 9832 <header></header> 9833 <div id="site-container"> 9834 <main></main> 9835 <div id="sidebar"></div> 9836 </div> 9837 <footer></footer> 9838 <script src="script.js"></script> 9839 </body> 9840 </html> 9841 html", true, true); 9842 9843 return document.requireSelector("main"); 9844 } 9845 9846 /// Renders a response as an HTTP error with associated html body 9847 void renderBasicError(Cgi cgi, int httpErrorCode) { 9848 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9849 auto c = htmlContainer(); 9850 c.innerText = getHttpCodeText(httpErrorCode); 9851 cgi.setResponseContentType("text/html; charset=utf-8"); 9852 cgi.write(c.parentDocument.toString(), true); 9853 } 9854 9855 template methodMeta(alias method) { 9856 enum methodMeta = null; 9857 } 9858 9859 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9860 switch(format) { 9861 case "html": 9862 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9863 break; 9864 case "json": 9865 import arsd.jsvar; 9866 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9867 var json; 9868 foreach(index, type; Types) { 9869 if(ret.contains == index) 9870 json = ret.payload[index]; 9871 } 9872 } else { 9873 var json = ret; 9874 } 9875 var envelope = json; // var.emptyObject; 9876 /* 9877 envelope.success = true; 9878 envelope.result = json; 9879 envelope.error = null; 9880 */ 9881 cgi.setResponseContentType("application/json"); 9882 cgi.write(envelope.toJson(), true); 9883 break; 9884 default: 9885 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9886 } 9887 } 9888 9889 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9890 /// in the default presenter - allowing the function to have full low-level control over the 9891 /// response. 9892 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9893 // nothing intentionally! 9894 } 9895 9896 /// Redirections are forwarded to [Cgi.setResponseLocation] 9897 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9898 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9899 } 9900 9901 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9902 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9903 cgi.setResponseStatus(getHttpCodeText(201)); 9904 if(ret.resourceUrl.length) 9905 cgi.header("Location: " ~ ret.resourceUrl); 9906 if(ret.refreshUrl.length) 9907 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9908 static if(!is(R == void)) 9909 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9910 } 9911 9912 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9913 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9914 bool outputted = false; 9915 foreach(index, type; Types) { 9916 if(ret.contains == index) { 9917 assert(!outputted); 9918 outputted = true; 9919 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9920 } 9921 } 9922 if(!outputted) 9923 assert(0); 9924 } 9925 9926 /++ 9927 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9928 +/ 9929 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9930 cgi.setCache(true); // not necessarily true but meh 9931 if(auto fn = ret.filename()) { 9932 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9933 } 9934 cgi.setResponseContentType(ret.contentType); 9935 cgi.write(ret.getData(), true); 9936 } 9937 9938 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9939 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9940 auto container = this.htmlContainer(); 9941 container.appendChild(formatReturnValueAsHtml(ret)); 9942 cgi.write(container.parentDocument.toString(), true); 9943 } 9944 9945 /++ 9946 9947 History: 9948 Added January 23, 2023 (dub v11.0) 9949 +/ 9950 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9951 switch(format) { 9952 case "html": 9953 presentExceptionAsHtml(cgi, t, meta); 9954 break; 9955 case "json": 9956 presentExceptionAsJsonImpl(cgi, t); 9957 break; 9958 default: 9959 } 9960 } 9961 9962 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 9963 cgi.setResponseStatus("500 Internal Server Error"); 9964 cgi.setResponseContentType("application/json"); 9965 import arsd.jsvar; 9966 var v = var.emptyObject; 9967 v.type = typeid(t).toString; 9968 v.msg = t.msg; 9969 v.fullString = t.toString(); 9970 cgi.write(v.toJson(), true); 9971 } 9972 9973 9974 /++ 9975 If you override this, you will need to cast the exception type `t` dynamically, 9976 but can then use the template arguments here to refer back to the function. 9977 9978 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9979 method on the live object. You could, in theory, change arguments and retry, but I 9980 provide that information mostly with the expectation that you will use them to make 9981 useful forms or richer error messages for the user. 9982 9983 History: 9984 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 9985 I removed this in favor of a `Meta` param. 9986 9987 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 9988 9989 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 9990 9991 If you used the func for something, move that something into your `methodMeta` template. 9992 9993 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 9994 enabling an easier implementation of [presentExceptionalReturn]. 9995 +/ 9996 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 9997 Form af; 9998 /+ 9999 foreach(attr; __traits(getAttributes, func)) { 10000 static if(__traits(isSame, attr, AutomaticForm)) { 10001 af = createAutomaticFormForFunction!(func)(dg); 10002 } 10003 } 10004 +/ 10005 presentExceptionAsHtmlImpl(cgi, t, af); 10006 } 10007 10008 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 10009 if(auto e = cast(ResourceNotFoundException) t) { 10010 auto container = this.htmlContainer(); 10011 10012 container.addChild("p", e.msg); 10013 10014 if(!cgi.outputtedResponseData) 10015 cgi.setResponseStatus("404 Not Found"); 10016 cgi.write(container.parentDocument.toString(), true); 10017 } else if(auto mae = cast(MissingArgumentException) t) { 10018 if(automaticForm is null) 10019 goto generic; 10020 auto container = this.htmlContainer(); 10021 if(cgi.requestMethod == Cgi.RequestMethod.POST) 10022 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 10023 container.appendChild(automaticForm); 10024 10025 cgi.write(container.parentDocument.toString(), true); 10026 } else { 10027 generic: 10028 auto container = this.htmlContainer(); 10029 10030 // import std.stdio; writeln(t.toString()); 10031 10032 container.appendChild(exceptionToElement(t)); 10033 10034 container.addChild("h4", "GET"); 10035 foreach(k, v; cgi.get) { 10036 auto deets = container.addChild("details"); 10037 deets.addChild("summary", k); 10038 deets.addChild("div", v); 10039 } 10040 10041 container.addChild("h4", "POST"); 10042 foreach(k, v; cgi.post) { 10043 auto deets = container.addChild("details"); 10044 deets.addChild("summary", k); 10045 deets.addChild("div", v); 10046 } 10047 10048 10049 if(!cgi.outputtedResponseData) 10050 cgi.setResponseStatus("500 Internal Server Error"); 10051 cgi.write(container.parentDocument.toString(), true); 10052 } 10053 } 10054 10055 Element exceptionToElement(Throwable t) { 10056 auto div = Element.make("div"); 10057 div.addClass("exception-display"); 10058 10059 div.addChild("p", t.msg); 10060 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10061 10062 auto pre = div.addChild("pre"); 10063 string s; 10064 s = t.toString(); 10065 Element currentBox; 10066 bool on = false; 10067 foreach(line; s.splitLines) { 10068 if(!on && line.startsWith("-----")) 10069 on = true; 10070 if(!on) continue; 10071 if(line.indexOf("arsd/") != -1) { 10072 if(currentBox is null) { 10073 currentBox = pre.addChild("details"); 10074 currentBox.addChild("summary", "Framework code"); 10075 } 10076 currentBox.addChild("span", line ~ "\n"); 10077 } else { 10078 pre.addChild("span", line ~ "\n"); 10079 currentBox = null; 10080 } 10081 } 10082 10083 return div; 10084 } 10085 10086 /++ 10087 Returns an element for a particular type 10088 +/ 10089 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10090 import std.traits; 10091 10092 auto div = Element.make("div"); 10093 div.addClass("form-field"); 10094 10095 static if(is(T : const Cgi.UploadedFile)) { 10096 Element lbl; 10097 if(displayName !is null) { 10098 lbl = div.addChild("label"); 10099 lbl.addChild("span", displayName, "label-text"); 10100 lbl.appendText(" "); 10101 } else { 10102 lbl = div; 10103 } 10104 auto i = lbl.addChild("input", name); 10105 i.attrs.name = name; 10106 i.attrs.type = "file"; 10107 i.attrs.multiple = "multiple"; 10108 } else static if(is(T == Cgi.UploadedFile)) { 10109 Element lbl; 10110 if(displayName !is null) { 10111 lbl = div.addChild("label"); 10112 lbl.addChild("span", displayName, "label-text"); 10113 lbl.appendText(" "); 10114 } else { 10115 lbl = div; 10116 } 10117 auto i = lbl.addChild("input", name); 10118 i.attrs.name = name; 10119 i.attrs.type = "file"; 10120 } else static if(is(T == enum)) { 10121 Element lbl; 10122 if(displayName !is null) { 10123 lbl = div.addChild("label"); 10124 lbl.addChild("span", displayName, "label-text"); 10125 lbl.appendText(" "); 10126 } else { 10127 lbl = div; 10128 } 10129 auto i = lbl.addChild("select", name); 10130 i.attrs.name = name; 10131 10132 foreach(memberName; __traits(allMembers, T)) 10133 i.addChild("option", memberName); 10134 10135 } else static if(is(T == struct)) { 10136 if(displayName !is null) 10137 div.addChild("span", displayName, "label-text"); 10138 auto fieldset = div.addChild("fieldset"); 10139 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10140 fieldset.addChild("input", name); 10141 foreach(idx, memberName; __traits(allMembers, T)) 10142 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10143 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10144 } 10145 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10146 Element lbl; 10147 if(displayName !is null) { 10148 lbl = div.addChild("label"); 10149 lbl.addChild("span", displayName, "label-text"); 10150 lbl.appendText(" "); 10151 } else { 10152 lbl = div; 10153 } 10154 Element i; 10155 if(udaSuggestion) { 10156 i = udaSuggestion(); 10157 lbl.appendChild(i); 10158 } else { 10159 i = lbl.addChild("input", name); 10160 } 10161 i.attrs.name = name; 10162 static if(isSomeString!T) 10163 i.attrs.type = "text"; 10164 else 10165 i.attrs.type = "number"; 10166 if(i.tagName == "textarea") 10167 i.textContent = to!string(T.init); 10168 else 10169 i.attrs.value = to!string(T.init); 10170 } else static if(is(T == bool)) { 10171 Element lbl; 10172 if(displayName !is null) { 10173 lbl = div.addChild("label"); 10174 lbl.addChild("span", displayName, "label-text"); 10175 lbl.appendText(" "); 10176 } else { 10177 lbl = div; 10178 } 10179 auto i = lbl.addChild("input", name); 10180 i.attrs.type = "checkbox"; 10181 i.attrs.value = "true"; 10182 i.attrs.name = name; 10183 } else static if(is(T == K[], K)) { 10184 auto templ = div.addChild("template"); 10185 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10186 if(displayName !is null) 10187 div.addChild("span", displayName, "label-text"); 10188 auto btn = div.addChild("button"); 10189 btn.addClass("add-array-button"); 10190 btn.attrs.type = "button"; 10191 btn.innerText = "Add"; 10192 btn.attrs.onclick = q{ 10193 var a = document.importNode(this.parentNode.firstChild.content, true); 10194 this.parentNode.insertBefore(a, this); 10195 }; 10196 } else static if(is(T == V[K], K, V)) { 10197 div.innerText = "assoc array not implemented for automatic form at this time"; 10198 } else { 10199 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10200 } 10201 10202 10203 return div; 10204 } 10205 10206 /// creates a form for gathering the function's arguments 10207 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10208 10209 auto form = cast(Form) Element.make("form"); 10210 10211 form.method = "POST"; // FIXME 10212 10213 form.addClass("automatic-form"); 10214 10215 string formDisplayName = beautify(__traits(identifier, method)); 10216 foreach(attr; __traits(getAttributes, method)) 10217 static if(is(typeof(attr) == DisplayName)) 10218 formDisplayName = attr.name; 10219 form.addChild("h3", formDisplayName); 10220 10221 import std.traits; 10222 10223 //Parameters!method params; 10224 //alias idents = ParameterIdentifierTuple!method; 10225 //alias defaults = ParameterDefaults!method; 10226 10227 static if(is(typeof(method) P == __parameters)) 10228 foreach(idx, _; P) {{ 10229 10230 alias param = P[idx .. idx + 1]; 10231 10232 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10233 string displayName = beautify(__traits(identifier, param)); 10234 Element function() element; 10235 foreach(attr; __traits(getAttributes, param)) { 10236 static if(is(typeof(attr) == DisplayName)) 10237 displayName = attr.name; 10238 else static if(is(typeof(attr) : typeof(element))) { 10239 element = attr; 10240 } 10241 } 10242 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10243 if(i.querySelector("input[type=file]") !is null) 10244 form.setAttribute("enctype", "multipart/form-data"); 10245 } 10246 }} 10247 10248 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10249 10250 return form; 10251 } 10252 10253 /// creates a form for gathering object members (for the REST object thing right now) 10254 Form createAutomaticFormForObject(T)(T obj) { 10255 auto form = cast(Form) Element.make("form"); 10256 10257 form.addClass("automatic-form"); 10258 10259 form.addChild("h3", beautify(__traits(identifier, T))); 10260 10261 import std.traits; 10262 10263 //Parameters!method params; 10264 //alias idents = ParameterIdentifierTuple!method; 10265 //alias defaults = ParameterDefaults!method; 10266 10267 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10268 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10269 string displayName = beautify(memberName); 10270 Element function() element; 10271 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10272 static if(is(typeof(attr) == DisplayName)) 10273 displayName = attr.name; 10274 else static if(is(typeof(attr) : typeof(element))) 10275 element = attr; 10276 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10277 10278 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10279 }}} 10280 10281 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10282 10283 return form; 10284 } 10285 10286 /// 10287 Element formatReturnValueAsHtml(T)(T t) { 10288 import std.traits; 10289 10290 static if(is(T == typeof(null))) { 10291 return Element.make("span"); 10292 } else static if(is(T : Element)) { 10293 return t; 10294 } else static if(is(T == MultipleResponses!Types, Types...)) { 10295 foreach(index, type; Types) { 10296 if(t.contains == index) 10297 return formatReturnValueAsHtml(t.payload[index]); 10298 } 10299 assert(0); 10300 } else static if(is(T == Paginated!E, E)) { 10301 auto e = Element.make("div").addClass("paginated-result"); 10302 e.appendChild(formatReturnValueAsHtml(t.items)); 10303 if(t.nextPageUrl.length) 10304 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10305 return e; 10306 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10307 return Element.make("span", to!string(t), "automatic-data-display"); 10308 } else static if(is(T == V[K], K, V)) { 10309 auto dl = Element.make("dl"); 10310 dl.addClass("automatic-data-display associative-array"); 10311 foreach(k, v; t) { 10312 dl.addChild("dt", to!string(k)); 10313 dl.addChild("dd", formatReturnValueAsHtml(v)); 10314 } 10315 return dl; 10316 } else static if(is(T == struct)) { 10317 auto dl = Element.make("dl"); 10318 dl.addClass("automatic-data-display struct"); 10319 10320 foreach(idx, memberName; __traits(allMembers, T)) 10321 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10322 dl.addChild("dt", beautify(memberName)); 10323 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10324 } 10325 10326 return dl; 10327 } else static if(is(T == bool)) { 10328 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10329 } else static if(is(T == E[], E)) { 10330 static if(is(E : RestObject!Proxy, Proxy)) { 10331 // treat RestObject similar to struct 10332 auto table = cast(Table) Element.make("table"); 10333 table.addClass("automatic-data-display"); 10334 string[] names; 10335 foreach(idx, memberName; __traits(derivedMembers, E)) 10336 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10337 names ~= beautify(memberName); 10338 } 10339 table.appendHeaderRow(names); 10340 10341 foreach(l; t) { 10342 auto tr = table.appendRow(); 10343 foreach(idx, memberName; __traits(derivedMembers, E)) 10344 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10345 static if(memberName == "id") { 10346 string val = to!string(__traits(getMember, l, memberName)); 10347 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10348 } else { 10349 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10350 } 10351 } 10352 } 10353 10354 return table; 10355 } else static if(is(E == struct)) { 10356 // an array of structs is kinda special in that I like 10357 // having those formatted as tables. 10358 auto table = cast(Table) Element.make("table"); 10359 table.addClass("automatic-data-display"); 10360 string[] names; 10361 foreach(idx, memberName; __traits(allMembers, E)) 10362 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10363 names ~= beautify(memberName); 10364 } 10365 table.appendHeaderRow(names); 10366 10367 foreach(l; t) { 10368 auto tr = table.appendRow(); 10369 foreach(idx, memberName; __traits(allMembers, E)) 10370 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10371 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10372 } 10373 } 10374 10375 return table; 10376 } else { 10377 // otherwise, I will just make a list. 10378 auto ol = Element.make("ol"); 10379 ol.addClass("automatic-data-display"); 10380 foreach(e; t) 10381 ol.addChild("li", formatReturnValueAsHtml(e)); 10382 return ol; 10383 } 10384 } else static if(is(T : Object)) { 10385 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10386 return Element.make("div", t.toHtml()); 10387 else 10388 return Element.make("div", t.toString()); 10389 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10390 10391 assert(0); 10392 } 10393 10394 } 10395 10396 /++ 10397 The base class for the [dispatcher] function and object support. 10398 +/ 10399 class WebObject { 10400 //protected Cgi cgi; 10401 10402 protected void initialize(Cgi cgi) { 10403 //this.cgi = cgi; 10404 } 10405 } 10406 10407 /++ 10408 Can return one of the given types, decided at runtime. The syntax 10409 is to declare all the possible types in the return value, then you 10410 can `return typeof(return)(...value...)` to construct it. 10411 10412 It has an auto-generated constructor for each value it can hold. 10413 10414 --- 10415 MultipleResponses!(Redirection, string) getData(int how) { 10416 if(how & 1) 10417 return typeof(return)(Redirection("http://dpldocs.info/")); 10418 else 10419 return typeof(return)("hi there!"); 10420 } 10421 --- 10422 10423 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10424 +/ 10425 struct MultipleResponses(T...) { 10426 private size_t contains; 10427 private union { 10428 private T payload; 10429 } 10430 10431 static foreach(index, type; T) 10432 public this(type t) { 10433 contains = index; 10434 payload[index] = t; 10435 } 10436 10437 /++ 10438 This is primarily for testing. It is your way of getting to the response. 10439 10440 Let's say you wanted to test that one holding a Redirection and a string actually 10441 holds a string, by name of "test": 10442 10443 --- 10444 auto valueToTest = your_test_function(); 10445 10446 valueToTest.visit( 10447 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10448 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10449 ); 10450 --- 10451 10452 History: 10453 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10454 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10455 +/ 10456 void visit(Handlers...)(Handlers handlers) { 10457 template findHandler(type, int count, HandlersToCheck...) { 10458 static if(HandlersToCheck.length == 0) 10459 enum findHandler = -1; 10460 else { 10461 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10462 enum findHandler = count; 10463 else 10464 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10465 } 10466 } 10467 foreach(index, type; T) { 10468 enum handlerIndex = findHandler!(type, 0, Handlers); 10469 static if(handlerIndex == -1) 10470 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10471 else { 10472 if(index == this.contains) 10473 handlers[handlerIndex](this.payload[index]); 10474 } 10475 } 10476 } 10477 10478 /+ 10479 auto toArsdJsvar()() { 10480 import arsd.jsvar; 10481 return var(null); 10482 } 10483 +/ 10484 } 10485 10486 // FIXME: implement this somewhere maybe 10487 struct RawResponse { 10488 int code; 10489 string[] headers; 10490 const(ubyte)[] responseBody; 10491 } 10492 10493 /++ 10494 You can return this from [WebObject] subclasses for redirections. 10495 10496 (though note the static types means that class must ALWAYS redirect if 10497 you return this directly. You might want to return [MultipleResponses] if it 10498 can be conditional) 10499 +/ 10500 struct Redirection { 10501 string to; /// The URL to redirect to. 10502 int code = 303; /// The HTTP code to return. 10503 } 10504 10505 /++ 10506 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10507 10508 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10509 the presenter in the dispatcher. 10510 10511 FIXME: explain this better 10512 10513 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10514 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10515 the runtime result of that is undefined. 10516 10517 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10518 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10519 10520 $(WARNING 10521 --- 10522 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10523 // even if you put different URL udas on it, the current code ignores them. 10524 void foo(int a) {} 10525 void foo(string a) {} 10526 --- 10527 ) 10528 10529 See_Also: [serveRestObject], [serveStaticFile] 10530 +/ 10531 auto serveApi(T)(string urlPrefix) { 10532 assert(urlPrefix[$ - 1] == '/'); 10533 return serveApiInternal!T(urlPrefix); 10534 } 10535 10536 private string nextPieceFromSlash(ref string remainingUrl) { 10537 if(remainingUrl.length == 0) 10538 return remainingUrl; 10539 int slash = 0; 10540 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10541 slash++; 10542 10543 // I am specifically passing `null` to differentiate it vs empty string 10544 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10545 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10546 // so if it is the last item, the dot can be used to load an alternative view 10547 // otherwise tho the dot is considered part of the identifier 10548 // FIXME 10549 10550 // again notice "" vs null here! 10551 if(slash == remainingUrl.length) 10552 remainingUrl = null; 10553 else 10554 remainingUrl = remainingUrl[slash + 1 .. $]; 10555 10556 return ident; 10557 } 10558 10559 /++ 10560 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10561 +/ 10562 enum AddTrailingSlash; 10563 /// ditto 10564 enum RemoveTrailingSlash; 10565 10566 private auto serveApiInternal(T)(string urlPrefix) { 10567 10568 import arsd.dom; 10569 import arsd.jsvar; 10570 10571 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10572 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10573 10574 try { 10575 // see duplicated code below by searching subresource_ctor 10576 // also see mustNotBeSetFromWebParams 10577 10578 static if(is(typeof(T.__ctor) P == __parameters)) { 10579 P params; 10580 10581 foreach(pidx, param; P) { 10582 static if(is(param : Cgi)) { 10583 static assert(!is(param == immutable)); 10584 cast() params[pidx] = cgi; 10585 } else static if(is(param == Session!D, D)) { 10586 static assert(!is(param == immutable)); 10587 cast() params[pidx] = cgi.getSessionObject!D(); 10588 10589 } else { 10590 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10591 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10592 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10593 static if(is(typeof(func(cgi)))) 10594 params[pidx] = func(cgi); 10595 else 10596 params[pidx] = func(); 10597 } 10598 } 10599 } else { 10600 10601 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10602 params[pidx] = param.getAutomaticallyForCgi(cgi); 10603 } else static if(is(param == string)) { 10604 auto ident = nextPieceFromSlash(remainingUrl); 10605 params[pidx] = ident; 10606 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10607 } 10608 } 10609 } 10610 10611 auto obj = new T(params); 10612 } else { 10613 auto obj = new T(); 10614 } 10615 10616 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10617 } catch(Throwable t) { 10618 switch(cgi.request("format", "html")) { 10619 case "html": 10620 static void dummy() {} 10621 presenter.presentExceptionAsHtml(cgi, t, null); 10622 return true; 10623 case "json": 10624 var envelope = var.emptyObject; 10625 envelope.success = false; 10626 envelope.result = null; 10627 envelope.error = t.toString(); 10628 cgi.setResponseContentType("application/json"); 10629 cgi.write(envelope.toJson(), true); 10630 return true; 10631 default: 10632 throw t; 10633 // return true; 10634 } 10635 // return true; 10636 } 10637 10638 assert(0); 10639 } 10640 10641 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10642 10643 obj.initialize(cgi); 10644 10645 /+ 10646 Overload rules: 10647 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10648 statically. 10649 10650 Moreover, some args vs no args can be overloaded dynamically. 10651 +/ 10652 10653 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10654 /+ 10655 auto orig = remainingUrl; 10656 assert(0, 10657 (orig is null ? "__null" : orig) 10658 ~ " .. " ~ 10659 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10660 +/ 10661 10662 if(methodNameFromUrl is null) 10663 methodNameFromUrl = "__null"; 10664 10665 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10666 10667 if(remainingUrl.length) 10668 hack ~= "/"; 10669 10670 switch(hack) { 10671 foreach(methodName; __traits(derivedMembers, T)) 10672 static if(methodName != "__ctor") 10673 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10674 static if(is(typeof(overload) P == __parameters)) 10675 static if(is(typeof(overload) R == return)) 10676 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10677 { 10678 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10679 case urlNameForMethod: 10680 10681 static if(is(R : WebObject)) { 10682 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10683 10684 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10685 10686 // subresource_ctor 10687 // also see mustNotBeSetFromWebParams 10688 10689 P params; 10690 10691 string ident; 10692 10693 foreach(pidx, param; P) { 10694 static if(is(param : Cgi)) { 10695 static assert(!is(param == immutable)); 10696 cast() params[pidx] = cgi; 10697 } else static if(is(param == typeof(presenter))) { 10698 cast() param[pidx] = presenter; 10699 } else static if(is(param == Session!D, D)) { 10700 static assert(!is(param == immutable)); 10701 cast() params[pidx] = cgi.getSessionObject!D(); 10702 } else { 10703 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10704 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10705 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10706 static if(is(typeof(func(cgi)))) 10707 params[pidx] = func(cgi); 10708 else 10709 params[pidx] = func(); 10710 } 10711 } 10712 } else { 10713 10714 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10715 params[pidx] = param.getAutomaticallyForCgi(cgi); 10716 } else static if(is(param == string)) { 10717 ident = nextPieceFromSlash(remainingUrl); 10718 if(ident is null) { 10719 // trailing slash mandated on subresources 10720 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10721 return true; 10722 } else { 10723 params[pidx] = ident; 10724 } 10725 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10726 } 10727 } 10728 } 10729 10730 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10731 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10732 } else { 10733 // 404 it if any url left - not a subresource means we don't get to play with that! 10734 if(remainingUrl.length) 10735 return false; 10736 10737 bool automaticForm; 10738 10739 foreach(attr; __traits(getAttributes, overload)) 10740 static if(is(attr == AddTrailingSlash)) { 10741 if(remainingUrl is null) { 10742 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10743 return true; 10744 } 10745 } else static if(is(attr == RemoveTrailingSlash)) { 10746 if(remainingUrl !is null) { 10747 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10748 return true; 10749 } 10750 10751 } else static if(__traits(isSame, AutomaticForm, attr)) { 10752 automaticForm = true; 10753 } 10754 10755 /+ 10756 int zeroArgOverload = -1; 10757 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10758 bool calledWithZeroArgs = true; 10759 foreach(k, v; cgi.get) 10760 if(k != "format") { 10761 calledWithZeroArgs = false; 10762 break; 10763 } 10764 foreach(k, v; cgi.post) 10765 if(k != "format") { 10766 calledWithZeroArgs = false; 10767 break; 10768 } 10769 10770 // first, we need to go through and see if there is an empty one, since that 10771 // changes inside. But otherwise, all the stuff I care about can be done via 10772 // simple looping (other improper overloads might be flagged for runtime semantic check) 10773 // 10774 // an argument of type Cgi is ignored for these purposes 10775 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10776 static if(is(typeof(overload) P == __parameters)) 10777 static if(P.length == 0) 10778 zeroArgOverload = cast(int) idx; 10779 else static if(P.length == 1 && is(P[0] : Cgi)) 10780 zeroArgOverload = cast(int) idx; 10781 }} 10782 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10783 bool overloadHasBeenCalled = false; 10784 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10785 bool callFunction = true; 10786 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10787 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10788 callFunction = false; 10789 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10790 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10791 callFunction = false; 10792 10793 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10794 10795 bool hadAnyMethodRestrictions = false; 10796 bool foundAcceptableMethod = false; 10797 foreach(attr; __traits(getAttributes, overload)) { 10798 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10799 hadAnyMethodRestrictions = true; 10800 if(attr == cgi.requestMethod) 10801 foundAcceptableMethod = true; 10802 } 10803 } 10804 10805 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10806 callFunction = false; 10807 10808 /+ 10809 The overloads we really want to allow are the sane ones 10810 from the web perspective. Which is likely on HTTP verbs, 10811 for the most part, but might also be potentially based on 10812 some args vs zero args, or on argument names. Can't really 10813 do argument types very reliable through the web though; those 10814 should probably be different URLs. 10815 10816 Even names I feel is better done inside the function, so I'm not 10817 going to support that here. But the HTTP verbs and zero vs some 10818 args makes sense - it lets you define custom forms pretty easily. 10819 10820 Moreover, I'm of the opinion that empty overload really only makes 10821 sense on GET for this case. On a POST, it is just a missing argument 10822 exception and that should be handled by the presenter. But meh, I'll 10823 let the user define that, D only allows one empty arg thing anyway 10824 so the method UDAs are irrelevant. 10825 +/ 10826 if(callFunction) 10827 +/ 10828 10829 auto format = cgi.request("format", defaultFormat!overload()); 10830 auto wantsFormFormat = format.startsWith("form-"); 10831 10832 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10833 // Should I still show the form on a json thing? idk... 10834 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10835 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10836 return true; 10837 } 10838 10839 try { 10840 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10841 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10842 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10843 } catch(Throwable t) { 10844 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10845 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10846 } 10847 return true; 10848 //}} 10849 10850 //cgi.header("Accept: POST"); // FIXME list the real thing 10851 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10852 //return true; 10853 } 10854 } 10855 } 10856 case "GET script.js": 10857 cgi.setResponseContentType("text/javascript"); 10858 cgi.gzipResponse = true; 10859 cgi.write(presenter.script(), true); 10860 return true; 10861 case "GET style.css": 10862 cgi.setResponseContentType("text/css"); 10863 cgi.gzipResponse = true; 10864 cgi.write(presenter.style(), true); 10865 return true; 10866 default: 10867 return false; 10868 } 10869 10870 assert(0); 10871 } 10872 return DispatcherDefinition!internalHandler(urlPrefix, false); 10873 } 10874 10875 string defaultFormat(alias method)() { 10876 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10877 foreach(attr; __traits(getAttributes, method)) { 10878 static if(is(typeof(attr) == DefaultFormat)) { 10879 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10880 return attr.value; 10881 } 10882 } 10883 return "html"; 10884 } 10885 10886 struct Paginated(T) { 10887 T[] items; 10888 string nextPageUrl; 10889 } 10890 10891 template urlNamesForMethod(alias method, string default_) { 10892 string[] helper() { 10893 auto verb = Cgi.RequestMethod.GET; 10894 bool foundVerb = false; 10895 bool foundNoun = false; 10896 10897 string def = default_; 10898 10899 bool hasAutomaticForm = false; 10900 10901 foreach(attr; __traits(getAttributes, method)) { 10902 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10903 verb = attr; 10904 if(foundVerb) 10905 assert(0, "Multiple http verbs on one function is not currently supported"); 10906 foundVerb = true; 10907 } 10908 static if(is(typeof(attr) == UrlName)) { 10909 if(foundNoun) 10910 assert(0, "Multiple url names on one function is not currently supported"); 10911 foundNoun = true; 10912 def = attr.name; 10913 } 10914 static if(__traits(isSame, attr, AutomaticForm)) { 10915 hasAutomaticForm = true; 10916 } 10917 } 10918 10919 if(def is null) 10920 def = "__null"; 10921 10922 string[] ret; 10923 10924 static if(is(typeof(method) R == return)) { 10925 static if(is(R : WebObject)) { 10926 def ~= "/"; 10927 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10928 ret ~= v ~ " " ~ def; 10929 } else { 10930 if(hasAutomaticForm) { 10931 ret ~= "GET " ~ def; 10932 ret ~= "POST " ~ def; 10933 } else { 10934 ret ~= to!string(verb) ~ " " ~ def; 10935 } 10936 } 10937 } else static assert(0); 10938 10939 return ret; 10940 } 10941 enum urlNamesForMethod = helper(); 10942 } 10943 10944 10945 enum AccessCheck { 10946 allowed, 10947 denied, 10948 nonExistant, 10949 } 10950 10951 enum Operation { 10952 show, 10953 create, 10954 replace, 10955 remove, 10956 update 10957 } 10958 10959 enum UpdateResult { 10960 accessDenied, 10961 noSuchResource, 10962 success, 10963 failure, 10964 unnecessary 10965 } 10966 10967 enum ValidationResult { 10968 valid, 10969 invalid 10970 } 10971 10972 10973 /++ 10974 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10975 10976 WARNING: this is not stable. 10977 +/ 10978 class RestObject(CRTP) : WebObject { 10979 10980 import arsd.dom; 10981 import arsd.jsvar; 10982 10983 /// Prepare the object to be shown. 10984 void show() {} 10985 /// ditto 10986 void show(string urlId) { 10987 load(urlId); 10988 show(); 10989 } 10990 10991 /// Override this to provide access control to this object. 10992 AccessCheck accessCheck(string urlId, Operation operation) { 10993 return AccessCheck.allowed; 10994 } 10995 10996 ValidationResult validate() { 10997 // FIXME 10998 return ValidationResult.valid; 10999 } 11000 11001 string getUrlSlug() { 11002 import std.conv; 11003 static if(is(typeof(CRTP.id))) 11004 return to!string((cast(CRTP) this).id); 11005 else 11006 return null; 11007 } 11008 11009 // The functions with more arguments are the low-level ones, 11010 // they forward to the ones with fewer arguments by default. 11011 11012 // POST on a parent collection - this is called from a collection class after the members are updated 11013 /++ 11014 Given a populated object, this creates a new entry. Returns the url identifier 11015 of the new object. 11016 +/ 11017 string create(scope void delegate() applyChanges) { 11018 applyChanges(); 11019 save(); 11020 return getUrlSlug(); 11021 } 11022 11023 void replace() { 11024 save(); 11025 } 11026 void replace(string urlId, scope void delegate() applyChanges) { 11027 load(urlId); 11028 applyChanges(); 11029 replace(); 11030 } 11031 11032 void update(string[] fieldList) { 11033 save(); 11034 } 11035 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11036 load(urlId); 11037 applyChanges(); 11038 update(fieldList); 11039 } 11040 11041 void remove() {} 11042 11043 void remove(string urlId) { 11044 load(urlId); 11045 remove(); 11046 } 11047 11048 abstract void load(string urlId); 11049 abstract void save(); 11050 11051 Element toHtml(Presenter)(Presenter presenter) { 11052 import arsd.dom; 11053 import std.conv; 11054 auto obj = cast(CRTP) this; 11055 auto div = Element.make("div"); 11056 div.addClass("Dclass_" ~ CRTP.stringof); 11057 div.dataset.url = getUrlSlug(); 11058 bool first = true; 11059 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11060 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11061 if(!first) div.addChild("br"); else first = false; 11062 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11063 } 11064 return div; 11065 } 11066 11067 var toJson() { 11068 import arsd.jsvar; 11069 var v = var.emptyObject(); 11070 auto obj = cast(CRTP) this; 11071 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11072 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11073 v[memberName] = __traits(getMember, obj, memberName); 11074 } 11075 return v; 11076 } 11077 11078 /+ 11079 auto structOf(this This) { 11080 11081 } 11082 +/ 11083 } 11084 11085 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11086 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11087 11088 /++ 11089 Base class for REST collections. 11090 +/ 11091 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11092 /// You might subclass this and use the cgi object's query params 11093 /// to implement a search filter, for example. 11094 /// 11095 /// FIXME: design a way to auto-generate that form 11096 /// (other than using the WebObject thing above lol 11097 // it'll prolly just be some searchParams UDA or maybe an enum. 11098 // 11099 // pagination too perhaps. 11100 // 11101 // and sorting too 11102 IndexResult index() { return IndexResult.init; } 11103 11104 string[] sortableFields() { return null; } 11105 string[] searchableFields() { return null; } 11106 11107 struct IndexResult { 11108 Obj[] results; 11109 11110 string[] sortableFields; 11111 11112 string previousPageIdentifier; 11113 string nextPageIdentifier; 11114 string firstPageIdentifier; 11115 string lastPageIdentifier; 11116 11117 int numberOfPages; 11118 } 11119 11120 override string create(scope void delegate() applyChanges) { assert(0); } 11121 override void load(string urlId) { assert(0); } 11122 override void save() { assert(0); } 11123 override void show() { 11124 index(); 11125 } 11126 override void show(string urlId) { 11127 show(); 11128 } 11129 11130 /// Proxy POST requests (create calls) to the child collection 11131 alias PostProxy = Obj; 11132 } 11133 11134 /++ 11135 Serves a REST object, similar to a Ruby on Rails resource. 11136 11137 You put data members in your class. cgi.d will automatically make something out of those. 11138 11139 It will call your constructor with the ID from the URL. This may be null. 11140 It will then populate the data members from the request. 11141 It will then call a method, if present, telling what happened. You don't need to write these! 11142 It finally returns a reply. 11143 11144 Your methods are passed a list of fields it actually set. 11145 11146 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11147 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11148 with relative linking. But meh.) 11149 11150 GET /items -> index. all values not set. 11151 GET /items/id -> get. only ID will be set, other params ignored. 11152 POST /items -> create. values set as given 11153 PUT /items/id -> replace. values set as given 11154 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11155 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11156 PATCH /items/id -> update. values set as given, list of changed fields passed 11157 or POST /items/id with cgi.post["_method"] == "PATCH" 11158 DELETE /items/id -> destroy. only ID guaranteed to be set 11159 or POST /items/id with cgi.post["_method"] == "DELETE" 11160 11161 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11162 redirect you away from it. 11163 11164 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11165 11166 I will also let you change the default, if you must. 11167 11168 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11169 11170 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11171 that follow the same thing. They may be individual resources or collections themselves. 11172 11173 Your class is expected to have at least the following methods: 11174 11175 FIXME: i kinda wanna add a routes object to the initialize call 11176 11177 create 11178 Create returns the new address on success, some code on failure. 11179 show 11180 index 11181 update 11182 remove 11183 11184 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11185 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11186 11187 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11188 11189 NOT IMPLEMENTED 11190 11191 11192 Really, a collection is a resource with a bunch of subresources. 11193 11194 GET /items 11195 index because it is GET on the top resource 11196 11197 GET /items/foo 11198 item but different than items? 11199 11200 class Items { 11201 11202 } 11203 11204 ... but meh, a collection can be automated. not worth making it 11205 a separate thing, let's look at a real example. Users has many 11206 items and a virtual one, /users/current. 11207 11208 the individual users have properties and two sub-resources: 11209 session, which is just one, and comments, a collection. 11210 11211 class User : RestObject!() { // no parent 11212 int id; 11213 string name; 11214 11215 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11216 // but you can override them to do it differently. 11217 11218 // any member which is of type RestObject can be linked automatically via href btw. 11219 11220 void show() {} 11221 void show(string urlId) {} // automated! GET of this specific thing 11222 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11223 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11224 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11225 void remove(string urlId) {} // DELETE 11226 11227 void load(string urlId) {} // the default implementation of show() populates the id, then 11228 11229 this() {} 11230 11231 mixin Subresource!Session; 11232 mixin Subresource!Comment; 11233 } 11234 11235 class Session : RestObject!() { 11236 // the parent object may not be fully constructed/loaded 11237 this(User parent) {} 11238 11239 } 11240 11241 class Comment : CollectionOf!Comment { 11242 this(User parent) {} 11243 } 11244 11245 class Users : CollectionOf!User { 11246 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11247 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11248 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11249 } 11250 11251 +/ 11252 auto serveRestObject(T)(string urlPrefix) { 11253 assert(urlPrefix[0] == '/'); 11254 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11255 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11256 string url = cgi.pathInfo[urlPrefix.length .. $]; 11257 11258 if(url.length && url[$ - 1] == '/') { 11259 // remove the final slash... 11260 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11261 return true; 11262 } 11263 11264 return restObjectServeHandler!T(cgi, presenter, url); 11265 } 11266 return DispatcherDefinition!internalHandler(urlPrefix, false); 11267 } 11268 11269 /+ 11270 /// Convenience method for serving a collection. It will be named the same 11271 /// as type T, just with an s at the end. If you need any further, just 11272 /// write the class yourself. 11273 auto serveRestCollectionOf(T)(string urlPrefix) { 11274 assert(urlPrefix[0] == '/'); 11275 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11276 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11277 } 11278 +/ 11279 11280 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11281 string urlId = null; 11282 if(url.length && url[0] == '/') { 11283 // asking for a subobject 11284 urlId = url[1 .. $]; 11285 foreach(idx, ch; urlId) { 11286 if(ch == '/') { 11287 urlId = urlId[0 .. idx]; 11288 break; 11289 } 11290 } 11291 } 11292 11293 // FIXME handle other subresources 11294 11295 static if(is(T : CollectionOf!(C), C)) { 11296 if(urlId !is null) { 11297 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11298 } 11299 } 11300 11301 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11302 11303 auto obj = new T(); 11304 obj.initialize(cgi); 11305 // FIXME: populate reflection info delegates 11306 11307 11308 // FIXME: I am not happy with this. 11309 switch(urlId) { 11310 case "script.js": 11311 cgi.setResponseContentType("text/javascript"); 11312 cgi.gzipResponse = true; 11313 cgi.write(presenter.script(), true); 11314 return true; 11315 case "style.css": 11316 cgi.setResponseContentType("text/css"); 11317 cgi.gzipResponse = true; 11318 cgi.write(presenter.style(), true); 11319 return true; 11320 default: 11321 // intentionally blank 11322 } 11323 11324 11325 11326 11327 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11328 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11329 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11330 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11331 } 11332 } 11333 void applyChanges() { 11334 applyChangesTemplate(cgi, obj); 11335 } 11336 11337 string[] modifiedList; 11338 11339 void writeObject(bool addFormLinks) { 11340 if(cgi.request("format") == "json") { 11341 cgi.setResponseContentType("application/json"); 11342 cgi.write(obj.toJson().toString, true); 11343 } else { 11344 auto container = presenter.htmlContainer(); 11345 if(addFormLinks) { 11346 static if(is(T : CollectionOf!(C), C)) 11347 container.appendHtml(` 11348 <form> 11349 <button type="submit" name="_method" value="POST">Create New</button> 11350 </form> 11351 `); 11352 else 11353 container.appendHtml(` 11354 <a href="..">Back</a> 11355 <form> 11356 <button type="submit" name="_method" value="PATCH">Edit</button> 11357 <button type="submit" name="_method" value="DELETE">Delete</button> 11358 </form> 11359 `); 11360 } 11361 container.appendChild(obj.toHtml(presenter)); 11362 cgi.write(container.parentDocument.toString, true); 11363 } 11364 } 11365 11366 // FIXME: I think I need a set type in here.... 11367 // it will be nice to pass sets of members. 11368 11369 try 11370 switch(cgi.requestMethod) { 11371 case Cgi.RequestMethod.GET: 11372 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11373 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11374 11375 // automatic forms here for usable basic auto site from browser. 11376 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11377 switch(cgi.request("_method", "GET")) { 11378 case "GET": 11379 static if(is(T : CollectionOf!(C), C)) { 11380 auto results = obj.index(); 11381 if(cgi.request("format", "html") == "html") { 11382 auto container = presenter.htmlContainer(); 11383 auto html = presenter.formatReturnValueAsHtml(results.results); 11384 container.appendHtml(` 11385 <form> 11386 <button type="submit" name="_method" value="POST">Create New</button> 11387 </form> 11388 `); 11389 11390 container.appendChild(html); 11391 cgi.write(container.parentDocument.toString, true); 11392 } else { 11393 cgi.setResponseContentType("application/json"); 11394 import arsd.jsvar; 11395 var json = var.emptyArray; 11396 foreach(r; results.results) { 11397 var o = var.emptyObject; 11398 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11399 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11400 o[memberName] = __traits(getMember, r, memberName); 11401 } 11402 11403 json ~= o; 11404 } 11405 cgi.write(json.toJson(), true); 11406 } 11407 } else { 11408 obj.show(urlId); 11409 writeObject(true); 11410 } 11411 break; 11412 case "PATCH": 11413 obj.load(urlId); 11414 goto case; 11415 case "PUT": 11416 case "POST": 11417 // an editing form for the object 11418 auto container = presenter.htmlContainer(); 11419 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11420 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11421 } else { 11422 auto form = presenter.createAutomaticFormForObject(obj); 11423 } 11424 form.attrs.method = "POST"; 11425 form.setValue("_method", cgi.request("_method", "GET")); 11426 container.appendChild(form); 11427 cgi.write(container.parentDocument.toString(), true); 11428 break; 11429 case "DELETE": 11430 // FIXME: a delete form for the object (can be phrased "are you sure?") 11431 auto container = presenter.htmlContainer(); 11432 container.appendHtml(` 11433 <form method="POST"> 11434 Are you sure you want to delete this item? 11435 <input type="hidden" name="_method" value="DELETE" /> 11436 <input type="submit" value="Yes, Delete It" /> 11437 </form> 11438 11439 `); 11440 cgi.write(container.parentDocument.toString(), true); 11441 break; 11442 default: 11443 cgi.write("bad method\n", true); 11444 } 11445 break; 11446 case Cgi.RequestMethod.POST: 11447 // this is to allow compatibility with HTML forms 11448 switch(cgi.request("_method", "POST")) { 11449 case "PUT": 11450 goto PUT; 11451 case "PATCH": 11452 goto PATCH; 11453 case "DELETE": 11454 goto DELETE; 11455 case "POST": 11456 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11457 auto p = new obj.PostProxy(); 11458 void specialApplyChanges() { 11459 applyChangesTemplate(cgi, p); 11460 } 11461 string n = p.create(&specialApplyChanges); 11462 } else { 11463 string n = obj.create(&applyChanges); 11464 } 11465 11466 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11467 cgi.setResponseLocation(newUrl); 11468 cgi.setResponseStatus("201 Created"); 11469 cgi.write(`The object has been created.`); 11470 break; 11471 default: 11472 cgi.write("bad method\n", true); 11473 } 11474 // FIXME this should be valid on the collection, but not the child.... 11475 // 303 See Other 11476 break; 11477 case Cgi.RequestMethod.PUT: 11478 PUT: 11479 obj.replace(urlId, &applyChanges); 11480 writeObject(false); 11481 break; 11482 case Cgi.RequestMethod.PATCH: 11483 PATCH: 11484 obj.update(urlId, &applyChanges, modifiedList); 11485 writeObject(false); 11486 break; 11487 case Cgi.RequestMethod.DELETE: 11488 DELETE: 11489 obj.remove(urlId); 11490 cgi.setResponseStatus("204 No Content"); 11491 break; 11492 default: 11493 // FIXME: OPTIONS, HEAD 11494 } 11495 catch(Throwable t) { 11496 presenter.presentExceptionAsHtml(cgi, t); 11497 } 11498 11499 return true; 11500 } 11501 11502 /+ 11503 struct SetOfFields(T) { 11504 private void[0][string] storage; 11505 void set(string what) { 11506 //storage[what] = 11507 } 11508 void unset(string what) {} 11509 void setAll() {} 11510 void unsetAll() {} 11511 bool isPresent(string what) { return false; } 11512 } 11513 +/ 11514 11515 /+ 11516 enum readonly; 11517 enum hideonindex; 11518 +/ 11519 11520 /++ 11521 Returns true if I recommend gzipping content of this type. You might 11522 want to call it from your Presenter classes before calling cgi.write. 11523 11524 --- 11525 cgi.setResponseContentType(yourContentType); 11526 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11527 cgi.write(yourData, true); 11528 --- 11529 11530 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11531 11532 11533 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11534 11535 History: 11536 Added January 28, 2023 (dub v11.0) 11537 +/ 11538 bool gzipRecommendedForContentType(string contentType) { 11539 if(contentType.startsWith("text/")) 11540 return true; 11541 if(contentType.startsWith("application/javascript")) 11542 return true; 11543 11544 return false; 11545 } 11546 11547 /++ 11548 Serves a static file. To be used with [dispatcher]. 11549 11550 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11551 +/ 11552 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11553 // https://baus.net/on-tcp_cork/ 11554 // man 2 sendfile 11555 assert(urlPrefix[0] == '/'); 11556 if(filename is null) 11557 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11558 if(contentType is null) { 11559 contentType = contentTypeFromFileExtension(filename); 11560 } 11561 11562 static struct DispatcherDetails { 11563 string filename; 11564 string contentType; 11565 } 11566 11567 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11568 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11569 cgi.setCache(true); 11570 cgi.setResponseContentType(details.contentType); 11571 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11572 cgi.write(std.file.read(details.filename), true); 11573 return true; 11574 } 11575 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11576 } 11577 11578 /++ 11579 Serves static data. To be used with [dispatcher]. 11580 11581 History: 11582 Added October 31, 2021 11583 +/ 11584 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11585 assert(urlPrefix[0] == '/'); 11586 if(contentType is null) { 11587 contentType = contentTypeFromFileExtension(urlPrefix); 11588 } 11589 11590 static struct DispatcherDetails { 11591 immutable(void)[] data; 11592 string contentType; 11593 } 11594 11595 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11596 cgi.setCache(true); 11597 cgi.setResponseContentType(details.contentType); 11598 cgi.write(details.data, true); 11599 return true; 11600 } 11601 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11602 } 11603 11604 string contentTypeFromFileExtension(string filename) { 11605 if(filename.endsWith(".png")) 11606 return "image/png"; 11607 if(filename.endsWith(".apng")) 11608 return "image/apng"; 11609 if(filename.endsWith(".svg")) 11610 return "image/svg+xml"; 11611 if(filename.endsWith(".jpg")) 11612 return "image/jpeg"; 11613 if(filename.endsWith(".html")) 11614 return "text/html"; 11615 if(filename.endsWith(".css")) 11616 return "text/css"; 11617 if(filename.endsWith(".js")) 11618 return "application/javascript"; 11619 if(filename.endsWith(".wasm")) 11620 return "application/wasm"; 11621 if(filename.endsWith(".mp3")) 11622 return "audio/mpeg"; 11623 if(filename.endsWith(".pdf")) 11624 return "application/pdf"; 11625 return null; 11626 } 11627 11628 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11629 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11630 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11631 assert(urlPrefix[0] == '/'); 11632 assert(urlPrefix[$-1] == '/'); 11633 11634 static struct DispatcherDetails { 11635 string directory; 11636 bool recursive; 11637 } 11638 11639 if(directory is null) 11640 directory = urlPrefix[1 .. $]; 11641 11642 if(directory.length == 0) 11643 directory = "./"; 11644 11645 assert(directory[$-1] == '/'); 11646 11647 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11648 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11649 11650 if(details.recursive) { 11651 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11652 if(file.indexOf("\\") != -1) 11653 return false; 11654 11655 import std.path; 11656 11657 file = std.path.buildNormalizedPath(file); 11658 enum upOneDir = ".." ~ std.path.dirSeparator; 11659 11660 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11661 if(file == ".." || file.startsWith(upOneDir)) 11662 return false; 11663 if(std.path.isAbsolute(file)) 11664 return false; 11665 11666 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11667 11668 // once it passes these filters it is probably ok. 11669 } else { 11670 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11671 return false; 11672 } 11673 11674 auto contentType = contentTypeFromFileExtension(file); 11675 11676 auto fn = details.directory ~ file; 11677 if(std.file.exists(fn)) { 11678 //if(contentType.indexOf("image/") == 0) 11679 //cgi.setCache(true); 11680 //else if(contentType.indexOf("audio/") == 0) 11681 cgi.setCache(true); 11682 cgi.setResponseContentType(contentType); 11683 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11684 cgi.write(std.file.read(fn), true); 11685 return true; 11686 } else { 11687 return false; 11688 } 11689 } 11690 11691 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11692 } 11693 11694 /++ 11695 Redirects one url to another 11696 11697 See_Also: [dispatcher], [serveStaticFile] 11698 +/ 11699 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11700 assert(urlPrefix[0] == '/'); 11701 static struct DispatcherDetails { 11702 string redirectTo; 11703 string code; 11704 } 11705 11706 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11707 cgi.setResponseLocation(details.redirectTo, true, details.code); 11708 return true; 11709 } 11710 11711 11712 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11713 } 11714 11715 /// Used exclusively with `dispatchTo` 11716 struct DispatcherData(Presenter) { 11717 Cgi cgi; /// You can use this cgi object. 11718 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11719 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11720 } 11721 11722 /++ 11723 Dispatches the URL to a specific function. 11724 +/ 11725 auto handleWith(alias handler)(string urlPrefix) { 11726 // cuz I'm too lazy to do it better right now 11727 static class Hack : WebObject { 11728 static import std.traits; 11729 @UrlName("") 11730 auto handle(std.traits.Parameters!handler args) { 11731 return handler(args); 11732 } 11733 } 11734 11735 return urlPrefix.serveApiInternal!Hack; 11736 } 11737 11738 /++ 11739 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11740 11741 --- 11742 bool other(DD)(DD dd) { 11743 return dd.dispatcher!( 11744 "/whatever".serveRedirect("/success"), 11745 "/api/".serveApi!MyClass 11746 ); 11747 } 11748 --- 11749 11750 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11751 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11752 Or, of course, you could just use the exact type in your own code. 11753 11754 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11755 good job. 11756 11757 11758 +/ 11759 auto dispatchTo(alias handler)(string urlPrefix) { 11760 assert(urlPrefix[0] == '/'); 11761 assert(urlPrefix[$-1] != '/'); 11762 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11763 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11764 } 11765 11766 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11767 } 11768 11769 /++ 11770 See [serveStaticFile] if you want to serve a file off disk. 11771 11772 History: 11773 Added January 28, 2023 (dub v11.0) 11774 +/ 11775 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11776 assert(urlPrefix[0] == '/'); 11777 11778 static struct DispatcherDetails { 11779 immutable(ubyte)[] data; 11780 string contentType; 11781 string filenameToSuggestAsDownload; 11782 } 11783 11784 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11785 cgi.setCache(true); 11786 cgi.setResponseContentType(details.contentType); 11787 if(details.filenameToSuggestAsDownload.length) 11788 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11789 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11790 cgi.write(details.data, true); 11791 return true; 11792 } 11793 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11794 } 11795 11796 /++ 11797 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11798 11799 History: 11800 Added January 28, 2023 (dub v11.0) 11801 +/ 11802 alias KeepExistingPresenter = typeof(null); 11803 11804 /++ 11805 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11806 this issues the given errorCode and stops processing. 11807 11808 --- 11809 bool hasAdminPermissions(Cgi cgi) { 11810 return true; 11811 } 11812 11813 mixin DispatcherMain!( 11814 "/admin".dispatchSubsection!( 11815 passFilterOrIssueError!(hasAdminPermissions, 403), 11816 KeepExistingPresenter, 11817 "/".serveApi!AdminFunctions 11818 ) 11819 ); 11820 --- 11821 11822 History: 11823 Added January 28, 2023 (dub v11.0) 11824 +/ 11825 template passFilterOrIssueError(alias filter, int errorCode) { 11826 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11827 if(filter(dd.cgi)) 11828 return true; 11829 dd.presenter.renderBasicError(dd.cgi, errorCode); 11830 return false; 11831 } 11832 } 11833 11834 /++ 11835 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11836 and then be dispatched to their own handlers. 11837 11838 --- 11839 /+ 11840 // a long-form filter function 11841 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11842 // you are permitted to call mutable methods on the Cgi object 11843 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11844 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11845 11846 if(checkPermissionOnRequest(dd.cgi)) { 11847 return true; // OK, allow processing to continue 11848 } else { 11849 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11850 return false; // and stop further processing into this subsection 11851 } 11852 } 11853 +/ 11854 11855 // but you can also do short-form filters: 11856 11857 bool permissionCheck(Cgi cgi) { 11858 return ("ok" in cgi.get) !is null; 11859 } 11860 11861 // handler for the subsection 11862 class AdminClass : WebObject { 11863 int foo() { return 5; } 11864 } 11865 11866 // handler for the main site 11867 class TheMainSite : WebObject {} 11868 11869 mixin DispatcherMain!( 11870 "/admin".dispatchSubsection!( 11871 // converts our short-form filter into a long-form filter 11872 passFilterOrIssueError!(permissionCheck, 403), 11873 // can use a new presenter if wanted for the subsection 11874 KeepExistingPresenter, 11875 // and then provide child route dispatchers 11876 "/".serveApi!AdminClass 11877 ), 11878 // and back to the top level 11879 "/".serveApi!TheMainSite 11880 ); 11881 --- 11882 11883 Note you can encapsulate sections in files like this: 11884 11885 --- 11886 auto adminDispatcher(string urlPrefix) { 11887 return urlPrefix.dispatchSubsection!( 11888 .... 11889 ); 11890 } 11891 11892 mixin DispatcherMain!( 11893 "/admin".adminDispatcher, 11894 // and so on 11895 ) 11896 --- 11897 11898 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11899 11900 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11901 11902 11903 History: 11904 Added January 28, 2023 (dub v11.0) 11905 +/ 11906 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11907 assert(urlPrefix[0] == '/'); 11908 assert(urlPrefix[$-1] != '/'); 11909 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11910 static if(!is(PreRequestFilter == typeof(null))) { 11911 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11912 return true; // we handled it by rejecting it 11913 } 11914 11915 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11916 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11917 } else { 11918 auto newPresenter = new NewPresenter(); 11919 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11920 } 11921 } 11922 11923 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11924 } 11925 11926 /++ 11927 A URL dispatcher. 11928 11929 --- 11930 if(cgi.dispatcher!( 11931 "/api/".serveApi!MyApiClass, 11932 "/objects/lol".serveRestObject!MyRestObject, 11933 "/file.js".serveStaticFile, 11934 "/admin/".dispatchTo!adminHandler 11935 )) return; 11936 --- 11937 11938 11939 You define a series of url prefixes followed by handlers. 11940 11941 You may want to do different pre- and post- processing there, for example, 11942 an authorization check and different page layout. You can use different 11943 presenters and different function chains. See [dispatchSubsection] for details. 11944 11945 [dispatchTo] will send the request to another function for handling. 11946 +/ 11947 template dispatcher(definitions...) { 11948 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11949 static if(is(Presenter == typeof(null))) { 11950 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11951 auto presenter = new GenericWebPresenter(); 11952 } else 11953 alias presenter = presenterArg; 11954 11955 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11956 } 11957 11958 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11959 // I can prolly make this more efficient later but meh. 11960 foreach(definition; definitions) { 11961 if(definition.rejectFurther) { 11962 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11963 auto ret = definition.handler( 11964 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11965 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11966 if(ret) 11967 return true; 11968 } 11969 } else if( 11970 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11971 // cgi.d dispatcher urls must be complete or have a /; 11972 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11973 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11974 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11975 ) { 11976 auto ret = definition.handler( 11977 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11978 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11979 if(ret) 11980 return true; 11981 } 11982 } 11983 return false; 11984 } 11985 } 11986 11987 }); 11988 11989 private struct StackBuffer { 11990 char[1024] initial = void; 11991 char[] buffer; 11992 size_t position; 11993 11994 this(int a) { 11995 buffer = initial[]; 11996 position = 0; 11997 } 11998 11999 void add(in char[] what) { 12000 if(position + what.length > buffer.length) 12001 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 12002 buffer[position .. position + what.length] = what[]; 12003 position += what.length; 12004 } 12005 12006 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 12007 add(w1); 12008 add(w2); 12009 add(w3); 12010 } 12011 12012 void add(long v) { 12013 char[16] buffer = void; 12014 auto pos = buffer.length; 12015 bool negative; 12016 if(v < 0) { 12017 negative = true; 12018 v = -v; 12019 } 12020 do { 12021 buffer[--pos] = cast(char) (v % 10 + '0'); 12022 v /= 10; 12023 } while(v); 12024 12025 if(negative) 12026 buffer[--pos] = '-'; 12027 12028 auto res = buffer[pos .. $]; 12029 12030 add(res[]); 12031 } 12032 12033 char[] get() @nogc { 12034 return buffer[0 .. position]; 12035 } 12036 } 12037 12038 // duplicated in http2.d 12039 private static string getHttpCodeText(int code) pure nothrow @nogc { 12040 switch(code) { 12041 case 200: return "200 OK"; 12042 case 201: return "201 Created"; 12043 case 202: return "202 Accepted"; 12044 case 203: return "203 Non-Authoritative Information"; 12045 case 204: return "204 No Content"; 12046 case 205: return "205 Reset Content"; 12047 case 206: return "206 Partial Content"; 12048 // 12049 case 300: return "300 Multiple Choices"; 12050 case 301: return "301 Moved Permanently"; 12051 case 302: return "302 Found"; 12052 case 303: return "303 See Other"; 12053 case 304: return "304 Not Modified"; 12054 case 305: return "305 Use Proxy"; 12055 case 307: return "307 Temporary Redirect"; 12056 case 308: return "308 Permanent Redirect"; 12057 12058 // 12059 case 400: return "400 Bad Request"; 12060 case 401: return "401 Unauthorized"; 12061 case 402: return "402 Payment Required"; 12062 case 403: return "403 Forbidden"; 12063 case 404: return "404 Not Found"; 12064 case 405: return "405 Method Not Allowed"; 12065 case 406: return "406 Not Acceptable"; 12066 case 407: return "407 Proxy Authentication Required"; 12067 case 408: return "408 Request Timeout"; 12068 case 409: return "409 Conflict"; 12069 case 410: return "410 Gone"; 12070 case 411: return "411 Length Required"; 12071 case 412: return "412 Precondition Failed"; 12072 case 413: return "413 Payload Too Large"; 12073 case 414: return "414 URI Too Long"; 12074 case 415: return "415 Unsupported Media Type"; 12075 case 416: return "416 Range Not Satisfiable"; 12076 case 417: return "417 Expectation Failed"; 12077 case 418: return "418 I'm a teapot"; 12078 case 421: return "421 Misdirected Request"; 12079 case 422: return "422 Unprocessable Entity (WebDAV)"; 12080 case 423: return "423 Locked (WebDAV)"; 12081 case 424: return "424 Failed Dependency (WebDAV)"; 12082 case 425: return "425 Too Early"; 12083 case 426: return "426 Upgrade Required"; 12084 case 428: return "428 Precondition Required"; 12085 case 431: return "431 Request Header Fields Too Large"; 12086 case 451: return "451 Unavailable For Legal Reasons"; 12087 12088 case 500: return "500 Internal Server Error"; 12089 case 501: return "501 Not Implemented"; 12090 case 502: return "502 Bad Gateway"; 12091 case 503: return "503 Service Unavailable"; 12092 case 504: return "504 Gateway Timeout"; 12093 case 505: return "505 HTTP Version Not Supported"; 12094 case 506: return "506 Variant Also Negotiates"; 12095 case 507: return "507 Insufficient Storage (WebDAV)"; 12096 case 508: return "508 Loop Detected (WebDAV)"; 12097 case 510: return "510 Not Extended"; 12098 case 511: return "511 Network Authentication Required"; 12099 // 12100 default: assert(0, "Unsupported http code"); 12101 } 12102 } 12103 12104 12105 /+ 12106 /++ 12107 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12108 12109 It relies on jsvar.d and dom.d. 12110 12111 12112 You can get javascript out of it to call. The generated functions need to look 12113 like 12114 12115 function name(a,b,c,d,e) { 12116 return _call("name", {"realName":a,"sds":b}); 12117 } 12118 12119 And _call returns an object you can call or set up or whatever. 12120 +/ 12121 bool apiDispatcher()(Cgi cgi) { 12122 import arsd.jsvar; 12123 import arsd.dom; 12124 } 12125 +/ 12126 version(linux) 12127 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12128 /* 12129 Copyright: Adam D. Ruppe, 2008 - 2023 12130 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12131 Authors: Adam D. Ruppe 12132 12133 Copyright Adam D. Ruppe 2008 - 2023. 12134 Distributed under the Boost Software License, Version 1.0. 12135 (See accompanying file LICENSE_1_0.txt or copy at 12136 http://www.boost.org/LICENSE_1_0.txt) 12137 */