1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 static import arsd.core; 490 version(Posix) 491 import arsd.core : makeNonBlocking; 492 493 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 494 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 495 496 /++ 497 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 498 +/ 499 version(Demo) 500 unittest { 501 import arsd.cgi; 502 503 mixin DispatcherMain!( 504 "/".serveStaticFileDirectory(null, true) 505 ); 506 } 507 508 /++ 509 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 510 +/ 511 version(Demo) 512 unittest { 513 import arsd.cgi; 514 515 void requestHandler(Cgi cgi) { 516 cgi.dispatcher!( 517 "/".serveStaticFileDirectory(null, true) 518 ); 519 } 520 521 // mixin GenericMain!requestHandler would add this function: 522 void main(string[] args) { 523 // this is all the content of [cgiMainImpl] which you can also call 524 525 // cgi.d embeds a few add on functions like real time event forwarders 526 // and session servers it can run in other processes. this spawns them, if needed. 527 if(tryAddonServers(args)) 528 return; 529 530 // cgi.d allows you to easily simulate http requests from the command line, 531 // without actually starting a server. this function will do that. 532 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 533 return; 534 535 RequestServer server; 536 // you can change the default port here if you like 537 // server.listeningPort = 9000; 538 539 // then call this to let the command line args override your default 540 server.configureFromCommandLine(args); 541 542 // here is where you could print out the listeningPort to the user if you wanted 543 544 // and serve the request(s) according to the compile configuration 545 server.serve!(requestHandler)(); 546 547 // or you could explicitly choose a serve mode like this: 548 // server.serveEmbeddedHttp!requestHandler(); 549 } 550 } 551 552 /++ 553 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 554 otherwise run through the rest of the internal mechanisms to call your functions without actually 555 spinning up a server. 556 +/ 557 version(Demo) 558 unittest { 559 import arsd.cgi; 560 561 void requestHandler(Cgi cgi) { 562 563 } 564 565 // D doesn't let me embed a unittest inside an example unittest 566 // so this is a function, but you can do it however in your real program 567 /* unittest */ void runTests() { 568 auto tester = new CgiTester(&requestHandler); 569 570 auto response = tester.GET("/"); 571 assert(response.code == 200); 572 } 573 } 574 575 static import std.file; 576 577 // for a single thread, linear request thing, use: 578 // -version=embedded_httpd_threads -version=cgi_no_threads 579 580 version(Posix) { 581 version(CRuntime_Musl) { 582 583 } else version(minimal) { 584 585 } else { 586 version(FreeBSD) { 587 // I never implemented the fancy stuff there either 588 } else { 589 version=with_breaking_cgi_features; 590 version=with_sendfd; 591 version=with_addon_servers; 592 } 593 } 594 } 595 596 version(Windows) { 597 version(minimal) { 598 599 } else { 600 // not too concerned about gdc here since the mingw version is fairly new as well 601 version=with_breaking_cgi_features; 602 } 603 } 604 605 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 606 void cloexec(int fd) { 607 version(Posix) { 608 import core.sys.posix.fcntl; 609 fcntl(fd, F_SETFD, FD_CLOEXEC); 610 } 611 } 612 613 void cloexec(Socket s) { 614 version(Posix) { 615 import core.sys.posix.fcntl; 616 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 617 } 618 } 619 620 version(embedded_httpd_hybrid) { 621 version=embedded_httpd_threads; 622 version(cgi_no_fork) {} else version(Posix) 623 version=cgi_use_fork; 624 version=cgi_use_fiber; 625 } 626 627 version(cgi_use_fork) 628 enum cgi_use_fork_default = true; 629 else 630 enum cgi_use_fork_default = false; 631 632 // the servers must know about the connections to talk to them; the interfaces are vital 633 version(with_addon_servers) 634 version=with_addon_servers_connections; 635 636 version(embedded_httpd) { 637 version(linux) 638 version=embedded_httpd_processes; 639 else { 640 version=embedded_httpd_threads; 641 } 642 643 /* 644 version(with_openssl) { 645 pragma(lib, "crypto"); 646 pragma(lib, "ssl"); 647 } 648 */ 649 } 650 651 version(embedded_httpd_processes) 652 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 653 654 version(embedded_httpd_threads) { 655 // unless the user overrides the default.. 656 version(cgi_session_server_process) 657 {} 658 else 659 version=cgi_embedded_sessions; 660 } 661 version(scgi) { 662 // unless the user overrides the default.. 663 version(cgi_session_server_process) 664 {} 665 else 666 version=cgi_embedded_sessions; 667 } 668 669 // fall back if the other is not defined so we can cleanly version it below 670 version(cgi_embedded_sessions) {} 671 else version=cgi_session_server_process; 672 673 674 version=cgi_with_websocket; 675 676 enum long defaultMaxContentLength = 5_000_000; 677 678 /* 679 680 To do a file download offer in the browser: 681 682 cgi.setResponseContentType("text/csv"); 683 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 684 */ 685 686 // FIXME: the location header is supposed to be an absolute url I guess. 687 688 // FIXME: would be cool to flush part of a dom document before complete 689 // somehow in here and dom.d. 690 691 692 // these are public so you can mixin GenericMain. 693 // FIXME: use a function level import instead! 694 public import std.string; 695 public import std.stdio; 696 public import std.conv; 697 import std.uri; 698 import std.uni; 699 import std.algorithm.comparison; 700 import std.algorithm.searching; 701 import std.exception; 702 import std.base64; 703 static import std.algorithm; 704 import std.datetime; 705 import std.range; 706 707 import std.process; 708 709 import std.zlib; 710 711 712 T[] consume(T)(T[] range, int count) { 713 if(count > range.length) 714 count = range.length; 715 return range[count..$]; 716 } 717 718 int locationOf(T)(T[] data, string item) { 719 const(ubyte[]) d = cast(const(ubyte[])) data; 720 const(ubyte[]) i = cast(const(ubyte[])) item; 721 722 // this is a vague sanity check to ensure we aren't getting insanely 723 // sized input that will infinite loop below. it should never happen; 724 // even huge file uploads ought to come in smaller individual pieces. 725 if(d.length > (int.max/2)) 726 throw new Exception("excessive block of input"); 727 728 for(int a = 0; a < d.length; a++) { 729 if(a + i.length > d.length) 730 return -1; 731 if(d[a..a+i.length] == i) 732 return a; 733 } 734 735 return -1; 736 } 737 738 /// If you are doing a custom cgi class, mixing this in can take care of 739 /// the required constructors for you 740 mixin template ForwardCgiConstructors() { 741 this(long maxContentLength = defaultMaxContentLength, 742 string[string] env = null, 743 const(ubyte)[] delegate() readdata = null, 744 void delegate(const(ubyte)[]) _rawDataOutput = null, 745 void delegate() _flush = null 746 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 747 748 this(string[] args) { super(args); } 749 750 this( 751 BufferedInputRange inputData, 752 string address, ushort _port, 753 int pathInfoStarts = 0, 754 bool _https = false, 755 void delegate(const(ubyte)[]) _rawDataOutput = null, 756 void delegate() _flush = null, 757 // this pointer tells if the connection is supposed to be closed after we handle this 758 bool* closeConnection = null) 759 { 760 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 761 } 762 763 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 764 } 765 766 /// thrown when a connection is closed remotely while we waiting on data from it 767 class ConnectionClosedException : Exception { 768 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 769 super(message, file, line, next); 770 } 771 } 772 773 774 version(Windows) { 775 // FIXME: ugly hack to solve stdin exception problems on Windows: 776 // reading stdin results in StdioException (Bad file descriptor) 777 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 778 private struct stdin { 779 struct ByChunk { // Replicates std.stdio.ByChunk 780 private: 781 ubyte[] chunk_; 782 public: 783 this(size_t size) 784 in { 785 assert(size, "size must be larger than 0"); 786 } 787 do { 788 chunk_ = new ubyte[](size); 789 popFront(); 790 } 791 792 @property bool empty() const { 793 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 794 } 795 @property nothrow ubyte[] front() { return chunk_; } 796 void popFront() { 797 enforce(!empty, "Cannot call popFront on empty range"); 798 chunk_ = stdin.rawRead(chunk_); 799 } 800 } 801 802 import core.sys.windows.windows; 803 static: 804 805 T[] rawRead(T)(T[] buf) { 806 uint bytesRead; 807 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 808 809 if (!result) { 810 auto err = GetLastError(); 811 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 812 return buf[0..0]; 813 // Some other error, throw it 814 815 char* buffer; 816 scope(exit) LocalFree(buffer); 817 818 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 819 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 820 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 821 throw new Exception(to!string(buffer)); 822 } 823 enforce(!(bytesRead % T.sizeof), "I/O error"); 824 return buf[0..bytesRead / T.sizeof]; 825 } 826 827 auto byChunk(size_t sz) { return ByChunk(sz); } 828 829 void close() { 830 std.stdio.stdin.close; 831 } 832 } 833 } 834 835 /// The main interface with the web request 836 class Cgi { 837 public: 838 /// the methods a request can be 839 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 840 // these are defined in the standard, but idk if they are useful for anything 841 OPTIONS, TRACE, CONNECT, 842 // These seem new, I have only recently seen them 843 PATCH, MERGE, 844 // this is an extension for when the method is not specified and you want to assume 845 CommandLine } 846 847 848 /+ 849 /++ 850 Cgi provides a per-request memory pool 851 852 +/ 853 void[] allocateMemory(size_t nBytes) { 854 855 } 856 857 /// ditto 858 void[] reallocateMemory(void[] old, size_t nBytes) { 859 860 } 861 862 /// ditto 863 void freeMemory(void[] memory) { 864 865 } 866 +/ 867 868 869 /* 870 import core.runtime; 871 auto args = Runtime.args(); 872 873 we can call the app a few ways: 874 875 1) set up the environment variables and call the app (manually simulating CGI) 876 2) simulate a call automatically: 877 ./app method 'uri' 878 879 for example: 880 ./app get /path?arg arg2=something 881 882 Anything on the uri is treated as query string etc 883 884 on get method, further args are appended to the query string (encoded automatically) 885 on post method, further args are done as post 886 887 888 @name means import from file "name". if name == -, it uses stdin 889 (so info=@- means set info to the value of stdin) 890 891 892 Other arguments include: 893 --cookie name=value (these are all concated together) 894 --header 'X-Something: cool' 895 --referrer 'something' 896 --port 80 897 --remote-address some.ip.address.here 898 --https yes 899 --user-agent 'something' 900 --userpass 'user:pass' 901 --authorization 'Basic base64encoded_user:pass' 902 --accept 'content' // FIXME: better example 903 --last-event-id 'something' 904 --host 'something.com' 905 906 Non-simulation arguments: 907 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 908 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 909 910 */ 911 912 /** Initializes it with command line arguments (for easy testing) */ 913 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 914 rawDataOutput = _rawDataOutput; 915 // these are all set locally so the loop works 916 // without triggering errors in dmd 2.064 917 // we go ahead and set them at the end of it to the this version 918 int port; 919 string referrer; 920 string remoteAddress; 921 string userAgent; 922 string authorization; 923 string origin; 924 string accept; 925 string lastEventId; 926 bool https; 927 string host; 928 RequestMethod requestMethod; 929 string requestUri; 930 string pathInfo; 931 string queryString; 932 933 bool lookingForMethod; 934 bool lookingForUri; 935 string nextArgIs; 936 937 string _cookie; 938 string _queryString; 939 string[][string] _post; 940 string[string] _headers; 941 942 string[] breakUp(string s) { 943 string k, v; 944 auto idx = s.indexOf("="); 945 if(idx == -1) { 946 k = s; 947 } else { 948 k = s[0 .. idx]; 949 v = s[idx + 1 .. $]; 950 } 951 952 return [k, v]; 953 } 954 955 lookingForMethod = true; 956 957 scriptName = args[0]; 958 scriptFileName = args[0]; 959 960 environmentVariables = cast(const) environment.toAA; 961 962 foreach(arg; args[1 .. $]) { 963 if(arg.startsWith("--")) { 964 nextArgIs = arg[2 .. $]; 965 } else if(nextArgIs.length) { 966 if (nextArgIs == "cookie") { 967 auto info = breakUp(arg); 968 if(_cookie.length) 969 _cookie ~= "; "; 970 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 971 } 972 else if (nextArgIs == "port") { 973 port = to!int(arg); 974 } 975 else if (nextArgIs == "referrer") { 976 referrer = arg; 977 } 978 else if (nextArgIs == "remote-address") { 979 remoteAddress = arg; 980 } 981 else if (nextArgIs == "user-agent") { 982 userAgent = arg; 983 } 984 else if (nextArgIs == "authorization") { 985 authorization = arg; 986 } 987 else if (nextArgIs == "userpass") { 988 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 989 } 990 else if (nextArgIs == "origin") { 991 origin = arg; 992 } 993 else if (nextArgIs == "accept") { 994 accept = arg; 995 } 996 else if (nextArgIs == "last-event-id") { 997 lastEventId = arg; 998 } 999 else if (nextArgIs == "https") { 1000 if(arg == "yes") 1001 https = true; 1002 } 1003 else if (nextArgIs == "header") { 1004 string thing, other; 1005 auto idx = arg.indexOf(":"); 1006 if(idx == -1) 1007 throw new Exception("need a colon in a http header"); 1008 thing = arg[0 .. idx]; 1009 other = arg[idx + 1.. $]; 1010 _headers[thing.strip.toLower()] = other.strip; 1011 } 1012 else if (nextArgIs == "host") { 1013 host = arg; 1014 } 1015 // else 1016 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1017 1018 nextArgIs = null; 1019 } else if(lookingForMethod) { 1020 lookingForMethod = false; 1021 lookingForUri = true; 1022 1023 if(arg.asLowerCase().equal("commandline")) 1024 requestMethod = RequestMethod.CommandLine; 1025 else 1026 requestMethod = to!RequestMethod(arg.toUpper()); 1027 } else if(lookingForUri) { 1028 lookingForUri = false; 1029 1030 requestUri = arg; 1031 1032 auto idx = arg.indexOf("?"); 1033 if(idx == -1) 1034 pathInfo = arg; 1035 else { 1036 pathInfo = arg[0 .. idx]; 1037 _queryString = arg[idx + 1 .. $]; 1038 } 1039 } else { 1040 // it is an argument of some sort 1041 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1042 auto parts = breakUp(arg); 1043 _post[parts[0]] ~= parts[1]; 1044 allPostNamesInOrder ~= parts[0]; 1045 allPostValuesInOrder ~= parts[1]; 1046 } else { 1047 if(_queryString.length) 1048 _queryString ~= "&"; 1049 auto parts = breakUp(arg); 1050 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1051 } 1052 } 1053 } 1054 1055 acceptsGzip = false; 1056 keepAliveRequested = false; 1057 requestHeaders = cast(immutable) _headers; 1058 1059 cookie = _cookie; 1060 cookiesArray = getCookieArray(); 1061 cookies = keepLastOf(cookiesArray); 1062 1063 queryString = _queryString; 1064 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1065 get = keepLastOf(getArray); 1066 1067 postArray = cast(immutable) _post; 1068 post = keepLastOf(_post); 1069 1070 // FIXME 1071 filesArray = null; 1072 files = null; 1073 1074 isCalledWithCommandLineArguments = true; 1075 1076 this.port = port; 1077 this.referrer = referrer; 1078 this.remoteAddress = remoteAddress; 1079 this.userAgent = userAgent; 1080 this.authorization = authorization; 1081 this.origin = origin; 1082 this.accept = accept; 1083 this.lastEventId = lastEventId; 1084 this.https = https; 1085 this.host = host; 1086 this.requestMethod = requestMethod; 1087 this.requestUri = requestUri; 1088 this.pathInfo = pathInfo; 1089 this.queryString = queryString; 1090 this.postBody = null; 1091 } 1092 1093 private { 1094 string[] allPostNamesInOrder; 1095 string[] allPostValuesInOrder; 1096 string[] allGetNamesInOrder; 1097 string[] allGetValuesInOrder; 1098 } 1099 1100 CgiConnectionHandle getOutputFileHandle() { 1101 return _outputFileHandle; 1102 } 1103 1104 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1105 1106 /** Initializes it using a CGI or CGI-like interface */ 1107 this(long maxContentLength = defaultMaxContentLength, 1108 // use this to override the environment variable listing 1109 in string[string] env = null, 1110 // and this should return a chunk of data. return empty when done 1111 const(ubyte)[] delegate() readdata = null, 1112 // finally, use this to do custom output if needed 1113 void delegate(const(ubyte)[]) _rawDataOutput = null, 1114 // to flush teh custom output 1115 void delegate() _flush = null 1116 ) 1117 { 1118 1119 // these are all set locally so the loop works 1120 // without triggering errors in dmd 2.064 1121 // we go ahead and set them at the end of it to the this version 1122 int port; 1123 string referrer; 1124 string remoteAddress; 1125 string userAgent; 1126 string authorization; 1127 string origin; 1128 string accept; 1129 string lastEventId; 1130 bool https; 1131 string host; 1132 RequestMethod requestMethod; 1133 string requestUri; 1134 string pathInfo; 1135 string queryString; 1136 1137 1138 1139 isCalledWithCommandLineArguments = false; 1140 rawDataOutput = _rawDataOutput; 1141 flushDelegate = _flush; 1142 auto getenv = delegate string(string var) { 1143 if(env is null) 1144 return std.process.environment.get(var); 1145 auto e = var in env; 1146 if(e is null) 1147 return null; 1148 return *e; 1149 }; 1150 1151 environmentVariables = env is null ? 1152 cast(const) environment.toAA : 1153 env; 1154 1155 // fetching all the request headers 1156 string[string] requestHeadersHere; 1157 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1158 if(k.startsWith("HTTP_")) { 1159 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1160 } 1161 } 1162 1163 this.requestHeaders = assumeUnique(requestHeadersHere); 1164 1165 requestUri = getenv("REQUEST_URI"); 1166 1167 cookie = getenv("HTTP_COOKIE"); 1168 cookiesArray = getCookieArray(); 1169 cookies = keepLastOf(cookiesArray); 1170 1171 referrer = getenv("HTTP_REFERER"); 1172 userAgent = getenv("HTTP_USER_AGENT"); 1173 remoteAddress = getenv("REMOTE_ADDR"); 1174 host = getenv("HTTP_HOST"); 1175 pathInfo = getenv("PATH_INFO"); 1176 1177 queryString = getenv("QUERY_STRING"); 1178 scriptName = getenv("SCRIPT_NAME"); 1179 { 1180 import core.runtime; 1181 auto sfn = getenv("SCRIPT_FILENAME"); 1182 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1183 } 1184 1185 bool iis = false; 1186 1187 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1188 if(requestUri.length == 0) { 1189 // IIS sometimes includes the script name as part of the path info - we don't want that 1190 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1191 pathInfo = pathInfo[scriptName.length .. $]; 1192 1193 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1194 1195 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1196 1197 // FIXME: this works for apache and iis... but what about others? 1198 } 1199 1200 1201 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1202 getArray = assumeUnique(ugh); 1203 get = keepLastOf(getArray); 1204 1205 1206 // NOTE: on shitpache, you need to specifically forward this 1207 authorization = getenv("HTTP_AUTHORIZATION"); 1208 // this is a hack because Apache is a shitload of fuck and 1209 // refuses to send the real header to us. Compatible 1210 // programs should send both the standard and X- versions 1211 1212 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1213 // unnecessary with mod_rewrite, so it is commented 1214 1215 //if(authorization.length == 0) // if the std is there, use it 1216 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1217 1218 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1219 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1220 port = to!int(getenv("SERVER_PORT")); 1221 else 1222 port = 0; // this was probably called from the command line 1223 1224 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1225 if(ae.length && ae.indexOf("gzip") != -1) 1226 acceptsGzip = true; 1227 1228 accept = getenv("HTTP_ACCEPT"); 1229 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1230 1231 auto ka = getenv("HTTP_CONNECTION"); 1232 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1233 keepAliveRequested = true; 1234 1235 auto or = getenv("HTTP_ORIGIN"); 1236 origin = or; 1237 1238 auto rm = getenv("REQUEST_METHOD"); 1239 if(rm.length) 1240 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1241 else 1242 requestMethod = RequestMethod.CommandLine; 1243 1244 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1245 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1246 1247 // FIXME: DOCUMENT_ROOT? 1248 1249 // FIXME: what about PUT? 1250 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1251 version(preserveData) // a hack to make forwarding simpler 1252 immutable(ubyte)[] data; 1253 size_t amountReceived = 0; 1254 auto contentType = getenv("CONTENT_TYPE"); 1255 1256 // FIXME: is this ever not going to be set? I guess it depends 1257 // on if the server de-chunks and buffers... seems like it has potential 1258 // to be slow if they did that. The spec says it is always there though. 1259 // And it has worked reliably for me all year in the live environment, 1260 // but some servers might be different. 1261 auto cls = getenv("CONTENT_LENGTH"); 1262 auto contentLength = to!size_t(cls.length ? cls : "0"); 1263 1264 immutable originalContentLength = contentLength; 1265 if(contentLength) { 1266 if(maxContentLength > 0 && contentLength > maxContentLength) { 1267 setResponseStatus("413 Request entity too large"); 1268 write("You tried to upload a file that is too large."); 1269 close(); 1270 throw new Exception("POST too large"); 1271 } 1272 prepareForIncomingDataChunks(contentType, contentLength); 1273 1274 1275 int processChunk(in ubyte[] chunk) { 1276 if(chunk.length > contentLength) { 1277 handleIncomingDataChunk(chunk[0..contentLength]); 1278 amountReceived += contentLength; 1279 contentLength = 0; 1280 return 1; 1281 } else { 1282 handleIncomingDataChunk(chunk); 1283 contentLength -= chunk.length; 1284 amountReceived += chunk.length; 1285 } 1286 if(contentLength == 0) 1287 return 1; 1288 1289 onRequestBodyDataReceived(amountReceived, originalContentLength); 1290 return 0; 1291 } 1292 1293 1294 if(readdata is null) { 1295 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1296 if(processChunk(chunk)) 1297 break; 1298 } else { 1299 // we have a custom data source.. 1300 auto chunk = readdata(); 1301 while(chunk.length) { 1302 if(processChunk(chunk)) 1303 break; 1304 chunk = readdata(); 1305 } 1306 } 1307 1308 onRequestBodyDataReceived(amountReceived, originalContentLength); 1309 postArray = assumeUnique(pps._post); 1310 filesArray = assumeUnique(pps._files); 1311 files = keepLastOf(filesArray); 1312 post = keepLastOf(postArray); 1313 this.postBody = pps.postBody; 1314 cleanUpPostDataState(); 1315 } 1316 1317 version(preserveData) 1318 originalPostData = data; 1319 } 1320 // fixme: remote_user script name 1321 1322 1323 this.port = port; 1324 this.referrer = referrer; 1325 this.remoteAddress = remoteAddress; 1326 this.userAgent = userAgent; 1327 this.authorization = authorization; 1328 this.origin = origin; 1329 this.accept = accept; 1330 this.lastEventId = lastEventId; 1331 this.https = https; 1332 this.host = host; 1333 this.requestMethod = requestMethod; 1334 this.requestUri = requestUri; 1335 this.pathInfo = pathInfo; 1336 this.queryString = queryString; 1337 } 1338 1339 /// Cleans up any temporary files. Do not use the object 1340 /// after calling this. 1341 /// 1342 /// NOTE: it is called automatically by GenericMain 1343 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1344 void dispose() { 1345 foreach(file; files) { 1346 if(!file.contentInMemory) 1347 if(std.file.exists(file.contentFilename)) 1348 std.file.remove(file.contentFilename); 1349 } 1350 } 1351 1352 private { 1353 struct PostParserState { 1354 string contentType; 1355 string boundary; 1356 string localBoundary; // the ones used at the end or something lol 1357 bool isMultipart; 1358 bool needsSavedBody; 1359 1360 ulong expectedLength; 1361 ulong contentConsumed; 1362 immutable(ubyte)[] buffer; 1363 1364 // multipart parsing state 1365 int whatDoWeWant; 1366 bool weHaveAPart; 1367 string[] thisOnesHeaders; 1368 immutable(ubyte)[] thisOnesData; 1369 1370 string postBody; 1371 1372 UploadedFile piece; 1373 bool isFile = false; 1374 1375 size_t memoryCommitted; 1376 1377 // do NOT keep mutable references to these anywhere! 1378 // I assume they are unique in the constructor once we're all done getting data. 1379 string[][string] _post; 1380 UploadedFile[][string] _files; 1381 } 1382 1383 PostParserState pps; 1384 } 1385 1386 /// This represents a file the user uploaded via a POST request. 1387 static struct UploadedFile { 1388 /// If you want to create one of these structs for yourself from some data, 1389 /// use this function. 1390 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1391 Cgi.UploadedFile f; 1392 f.filename = name; 1393 f.content = cast(immutable(ubyte)[]) data; 1394 f.contentInMemory = true; 1395 return f; 1396 } 1397 1398 string name; /// The name of the form element. 1399 string filename; /// The filename the user set. 1400 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1401 1402 /** 1403 For small files, cgi.d will buffer the uploaded file in memory, and make it 1404 directly accessible to you through the content member. I find this very convenient 1405 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1406 often want to inspect and modify the file anyway!) 1407 1408 I find the file is very large, it is undesirable to eat that much memory just 1409 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1410 to the constructor so they are accepted, cgi.d will write the content to a temporary 1411 file that you can re-read later. 1412 1413 You can override this behavior by subclassing Cgi and overriding the protected 1414 handlePostChunk method. Note that the object is not initialized when you 1415 write that method - the http headers are available, but the cgi.post method 1416 is not. You may parse the file as it streams in using this method. 1417 1418 1419 Anyway, if the file is small enough to be in memory, contentInMemory will be 1420 set to true, and the content is available in the content member. 1421 1422 If not, contentInMemory will be set to false, and the content saved in a file, 1423 whose name will be available in the contentFilename member. 1424 1425 1426 Tip: if you know you are always dealing with small files, and want the convenience 1427 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1428 a large file comes in, it simply throws an exception (and HTTP error response) 1429 instead of trying to handle it. 1430 1431 The default value of maxContentLength in the constructor is for small files. 1432 */ 1433 bool contentInMemory = true; // the default ought to always be true 1434 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1435 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1436 1437 /// 1438 ulong fileSize() const { 1439 if(contentInMemory) 1440 return content.length; 1441 import std.file; 1442 return std.file.getSize(contentFilename); 1443 1444 } 1445 1446 /// 1447 void writeToFile(string filenameToSaveTo) const { 1448 import std.file; 1449 if(contentInMemory) 1450 std.file.write(filenameToSaveTo, content); 1451 else 1452 std.file.rename(contentFilename, filenameToSaveTo); 1453 } 1454 } 1455 1456 // given a content type and length, decide what we're going to do with the data.. 1457 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1458 pps.expectedLength = contentLength; 1459 1460 auto terminator = contentType.indexOf(";"); 1461 if(terminator == -1) 1462 terminator = contentType.length; 1463 1464 pps.contentType = contentType[0 .. terminator]; 1465 auto b = contentType[terminator .. $]; 1466 if(b.length) { 1467 auto idx = b.indexOf("boundary="); 1468 if(idx != -1) { 1469 pps.boundary = b[idx + "boundary=".length .. $]; 1470 pps.localBoundary = "\r\n--" ~ pps.boundary; 1471 } 1472 } 1473 1474 // while a content type SHOULD be sent according to the RFC, it is 1475 // not required. We're told we SHOULD guess by looking at the content 1476 // but it seems to me that this only happens when it is urlencoded. 1477 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1478 pps.isMultipart = false; 1479 pps.needsSavedBody = false; 1480 } else if(pps.contentType == "multipart/form-data") { 1481 pps.isMultipart = true; 1482 enforce(pps.boundary.length, "no boundary"); 1483 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1484 // save the body so the application can handle it 1485 pps.isMultipart = false; 1486 pps.needsSavedBody = true; 1487 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1488 // save the body so the application can handle it 1489 pps.needsSavedBody = true; 1490 pps.isMultipart = false; 1491 } else { 1492 // the rest is 100% handled by the application. just save the body and send it to them 1493 pps.needsSavedBody = true; 1494 pps.isMultipart = false; 1495 } 1496 } 1497 1498 // handles streaming POST data. If you handle some other content type, you should 1499 // override this. If the data isn't the content type you want, you ought to call 1500 // super.handleIncomingDataChunk so regular forms and files still work. 1501 1502 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1503 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1504 // input anyway, so I'm not going to get too worked up about it right now. 1505 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1506 if(chunk.length == 0) 1507 return; 1508 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1509 // if we're passed big chunks, it might throw unnecessarily. 1510 // just pass it smaller chunks at a time. 1511 if(pps.isMultipart) { 1512 // multipart/form-data 1513 1514 1515 // FIXME: this might want to be factored out and factorized 1516 // need to make sure the stream hooks actually work. 1517 void pieceHasNewContent() { 1518 // we just grew the piece's buffer. Do we have to switch to file backing? 1519 if(pps.piece.contentInMemory) { 1520 if(pps.piece.content.length <= 10 * 1024 * 1024) 1521 // meh, I'm ok with it. 1522 return; 1523 else { 1524 // this is too big. 1525 if(!pps.isFile) 1526 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1527 else { 1528 // a file this large is probably acceptable though... let's use a backing file. 1529 pps.piece.contentInMemory = false; 1530 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1531 1532 int count = 0; 1533 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1534 // odds are this loop will never be entered, but we want it just in case. 1535 while(std.file.exists(pps.piece.contentFilename)) { 1536 count++; 1537 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1538 } 1539 // I hope this creates the file pretty quickly, or the loop might be useless... 1540 // FIXME: maybe I should write some kind of custom transaction here. 1541 std.file.write(pps.piece.contentFilename, pps.piece.content); 1542 1543 pps.piece.content = null; 1544 } 1545 } 1546 } else { 1547 // it's already in a file, so just append it to what we have 1548 if(pps.piece.content.length) { 1549 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1550 std.file.append(pps.piece.contentFilename, pps.piece.content); 1551 pps.piece.content = null; 1552 } 1553 } 1554 } 1555 1556 1557 void commitPart() { 1558 if(!pps.weHaveAPart) 1559 return; 1560 1561 pieceHasNewContent(); // be sure the new content is handled every time 1562 1563 if(pps.isFile) { 1564 // I'm not sure if other environments put files in post or not... 1565 // I used to not do it, but I think I should, since it is there... 1566 pps._post[pps.piece.name] ~= pps.piece.filename; 1567 pps._files[pps.piece.name] ~= pps.piece; 1568 1569 allPostNamesInOrder ~= pps.piece.name; 1570 allPostValuesInOrder ~= pps.piece.filename; 1571 } else { 1572 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1573 1574 allPostNamesInOrder ~= pps.piece.name; 1575 allPostValuesInOrder ~= cast(string) pps.piece.content; 1576 } 1577 1578 /* 1579 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1580 pps.piece.content.length < 1000 1581 ? 1582 to!string(pps.piece.content) 1583 : 1584 "too long"); 1585 */ 1586 1587 // FIXME: the limit here 1588 pps.memoryCommitted += pps.piece.content.length; 1589 1590 pps.weHaveAPart = false; 1591 pps.whatDoWeWant = 1; 1592 pps.thisOnesHeaders = null; 1593 pps.thisOnesData = null; 1594 1595 pps.piece = UploadedFile.init; 1596 pps.isFile = false; 1597 } 1598 1599 void acceptChunk() { 1600 pps.buffer ~= chunk; 1601 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1602 } 1603 1604 immutable(ubyte)[] consume(size_t howMuch) { 1605 pps.contentConsumed += howMuch; 1606 auto ret = pps.buffer[0 .. howMuch]; 1607 pps.buffer = pps.buffer[howMuch .. $]; 1608 return ret; 1609 } 1610 1611 dataConsumptionLoop: do { 1612 switch(pps.whatDoWeWant) { 1613 default: assert(0); 1614 case 0: 1615 acceptChunk(); 1616 // the format begins with two extra leading dashes, then we should be at the boundary 1617 if(pps.buffer.length < 2) 1618 return; 1619 assert(pps.buffer[0] == '-', "no leading dash"); 1620 consume(1); 1621 assert(pps.buffer[0] == '-', "no second leading dash"); 1622 consume(1); 1623 1624 pps.whatDoWeWant = 1; 1625 goto case 1; 1626 /* fallthrough */ 1627 case 1: // looking for headers 1628 // here, we should be lined up right at the boundary, which is followed by a \r\n 1629 1630 // want to keep the buffer under control in case we're under attack 1631 //stderr.writeln("here once"); 1632 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1633 // throw new Exception("wtf is up with the huge mime part headers"); 1634 1635 acceptChunk(); 1636 1637 if(pps.buffer.length < pps.boundary.length) 1638 return; // not enough data, since there should always be a boundary here at least 1639 1640 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1641 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1642 // we *should* be at the end here! 1643 assert(pps.buffer[0] == '-'); 1644 consume(1); 1645 assert(pps.buffer[0] == '-'); 1646 consume(1); 1647 1648 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1649 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1650 "not lined up on boundary " ~ pps.boundary); 1651 consume(pps.boundary.length); 1652 1653 assert(pps.buffer[0] == '-'); 1654 consume(1); 1655 assert(pps.buffer[0] == '-'); 1656 consume(1); 1657 1658 assert(pps.buffer[0] == '\r'); 1659 consume(1); 1660 assert(pps.buffer[0] == '\n'); 1661 consume(1); 1662 1663 assert(pps.buffer.length == 0); 1664 assert(pps.contentConsumed == pps.expectedLength); 1665 break dataConsumptionLoop; // we're done! 1666 } else { 1667 // we're not done yet. We should be lined up on a boundary. 1668 1669 // But, we want to ensure the headers are here before we consume anything! 1670 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1671 if(headerEndLocation == -1) 1672 return; // they *should* all be here, so we can handle them all at once. 1673 1674 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1675 "not lined up on boundary " ~ pps.boundary); 1676 1677 consume(pps.boundary.length); 1678 // the boundary is always followed by a \r\n 1679 assert(pps.buffer[0] == '\r'); 1680 consume(1); 1681 assert(pps.buffer[0] == '\n'); 1682 consume(1); 1683 } 1684 1685 // re-running since by consuming the boundary, we invalidate the old index. 1686 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1687 assert(headerEndLocation >= 0, "no header"); 1688 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1689 1690 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1691 1692 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1693 1694 // now we'll parse the headers 1695 foreach(h; pps.thisOnesHeaders) { 1696 auto p = h.indexOf(":"); 1697 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1698 string hn = h[0..p]; 1699 string hv = h[p+2..$]; 1700 1701 switch(hn.toLower) { 1702 default: assert(0); 1703 case "content-disposition": 1704 auto info = hv.split("; "); 1705 foreach(i; info[1..$]) { // skipping the form-data 1706 auto o = i.split("="); // FIXME 1707 string pn = o[0]; 1708 string pv = o[1][1..$-1]; 1709 1710 if(pn == "name") { 1711 pps.piece.name = pv; 1712 } else if (pn == "filename") { 1713 pps.piece.filename = pv; 1714 pps.isFile = true; 1715 } 1716 } 1717 break; 1718 case "content-type": 1719 pps.piece.contentType = hv; 1720 break; 1721 } 1722 } 1723 1724 pps.whatDoWeWant++; // move to the next step - the data 1725 break; 1726 case 2: 1727 // when we get here, pps.buffer should contain our first chunk of data 1728 1729 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1730 throw new Exception("wtf is up with the huge mime part buffer"); 1731 1732 acceptChunk(); 1733 1734 // so the trick is, we want to process all the data up to the boundary, 1735 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1736 // want to wait for the next chunk. We start by looking for the whole boundary 1737 // in the buffer somewhere. 1738 1739 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1740 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1741 if(boundaryLocation != -1) { 1742 // this is easy - we can see it in it's entirety! 1743 1744 pps.piece.content ~= consume(boundaryLocation); 1745 1746 assert(pps.buffer[0] == '\r'); 1747 consume(1); 1748 assert(pps.buffer[0] == '\n'); 1749 consume(1); 1750 assert(pps.buffer[0] == '-'); 1751 consume(1); 1752 assert(pps.buffer[0] == '-'); 1753 consume(1); 1754 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1755 pps.weHaveAPart = true; 1756 pps.whatDoWeWant = 1; // back to getting headers for the next part 1757 1758 commitPart(); // we're done here 1759 } else { 1760 // we can't see the whole thing, but what if there's a partial boundary? 1761 1762 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1763 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1764 bool potentialBoundaryFound = false; 1765 1766 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1767 // we grow the boundary a bit each time. If we think it looks the 1768 // same, better pull another chunk to be sure it's not the end. 1769 // Starting small because exiting the loop early is desirable, since 1770 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1771 // the best we can do. 1772 if(a > pps.buffer.length) 1773 break; // FIXME: is this right? 1774 assert(a <= pps.buffer.length); 1775 assert(a > 0); 1776 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1777 // ok, there *might* be a boundary here, so let's 1778 // not treat the end as data yet. The rest is good to 1779 // use though, since if there was a boundary there, we'd 1780 // have handled it up above after locationOf. 1781 1782 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1783 consume(pps.buffer.length - a); 1784 pieceHasNewContent(); 1785 potentialBoundaryFound = true; 1786 break boundaryCheck; 1787 } 1788 } 1789 1790 if(!potentialBoundaryFound) { 1791 // we can consume the whole thing 1792 pps.piece.content ~= pps.buffer; 1793 pieceHasNewContent(); 1794 consume(pps.buffer.length); 1795 } else { 1796 // we found a possible boundary, but there was 1797 // insufficient data to be sure. 1798 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1799 1800 return; // wait for the next chunk. 1801 } 1802 } 1803 } 1804 } while(pps.buffer.length); 1805 1806 // btw all boundaries except the first should have a \r\n before them 1807 } else { 1808 // application/x-www-form-urlencoded and application/json 1809 1810 // not using maxContentLength because that might be cranked up to allow 1811 // large file uploads. We can handle them, but a huge post[] isn't any good. 1812 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1813 throw new Exception("wtf is up with such a gigantic form submission????"); 1814 1815 pps.buffer ~= chunk; 1816 1817 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1818 if(pps.buffer.length == pps.expectedLength) { 1819 if(pps.needsSavedBody) 1820 pps.postBody = cast(string) pps.buffer; 1821 else 1822 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1823 version(preserveData) 1824 originalPostData = pps.buffer; 1825 } else { 1826 // just for debugging 1827 } 1828 } 1829 } 1830 1831 protected void cleanUpPostDataState() { 1832 pps = PostParserState.init; 1833 } 1834 1835 /// you can override this function to somehow react 1836 /// to an upload in progress. 1837 /// 1838 /// Take note that parts of the CGI object is not yet 1839 /// initialized! Stuff from HTTP headers, including get[], is usable. 1840 /// But, none of post[] is usable, and you cannot write here. That's 1841 /// why this method is const - mutating the object won't do much anyway. 1842 /// 1843 /// My idea here was so you can output a progress bar or 1844 /// something to a cooperative client (see arsd.rtud for a potential helper) 1845 /// 1846 /// The default is to do nothing. Subclass cgi and use the 1847 /// CustomCgiMain mixin to do something here. 1848 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1849 // This space intentionally left blank. 1850 } 1851 1852 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1853 /// *closeConnection will be set to true if you should close the connection after handling this request 1854 this(BufferedInputRange ir, bool* closeConnection) { 1855 isCalledWithCommandLineArguments = false; 1856 import al = std.algorithm; 1857 1858 immutable(ubyte)[] data; 1859 1860 void rdo(const(ubyte)[] d) { 1861 //import std.stdio; writeln(d); 1862 sendAll(ir.source, d); 1863 } 1864 1865 auto ira = ir.source.remoteAddress(); 1866 auto irLocalAddress = ir.source.localAddress(); 1867 1868 ushort port = 80; 1869 if(auto ia = cast(InternetAddress) irLocalAddress) { 1870 port = ia.port; 1871 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1872 port = ia.port; 1873 } 1874 1875 // that check for UnixAddress is to work around a Phobos bug 1876 // see: https://github.com/dlang/phobos/pull/7383 1877 // but this might be more useful anyway tbh for this case 1878 version(Posix) 1879 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1880 else 1881 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1882 } 1883 1884 /** 1885 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1886 1887 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1888 1889 Params: 1890 inputData = the incoming data, including headers and other raw http data. 1891 When the constructor exits, it will leave this range exactly at the start of 1892 the next request on the connection (if there is one). 1893 1894 address = the IP address of the remote user 1895 _port = the port number of the connection 1896 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1897 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1898 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1899 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1900 closeConnection = if the request asks to close the connection, *closeConnection == true. 1901 */ 1902 this( 1903 BufferedInputRange inputData, 1904 // string[] headers, immutable(ubyte)[] data, 1905 string address, ushort _port, 1906 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1907 bool _https = false, 1908 void delegate(const(ubyte)[]) _rawDataOutput = null, 1909 void delegate() _flush = null, 1910 // this pointer tells if the connection is supposed to be closed after we handle this 1911 bool* closeConnection = null) 1912 { 1913 // these are all set locally so the loop works 1914 // without triggering errors in dmd 2.064 1915 // we go ahead and set them at the end of it to the this version 1916 int port; 1917 string referrer; 1918 string remoteAddress; 1919 string userAgent; 1920 string authorization; 1921 string origin; 1922 string accept; 1923 string lastEventId; 1924 bool https; 1925 string host; 1926 RequestMethod requestMethod; 1927 string requestUri; 1928 string pathInfo; 1929 string queryString; 1930 string scriptName; 1931 string[string] get; 1932 string[][string] getArray; 1933 bool keepAliveRequested; 1934 bool acceptsGzip; 1935 string cookie; 1936 1937 1938 1939 environmentVariables = cast(const) environment.toAA; 1940 1941 idlol = inputData; 1942 1943 isCalledWithCommandLineArguments = false; 1944 1945 https = _https; 1946 port = _port; 1947 1948 rawDataOutput = _rawDataOutput; 1949 flushDelegate = _flush; 1950 nph = true; 1951 1952 remoteAddress = address; 1953 1954 // streaming parser 1955 import al = std.algorithm; 1956 1957 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1958 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1959 while(idx == -1) { 1960 inputData.popFront(0); 1961 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1962 } 1963 1964 assert(idx != -1); 1965 1966 1967 string contentType = ""; 1968 string[string] requestHeadersHere; 1969 1970 size_t contentLength; 1971 1972 bool isChunked; 1973 1974 { 1975 import core.runtime; 1976 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 1977 } 1978 1979 1980 int headerNumber = 0; 1981 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 1982 if(line.length) { 1983 headerNumber++; 1984 auto header = cast(string) line.idup; 1985 if(headerNumber == 1) { 1986 // request line 1987 auto parts = al.splitter(header, " "); 1988 if(parts.front == "PRI") { 1989 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 1990 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 1991 // to bring me benefit) 1992 throw new HttpVersionNotSupportedException(); 1993 } 1994 requestMethod = to!RequestMethod(parts.front); 1995 parts.popFront(); 1996 requestUri = parts.front; 1997 1998 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 1999 scriptName = requestUri[0 .. pathInfoStarts]; 2000 2001 auto question = requestUri.indexOf("?"); 2002 if(question == -1) { 2003 queryString = ""; 2004 // FIXME: double check, this might be wrong since it could be url encoded 2005 pathInfo = requestUri[pathInfoStarts..$]; 2006 } else { 2007 queryString = requestUri[question+1..$]; 2008 pathInfo = requestUri[pathInfoStarts..question]; 2009 } 2010 2011 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2012 getArray = cast(string[][string]) assumeUnique(ugh); 2013 2014 if(header.indexOf("HTTP/1.0") != -1) { 2015 http10 = true; 2016 autoBuffer = true; 2017 if(closeConnection) { 2018 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2019 *closeConnection = true; 2020 } 2021 } 2022 } else { 2023 // other header 2024 auto colon = header.indexOf(":"); 2025 if(colon == -1) 2026 throw new Exception("HTTP headers should have a colon!"); 2027 string name = header[0..colon].toLower; 2028 string value = header[colon+2..$]; // skip the colon and the space 2029 2030 requestHeadersHere[name] = value; 2031 2032 if (name == "accept") { 2033 accept = value; 2034 } 2035 else if (name == "origin") { 2036 origin = value; 2037 } 2038 else if (name == "connection") { 2039 if(value == "close" && closeConnection) 2040 *closeConnection = true; 2041 if(value.asLowerCase().canFind("keep-alive")) { 2042 keepAliveRequested = true; 2043 2044 // on http 1.0, the connection is closed by default, 2045 // but not if they request keep-alive. then we don't close 2046 // anymore - undoing the set above 2047 if(http10 && closeConnection) { 2048 *closeConnection = false; 2049 } 2050 } 2051 } 2052 else if (name == "transfer-encoding") { 2053 if(value == "chunked") 2054 isChunked = true; 2055 } 2056 else if (name == "last-event-id") { 2057 lastEventId = value; 2058 } 2059 else if (name == "authorization") { 2060 authorization = value; 2061 } 2062 else if (name == "content-type") { 2063 contentType = value; 2064 } 2065 else if (name == "content-length") { 2066 contentLength = to!size_t(value); 2067 } 2068 else if (name == "x-forwarded-for") { 2069 remoteAddress = value; 2070 } 2071 else if (name == "x-forwarded-host" || name == "host") { 2072 if(name != "host" || host is null) 2073 host = value; 2074 } 2075 // FIXME: https://tools.ietf.org/html/rfc7239 2076 else if (name == "accept-encoding") { 2077 if(value.indexOf("gzip") != -1) 2078 acceptsGzip = true; 2079 } 2080 else if (name == "user-agent") { 2081 userAgent = value; 2082 } 2083 else if (name == "referer") { 2084 referrer = value; 2085 } 2086 else if (name == "cookie") { 2087 cookie ~= value; 2088 } else if(name == "expect") { 2089 if(value == "100-continue") { 2090 // FIXME we should probably give user code a chance 2091 // to process and reject but that needs to be virtual, 2092 // perhaps part of the CGI redesign. 2093 2094 // FIXME: if size is > max content length it should 2095 // also fail at this point. 2096 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2097 2098 // FIXME: let the user write out 103 early hints too 2099 } 2100 } 2101 // else 2102 // ignore it 2103 2104 } 2105 } 2106 2107 inputData.consume(idx + 4); 2108 // done 2109 2110 requestHeaders = assumeUnique(requestHeadersHere); 2111 2112 ByChunkRange dataByChunk; 2113 2114 // reading Content-Length type data 2115 // We need to read up the data we have, and write it out as a chunk. 2116 if(!isChunked) { 2117 dataByChunk = byChunk(inputData, contentLength); 2118 } else { 2119 // chunked requests happen, but not every day. Since we need to know 2120 // the content length (for now, maybe that should change), we'll buffer 2121 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2122 auto data = dechunk(inputData); 2123 2124 // set the range here 2125 dataByChunk = byChunk(data); 2126 contentLength = data.length; 2127 } 2128 2129 assert(dataByChunk !is null); 2130 2131 if(contentLength) { 2132 prepareForIncomingDataChunks(contentType, contentLength); 2133 foreach(dataChunk; dataByChunk) { 2134 handleIncomingDataChunk(dataChunk); 2135 } 2136 postArray = assumeUnique(pps._post); 2137 filesArray = assumeUnique(pps._files); 2138 files = keepLastOf(filesArray); 2139 post = keepLastOf(postArray); 2140 postBody = pps.postBody; 2141 cleanUpPostDataState(); 2142 } 2143 2144 this.port = port; 2145 this.referrer = referrer; 2146 this.remoteAddress = remoteAddress; 2147 this.userAgent = userAgent; 2148 this.authorization = authorization; 2149 this.origin = origin; 2150 this.accept = accept; 2151 this.lastEventId = lastEventId; 2152 this.https = https; 2153 this.host = host; 2154 this.requestMethod = requestMethod; 2155 this.requestUri = requestUri; 2156 this.pathInfo = pathInfo; 2157 this.queryString = queryString; 2158 2159 this.scriptName = scriptName; 2160 this.get = keepLastOf(getArray); 2161 this.getArray = cast(immutable) getArray; 2162 this.keepAliveRequested = keepAliveRequested; 2163 this.acceptsGzip = acceptsGzip; 2164 this.cookie = cookie; 2165 2166 cookiesArray = getCookieArray(); 2167 cookies = keepLastOf(cookiesArray); 2168 2169 } 2170 BufferedInputRange idlol; 2171 2172 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2173 string[string] ca; 2174 foreach(k, v; arr) 2175 ca[k] = v[$-1]; 2176 2177 return assumeUnique(ca); 2178 } 2179 2180 // FIXME duplication 2181 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2182 UploadedFile[string] ca; 2183 foreach(k, v; arr) 2184 ca[k] = v[$-1]; 2185 2186 return assumeUnique(ca); 2187 } 2188 2189 2190 private immutable(string[][string]) getCookieArray() { 2191 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2192 return assumeUnique(forTheLoveOfGod); 2193 } 2194 2195 /// Very simple method to require a basic auth username and password. 2196 /// If the http request doesn't include the required credentials, it throws a 2197 /// HTTP 401 error, and an exception. 2198 /// 2199 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 2200 /// the user's credentials are sent in plain text on every request. 2201 /// 2202 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2203 /// application. Either use Apache's built in methods for basic authentication, or add 2204 /// something along these lines to your server configuration: 2205 /// 2206 /// RewriteEngine On 2207 /// RewriteCond %{HTTP:Authorization} ^(.*) 2208 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2209 /// 2210 /// To ensure the necessary data is available to cgi.d. 2211 void requireBasicAuth(string user, string pass, string message = null) { 2212 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2213 setResponseStatus("401 Authorization Required"); 2214 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 2215 close(); 2216 throw new Exception("Not authorized; got " ~ authorization); 2217 } 2218 } 2219 2220 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2221 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2222 /// Use setResponseExpires and updateResponseExpires for more control 2223 void setCache(bool allowCaching) { 2224 noCache = !allowCaching; 2225 } 2226 2227 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2228 /// who can accept it 2229 bool gzipResponse; 2230 2231 immutable bool acceptsGzip; 2232 immutable bool keepAliveRequested; 2233 2234 /// Set to true if and only if this was initialized with command line arguments 2235 immutable bool isCalledWithCommandLineArguments; 2236 2237 /// This gets a full url for the current request, including port, protocol, host, path, and query 2238 string getCurrentCompleteUri() const { 2239 ushort defaultPort = https ? 443 : 80; 2240 2241 string uri = "http"; 2242 if(https) 2243 uri ~= "s"; 2244 uri ~= "://"; 2245 uri ~= host; 2246 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2247 version(none) 2248 if(!(!port || port == defaultPort)) { 2249 uri ~= ":"; 2250 uri ~= to!string(port); 2251 } 2252 +/ 2253 uri ~= requestUri; 2254 return uri; 2255 } 2256 2257 /// You can override this if your site base url isn't the same as the script name 2258 string logicalScriptName() const { 2259 return scriptName; 2260 } 2261 2262 /++ 2263 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2264 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2265 Note setResponseStatus() must be called *before* you write() any data to the output. 2266 2267 History: 2268 The `int` overload was added on January 11, 2021. 2269 +/ 2270 void setResponseStatus(string status) { 2271 assert(!outputtedResponseData); 2272 responseStatus = status; 2273 } 2274 /// ditto 2275 void setResponseStatus(int statusCode) { 2276 setResponseStatus(getHttpCodeText(statusCode)); 2277 } 2278 private string responseStatus = null; 2279 2280 /// Returns true if it is still possible to output headers 2281 bool canOutputHeaders() { 2282 return !isClosed && !outputtedResponseData; 2283 } 2284 2285 /// Sets the location header, which the browser will redirect the user to automatically. 2286 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2287 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2288 void setResponseLocation(string uri, bool important = true, string status = null) { 2289 if(!important && isCurrentResponseLocationImportant) 2290 return; // important redirects always override unimportant ones 2291 2292 if(uri is null) { 2293 responseStatus = "200 OK"; 2294 responseLocation = null; 2295 isCurrentResponseLocationImportant = important; 2296 return; // this just cancels the redirect 2297 } 2298 2299 assert(!outputtedResponseData); 2300 if(status is null) 2301 responseStatus = "302 Found"; 2302 else 2303 responseStatus = status; 2304 2305 responseLocation = uri.strip; 2306 isCurrentResponseLocationImportant = important; 2307 } 2308 protected string responseLocation = null; 2309 private bool isCurrentResponseLocationImportant = false; 2310 2311 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2312 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2313 /// Note: the when parameter is different than setCookie's expire parameter. 2314 void setResponseExpires(long when, bool isPublic = false) { 2315 responseExpires = when; 2316 setCache(true); // need to enable caching so the date has meaning 2317 2318 responseIsPublic = isPublic; 2319 responseExpiresRelative = false; 2320 } 2321 2322 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2323 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2324 responseExpires = whenFromNow; 2325 setCache(true); // need to enable caching so the date has meaning 2326 2327 responseIsPublic = isPublic; 2328 responseExpiresRelative = true; 2329 } 2330 private long responseExpires = long.min; 2331 private bool responseIsPublic = false; 2332 private bool responseExpiresRelative = false; 2333 2334 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2335 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2336 /// output as a whole is as cacheable as the least cachable part in the chain. 2337 2338 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2339 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2340 void updateResponseExpires(long when, bool isPublic) { 2341 if(responseExpires == long.min) 2342 setResponseExpires(when, isPublic); 2343 else if(when < responseExpires) 2344 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2345 } 2346 2347 /* 2348 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2349 /// Should generally be false if the user is logged in. It assumes private cache only. 2350 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2351 void setPublicCaching(bool allowPublicCaches) { 2352 publicCaching = allowPublicCaches; 2353 } 2354 private bool publicCaching = false; 2355 */ 2356 2357 /++ 2358 History: 2359 Added January 11, 2021 2360 +/ 2361 enum SameSitePolicy { 2362 Lax, 2363 Strict, 2364 None 2365 } 2366 2367 /++ 2368 Sets an HTTP cookie, automatically encoding the data to the correct string. 2369 expiresIn is how many milliseconds in the future the cookie will expire. 2370 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2371 Note setCookie() must be called *before* you write() any data to the output. 2372 2373 History: 2374 Parameter `sameSitePolicy` was added on January 11, 2021. 2375 +/ 2376 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2377 assert(!outputtedResponseData); 2378 string cookie = std.uri.encodeComponent(name) ~ "="; 2379 cookie ~= std.uri.encodeComponent(data); 2380 if(path !is null) 2381 cookie ~= "; path=" ~ path; 2382 // FIXME: should I just be using max-age here? (also in cache below) 2383 if(expiresIn != 0) 2384 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2385 if(domain !is null) 2386 cookie ~= "; domain=" ~ domain; 2387 if(secure == true) 2388 cookie ~= "; Secure"; 2389 if(httpOnly == true ) 2390 cookie ~= "; HttpOnly"; 2391 final switch(sameSitePolicy) { 2392 case SameSitePolicy.Lax: 2393 cookie ~= "; SameSite=Lax"; 2394 break; 2395 case SameSitePolicy.Strict: 2396 cookie ~= "; SameSite=Strict"; 2397 break; 2398 case SameSitePolicy.None: 2399 cookie ~= "; SameSite=None"; 2400 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2401 break; 2402 } 2403 2404 if(auto idx = name in cookieIndexes) { 2405 responseCookies[*idx] = cookie; 2406 } else { 2407 cookieIndexes[name] = responseCookies.length; 2408 responseCookies ~= cookie; 2409 } 2410 } 2411 private string[] responseCookies; 2412 private size_t[string] cookieIndexes; 2413 2414 /// Clears a previously set cookie with the given name, path, and domain. 2415 void clearCookie(string name, string path = null, string domain = null) { 2416 assert(!outputtedResponseData); 2417 setCookie(name, "", 1, path, domain); 2418 } 2419 2420 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2421 void setResponseContentType(string ct) { 2422 assert(!outputtedResponseData); 2423 responseContentType = ct; 2424 } 2425 private string responseContentType = null; 2426 2427 /// Adds a custom header. It should be the name: value, but without any line terminator. 2428 /// For example: header("X-My-Header: Some value"); 2429 /// Note you should use the specialized functions in this object if possible to avoid 2430 /// duplicates in the output. 2431 void header(string h) { 2432 customHeaders ~= h; 2433 } 2434 2435 /++ 2436 I named the original function `header` after PHP, but this pattern more fits 2437 the rest of the Cgi object. 2438 2439 Either name are allowed. 2440 2441 History: 2442 Alias added June 17, 2022. 2443 +/ 2444 alias setResponseHeader = header; 2445 2446 private string[] customHeaders; 2447 private bool websocketMode; 2448 2449 void flushHeaders(const(void)[] t, bool isAll = false) { 2450 StackBuffer buffer = StackBuffer(0); 2451 2452 prepHeaders(t, isAll, &buffer); 2453 2454 if(rawDataOutput !is null) 2455 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2456 else { 2457 stdout.rawWrite(buffer.get()); 2458 } 2459 } 2460 2461 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2462 string terminator = "\n"; 2463 if(rawDataOutput !is null) 2464 terminator = "\r\n"; 2465 2466 if(responseStatus !is null) { 2467 if(nph) { 2468 if(http10) 2469 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2470 else 2471 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2472 } else 2473 buffer.add("Status: ", responseStatus, terminator); 2474 } else if (nph) { 2475 if(http10) 2476 buffer.add("HTTP/1.0 200 OK", terminator); 2477 else 2478 buffer.add("HTTP/1.1 200 OK", terminator); 2479 } 2480 2481 if(websocketMode) 2482 goto websocket; 2483 2484 if(nph) { // we're responsible for setting the date too according to http 1.1 2485 char[29] db = void; 2486 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2487 buffer.add("Date: ", db[], terminator); 2488 } 2489 2490 // FIXME: what if the user wants to set his own content-length? 2491 // The custom header function can do it, so maybe that's best. 2492 // Or we could reuse the isAll param. 2493 if(responseLocation !is null) { 2494 buffer.add("Location: ", responseLocation, terminator); 2495 } 2496 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2497 if(responseExpiresRelative) { 2498 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2499 buffer.add(responseExpires); 2500 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2501 } else { 2502 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2503 char[29] db = void; 2504 printDateToBuffer(cast(DateTime) expires, db[]); 2505 buffer.add("Expires: ", db[], terminator); 2506 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2507 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2508 buffer.add(terminator); 2509 } 2510 } 2511 if(responseCookies !is null && responseCookies.length > 0) { 2512 foreach(c; responseCookies) 2513 buffer.add("Set-Cookie: ", c, terminator); 2514 } 2515 if(noCache) { // we specifically do not want caching (this is actually the default) 2516 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2517 buffer.add("Expires: 0", terminator); 2518 buffer.add("Pragma: no-cache", terminator); 2519 } else { 2520 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2521 buffer.add("Cache-Control: public", terminator); 2522 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2523 } 2524 } 2525 if(responseContentType !is null) { 2526 buffer.add("Content-Type: ", responseContentType, terminator); 2527 } else 2528 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2529 2530 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2531 buffer.add("Content-Encoding: gzip", terminator); 2532 } 2533 2534 2535 if(!isAll) { 2536 if(nph && !http10) { 2537 buffer.add("Transfer-Encoding: chunked", terminator); 2538 responseChunked = true; 2539 } 2540 } else { 2541 buffer.add("Content-Length: "); 2542 buffer.add(t.length); 2543 buffer.add(terminator); 2544 if(nph && keepAliveRequested) { 2545 buffer.add("Connection: Keep-Alive", terminator); 2546 } 2547 } 2548 2549 websocket: 2550 2551 foreach(hd; customHeaders) 2552 buffer.add(hd, terminator); 2553 2554 // FIXME: what about duplicated headers? 2555 2556 // end of header indicator 2557 buffer.add(terminator); 2558 2559 outputtedResponseData = true; 2560 } 2561 2562 /// Writes the data to the output, flushing headers if they have not yet been sent. 2563 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2564 assert(!closed, "Output has already been closed"); 2565 2566 StackBuffer buffer = StackBuffer(0); 2567 2568 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2569 // actually gzip the data here 2570 2571 auto c = new Compress(HeaderFormat.gzip); // want gzip 2572 2573 auto data = c.compress(t); 2574 data ~= c.flush(); 2575 2576 // std.file.write("/tmp/last-item", data); 2577 2578 t = data; 2579 } 2580 2581 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2582 prepHeaders(t, isAll, &buffer); 2583 } 2584 2585 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2586 if (autoBuffer && !isAll) { 2587 outputBuffer ~= cast(ubyte[]) t; 2588 } 2589 if(!autoBuffer || isAll) { 2590 if(rawDataOutput !is null) 2591 if(nph && responseChunked) { 2592 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2593 // we're making the chunk here instead of in a function 2594 // to avoid unneeded gc pressure 2595 buffer.add(toHex(t.length)); 2596 buffer.add("\r\n"); 2597 buffer.add(cast(char[]) t, "\r\n"); 2598 } else { 2599 buffer.add(cast(char[]) t); 2600 } 2601 else 2602 buffer.add(cast(char[]) t); 2603 } 2604 } 2605 2606 if(rawDataOutput !is null) 2607 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2608 else 2609 stdout.rawWrite(buffer.get()); 2610 2611 if(maybeAutoClose && isAll) 2612 close(); // if you say it is all, that means we're definitely done 2613 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2614 } 2615 2616 /++ 2617 Convenience method to set content type to json and write the string as the complete response. 2618 2619 History: 2620 Added January 16, 2020 2621 +/ 2622 void writeJson(string json) { 2623 this.setResponseContentType("application/json"); 2624 this.write(json, true); 2625 } 2626 2627 /// Flushes the pending buffer, leaving the connection open so you can send more. 2628 void flush() { 2629 if(rawDataOutput is null) 2630 stdout.flush(); 2631 else if(flushDelegate !is null) 2632 flushDelegate(); 2633 } 2634 2635 version(autoBuffer) 2636 bool autoBuffer = true; 2637 else 2638 bool autoBuffer = false; 2639 ubyte[] outputBuffer; 2640 2641 /// Flushes the buffers to the network, signifying that you are done. 2642 /// You should always call this explicitly when you are done outputting data. 2643 void close() { 2644 if(closed) 2645 return; // don't double close 2646 2647 if(!outputtedResponseData) 2648 write("", true, false); 2649 2650 // writing auto buffered data 2651 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2652 if(!nph) 2653 stdout.rawWrite(outputBuffer); 2654 else 2655 write(outputBuffer, true, false); // tell it this is everything 2656 } 2657 2658 // closing the last chunk... 2659 if(nph && rawDataOutput !is null && responseChunked) 2660 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2661 2662 if(flushDelegate) 2663 flushDelegate(); 2664 2665 closed = true; 2666 } 2667 2668 // Closes without doing anything, shouldn't be used often 2669 void rawClose() { 2670 closed = true; 2671 } 2672 2673 /++ 2674 Gets a request variable as a specific type, or the default value of it isn't there 2675 or isn't convertible to the request type. 2676 2677 Checks both GET and POST variables, preferring the POST variable, if available. 2678 2679 A nice trick is using the default value to choose the type: 2680 2681 --- 2682 /* 2683 The return value will match the type of the default. 2684 Here, I gave 10 as a default, so the return value will 2685 be an int. 2686 2687 If the user-supplied value cannot be converted to the 2688 requested type, you will get the default value back. 2689 */ 2690 int a = cgi.request("number", 10); 2691 2692 if(cgi.get["number"] == "11") 2693 assert(a == 11); // conversion succeeds 2694 2695 if("number" !in cgi.get) 2696 assert(a == 10); // no value means you can't convert - give the default 2697 2698 if(cgi.get["number"] == "twelve") 2699 assert(a == 10); // conversion from string to int would fail, so we get the default 2700 --- 2701 2702 You can use an enum as an easy whitelist, too: 2703 2704 --- 2705 enum Operations { 2706 add, remove, query 2707 } 2708 2709 auto op = cgi.request("op", Operations.query); 2710 2711 if(cgi.get["op"] == "add") 2712 assert(op == Operations.add); 2713 if(cgi.get["op"] == "remove") 2714 assert(op == Operations.remove); 2715 if(cgi.get["op"] == "query") 2716 assert(op == Operations.query); 2717 2718 if(cgi.get["op"] == "random string") 2719 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2720 --- 2721 +/ 2722 T request(T = string)(in string name, in T def = T.init) const nothrow { 2723 try { 2724 return 2725 (name in post) ? to!T(post[name]) : 2726 (name in get) ? to!T(get[name]) : 2727 def; 2728 } catch(Exception e) { return def; } 2729 } 2730 2731 /// Is the output already closed? 2732 bool isClosed() const { 2733 return closed; 2734 } 2735 2736 /++ 2737 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2738 +/ 2739 Session!Data getSessionObject(Data)() { 2740 if(testInProcess !is null) { 2741 // test mode 2742 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2743 if(obj !is null) 2744 return cast(typeof(return)) obj; 2745 else { 2746 auto o = new MockSession!Data(); 2747 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2748 return o; 2749 } 2750 } else { 2751 // normal operation 2752 return new BasicDataServerSession!Data(this); 2753 } 2754 } 2755 2756 // if it is in test mode; triggers mock sessions. Used by CgiTester 2757 version(with_breaking_cgi_features) 2758 private CgiTester testInProcess; 2759 2760 /* Hooks for redirecting input and output */ 2761 private void delegate(const(ubyte)[]) rawDataOutput = null; 2762 private void delegate() flushDelegate = null; 2763 2764 /* This info is used when handling a more raw HTTP protocol */ 2765 private bool nph; 2766 private bool http10; 2767 private bool closed; 2768 private bool responseChunked = false; 2769 2770 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2771 immutable(ubyte)[] originalPostData; 2772 2773 /++ 2774 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2775 2776 It is intended to be used for JSON and XML request content types, but also may be used 2777 for other content types your application can handle. But it will NOT be populated 2778 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2779 parsed into the post and postArray members. 2780 2781 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2782 will be discarded to the client with an error. This helps keep this array from being exploded in size 2783 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2784 client in certain build modes.) 2785 2786 History: 2787 Added January 5, 2021 2788 Documented February 21, 2023 (dub v11.0) 2789 +/ 2790 public immutable string postBody; 2791 alias postJson = postBody; // old name 2792 2793 /* Internal state flags */ 2794 private bool outputtedResponseData; 2795 private bool noCache = true; 2796 2797 const(string[string]) environmentVariables; 2798 2799 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2800 partially because it logically is (your code doesn't change what the user requested...) 2801 and partially because I hate how bad programs in PHP change those superglobals to do 2802 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2803 2804 For some of these, you'll want to refer to the http or cgi specs for more details. 2805 */ 2806 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2807 2808 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2809 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2810 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2811 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2812 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2813 immutable(char[]) scriptFileName; /// The physical filename of your script 2814 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2815 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2816 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2817 2818 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2819 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2820 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2821 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2822 2823 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2824 */ 2825 immutable(char[]) referrer; 2826 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2827 2828 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2829 2830 immutable bool https; /// Was the request encrypted via https? 2831 immutable int port; /// On what TCP port number did the server receive the request? 2832 2833 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2834 2835 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2836 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2837 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2838 2839 /** 2840 Represents user uploaded files. 2841 2842 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2843 */ 2844 immutable(UploadedFile[][string]) filesArray; 2845 immutable(UploadedFile[string]) files; 2846 2847 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2848 /// the order of the arrays is the order the data arrives 2849 immutable(string[][string]) getArray; /// like get, but an array of values per name 2850 immutable(string[][string]) postArray; /// ditto for post 2851 immutable(string[][string]) cookiesArray; /// ditto for cookies 2852 2853 // convenience function for appending to a uri without extra ? 2854 // matches the name and effect of javascript's location.search property 2855 string search() const { 2856 if(queryString.length) 2857 return "?" ~ queryString; 2858 return ""; 2859 } 2860 2861 // FIXME: what about multiple files with the same name? 2862 private: 2863 //RequestMethod _requestMethod; 2864 } 2865 2866 /// use this for testing or other isolated things when you want it to be no-ops 2867 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2868 // we want to ignore, not use stdout 2869 if(outputSink is null) 2870 outputSink = delegate void(const(ubyte)[]) { }; 2871 2872 string[string] env; 2873 env["REQUEST_METHOD"] = to!string(method); 2874 env["CONTENT_LENGTH"] = to!string(data.length); 2875 2876 auto cgi = new Cgi( 2877 0, 2878 env, 2879 { return data; }, 2880 outputSink, 2881 null); 2882 2883 return cgi; 2884 } 2885 2886 /++ 2887 A helper test class for request handler unittests. 2888 +/ 2889 version(with_breaking_cgi_features) 2890 class CgiTester { 2891 private { 2892 SessionObject[TypeInfo] mockSessions; 2893 SessionObject getSessionOverride(TypeInfo ti) { 2894 if(auto o = ti in mockSessions) 2895 return *o; 2896 else 2897 return null; 2898 } 2899 void setSessionOverride(TypeInfo ti, SessionObject so) { 2900 mockSessions[ti] = so; 2901 } 2902 } 2903 2904 /++ 2905 Gets (and creates if necessary) a mock session object for this test. Note 2906 it will be the same one used for any test operations through this CgiTester instance. 2907 +/ 2908 Session!Data getSessionObject(Data)() { 2909 auto obj = getSessionOverride(typeid(typeof(return))); 2910 if(obj !is null) 2911 return cast(typeof(return)) obj; 2912 else { 2913 auto o = new MockSession!Data(); 2914 setSessionOverride(typeid(typeof(return)), o); 2915 return o; 2916 } 2917 } 2918 2919 /++ 2920 Pass a reference to your request handler when creating the tester. 2921 +/ 2922 this(void function(Cgi) requestHandler) { 2923 this.requestHandler = requestHandler; 2924 } 2925 2926 /++ 2927 You can check response information with these methods after you call the request handler. 2928 +/ 2929 struct Response { 2930 int code; 2931 string[string] headers; 2932 string responseText; 2933 ubyte[] responseBody; 2934 } 2935 2936 /++ 2937 Executes a test request on your request handler, and returns the response. 2938 2939 Params: 2940 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2941 args = additional arguments. Same format as cgi's command line handler. 2942 +/ 2943 Response GET(string url, string[] args = null) { 2944 return executeTest("GET", url, args); 2945 } 2946 /// ditto 2947 Response POST(string url, string[] args = null) { 2948 return executeTest("POST", url, args); 2949 } 2950 2951 /// ditto 2952 Response executeTest(string method, string url, string[] args) { 2953 ubyte[] outputtedRawData; 2954 void outputSink(const(ubyte)[] data) { 2955 outputtedRawData ~= data; 2956 } 2957 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2958 cgi.testInProcess = this; 2959 scope(exit) cgi.dispose(); 2960 2961 requestHandler(cgi); 2962 2963 cgi.close(); 2964 2965 Response response; 2966 2967 if(outputtedRawData.length) { 2968 enum LINE = "\r\n"; 2969 2970 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2971 assert(idx != -1, to!string(outputtedRawData)); 2972 auto headers = cast(string) outputtedRawData[0 .. idx]; 2973 response.code = 200; 2974 while(headers.length) { 2975 auto i = headers.locationOf(LINE); 2976 if(i == -1) i = cast(int) headers.length; 2977 2978 auto header = headers[0 .. i]; 2979 2980 auto c = header.locationOf(":"); 2981 if(c != -1) { 2982 auto name = header[0 .. c]; 2983 auto value = header[c + 2 ..$]; 2984 2985 if(name == "Status") 2986 response.code = value[0 .. value.locationOf(" ")].to!int; 2987 2988 response.headers[name] = value; 2989 } else { 2990 assert(0); 2991 } 2992 2993 if(i != headers.length) 2994 i += 2; 2995 headers = headers[i .. $]; 2996 } 2997 response.responseBody = outputtedRawData[idx + 4 .. $]; 2998 response.responseText = cast(string) response.responseBody; 2999 } 3000 3001 return response; 3002 } 3003 3004 private void function(Cgi) requestHandler; 3005 } 3006 3007 3008 // should this be a separate module? Probably, but that's a hassle. 3009 3010 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3011 string makeDataUrl(string mimeType, in void[] data) { 3012 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3013 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3014 } 3015 3016 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3017 /// Represents a url that can be broken down or built up through properties 3018 struct Uri { 3019 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3020 3021 // scheme//userinfo@host:port/path?query#fragment 3022 3023 string scheme; /// e.g. "http" in "http://example.com/" 3024 string userinfo; /// the username (and possibly a password) in the uri 3025 string host; /// the domain name 3026 int port; /// port number, if given. Will be zero if a port was not explicitly given 3027 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3028 string query; /// the stuff after the ? in a uri 3029 string fragment; /// the stuff after the # in a uri. 3030 3031 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3032 // the decode ones need to keep different names anyway because we can't overload on return values... 3033 static string encode(string s) { return std.uri.encodeComponent(s); } 3034 static string encode(string[string] s) { return encodeVariables(s); } 3035 static string encode(string[][string] s) { return encodeVariables(s); } 3036 3037 /// Breaks down a uri string to its components 3038 this(string uri) { 3039 reparse(uri); 3040 } 3041 3042 private void reparse(string uri) { 3043 // from RFC 3986 3044 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3045 // it was a nice experiment but just not worth it. 3046 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3047 /* 3048 Captures: 3049 0 = whole url 3050 1 = scheme, with : 3051 2 = scheme, no : 3052 3 = authority, with // 3053 4 = authority, no // 3054 5 = path 3055 6 = query string, with ? 3056 7 = query string, no ? 3057 8 = anchor, with # 3058 9 = anchor, no # 3059 */ 3060 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3061 // instead, I will DIY and cut that down to 0.6s on the same computer. 3062 /* 3063 3064 Note that authority is 3065 user:password@domain:port 3066 where the user:password@ part is optional, and the :port is optional. 3067 3068 Regex translation: 3069 3070 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3071 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3072 Path cannot have any ? or # in it. It is optional. 3073 Query must start with ? and must not have # in it. It is optional. 3074 Anchor must start with # and can have anything else in it to end of string. It is optional. 3075 */ 3076 3077 this = Uri.init; // reset all state 3078 3079 // empty uri = nothing special 3080 if(uri.length == 0) { 3081 return; 3082 } 3083 3084 size_t idx; 3085 3086 scheme_loop: foreach(char c; uri[idx .. $]) { 3087 switch(c) { 3088 case ':': 3089 case '/': 3090 case '?': 3091 case '#': 3092 break scheme_loop; 3093 default: 3094 } 3095 idx++; 3096 } 3097 3098 if(idx == 0 && uri[idx] == ':') { 3099 // this is actually a path! we skip way ahead 3100 goto path_loop; 3101 } 3102 3103 if(idx == uri.length) { 3104 // the whole thing is a path, apparently 3105 path = uri; 3106 return; 3107 } 3108 3109 if(idx > 0 && uri[idx] == ':') { 3110 scheme = uri[0 .. idx]; 3111 idx++; 3112 } else { 3113 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3114 idx = 0; 3115 } 3116 3117 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3118 // we have an authority.... 3119 idx += 2; 3120 3121 auto authority_start = idx; 3122 authority_loop: foreach(char c; uri[idx .. $]) { 3123 switch(c) { 3124 case '/': 3125 case '?': 3126 case '#': 3127 break authority_loop; 3128 default: 3129 } 3130 idx++; 3131 } 3132 3133 auto authority = uri[authority_start .. idx]; 3134 3135 auto idx2 = authority.indexOf("@"); 3136 if(idx2 != -1) { 3137 userinfo = authority[0 .. idx2]; 3138 authority = authority[idx2 + 1 .. $]; 3139 } 3140 3141 if(authority.length && authority[0] == '[') { 3142 // ipv6 address special casing 3143 idx2 = authority.indexOf(']'); 3144 if(idx2 != -1) { 3145 auto end = authority[idx2 + 1 .. $]; 3146 if(end.length && end[0] == ':') 3147 idx2 = idx2 + 1; 3148 else 3149 idx2 = -1; 3150 } 3151 } else { 3152 idx2 = authority.indexOf(":"); 3153 } 3154 3155 if(idx2 == -1) { 3156 port = 0; // 0 means not specified; we should use the default for the scheme 3157 host = authority; 3158 } else { 3159 host = authority[0 .. idx2]; 3160 if(idx2 + 1 < authority.length) 3161 port = to!int(authority[idx2 + 1 .. $]); 3162 else 3163 port = 0; 3164 } 3165 } 3166 3167 path_loop: 3168 auto path_start = idx; 3169 3170 foreach(char c; uri[idx .. $]) { 3171 if(c == '?' || c == '#') 3172 break; 3173 idx++; 3174 } 3175 3176 path = uri[path_start .. idx]; 3177 3178 if(idx == uri.length) 3179 return; // nothing more to examine... 3180 3181 if(uri[idx] == '?') { 3182 idx++; 3183 auto query_start = idx; 3184 foreach(char c; uri[idx .. $]) { 3185 if(c == '#') 3186 break; 3187 idx++; 3188 } 3189 query = uri[query_start .. idx]; 3190 } 3191 3192 if(idx < uri.length && uri[idx] == '#') { 3193 idx++; 3194 fragment = uri[idx .. $]; 3195 } 3196 3197 // uriInvalidated = false; 3198 } 3199 3200 private string rebuildUri() const { 3201 string ret; 3202 if(scheme.length) 3203 ret ~= scheme ~ ":"; 3204 if(userinfo.length || host.length) 3205 ret ~= "//"; 3206 if(userinfo.length) 3207 ret ~= userinfo ~ "@"; 3208 if(host.length) 3209 ret ~= host; 3210 if(port) 3211 ret ~= ":" ~ to!string(port); 3212 3213 ret ~= path; 3214 3215 if(query.length) 3216 ret ~= "?" ~ query; 3217 3218 if(fragment.length) 3219 ret ~= "#" ~ fragment; 3220 3221 // uri = ret; 3222 // uriInvalidated = false; 3223 return ret; 3224 } 3225 3226 /// Converts the broken down parts back into a complete string 3227 string toString() const { 3228 // if(uriInvalidated) 3229 return rebuildUri(); 3230 } 3231 3232 /// Returns a new absolute Uri given a base. It treats this one as 3233 /// relative where possible, but absolute if not. (If protocol, domain, or 3234 /// other info is not set, the new one inherits it from the base.) 3235 /// 3236 /// Browsers use a function like this to figure out links in html. 3237 Uri basedOn(in Uri baseUrl) const { 3238 Uri n = this; // copies 3239 if(n.scheme == "data") 3240 return n; 3241 // n.uriInvalidated = true; // make sure we regenerate... 3242 3243 // userinfo is not inherited... is this wrong? 3244 3245 // if anything is given in the existing url, we don't use the base anymore. 3246 if(n.scheme.empty) { 3247 n.scheme = baseUrl.scheme; 3248 if(n.host.empty) { 3249 n.host = baseUrl.host; 3250 if(n.port == 0) { 3251 n.port = baseUrl.port; 3252 if(n.path.length > 0 && n.path[0] != '/') { 3253 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3254 if(b.length == 0) 3255 b = "/"; 3256 n.path = b ~ n.path; 3257 } else if(n.path.length == 0) { 3258 n.path = baseUrl.path; 3259 } 3260 } 3261 } 3262 } 3263 3264 n.removeDots(); 3265 3266 return n; 3267 } 3268 3269 void removeDots() { 3270 auto parts = this.path.split("/"); 3271 string[] toKeep; 3272 foreach(part; parts) { 3273 if(part == ".") { 3274 continue; 3275 } else if(part == "..") { 3276 //if(toKeep.length > 1) 3277 toKeep = toKeep[0 .. $-1]; 3278 //else 3279 //toKeep = [""]; 3280 continue; 3281 } else { 3282 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3283 //continue; // skip a `//` situation 3284 toKeep ~= part; 3285 } 3286 } 3287 3288 auto path = toKeep.join("/"); 3289 if(path.length && path[0] != '/') 3290 path = "/" ~ path; 3291 3292 this.path = path; 3293 } 3294 3295 unittest { 3296 auto uri = Uri("test.html"); 3297 assert(uri.path == "test.html"); 3298 uri = Uri("path/1/lol"); 3299 assert(uri.path == "path/1/lol"); 3300 uri = Uri("http://me@example.com"); 3301 assert(uri.scheme == "http"); 3302 assert(uri.userinfo == "me"); 3303 assert(uri.host == "example.com"); 3304 uri = Uri("http://example.com/#a"); 3305 assert(uri.scheme == "http"); 3306 assert(uri.host == "example.com"); 3307 assert(uri.fragment == "a"); 3308 uri = Uri("#foo"); 3309 assert(uri.fragment == "foo"); 3310 uri = Uri("?lol"); 3311 assert(uri.query == "lol"); 3312 uri = Uri("#foo?lol"); 3313 assert(uri.fragment == "foo?lol"); 3314 uri = Uri("?lol#foo"); 3315 assert(uri.fragment == "foo"); 3316 assert(uri.query == "lol"); 3317 3318 uri = Uri("http://127.0.0.1/"); 3319 assert(uri.host == "127.0.0.1"); 3320 assert(uri.port == 0); 3321 3322 uri = Uri("http://127.0.0.1:123/"); 3323 assert(uri.host == "127.0.0.1"); 3324 assert(uri.port == 123); 3325 3326 uri = Uri("http://[ff:ff::0]/"); 3327 assert(uri.host == "[ff:ff::0]"); 3328 3329 uri = Uri("http://[ff:ff::0]:123/"); 3330 assert(uri.host == "[ff:ff::0]"); 3331 assert(uri.port == 123); 3332 } 3333 3334 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3335 // the possibilities. 3336 unittest { 3337 auto url = Uri("cool.html"); // checking relative links 3338 3339 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3340 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3341 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3342 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3343 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3344 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3345 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3346 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3347 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3348 3349 url = Uri("/something/cool.html"); // same server, different path 3350 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3351 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3352 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3353 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3354 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3355 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3356 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3357 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3358 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3359 3360 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3361 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3362 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3363 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3364 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3365 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3366 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3367 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3368 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3369 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3370 3371 url = Uri("/test/bar"); 3372 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3373 assert(Uri("../").basedOn(url) == "/"); 3374 3375 url = Uri("http://example.com/"); 3376 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3377 3378 //auto uriBefore = url; 3379 url = Uri("#anchor"); // everything should remain the same except the anchor 3380 //uriBefore.anchor = "anchor"); 3381 //assert(url == uriBefore); 3382 3383 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3384 3385 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3386 3387 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3388 3389 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3390 3391 // FIXME: add something for port too 3392 } 3393 3394 // these are like javascript's location.search and location.hash 3395 string search() const { 3396 return query.length ? ("?" ~ query) : ""; 3397 } 3398 string hash() const { 3399 return fragment.length ? ("#" ~ fragment) : ""; 3400 } 3401 } 3402 3403 3404 /* 3405 for session, see web.d 3406 */ 3407 3408 /// breaks down a url encoded string 3409 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3410 auto vars = data.split(separator); 3411 string[][string] _get; 3412 foreach(var; vars) { 3413 auto equal = var.indexOf("="); 3414 string name; 3415 string value; 3416 if(equal == -1) { 3417 name = decodeComponent(var); 3418 value = ""; 3419 } else { 3420 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3421 // stupid + -> space conversion. 3422 name = decodeComponent(var[0..equal].replace("+", " ")); 3423 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3424 } 3425 3426 _get[name] ~= value; 3427 if(namesInOrder) 3428 (*namesInOrder) ~= name; 3429 if(valuesInOrder) 3430 (*valuesInOrder) ~= value; 3431 } 3432 return _get; 3433 } 3434 3435 /// breaks down a url encoded string, but only returns the last value of any array 3436 string[string] decodeVariablesSingle(string data) { 3437 string[string] va; 3438 auto varArray = decodeVariables(data); 3439 foreach(k, v; varArray) 3440 va[k] = v[$-1]; 3441 3442 return va; 3443 } 3444 3445 /// url encodes the whole string 3446 string encodeVariables(in string[string] data) { 3447 string ret; 3448 3449 bool outputted = false; 3450 foreach(k, v; data) { 3451 if(outputted) 3452 ret ~= "&"; 3453 else 3454 outputted = true; 3455 3456 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3457 } 3458 3459 return ret; 3460 } 3461 3462 /// url encodes a whole string 3463 string encodeVariables(in string[][string] data) { 3464 string ret; 3465 3466 bool outputted = false; 3467 foreach(k, arr; data) { 3468 foreach(v; arr) { 3469 if(outputted) 3470 ret ~= "&"; 3471 else 3472 outputted = true; 3473 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3474 } 3475 } 3476 3477 return ret; 3478 } 3479 3480 /// Encodes all but the explicitly unreserved characters per rfc 3986 3481 /// Alphanumeric and -_.~ are the only ones left unencoded 3482 /// name is borrowed from php 3483 string rawurlencode(in char[] data) { 3484 string ret; 3485 ret.reserve(data.length * 2); 3486 foreach(char c; data) { 3487 if( 3488 (c >= 'a' && c <= 'z') || 3489 (c >= 'A' && c <= 'Z') || 3490 (c >= '0' && c <= '9') || 3491 c == '-' || c == '_' || c == '.' || c == '~') 3492 { 3493 ret ~= c; 3494 } else { 3495 ret ~= '%'; 3496 // since we iterate on char, this should give us the octets of the full utf8 string 3497 ret ~= toHexUpper(c); 3498 } 3499 } 3500 3501 return ret; 3502 } 3503 3504 3505 // http helper functions 3506 3507 // for chunked responses (which embedded http does whenever possible) 3508 version(none) // this is moved up above to avoid making a copy of the data 3509 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3510 const(ubyte)[] ret; 3511 3512 ret = cast(const(ubyte)[]) toHex(data.length); 3513 ret ~= cast(const(ubyte)[]) "\r\n"; 3514 ret ~= data; 3515 ret ~= cast(const(ubyte)[]) "\r\n"; 3516 3517 return ret; 3518 } 3519 3520 string toHex(long num) { 3521 string ret; 3522 while(num) { 3523 int v = num % 16; 3524 num /= 16; 3525 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3526 ret ~= d; 3527 } 3528 3529 return to!string(array(ret.retro)); 3530 } 3531 3532 string toHexUpper(long num) { 3533 string ret; 3534 while(num) { 3535 int v = num % 16; 3536 num /= 16; 3537 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3538 ret ~= d; 3539 } 3540 3541 if(ret.length == 1) 3542 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3543 3544 return to!string(array(ret.retro)); 3545 } 3546 3547 3548 // the generic mixins 3549 3550 /++ 3551 Use this instead of writing your own main 3552 3553 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3554 +/ 3555 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3556 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3557 } 3558 3559 /++ 3560 Boilerplate mixin for a main function that uses the [dispatcher] function. 3561 3562 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3563 3564 History: 3565 Added July 9, 2021 3566 +/ 3567 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3568 /// forwards to [CustomCgiDispatcherMain] with default args 3569 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3570 } 3571 3572 /// ditto 3573 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3574 class GenericPresenter : WebPresenter!GenericPresenter {} 3575 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3576 } 3577 3578 /++ 3579 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3580 3581 History: 3582 Added May 13, 2023 (dub v11.0) 3583 +/ 3584 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3585 /++ 3586 Handler to the generated presenter you can use from your objects, etc. 3587 +/ 3588 Presenter activePresenter; 3589 3590 /++ 3591 Request handler that creates the presenter then forwards to the [dispatcher] function. 3592 Renders 404 if the dispatcher did not handle the request. 3593 3594 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3595 +/ 3596 void handler(Cgi cgi) { 3597 auto presenter = new Presenter; 3598 activePresenter = presenter; 3599 scope(exit) activePresenter = null; 3600 3601 if(cgi.dispatcher!DispatcherArgs(presenter)) 3602 return; 3603 3604 switch(cgi.pathInfo) { 3605 case "/style.css": 3606 cgi.setCache(true); 3607 cgi.setResponseContentType("text/css"); 3608 cgi.write(presenter.style(), true); 3609 break; 3610 case "/script.js": 3611 cgi.setCache(true); 3612 cgi.setResponseContentType("application/javascript"); 3613 cgi.write(presenter.script(), true); 3614 break; 3615 default: 3616 presenter.renderBasicError(cgi, 404); 3617 } 3618 } 3619 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3620 } 3621 3622 /// ditto 3623 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3624 class GenericPresenter : WebPresenter!GenericPresenter {} 3625 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3626 3627 } 3628 3629 private string simpleHtmlEncode(string s) { 3630 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3631 } 3632 3633 string messageFromException(Throwable t) { 3634 string message; 3635 if(t !is null) { 3636 debug message = t.toString(); 3637 else message = "An unexpected error has occurred."; 3638 } else { 3639 message = "Unknown error"; 3640 } 3641 return message; 3642 } 3643 3644 string plainHttpError(bool isCgi, string type, Throwable t) { 3645 auto message = messageFromException(t); 3646 message = simpleHtmlEncode(message); 3647 3648 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3649 isCgi ? "Status:" : "HTTP/1.1", 3650 type, message.length, message); 3651 } 3652 3653 // returns true if we were able to recover reasonably 3654 bool handleException(Cgi cgi, Throwable t) { 3655 if(cgi.isClosed) { 3656 // if the channel has been explicitly closed, we can't handle it here 3657 return true; 3658 } 3659 3660 if(cgi.outputtedResponseData) { 3661 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3662 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3663 } else { 3664 // no headers are sent, we can send a full blown error and recover 3665 cgi.setCache(false); 3666 cgi.setResponseContentType("text/html"); 3667 cgi.setResponseLocation(null); // cancel the redirect 3668 cgi.setResponseStatus("500 Internal Server Error"); 3669 cgi.write(simpleHtmlEncode(messageFromException(t))); 3670 cgi.close(); 3671 return true; 3672 } 3673 } 3674 3675 bool isCgiRequestMethod(string s) { 3676 s = s.toUpper(); 3677 if(s == "COMMANDLINE") 3678 return true; 3679 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3680 if(s == member) 3681 return true; 3682 return false; 3683 } 3684 3685 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3686 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3687 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3688 void main(string[] args) { 3689 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3690 } 3691 } 3692 3693 version(embedded_httpd_processes) 3694 __gshared int processPoolSize = 8; 3695 3696 // Returns true if run. You should exit the program after that. 3697 bool tryAddonServers(string[] args) { 3698 if(args.length > 1) { 3699 // run the special separate processes if needed 3700 switch(args[1]) { 3701 case "--websocket-server": 3702 version(with_addon_servers) 3703 websocketServers[args[2]](args[3 .. $]); 3704 else 3705 printf("Add-on servers not compiled in.\n"); 3706 return true; 3707 case "--websocket-servers": 3708 import core.demangle; 3709 version(with_addon_servers_connections) 3710 foreach(k, v; websocketServers) 3711 writeln(k, "\t", demangle(k)); 3712 return true; 3713 case "--session-server": 3714 version(with_addon_servers) 3715 runSessionServer(); 3716 else 3717 printf("Add-on servers not compiled in.\n"); 3718 return true; 3719 case "--event-server": 3720 version(with_addon_servers) 3721 runEventServer(); 3722 else 3723 printf("Add-on servers not compiled in.\n"); 3724 return true; 3725 case "--timer-server": 3726 version(with_addon_servers) 3727 runTimerServer(); 3728 else 3729 printf("Add-on servers not compiled in.\n"); 3730 return true; 3731 case "--timed-jobs": 3732 import core.demangle; 3733 version(with_addon_servers_connections) 3734 foreach(k, v; scheduledJobHandlers) 3735 writeln(k, "\t", demangle(k)); 3736 return true; 3737 case "--timed-job": 3738 scheduledJobHandlers[args[2]](args[3 .. $]); 3739 return true; 3740 default: 3741 // intentionally blank - do nothing and carry on to run normally 3742 } 3743 } 3744 return false; 3745 } 3746 3747 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3748 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3749 // we support command line thing for easy testing everywhere 3750 // it needs to be called ./app method uri [other args...] 3751 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3752 Cgi cgi = new CustomCgi(args); 3753 scope(exit) cgi.dispose(); 3754 fun(cgi); 3755 cgi.close(); 3756 return true; 3757 } 3758 return false; 3759 } 3760 3761 /++ 3762 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3763 3764 As of version 11 (released August 2023), you can also make things like this: 3765 3766 --- 3767 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3768 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3769 3770 // can also: 3771 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3772 3773 // NOT IMPLEMENTED YET 3774 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3775 3776 foreach(listenSpec; server.listenSpecs) { 3777 // you can check what it actually bound to here and see your assigned ports 3778 } 3779 3780 // NOT IMPLEMENTED YET 3781 // server.start!handler(); // starts and runs in the arsd.core event loop 3782 3783 server.serve!handler(); // blocks the thread until the server exits 3784 --- 3785 3786 History: 3787 Added Sept 26, 2020 (release version 8.5). 3788 3789 The `listenSpec` member was added July 31, 2023. 3790 +/ 3791 struct RequestServer { 3792 /++ 3793 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3794 +/ 3795 string listeningHost = defaultListeningHost(); 3796 /// ditto 3797 ushort listeningPort = defaultListeningPort(); 3798 3799 static struct ListenSpec { 3800 enum Protocol { 3801 http, 3802 https, 3803 scgi 3804 } 3805 Protocol protocol; 3806 3807 enum AddressType { 3808 ip, 3809 unix, 3810 abstract_ 3811 } 3812 AddressType addressType; 3813 3814 string address; 3815 ushort port; 3816 } 3817 3818 /++ 3819 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3820 3821 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3822 3823 `http://localhost:8080` 3824 3825 `http://unix:filename/here` 3826 3827 `scgi://abstract:/name/here` 3828 3829 `http://[::1]:4444` 3830 3831 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3832 3833 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3834 3835 `localhost:8080` serves the default protocol. 3836 3837 `8080` or `:8080` assumes default protocol on localhost. 3838 3839 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3840 3841 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3842 3843 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3844 3845 $(PITFALL 3846 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3847 ) 3848 3849 Bugs: 3850 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3851 3852 History: 3853 Added July 31, 2023 (dub v11.0) 3854 +/ 3855 string[] listenSpec; 3856 3857 /++ 3858 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3859 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3860 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3861 [stop] may not work as well. 3862 3863 History: 3864 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3865 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3866 compatibility. 3867 +/ 3868 bool useFork = cgi_use_fork_default; 3869 3870 /++ 3871 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3872 default based on the number of cpus modified by the server mode. 3873 3874 History: 3875 Added August 12, 2022 (dub v10.9) 3876 +/ 3877 int numberOfThreads = 0; 3878 3879 /++ 3880 Creates a server configured to listen to multiple URLs. 3881 3882 History: 3883 Added July 31, 2023 (dub v11.0) 3884 +/ 3885 this(string[] listenTo) { 3886 this.listenSpec = listenTo; 3887 } 3888 3889 /// Creates a server object configured to listen on a single host and port. 3890 this(string defaultHost, ushort defaultPort) { 3891 this.listeningHost = defaultHost; 3892 this.listeningPort = defaultPort; 3893 } 3894 3895 /// ditto 3896 this(ushort defaultPort) { 3897 listeningPort = defaultPort; 3898 } 3899 3900 /++ 3901 Reads the command line arguments into the values here. 3902 3903 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3904 3905 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 3906 +/ 3907 void configureFromCommandLine(string[] args) { 3908 bool portOrHostFound = false; 3909 3910 bool foundPort = false; 3911 bool foundHost = false; 3912 bool foundUid = false; 3913 bool foundGid = false; 3914 bool foundListen = false; 3915 foreach(arg; args) { 3916 if(foundPort) { 3917 listeningPort = to!ushort(arg); 3918 portOrHostFound = true; 3919 foundPort = false; 3920 continue; 3921 } 3922 if(foundHost) { 3923 listeningHost = arg; 3924 portOrHostFound = true; 3925 foundHost = false; 3926 continue; 3927 } 3928 if(foundUid) { 3929 privilegesDropToUid = to!uid_t(arg); 3930 foundUid = false; 3931 continue; 3932 } 3933 if(foundGid) { 3934 privilegesDropToGid = to!gid_t(arg); 3935 foundGid = false; 3936 continue; 3937 } 3938 if(foundListen) { 3939 this.listenSpec ~= arg; 3940 foundListen = false; 3941 continue; 3942 } 3943 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3944 foundHost = true; 3945 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3946 foundPort = true; 3947 else if(arg == "--uid") 3948 foundUid = true; 3949 else if(arg == "--gid") 3950 foundGid = true; 3951 else if(arg == "--listen") 3952 foundListen = true; 3953 } 3954 3955 if(portOrHostFound && listenSpec.length) { 3956 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 3957 } 3958 } 3959 3960 version(Windows) { 3961 private alias uid_t = int; 3962 private alias gid_t = int; 3963 } 3964 3965 /// user (uid) to drop privileges to 3966 /// 0 … do nothing 3967 uid_t privilegesDropToUid = 0; 3968 /// group (gid) to drop privileges to 3969 /// 0 … do nothing 3970 gid_t privilegesDropToGid = 0; 3971 3972 private void dropPrivileges() { 3973 version(Posix) { 3974 import core.sys.posix.unistd; 3975 3976 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 3977 throw new Exception("Dropping privileges via setgid() failed."); 3978 3979 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 3980 throw new Exception("Dropping privileges via setuid() failed."); 3981 } 3982 else { 3983 // FIXME: Windows? 3984 //pragma(msg, "Dropping privileges is not implemented for this platform"); 3985 } 3986 3987 // done, set zero 3988 privilegesDropToGid = 0; 3989 privilegesDropToUid = 0; 3990 } 3991 3992 /++ 3993 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 3994 3995 History: 3996 Added Oct 10, 2020. 3997 Example: 3998 3999 --- 4000 import arsd.cgi; 4001 void main() { 4002 RequestServer server = RequestServer("127.0.0.1", 6789); 4003 string oauthCode; 4004 string oauthScope; 4005 server.serveHttpOnce!((cgi) { 4006 oauthCode = cgi.request("code"); 4007 oauthScope = cgi.request("scope"); 4008 cgi.write("Thank you, please return to the application."); 4009 }); 4010 // use the code and scope given 4011 } 4012 --- 4013 +/ 4014 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4015 import std.socket; 4016 4017 bool tcp; 4018 void delegate() cleanup; 4019 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4020 auto connection = socket.accept(); 4021 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4022 4023 if(cleanup) 4024 cleanup(); 4025 } 4026 4027 /++ 4028 Starts serving requests according to the current configuration. 4029 +/ 4030 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4031 version(netman_httpd) { 4032 // Obsolete! 4033 4034 import arsd.httpd; 4035 // what about forwarding the other constructor args? 4036 // this probably needs a whole redoing... 4037 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4038 return; 4039 } else 4040 version(embedded_httpd_processes) { 4041 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4042 } else 4043 version(embedded_httpd_threads) { 4044 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4045 } else 4046 version(scgi) { 4047 serveScgi!(fun, CustomCgi, maxContentLength)(); 4048 } else 4049 version(fastcgi) { 4050 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4051 } else 4052 version(stdio_http) { 4053 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4054 } else { 4055 //version=plain_cgi; 4056 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4057 } 4058 } 4059 4060 /++ 4061 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4062 4063 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4064 +/ 4065 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4066 globalStopFlag = false; 4067 static if(__traits(isStaticFunction, fun)) 4068 alias funToUse = fun; 4069 else 4070 void funToUse(CustomCgi cgi) { 4071 static if(__VERSION__ > 2097) 4072 __traits(child, _this, fun)(cgi); 4073 else static assert(0, "Not implemented in your compiler version!"); 4074 } 4075 auto manager = this.listenSpec is null ? 4076 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4077 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4078 manager.listen(); 4079 } 4080 4081 /++ 4082 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4083 +/ 4084 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4085 globalStopFlag = false; 4086 auto manager = this.listenSpec is null ? 4087 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4088 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4089 manager.listen(); 4090 } 4091 4092 /++ 4093 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4094 4095 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4096 4097 History: 4098 Added May 29, 2021 4099 +/ 4100 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4101 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4102 } 4103 4104 /++ 4105 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4106 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4107 +/ 4108 enum ForceStop { 4109 /++ 4110 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4111 +/ 4112 afterQueuedRequestsComplete, 4113 /++ 4114 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4115 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4116 +/ 4117 afterCurrentRequestsComplete, 4118 /++ 4119 Partial response writes will throw an exception, cancelling any streaming response, but complete 4120 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4121 +/ 4122 cancelStreamingRequestsEarly, 4123 /++ 4124 All writes will throw. 4125 +/ 4126 cancelAllRequestsEarly, 4127 /++ 4128 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4129 +/ 4130 forciblyTerminate, 4131 } 4132 4133 version(embedded_httpd_processes) {} else 4134 /++ 4135 Stops serving after the current requests are completed. 4136 4137 Bugs: 4138 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4139 on Windows however). Only partially implemented on non-Linux posix systems. 4140 4141 You might also try SIGINT perhaps. 4142 4143 The stopPriority is not yet fully implemented. 4144 +/ 4145 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4146 globalStopFlag = true; 4147 4148 version(Posix) { 4149 if(cancelfd > 0) { 4150 ulong a = 1; 4151 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4152 } 4153 } 4154 version(Windows) { 4155 if(iocp) { 4156 foreach(i; 0 .. 16) // FIXME 4157 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4158 } 4159 } 4160 } 4161 } 4162 4163 private alias AliasSeq(T...) = T; 4164 4165 version(with_breaking_cgi_features) 4166 mixin(q{ 4167 template ThisFor(alias t) { 4168 static if(__traits(isStaticFunction, t)) { 4169 alias ThisFor = AliasSeq!(); 4170 } else { 4171 alias ThisFor = __traits(parent, t); 4172 } 4173 } 4174 }); 4175 else 4176 alias ThisFor(alias t) = AliasSeq!(); 4177 4178 private __gshared bool globalStopFlag = false; 4179 4180 version(embedded_httpd_processes) 4181 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4182 import core.sys.posix.unistd; 4183 import core.sys.posix.sys.socket; 4184 import core.sys.posix.netinet.in_; 4185 //import std.c.linux.socket; 4186 4187 int sock = socket(AF_INET, SOCK_STREAM, 0); 4188 if(sock == -1) 4189 throw new Exception("socket"); 4190 4191 cloexec(sock); 4192 4193 { 4194 4195 sockaddr_in addr; 4196 addr.sin_family = AF_INET; 4197 addr.sin_port = htons(params.listeningPort); 4198 auto lh = params.listeningHost; 4199 if(lh.length) { 4200 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4201 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4202 } else 4203 addr.sin_addr.s_addr = INADDR_ANY; 4204 4205 // HACKISH 4206 int on = 1; 4207 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4208 // end hack 4209 4210 4211 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4212 close(sock); 4213 throw new Exception("bind"); 4214 } 4215 4216 // FIXME: if this queue is full, it will just ignore it 4217 // and wait for the client to retransmit it. This is an 4218 // obnoxious timeout condition there. 4219 if(sock.listen(128) == -1) { 4220 close(sock); 4221 throw new Exception("listen"); 4222 } 4223 params.dropPrivileges(); 4224 } 4225 4226 version(embedded_httpd_processes_accept_after_fork) {} else { 4227 int pipeReadFd; 4228 int pipeWriteFd; 4229 4230 { 4231 int[2] pipeFd; 4232 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4233 import core.stdc.errno; 4234 throw new Exception("pipe failed " ~ to!string(errno)); 4235 } 4236 4237 pipeReadFd = pipeFd[0]; 4238 pipeWriteFd = pipeFd[1]; 4239 } 4240 } 4241 4242 4243 int processCount; 4244 pid_t newPid; 4245 reopen: 4246 while(processCount < processPoolSize) { 4247 newPid = fork(); 4248 if(newPid == 0) { 4249 // start serving on the socket 4250 //ubyte[4096] backingBuffer; 4251 for(;;) { 4252 bool closeConnection; 4253 uint i; 4254 sockaddr addr; 4255 i = addr.sizeof; 4256 version(embedded_httpd_processes_accept_after_fork) { 4257 int s = accept(sock, &addr, &i); 4258 int opt = 1; 4259 import core.sys.posix.netinet.tcp; 4260 // the Cgi class does internal buffering, so disabling this 4261 // helps with latency in many cases... 4262 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4263 cloexec(s); 4264 } else { 4265 int s; 4266 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4267 if(readret != s.sizeof) { 4268 import core.stdc.errno; 4269 throw new Exception("pipe read failed " ~ to!string(errno)); 4270 } 4271 4272 //writeln("process ", getpid(), " got socket ", s); 4273 } 4274 4275 try { 4276 4277 if(s == -1) 4278 throw new Exception("accept"); 4279 4280 scope(failure) close(s); 4281 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4282 auto ir = new BufferedInputRange(s); 4283 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4284 4285 while(!ir.empty) { 4286 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4287 4288 Cgi cgi; 4289 try { 4290 cgi = new CustomCgi(ir, &closeConnection); 4291 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4292 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4293 if(processPoolSize <= 1) 4294 closeConnection = true; 4295 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4296 } catch(HttpVersionNotSupportedException he) { 4297 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4298 closeConnection = true; 4299 break; 4300 } catch(Throwable t) { 4301 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4302 // anyway let's kill the connection 4303 version(CRuntime_Musl) { 4304 // LockingTextWriter fails here 4305 // so working around it 4306 auto estr = t.toString(); 4307 stderr.rawWrite(estr); 4308 stderr.rawWrite("\n"); 4309 } else 4310 stderr.writeln(t.toString()); 4311 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4312 closeConnection = true; 4313 break; 4314 } 4315 assert(cgi !is null); 4316 scope(exit) 4317 cgi.dispose(); 4318 4319 try { 4320 fun(cgi); 4321 cgi.close(); 4322 if(cgi.websocketMode) 4323 closeConnection = true; 4324 } catch(ConnectionException ce) { 4325 closeConnection = true; 4326 } catch(Throwable t) { 4327 // a processing error can be recovered from 4328 version(CRuntime_Musl) { 4329 // LockingTextWriter fails here 4330 // so working around it 4331 auto estr = t.toString(); 4332 stderr.rawWrite(estr); 4333 } else { 4334 stderr.writeln(t.toString); 4335 } 4336 if(!handleException(cgi, t)) 4337 closeConnection = true; 4338 } 4339 4340 if(closeConnection) { 4341 ir.source.close(); 4342 break; 4343 } else { 4344 if(!ir.empty) 4345 ir.popFront(); // get the next 4346 else if(ir.sourceClosed) { 4347 ir.source.close(); 4348 } 4349 } 4350 } 4351 4352 ir.source.close(); 4353 } catch(Throwable t) { 4354 version(CRuntime_Musl) {} else 4355 debug writeln(t); 4356 // most likely cause is a timeout 4357 } 4358 } 4359 } else if(newPid < 0) { 4360 throw new Exception("fork failed"); 4361 } else { 4362 processCount++; 4363 } 4364 } 4365 4366 // the parent should wait for its children... 4367 if(newPid) { 4368 import core.sys.posix.sys.wait; 4369 4370 version(embedded_httpd_processes_accept_after_fork) {} else { 4371 import core.sys.posix.sys.select; 4372 int[] fdQueue; 4373 while(true) { 4374 // writeln("select call"); 4375 int nfds = pipeWriteFd; 4376 if(sock > pipeWriteFd) 4377 nfds = sock; 4378 nfds += 1; 4379 fd_set read_fds; 4380 fd_set write_fds; 4381 FD_ZERO(&read_fds); 4382 FD_ZERO(&write_fds); 4383 FD_SET(sock, &read_fds); 4384 if(fdQueue.length) 4385 FD_SET(pipeWriteFd, &write_fds); 4386 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4387 if(ret == -1) { 4388 import core.stdc.errno; 4389 if(errno == EINTR) 4390 goto try_wait; 4391 else 4392 throw new Exception("wtf select"); 4393 } 4394 4395 int s = -1; 4396 if(FD_ISSET(sock, &read_fds)) { 4397 uint i; 4398 sockaddr addr; 4399 i = addr.sizeof; 4400 s = accept(sock, &addr, &i); 4401 cloexec(s); 4402 import core.sys.posix.netinet.tcp; 4403 int opt = 1; 4404 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4405 } 4406 4407 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4408 if(s == -1 && fdQueue.length) { 4409 s = fdQueue[0]; 4410 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4411 } 4412 write_fd(pipeWriteFd, &s, s.sizeof, s); 4413 close(s); // we are done with it, let the other process take ownership 4414 } else 4415 fdQueue ~= s; 4416 } 4417 } 4418 4419 try_wait: 4420 4421 int status; 4422 while(-1 != wait(&status)) { 4423 version(CRuntime_Musl) {} else { 4424 import std.stdio; writeln("Process died ", status); 4425 } 4426 processCount--; 4427 goto reopen; 4428 } 4429 close(sock); 4430 } 4431 } 4432 4433 version(fastcgi) 4434 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4435 // SetHandler fcgid-script 4436 FCGX_Stream* input, output, error; 4437 FCGX_ParamArray env; 4438 4439 4440 4441 const(ubyte)[] getFcgiChunk() { 4442 const(ubyte)[] ret; 4443 while(FCGX_HasSeenEOF(input) != -1) 4444 ret ~= cast(ubyte) FCGX_GetChar(input); 4445 return ret; 4446 } 4447 4448 void writeFcgi(const(ubyte)[] data) { 4449 FCGX_PutStr(data.ptr, data.length, output); 4450 } 4451 4452 void doARequest() { 4453 string[string] fcgienv; 4454 4455 for(auto e = env; e !is null && *e !is null; e++) { 4456 string cur = to!string(*e); 4457 auto idx = cur.indexOf("="); 4458 string name, value; 4459 if(idx == -1) 4460 name = cur; 4461 else { 4462 name = cur[0 .. idx]; 4463 value = cur[idx + 1 .. $]; 4464 } 4465 4466 fcgienv[name] = value; 4467 } 4468 4469 void flushFcgi() { 4470 FCGX_FFlush(output); 4471 } 4472 4473 Cgi cgi; 4474 try { 4475 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4476 } catch(Throwable t) { 4477 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4478 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4479 return; //continue; 4480 } 4481 assert(cgi !is null); 4482 scope(exit) cgi.dispose(); 4483 try { 4484 fun(cgi); 4485 cgi.close(); 4486 } catch(Throwable t) { 4487 // log it to the error stream 4488 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4489 // handle it for the user, if we can 4490 if(!handleException(cgi, t)) 4491 return; // continue; 4492 } 4493 } 4494 4495 auto lp = params.listeningPort; 4496 auto host = params.listeningHost; 4497 4498 FCGX_Request request; 4499 if(lp || !host.empty) { 4500 // if a listening port was specified on the command line, we want to spawn ourself 4501 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4502 FCGX_Init(); 4503 4504 int sock; 4505 4506 if(host.startsWith("unix:")) { 4507 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4508 } else if(host.startsWith("abstract:")) { 4509 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4510 } else { 4511 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4512 } 4513 4514 if(sock < 0) 4515 throw new Exception("Couldn't listen on the port"); 4516 FCGX_InitRequest(&request, sock, 0); 4517 while(FCGX_Accept_r(&request) >= 0) { 4518 input = request.inStream; 4519 output = request.outStream; 4520 error = request.errStream; 4521 env = request.envp; 4522 doARequest(); 4523 } 4524 } else { 4525 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4526 // using the version with a global variable since we are separate processes anyway 4527 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4528 doARequest(); 4529 } 4530 } 4531 } 4532 4533 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4534 ushort defaultListeningPort() { 4535 version(netman_httpd) 4536 return 8080; 4537 else version(embedded_httpd_processes) 4538 return 8085; 4539 else version(embedded_httpd_threads) 4540 return 8085; 4541 else version(scgi) 4542 return 4000; 4543 else 4544 return 0; 4545 } 4546 4547 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4548 string defaultListeningHost() { 4549 version(netman_httpd) 4550 return null; 4551 else version(embedded_httpd_processes) 4552 return null; 4553 else version(embedded_httpd_threads) 4554 return null; 4555 else version(scgi) 4556 return "127.0.0.1"; 4557 else 4558 return null; 4559 4560 } 4561 4562 /++ 4563 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4564 4565 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4566 4567 Params: 4568 fun = Your request handler 4569 CustomCgi = a subclass of Cgi, if you wise to customize it further 4570 maxContentLength = max POST size you want to allow 4571 args = command-line arguments 4572 4573 History: 4574 Documented Sept 26, 2020. 4575 +/ 4576 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4577 if(tryAddonServers(args)) 4578 return; 4579 4580 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4581 return; 4582 4583 RequestServer server; 4584 // you can change the port here if you like 4585 // server.listeningPort = 9000; 4586 4587 // then call this to let the command line args override your default 4588 server.configureFromCommandLine(args); 4589 4590 // and serve the request(s). 4591 server.serve!(fun, CustomCgi, maxContentLength)(); 4592 } 4593 4594 //version(plain_cgi) 4595 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4596 // standard CGI is the default version 4597 4598 4599 // Set stdin to binary mode if necessary to avoid mangled newlines 4600 // the fact that stdin is global means this could be trouble but standard cgi request 4601 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4602 version(Windows) { 4603 version(Win64) 4604 _setmode(std.stdio.stdin.fileno(), 0x8000); 4605 else 4606 setmode(std.stdio.stdin.fileno(), 0x8000); 4607 } 4608 4609 Cgi cgi; 4610 try { 4611 cgi = new CustomCgi(maxContentLength); 4612 version(Posix) 4613 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4614 else version(Windows) 4615 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4616 else static assert(0); 4617 } catch(Throwable t) { 4618 version(CRuntime_Musl) { 4619 // LockingTextWriter fails here 4620 // so working around it 4621 auto s = t.toString(); 4622 stderr.rawWrite(s); 4623 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4624 } else { 4625 stderr.writeln(t.msg); 4626 // the real http server will probably handle this; 4627 // most likely, this is a bug in Cgi. But, oh well. 4628 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4629 } 4630 return; 4631 } 4632 assert(cgi !is null); 4633 scope(exit) cgi.dispose(); 4634 4635 try { 4636 fun(cgi); 4637 cgi.close(); 4638 } catch (Throwable t) { 4639 version(CRuntime_Musl) { 4640 // LockingTextWriter fails here 4641 // so working around it 4642 auto s = t.msg; 4643 stderr.rawWrite(s); 4644 } else { 4645 stderr.writeln(t.msg); 4646 } 4647 if(!handleException(cgi, t)) 4648 return; 4649 } 4650 } 4651 4652 private __gshared int cancelfd = -1; 4653 4654 /+ 4655 The event loop for embedded_httpd_threads will prolly fiber dispatch 4656 cgi constructors too, so slow posts will not monopolize a worker thread. 4657 4658 May want to provide the worker task system just need to ensure all the fibers 4659 has a big enough stack for real work... would also ideally like to reuse them. 4660 4661 4662 So prolly bir would switch it to nonblocking. If it would block, it epoll 4663 registers one shot with this existing fiber to take it over. 4664 4665 new connection comes in. it picks a fiber off the free list, 4666 or if there is none, it creates a new one. this fiber handles 4667 this connection the whole time. 4668 4669 epoll triggers the fiber when something comes in. it is called by 4670 a random worker thread, it might change at any time. at least during 4671 the constructor. maybe into the main body it will stay tied to a thread 4672 just so TLS stuff doesn't randomly change in the middle. but I could 4673 specify if you yield all bets are off. 4674 4675 when the request is finished, if there's more data buffered, it just 4676 keeps going. if there is no more data buffered, it epoll ctls to 4677 get triggered when more data comes in. all one shot. 4678 4679 when a connection is closed, the fiber returns and is then reset 4680 and added to the free list. if the free list is full, the fiber is 4681 just freed, this means it will balloon to a certain size but not generally 4682 grow beyond that unless the activity keeps going. 4683 4684 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4685 4686 So the fiber has its own magic methods to read and write. if they would block, it registers 4687 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4688 4689 basically you issue the command and it tells you when it is done 4690 4691 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4692 4693 +/ 4694 4695 /++ 4696 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4697 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4698 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4699 4700 History: 4701 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4702 +/ 4703 version(cgi_use_fiber) 4704 __gshared size_t fiberStackSize = 4096 * 100; 4705 4706 version(cgi_use_fiber) 4707 class CgiFiber : Fiber { 4708 private void function(Socket) f_handler; 4709 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4710 f_handler(s); 4711 } 4712 this(void function(Socket) handler) { 4713 this.f_handler = handler; 4714 this(&f_handler_dg); 4715 } 4716 4717 this(void delegate(Socket) handler) { 4718 this.handler = handler; 4719 super(&run, fiberStackSize); 4720 } 4721 4722 Socket connection; 4723 void delegate(Socket) handler; 4724 4725 void run() { 4726 handler(connection); 4727 } 4728 4729 void delegate() postYield; 4730 4731 private void setPostYield(scope void delegate() py) @nogc { 4732 postYield = cast(void delegate()) py; 4733 } 4734 4735 void proceed() { 4736 try { 4737 call(); 4738 auto py = postYield; 4739 postYield = null; 4740 if(py !is null) 4741 py(); 4742 } catch(Exception e) { 4743 if(connection) 4744 connection.close(); 4745 goto terminate; 4746 } 4747 4748 if(state == State.TERM) { 4749 terminate: 4750 import core.memory; 4751 GC.removeRoot(cast(void*) this); 4752 } 4753 } 4754 } 4755 4756 version(cgi_use_fiber) 4757 version(Windows) { 4758 4759 extern(Windows) private { 4760 4761 import core.sys.windows.mswsock; 4762 4763 alias GROUP=uint; 4764 alias LPWSAPROTOCOL_INFOW = void*; 4765 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4766 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4767 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4768 4769 struct WSABUF { 4770 ULONG len; 4771 CHAR *buf; 4772 } 4773 alias LPWSABUF = WSABUF*; 4774 4775 alias WSAOVERLAPPED = OVERLAPPED; 4776 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4777 /+ 4778 4779 alias LPFN_ACCEPTEX = 4780 BOOL 4781 function( 4782 SOCKET sListenSocket, 4783 SOCKET sAcceptSocket, 4784 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4785 void* lpOutputBuffer, 4786 WORD dwReceiveDataLength, 4787 WORD dwLocalAddressLength, 4788 WORD dwRemoteAddressLength, 4789 LPDWORD lpdwBytesReceived, 4790 LPOVERLAPPED lpOverlapped 4791 ); 4792 4793 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4794 +/ 4795 4796 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4797 } 4798 4799 private class PseudoblockingOverlappedSocket : Socket { 4800 SOCKET handle; 4801 4802 CgiFiber fiber; 4803 4804 this(AddressFamily af, SocketType st) { 4805 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4806 if(!handle) 4807 throw new Exception("WSASocketW"); 4808 this.handle = handle; 4809 4810 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4811 4812 if(iocp is null) { 4813 writeln(GetLastError()); 4814 throw new Exception("CreateIoCompletionPort"); 4815 } 4816 4817 super(cast(socket_t) handle, af); 4818 } 4819 this() pure nothrow @trusted { assert(0); } 4820 4821 override void blocking(bool) {} // meaningless to us, just ignore it. 4822 4823 protected override Socket accepting() pure nothrow { 4824 assert(0); 4825 } 4826 4827 bool addressesParsed; 4828 Address la; 4829 Address ra; 4830 4831 private void populateAddresses() { 4832 if(addressesParsed) 4833 return; 4834 addressesParsed = true; 4835 4836 int lalen, ralen; 4837 4838 sockaddr_in* la; 4839 sockaddr_in* ra; 4840 4841 lpfnGetAcceptExSockaddrs( 4842 scratchBuffer.ptr, 4843 0, // same as in the AcceptEx call! 4844 sockaddr_in.sizeof + 16, 4845 sockaddr_in.sizeof + 16, 4846 cast(sockaddr**) &la, 4847 &lalen, 4848 cast(sockaddr**) &ra, 4849 &ralen 4850 ); 4851 4852 if(la) 4853 this.la = new InternetAddress(*la); 4854 if(ra) 4855 this.ra = new InternetAddress(*ra); 4856 4857 } 4858 4859 override @property @trusted Address localAddress() { 4860 populateAddresses(); 4861 return la; 4862 } 4863 override @property @trusted Address remoteAddress() { 4864 populateAddresses(); 4865 return ra; 4866 } 4867 4868 PseudoblockingOverlappedSocket accepted; 4869 4870 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4871 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4872 4873 override Socket accept() @trusted { 4874 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4875 4876 if(lpfnAcceptEx is null) { 4877 DWORD dwBytes; 4878 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4879 4880 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4881 &GuidAcceptEx, GuidAcceptEx.sizeof, 4882 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4883 &dwBytes, null, null); 4884 4885 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4886 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4887 &GuidAcceptEx, GuidAcceptEx.sizeof, 4888 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4889 &dwBytes, null, null); 4890 4891 } 4892 4893 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4894 accepted = pfa; 4895 4896 SOCKET pendingForAccept = pfa.handle; 4897 DWORD ignored; 4898 4899 auto ret = lpfnAcceptEx(handle, 4900 pendingForAccept, 4901 // buffer to receive up front 4902 pfa.scratchBuffer.ptr, 4903 0, 4904 // size of local and remote addresses. normally + 16. 4905 sockaddr_in.sizeof + 16, 4906 sockaddr_in.sizeof + 16, 4907 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4908 &overlapped 4909 ); 4910 4911 return pfa; 4912 } 4913 4914 override void connect(Address to) { assert(0); } 4915 4916 DWORD lastAnswer; 4917 ubyte[1024] scratchBuffer; 4918 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4919 4920 WSABUF[1] buffer; 4921 OVERLAPPED overlapped; 4922 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4923 overlapped = overlapped.init; 4924 buffer[0].len = cast(DWORD) buf.length; 4925 buffer[0].buf = cast(CHAR*) buf.ptr; 4926 fiber.setPostYield( () { 4927 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4928 if(GetLastError() != 997) { 4929 //throw new Exception("WSASend fail"); 4930 } 4931 } 4932 }); 4933 4934 Fiber.yield(); 4935 return lastAnswer; 4936 } 4937 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4938 overlapped = overlapped.init; 4939 buffer[0].len = cast(DWORD) buf.length; 4940 buffer[0].buf = cast(CHAR*) buf.ptr; 4941 4942 DWORD flags2 = 0; 4943 4944 fiber.setPostYield(() { 4945 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4946 if(GetLastError() != 997) { 4947 //writeln("WSARecv ", WSAGetLastError()); 4948 //throw new Exception("WSARecv fail"); 4949 } 4950 } 4951 }); 4952 4953 Fiber.yield(); 4954 return lastAnswer; 4955 } 4956 4957 // I might go back and implement these for udp things. 4958 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4959 assert(0); 4960 } 4961 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4962 assert(0); 4963 } 4964 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4965 assert(0); 4966 } 4967 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4968 assert(0); 4969 } 4970 4971 // lol overload sets 4972 alias send = typeof(super).send; 4973 alias receive = typeof(super).receive; 4974 alias sendTo = typeof(super).sendTo; 4975 alias receiveFrom = typeof(super).receiveFrom; 4976 4977 } 4978 } 4979 4980 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 4981 assert(connection !is null); 4982 version(cgi_use_fiber) { 4983 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 4984 4985 version(Windows) { 4986 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 4987 } 4988 4989 import core.memory; 4990 GC.addRoot(cast(void*) fiber); 4991 fiber.connection = connection; 4992 fiber.proceed(); 4993 } else { 4994 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 4995 } 4996 } 4997 4998 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 4999 scope(failure) { 5000 // catch all for other errors 5001 try { 5002 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5003 connection.close(); 5004 } catch(Exception e) {} // swallow it, we're aborting anyway. 5005 } 5006 5007 bool closeConnection = alwaysCloseConnection; 5008 5009 /+ 5010 ubyte[4096] inputBuffer = void; 5011 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5012 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5013 5014 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5015 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5016 ir.__ctor(connection, inputBuffer[], true); 5017 +/ 5018 5019 auto ir = new BufferedInputRange(connection); 5020 5021 while(!ir.empty) { 5022 5023 if(ir.view.length == 0) { 5024 ir.popFront(); 5025 if(ir.sourceClosed) { 5026 connection.close(); 5027 closeConnection = true; 5028 break; 5029 } 5030 } 5031 5032 Cgi cgi; 5033 try { 5034 cgi = new CustomCgi(ir, &closeConnection); 5035 // There's a bunch of these casts around because the type matches up with 5036 // the -version=.... specifiers, just you can also create a RequestServer 5037 // and instantiate the things where the types don't match up. It isn't exactly 5038 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5039 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5040 } catch(ConnectionClosedException ce) { 5041 closeConnection = true; 5042 break; 5043 } catch(ConnectionException ce) { 5044 // broken pipe or something, just abort the connection 5045 closeConnection = true; 5046 break; 5047 } catch(HttpVersionNotSupportedException ve) { 5048 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5049 closeConnection = true; 5050 break; 5051 } catch(Throwable t) { 5052 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5053 // anyway let's kill the connection 5054 version(CRuntime_Musl) { 5055 stderr.rawWrite(t.toString()); 5056 stderr.rawWrite("\n"); 5057 } else { 5058 stderr.writeln(t.toString()); 5059 } 5060 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5061 closeConnection = true; 5062 break; 5063 } 5064 assert(cgi !is null); 5065 scope(exit) 5066 cgi.dispose(); 5067 5068 try { 5069 fun(cgi); 5070 cgi.close(); 5071 if(cgi.websocketMode) 5072 closeConnection = true; 5073 } catch(ConnectionException ce) { 5074 // broken pipe or something, just abort the connection 5075 closeConnection = true; 5076 } catch(ConnectionClosedException ce) { 5077 // broken pipe or something, just abort the connection 5078 closeConnection = true; 5079 } catch(Throwable t) { 5080 // a processing error can be recovered from 5081 version(CRuntime_Musl) {} else 5082 stderr.writeln(t.toString); 5083 if(!handleException(cgi, t)) 5084 closeConnection = true; 5085 } 5086 5087 if(globalStopFlag) 5088 closeConnection = true; 5089 5090 if(closeConnection || alwaysCloseConnection) { 5091 connection.shutdown(SocketShutdown.BOTH); 5092 connection.close(); 5093 ir.dispose(); 5094 closeConnection = false; // don't reclose after loop 5095 break; 5096 } else { 5097 if(ir.front.length) { 5098 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5099 } else if(ir.sourceClosed) { 5100 ir.source.shutdown(SocketShutdown.BOTH); 5101 ir.source.close(); 5102 ir.dispose(); 5103 closeConnection = false; 5104 } else { 5105 continue; 5106 // break; // this was for a keepalive experiment 5107 } 5108 } 5109 } 5110 5111 if(closeConnection) { 5112 connection.shutdown(SocketShutdown.BOTH); 5113 connection.close(); 5114 ir.dispose(); 5115 } 5116 5117 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5118 } 5119 5120 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5121 // and now we can buffer 5122 scope(failure) 5123 connection.close(); 5124 5125 import al = std.algorithm; 5126 5127 size_t size; 5128 5129 string[string] headers; 5130 5131 auto range = new BufferedInputRange(connection); 5132 more_data: 5133 auto chunk = range.front(); 5134 // waiting for colon for header length 5135 auto idx = indexOf(cast(string) chunk, ':'); 5136 if(idx == -1) { 5137 try { 5138 range.popFront(); 5139 } catch(Exception e) { 5140 // it is just closed, no big deal 5141 connection.close(); 5142 return; 5143 } 5144 goto more_data; 5145 } 5146 5147 size = to!size_t(cast(string) chunk[0 .. idx]); 5148 chunk = range.consume(idx + 1); 5149 // reading headers 5150 if(chunk.length < size) 5151 range.popFront(0, size + 1); 5152 // we are now guaranteed to have enough 5153 chunk = range.front(); 5154 assert(chunk.length > size); 5155 5156 idx = 0; 5157 string key; 5158 string value; 5159 foreach(part; al.splitter(chunk, '\0')) { 5160 if(idx & 1) { // odd is value 5161 value = cast(string)(part.idup); 5162 headers[key] = value; // commit 5163 } else 5164 key = cast(string)(part.idup); 5165 idx++; 5166 } 5167 5168 enforce(chunk[size] == ','); // the terminator 5169 5170 range.consume(size + 1); 5171 // reading data 5172 // this will be done by Cgi 5173 5174 const(ubyte)[] getScgiChunk() { 5175 // we are already primed 5176 auto data = range.front(); 5177 if(data.length == 0 && !range.sourceClosed) { 5178 range.popFront(0); 5179 data = range.front(); 5180 } else if (range.sourceClosed) 5181 range.source.close(); 5182 5183 return data; 5184 } 5185 5186 void writeScgi(const(ubyte)[] data) { 5187 sendAll(connection, data); 5188 } 5189 5190 void flushScgi() { 5191 // I don't *think* I have to do anything.... 5192 } 5193 5194 Cgi cgi; 5195 try { 5196 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5197 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5198 } catch(Throwable t) { 5199 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5200 connection.close(); 5201 return; // this connection is dead 5202 } 5203 assert(cgi !is null); 5204 scope(exit) cgi.dispose(); 5205 try { 5206 fun(cgi); 5207 cgi.close(); 5208 connection.close(); 5209 } catch(Throwable t) { 5210 // no std err 5211 if(!handleException(cgi, t)) { 5212 connection.close(); 5213 return; 5214 } else { 5215 connection.close(); 5216 return; 5217 } 5218 } 5219 } 5220 5221 string printDate(DateTime date) { 5222 char[29] buffer = void; 5223 printDateToBuffer(date, buffer[]); 5224 return buffer.idup; 5225 } 5226 5227 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5228 assert(buffer.length >= 29); 5229 // 29 static length ? 5230 5231 static immutable daysOfWeek = [ 5232 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5233 ]; 5234 5235 static immutable months = [ 5236 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5237 ]; 5238 5239 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5240 buffer[3 .. 5] = ", "; 5241 buffer[5] = date.day / 10 + '0'; 5242 buffer[6] = date.day % 10 + '0'; 5243 buffer[7] = ' '; 5244 buffer[8 .. 11] = months[date.month]; 5245 buffer[11] = ' '; 5246 auto y = date.year; 5247 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5248 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5249 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5250 buffer[15] = cast(char) (y + '0'); 5251 buffer[16] = ' '; 5252 buffer[17] = date.hour / 10 + '0'; 5253 buffer[18] = date.hour % 10 + '0'; 5254 buffer[19] = ':'; 5255 buffer[20] = date.minute / 10 + '0'; 5256 buffer[21] = date.minute % 10 + '0'; 5257 buffer[22] = ':'; 5258 buffer[23] = date.second / 10 + '0'; 5259 buffer[24] = date.second % 10 + '0'; 5260 buffer[25 .. $] = " GMT"; 5261 5262 return 29; 5263 } 5264 5265 5266 // Referencing this gigantic typeid seems to remind the compiler 5267 // to actually put the symbol in the object file. I guess the immutable 5268 // assoc array array isn't actually included in druntime 5269 void hackAroundLinkerError() { 5270 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5271 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5272 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5273 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5274 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5275 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5276 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5277 // this is getting kinda ridiculous btw. Moving assoc arrays 5278 // to the library is the pain that keeps on coming. 5279 5280 // eh this broke the build on the work server 5281 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5282 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5283 } 5284 5285 5286 5287 5288 5289 version(fastcgi) { 5290 pragma(lib, "fcgi"); 5291 5292 static if(size_t.sizeof == 8) // 64 bit 5293 alias long c_int; 5294 else 5295 alias int c_int; 5296 5297 extern(C) { 5298 struct FCGX_Stream { 5299 ubyte* rdNext; 5300 ubyte* wrNext; 5301 ubyte* stop; 5302 ubyte* stopUnget; 5303 c_int isReader; 5304 c_int isClosed; 5305 c_int wasFCloseCalled; 5306 c_int FCGI_errno; 5307 void* function(FCGX_Stream* stream) fillBuffProc; 5308 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5309 void* data; 5310 } 5311 5312 // note: this is meant to be opaque, so don't access it directly 5313 struct FCGX_Request { 5314 int requestId; 5315 int role; 5316 FCGX_Stream* inStream; 5317 FCGX_Stream* outStream; 5318 FCGX_Stream* errStream; 5319 char** envp; 5320 void* paramsPtr; 5321 int ipcFd; 5322 int isBeginProcessed; 5323 int keepConnection; 5324 int appStatus; 5325 int nWriters; 5326 int flags; 5327 int listen_sock; 5328 } 5329 5330 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5331 void FCGX_Init(); 5332 5333 int FCGX_Accept_r(FCGX_Request *request); 5334 5335 5336 alias char** FCGX_ParamArray; 5337 5338 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5339 c_int FCGX_GetChar(FCGX_Stream* stream); 5340 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5341 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5342 c_int FCGX_FFlush(FCGX_Stream *stream); 5343 5344 int FCGX_OpenSocket(in char*, int); 5345 } 5346 } 5347 5348 5349 /* This might go int a separate module eventually. It is a network input helper class. */ 5350 5351 import std.socket; 5352 5353 version(cgi_use_fiber) { 5354 import core.thread; 5355 5356 version(linux) { 5357 import core.sys.linux.epoll; 5358 5359 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5360 } else version(Windows) { 5361 // declaring the iocp thing below... 5362 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5363 } 5364 5365 version(Windows) 5366 __gshared HANDLE iocp; 5367 5368 version(cgi_use_fiber) { 5369 version(linux) 5370 private enum WakeupEvent { 5371 Read = EPOLLIN, 5372 Write = EPOLLOUT 5373 } 5374 else version(Windows) 5375 private enum WakeupEvent { 5376 Read, Write 5377 } 5378 else static assert(0); 5379 } 5380 5381 version(cgi_use_fiber) 5382 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5383 5384 // static cast since I know what i have in here and don't want to pay for dynamic cast 5385 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5386 5387 version(linux) { 5388 f.setPostYield = () { 5389 if(*registered) { 5390 // rearm 5391 epoll_event evt; 5392 evt.events = e | EPOLLONESHOT; 5393 evt.data.ptr = cast(void*) f; 5394 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5395 throw new Exception("epoll_ctl"); 5396 } else { 5397 // initial registration 5398 *registered = true ; 5399 int fd = source.handle; 5400 epoll_event evt; 5401 evt.events = e | EPOLLONESHOT; 5402 evt.data.ptr = cast(void*) f; 5403 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5404 throw new Exception("epoll_ctl"); 5405 } 5406 }; 5407 5408 Fiber.yield(); 5409 5410 f.setPostYield(null); 5411 } else version(Windows) { 5412 Fiber.yield(); 5413 } 5414 else static assert(0); 5415 } 5416 5417 version(cgi_use_fiber) 5418 void unregisterSource(Socket s) { 5419 version(linux) { 5420 epoll_event evt; 5421 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5422 } else version(Windows) { 5423 // intentionally blank 5424 } 5425 else static assert(0); 5426 } 5427 5428 // it is a class primarily for reference semantics 5429 // I might change this interface 5430 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5431 class BufferedInputRange { 5432 version(Posix) 5433 this(int source, ubyte[] buffer = null) { 5434 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5435 } 5436 5437 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5438 // if they connect but never send stuff to us, we don't want it wasting the process 5439 // so setting a time out 5440 version(cgi_use_fiber) 5441 source.blocking = false; 5442 else 5443 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5444 5445 this.source = source; 5446 if(buffer is null) { 5447 underlyingBuffer = new ubyte[4096]; 5448 this.allowGrowth = true; 5449 } else { 5450 underlyingBuffer = buffer; 5451 this.allowGrowth = allowGrowth; 5452 } 5453 5454 assert(underlyingBuffer.length); 5455 5456 // we assume view.ptr is always inside underlyingBuffer 5457 view = underlyingBuffer[0 .. 0]; 5458 5459 popFront(); // prime 5460 } 5461 5462 version(cgi_use_fiber) { 5463 bool registered; 5464 } 5465 5466 void dispose() { 5467 version(cgi_use_fiber) { 5468 if(registered) 5469 unregisterSource(source); 5470 } 5471 } 5472 5473 /** 5474 A slight difference from regular ranges is you can give it the maximum 5475 number of bytes to consume. 5476 5477 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5478 consume() yourself and use a regular foreach, it will infinitely loop! 5479 5480 The default is to do what a normal range does, and consume the whole buffer 5481 and wait for additional input. 5482 5483 You can also specify 0, to append to the buffer, or any other number 5484 to remove the front n bytes and wait for more. 5485 */ 5486 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5487 if(sourceClosed) 5488 throw new ConnectionClosedException("can't get any more data from a closed source"); 5489 if(!skipConsume) 5490 consume(maxBytesToConsume); 5491 5492 // we might have to grow the buffer 5493 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5494 if(allowGrowth) { 5495 //import std.stdio; writeln("growth"); 5496 auto viewStart = view.ptr - underlyingBuffer.ptr; 5497 size_t growth = 4096; 5498 // make sure we have enough for what we're being asked for 5499 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5500 growth = minBytesToSettleFor - underlyingBuffer.length; 5501 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5502 underlyingBuffer.length += growth; 5503 view = underlyingBuffer[viewStart .. view.length]; 5504 } else 5505 throw new Exception("No room left in the buffer"); 5506 } 5507 5508 do { 5509 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5510 try_again: 5511 auto ret = source.receive(freeSpace); 5512 if(ret == Socket.ERROR) { 5513 if(wouldHaveBlocked()) { 5514 version(cgi_use_fiber) { 5515 registerEventWakeup(®istered, source, WakeupEvent.Read); 5516 goto try_again; 5517 } else { 5518 // gonna treat a timeout here as a close 5519 sourceClosed = true; 5520 return; 5521 } 5522 } 5523 version(Posix) { 5524 import core.stdc.errno; 5525 if(errno == EINTR || errno == EAGAIN) { 5526 goto try_again; 5527 } 5528 if(errno == ECONNRESET) { 5529 sourceClosed = true; 5530 return; 5531 } 5532 } 5533 throw new Exception(lastSocketError); // FIXME 5534 } 5535 if(ret == 0) { 5536 sourceClosed = true; 5537 return; 5538 } 5539 5540 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5541 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5542 //import std.stdio; writeln(cast(string) view); 5543 } while(view.length < minBytesToSettleFor); 5544 } 5545 5546 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5547 /// You might want to idup the data you are consuming if you store it, since it may 5548 /// be overwritten on the new popFront. 5549 /// 5550 /// You do not need to call this if you always want to wait for more data when you 5551 /// consume some. 5552 ubyte[] consume(size_t bytes) { 5553 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5554 view = view[bytes > $ ? $ : bytes .. $]; 5555 if(view.length == 0) { 5556 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5557 /* 5558 writeln("HERE"); 5559 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5560 writeln(cast(string)front); 5561 writeln("DONE"); 5562 */ 5563 } 5564 return front; 5565 } 5566 5567 bool empty() { 5568 return sourceClosed && view.length == 0; 5569 } 5570 5571 ubyte[] front() { 5572 return view; 5573 } 5574 5575 invariant() { 5576 assert(view.ptr >= underlyingBuffer.ptr); 5577 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5578 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5579 } 5580 5581 ubyte[] underlyingBuffer; 5582 bool allowGrowth; 5583 ubyte[] view; 5584 Socket source; 5585 bool sourceClosed; 5586 } 5587 5588 private class FakeSocketForStdin : Socket { 5589 import std.stdio; 5590 5591 this() { 5592 5593 } 5594 5595 private bool closed; 5596 5597 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5598 if(closed) 5599 throw new Exception("Closed"); 5600 return stdin.rawRead(buffer).length; 5601 } 5602 5603 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5604 if(closed) 5605 throw new Exception("Closed"); 5606 stdout.rawWrite(buffer); 5607 return buffer.length; 5608 } 5609 5610 override void close() @trusted scope { 5611 (cast(void delegate() @nogc nothrow) &realClose)(); 5612 } 5613 5614 override void shutdown(SocketShutdown s) { 5615 // FIXME 5616 } 5617 5618 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5619 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5620 5621 override @property @trusted Address remoteAddress() { return null; } 5622 override @property @trusted Address localAddress() { return null; } 5623 5624 void realClose() { 5625 closed = true; 5626 try { 5627 stdin.close(); 5628 stdout.close(); 5629 } catch(Exception e) { 5630 5631 } 5632 } 5633 } 5634 5635 import core.sync.semaphore; 5636 import core.atomic; 5637 5638 /** 5639 To use this thing: 5640 5641 --- 5642 void handler(Socket s) { do something... } 5643 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5644 manager.listen(); 5645 --- 5646 5647 The 4th parameter is optional. 5648 5649 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5650 comes in, you will get control. You can just continue; though to fetch more. 5651 5652 5653 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5654 */ 5655 class ListeningConnectionManager { 5656 Semaphore semaphore; 5657 Socket[256] queue; 5658 shared(ubyte) nextIndexFront; 5659 ubyte nextIndexBack; 5660 shared(int) queueLength; 5661 5662 Socket acceptCancelable() { 5663 version(Posix) { 5664 import core.sys.posix.sys.select; 5665 fd_set read_fds; 5666 FD_ZERO(&read_fds); 5667 int max = 0; 5668 foreach(listener; listeners) { 5669 FD_SET(listener.handle, &read_fds); 5670 if(listener.handle > max) 5671 max = listener.handle; 5672 } 5673 if(cancelfd != -1) { 5674 FD_SET(cancelfd, &read_fds); 5675 if(cancelfd > max) 5676 max = cancelfd; 5677 } 5678 auto ret = select(max + 1, &read_fds, null, null, null); 5679 if(ret == -1) { 5680 import core.stdc.errno; 5681 if(errno == EINTR) 5682 return null; 5683 else 5684 throw new Exception("wtf select"); 5685 } 5686 5687 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5688 return null; 5689 } 5690 5691 foreach(listener; listeners) { 5692 if(FD_ISSET(listener.handle, &read_fds)) 5693 return listener.accept(); 5694 } 5695 5696 return null; 5697 } else { 5698 5699 auto check = new SocketSet(); 5700 5701 keep_looping: 5702 check.reset(); 5703 foreach(listener; listeners) 5704 check.add(listener); 5705 5706 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5707 auto got = Socket.select(check, null, null, 3.seconds); 5708 if(got > 0) 5709 foreach(listener; listeners) 5710 if(check.isSet(listener)) 5711 return listener.accept(); 5712 if(globalStopFlag) 5713 return null; 5714 else 5715 goto keep_looping; 5716 } 5717 } 5718 5719 int defaultNumberOfThreads() { 5720 import std.parallelism; 5721 version(cgi_use_fiber) { 5722 return totalCPUs * 1 + 1; 5723 } else { 5724 // I times 4 here because there's a good chance some will be blocked on i/o. 5725 return totalCPUs * 4; 5726 } 5727 5728 } 5729 5730 void listen() { 5731 shared(int) loopBroken; 5732 5733 version(Posix) { 5734 import core.sys.posix.signal; 5735 signal(SIGPIPE, SIG_IGN); 5736 } 5737 5738 version(linux) { 5739 if(cancelfd == -1) 5740 cancelfd = eventfd(0, 0); 5741 } 5742 5743 version(cgi_no_threads) { 5744 // NEVER USE THIS 5745 // it exists only for debugging and other special occasions 5746 5747 // the thread mode is faster and less likely to stall the whole 5748 // thing when a request is slow 5749 while(!loopBroken && !globalStopFlag) { 5750 auto sn = acceptCancelable(); 5751 if(sn is null) continue; 5752 cloexec(sn); 5753 try { 5754 handler(sn); 5755 } catch(Exception e) { 5756 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5757 sn.close(); 5758 } 5759 } 5760 } else { 5761 5762 if(useFork) { 5763 version(linux) { 5764 //asm { int 3; } 5765 fork(); 5766 } 5767 } 5768 5769 version(cgi_use_fiber) { 5770 5771 version(Windows) { 5772 // please note these are overlapped sockets! so the accept just kicks things off 5773 foreach(listener; listeners) 5774 listener.accept(); 5775 } 5776 5777 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5778 foreach(i, ref thread; threads) { 5779 thread = new WorkerThread(this, handler, cast(int) i); 5780 thread.start(); 5781 } 5782 5783 bool fiber_crash_check() { 5784 bool hasAnyRunning; 5785 foreach(thread; threads) { 5786 if(!thread.isRunning) { 5787 thread.join(); 5788 } else hasAnyRunning = true; 5789 } 5790 5791 return (!hasAnyRunning); 5792 } 5793 5794 5795 while(!globalStopFlag) { 5796 Thread.sleep(1.seconds); 5797 if(fiber_crash_check()) 5798 break; 5799 } 5800 5801 } else { 5802 semaphore = new Semaphore(); 5803 5804 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5805 foreach(i, ref thread; threads) { 5806 thread = new ConnectionThread(this, handler, cast(int) i); 5807 thread.start(); 5808 } 5809 5810 while(!loopBroken && !globalStopFlag) { 5811 Socket sn; 5812 5813 bool crash_check() { 5814 bool hasAnyRunning; 5815 foreach(thread; threads) { 5816 if(!thread.isRunning) { 5817 thread.join(); 5818 } else hasAnyRunning = true; 5819 } 5820 5821 return (!hasAnyRunning); 5822 } 5823 5824 5825 void accept_new_connection() { 5826 sn = acceptCancelable(); 5827 if(sn is null) return; 5828 cloexec(sn); 5829 if(tcp) { 5830 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5831 // on the socket because we do some buffering internally. I think this helps, 5832 // certainly does for small requests, and I think it does for larger ones too 5833 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5834 5835 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5836 } 5837 } 5838 5839 void existing_connection_new_data() { 5840 // wait until a slot opens up 5841 // int waited = 0; 5842 while(queueLength >= queue.length) { 5843 Thread.sleep(1.msecs); 5844 // waited ++; 5845 } 5846 // if(waited) {import std.stdio; writeln(waited);} 5847 synchronized(this) { 5848 queue[nextIndexBack] = sn; 5849 nextIndexBack++; 5850 atomicOp!"+="(queueLength, 1); 5851 } 5852 semaphore.notify(); 5853 } 5854 5855 5856 accept_new_connection(); 5857 if(sn !is null) 5858 existing_connection_new_data(); 5859 else if(sn is null && globalStopFlag) { 5860 foreach(thread; threads) { 5861 semaphore.notify(); 5862 } 5863 Thread.sleep(50.msecs); 5864 } 5865 5866 if(crash_check()) 5867 break; 5868 } 5869 } 5870 5871 // FIXME: i typically stop this with ctrl+c which never 5872 // actually gets here. i need to do a sigint handler. 5873 if(cleanup) 5874 cleanup(); 5875 } 5876 } 5877 5878 //version(linux) 5879 //int epoll_fd; 5880 5881 bool tcp; 5882 void delegate() cleanup; 5883 5884 private void function(Socket) fhandler; 5885 private void dg_handler(Socket s) { 5886 fhandler(s); 5887 } 5888 5889 5890 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5891 fhandler = handler; 5892 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 5893 } 5894 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5895 string[] host; 5896 ushort[] port; 5897 5898 foreach(spec; listenSpec) { 5899 /+ 5900 The format: 5901 5902 protocol:// 5903 address_spec 5904 5905 Protocol is optional. Must be http, https, scgi, or fastcgi. 5906 5907 address_spec is either: 5908 ipv4 address : port 5909 [ipv6 address] : port 5910 unix:filename 5911 abstract:name 5912 port <which is tcp but on any interface> 5913 +/ 5914 5915 string protocol; 5916 string address_spec; 5917 5918 auto protocolIdx = spec.indexOf("://"); 5919 if(protocolIdx != -1) { 5920 protocol = spec[0 .. protocolIdx]; 5921 address_spec = spec[protocolIdx + "://".length .. $]; 5922 } else { 5923 address_spec = spec; 5924 } 5925 5926 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 5927 host ~= address_spec; 5928 port ~= 0; 5929 } else { 5930 auto idx = address_spec.lastIndexOf(":"); 5931 if(idx == -1) { 5932 host ~= null; 5933 } else { 5934 auto as = address_spec[0 .. idx]; 5935 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 5936 as = as[1 .. $-1]; 5937 host ~= as; 5938 } 5939 port ~= address_spec[idx + 1 .. $].to!ushort; 5940 } 5941 5942 } 5943 5944 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 5945 } 5946 5947 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5948 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5949 } 5950 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5951 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5952 } 5953 5954 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5955 fhandler = handler; 5956 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5957 } 5958 5959 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5960 assert(host.length == port.length); 5961 5962 this.handler = handler; 5963 this.useFork = useFork; 5964 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5965 5966 listeners.reserve(host.length); 5967 5968 foreach(i; 0 .. host.length) 5969 if(host[i] == "localhost") { 5970 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 5971 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 5972 } else { 5973 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 5974 } 5975 5976 version(cgi_use_fiber) 5977 if(useFork) { 5978 foreach(listener; listeners) 5979 listener.blocking = false; 5980 } 5981 5982 // this is the UI control thread and thus gets more priority 5983 Thread.getThis.priority = Thread.PRIORITY_MAX; 5984 } 5985 5986 Socket[] listeners; 5987 void delegate(Socket) handler; 5988 5989 immutable bool useFork; 5990 int numberOfThreads; 5991 } 5992 5993 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 5994 Socket listener; 5995 if(host.startsWith("unix:")) { 5996 version(Posix) { 5997 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5998 cloexec(listener); 5999 string filename = host["unix:".length .. $].idup; 6000 listener.bind(new UnixAddress(filename)); 6001 cleanup = delegate() { 6002 listener.close(); 6003 import std.file; 6004 remove(filename); 6005 }; 6006 tcp = false; 6007 } else { 6008 throw new Exception("unix sockets not supported on this system"); 6009 } 6010 } else if(host.startsWith("abstract:")) { 6011 version(linux) { 6012 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6013 cloexec(listener); 6014 string filename = "\0" ~ host["abstract:".length .. $]; 6015 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6016 listener.bind(new UnixAddress(filename)); 6017 tcp = false; 6018 } else { 6019 throw new Exception("abstract unix sockets not supported on this system"); 6020 } 6021 } else { 6022 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6023 version(cgi_use_fiber) { 6024 version(Windows) 6025 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6026 else 6027 listener = new Socket(address.addressFamily, SocketType.STREAM); 6028 } else { 6029 listener = new Socket(address.addressFamily, SocketType.STREAM); 6030 } 6031 cloexec(listener); 6032 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6033 listener.bind(address); 6034 cleanup = delegate() { 6035 listener.close(); 6036 }; 6037 tcp = true; 6038 } 6039 6040 listener.listen(backQueue); 6041 6042 if (dropPrivs !is null) // can be null, backwards compatibility 6043 dropPrivs(); 6044 6045 return listener; 6046 } 6047 6048 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6049 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6050 if(data.length == 0) return; 6051 ptrdiff_t amount; 6052 //import std.stdio; writeln("***",cast(string) data,"///"); 6053 do { 6054 amount = s.send(data); 6055 if(amount == Socket.ERROR) { 6056 version(cgi_use_fiber) { 6057 if(wouldHaveBlocked()) { 6058 bool registered = true; 6059 registerEventWakeup(®istered, s, WakeupEvent.Write); 6060 continue; 6061 } 6062 } 6063 throw new ConnectionException(s, lastSocketError, file, line); 6064 } 6065 assert(amount > 0); 6066 6067 data = data[amount .. $]; 6068 } while(data.length); 6069 } 6070 6071 class ConnectionException : Exception { 6072 Socket socket; 6073 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6074 this.socket = s; 6075 super(msg, file, line); 6076 } 6077 } 6078 6079 class HttpVersionNotSupportedException : Exception { 6080 this(string file = __FILE__, size_t line = __LINE__) { 6081 super("HTTP Version Not Supported", file, line); 6082 } 6083 } 6084 6085 alias void delegate(Socket) CMT; 6086 6087 import core.thread; 6088 /+ 6089 cgi.d now uses a hybrid of event i/o and threads at the top level. 6090 6091 Top level thread is responsible for accepting sockets and selecting on them. 6092 6093 It then indicates to a child that a request is pending, and any random worker 6094 thread that is free handles it. It goes into blocking mode and handles that 6095 http request to completion. 6096 6097 At that point, it goes back into the waiting queue. 6098 6099 6100 This concept is only implemented on Linux. On all other systems, it still 6101 uses the worker threads and semaphores (which is perfectly fine for a lot of 6102 things! Just having a great number of keep-alive connections will break that.) 6103 6104 6105 So the algorithm is: 6106 6107 select(accept, event, pending) 6108 if accept -> send socket to free thread, if any. if not, add socket to queue 6109 if event -> send the signaling thread a socket from the queue, if not, mark it free 6110 - event might block until it can be *written* to. it is a fifo sending socket fds! 6111 6112 A worker only does one http request at a time, then signals its availability back to the boss. 6113 6114 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6115 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6116 actual FD will not be kept out here. 6117 6118 So: 6119 queue = sockets we know are ready to read now, but no worker thread is available 6120 idle list = worker threads not doing anything else. they signal back and forth 6121 6122 the workers all read off the event fd. This is the semaphore wait 6123 6124 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6125 it puts it in the queue and writes to the event fd. 6126 6127 The child could put the socket back in the epoll thing itself. 6128 6129 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6130 +/ 6131 class ConnectionThread : Thread { 6132 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6133 this.lcm = lcm; 6134 this.dg = dg; 6135 this.myThreadNumber = myThreadNumber; 6136 super(&run); 6137 } 6138 6139 void run() { 6140 while(true) { 6141 // so if there's a bunch of idle keep-alive connections, it can 6142 // consume all the worker threads... just sitting there. 6143 lcm.semaphore.wait(); 6144 if(globalStopFlag) 6145 return; 6146 Socket socket; 6147 synchronized(lcm) { 6148 auto idx = lcm.nextIndexFront; 6149 socket = lcm.queue[idx]; 6150 lcm.queue[idx] = null; 6151 atomicOp!"+="(lcm.nextIndexFront, 1); 6152 atomicOp!"-="(lcm.queueLength, 1); 6153 } 6154 try { 6155 //import std.stdio; writeln(myThreadNumber, " taking it"); 6156 dg(socket); 6157 /+ 6158 if(socket.isAlive) { 6159 // process it more later 6160 version(linux) { 6161 import core.sys.linux.epoll; 6162 epoll_event ev; 6163 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6164 ev.data.fd = socket.handle; 6165 import std.stdio; writeln("adding"); 6166 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6167 if(errno == EEXIST) { 6168 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6169 ev.data.fd = socket.handle; 6170 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6171 throw new Exception("epoll_ctl " ~ to!string(errno)); 6172 } else 6173 throw new Exception("epoll_ctl " ~ to!string(errno)); 6174 } 6175 //import std.stdio; writeln("keep alive"); 6176 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6177 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6178 } else { 6179 continue; // hope it times out in a reasonable amount of time... 6180 } 6181 } 6182 +/ 6183 } catch(ConnectionClosedException e) { 6184 // can just ignore this, it is fairly normal 6185 socket.close(); 6186 } catch(Throwable e) { 6187 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6188 socket.close(); 6189 } 6190 } 6191 } 6192 6193 ListeningConnectionManager lcm; 6194 CMT dg; 6195 int myThreadNumber; 6196 } 6197 6198 version(cgi_use_fiber) 6199 class WorkerThread : Thread { 6200 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6201 this.lcm = lcm; 6202 this.dg = dg; 6203 this.myThreadNumber = myThreadNumber; 6204 super(&run); 6205 } 6206 6207 version(Windows) 6208 void run() { 6209 auto timeout = INFINITE; 6210 PseudoblockingOverlappedSocket key; 6211 OVERLAPPED* overlapped; 6212 DWORD bytes; 6213 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6214 if(key is null) 6215 continue; 6216 key.lastAnswer = bytes; 6217 if(key.fiber) { 6218 key.fiber.proceed(); 6219 } else { 6220 // we have a new connection, issue the first receive on it and issue the next accept 6221 6222 auto sn = key.accepted; 6223 6224 key.accept(); 6225 6226 cloexec(sn); 6227 if(lcm.tcp) { 6228 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6229 // on the socket because we do some buffering internally. I think this helps, 6230 // certainly does for small requests, and I think it does for larger ones too 6231 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6232 6233 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6234 } 6235 6236 dg(sn); 6237 } 6238 } 6239 //SleepEx(INFINITE, TRUE); 6240 } 6241 6242 version(linux) 6243 void run() { 6244 6245 import core.sys.linux.epoll; 6246 epfd = epoll_create1(EPOLL_CLOEXEC); 6247 if(epfd == -1) 6248 throw new Exception("epoll_create1 " ~ to!string(errno)); 6249 scope(exit) { 6250 import core.sys.posix.unistd; 6251 close(epfd); 6252 } 6253 6254 { 6255 epoll_event ev; 6256 ev.events = EPOLLIN; 6257 ev.data.fd = cancelfd; 6258 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6259 } 6260 6261 foreach(listener; lcm.listeners) { 6262 epoll_event ev; 6263 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6264 ev.data.fd = listener.handle; 6265 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6266 throw new Exception("epoll_ctl " ~ to!string(errno)); 6267 } 6268 6269 6270 6271 while(!globalStopFlag) { 6272 Socket sn; 6273 6274 epoll_event[64] events; 6275 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6276 if(nfds == -1) { 6277 if(errno == EINTR) 6278 continue; 6279 throw new Exception("epoll_wait " ~ to!string(errno)); 6280 } 6281 6282 outer: foreach(idx; 0 .. nfds) { 6283 auto flags = events[idx].events; 6284 6285 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6286 globalStopFlag = true; 6287 //import std.stdio; writeln("exit heard"); 6288 break; 6289 } else { 6290 foreach(listener; lcm.listeners) { 6291 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6292 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6293 // this try/catch is because it is set to non-blocking mode 6294 // and Phobos' stupid api throws an exception instead of returning 6295 // if it would block. Why would it block? because a forked process 6296 // might have beat us to it, but the wakeup event thundered our herds. 6297 try 6298 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6299 catch(SocketAcceptException e) { continue outer; } 6300 6301 cloexec(sn); 6302 if(lcm.tcp) { 6303 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6304 // on the socket because we do some buffering internally. I think this helps, 6305 // certainly does for small requests, and I think it does for larger ones too 6306 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6307 6308 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6309 } 6310 6311 dg(sn); 6312 continue outer; 6313 } else { 6314 // writeln(events[idx].data.ptr); 6315 } 6316 } 6317 6318 if(cast(size_t) events[idx].data.ptr < 1024) { 6319 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6320 } 6321 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6322 fiber.proceed(); 6323 } 6324 } 6325 } 6326 } 6327 6328 ListeningConnectionManager lcm; 6329 CMT dg; 6330 int myThreadNumber; 6331 } 6332 6333 6334 /* Done with network helper */ 6335 6336 /* Helpers for doing temporary files. Used both here and in web.d */ 6337 6338 version(Windows) { 6339 import core.sys.windows.windows; 6340 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6341 alias GetTempPathW GetTempPath; 6342 } 6343 6344 version(Posix) { 6345 static import linux = core.sys.posix.unistd; 6346 } 6347 6348 string getTempDirectory() { 6349 string path; 6350 version(Windows) { 6351 wchar[1024] buffer; 6352 auto len = GetTempPath(1024, buffer.ptr); 6353 if(len == 0) 6354 throw new Exception("couldn't find a temporary path"); 6355 6356 auto b = buffer[0 .. len]; 6357 6358 path = to!string(b); 6359 } else 6360 path = "/tmp/"; 6361 6362 return path; 6363 } 6364 6365 6366 // I like std.date. These functions help keep my old code and data working with phobos changing. 6367 6368 long sysTimeToDTime(in SysTime sysTime) { 6369 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6370 } 6371 6372 long dateTimeToDTime(in DateTime dt) { 6373 return sysTimeToDTime(cast(SysTime) dt); 6374 } 6375 6376 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6377 return sysTimeToDTime(Clock.currTime(UTC())); 6378 } 6379 6380 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6381 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6382 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6383 return SysTime(hnsecs, tz); 6384 } 6385 6386 6387 6388 // this is a helper to read HTTP transfer-encoding: chunked responses 6389 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6390 immutable(ubyte)[] ret; 6391 6392 another_chunk: 6393 // If here, we are at the beginning of a chunk. 6394 auto a = ir.front(); 6395 int chunkSize; 6396 int loc = locationOf(a, "\r\n"); 6397 while(loc == -1) { 6398 ir.popFront(); 6399 a = ir.front(); 6400 loc = locationOf(a, "\r\n"); 6401 } 6402 6403 string hex; 6404 hex = ""; 6405 for(int i = 0; i < loc; i++) { 6406 char c = a[i]; 6407 if(c >= 'A' && c <= 'Z') 6408 c += 0x20; 6409 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6410 hex ~= c; 6411 } else { 6412 break; 6413 } 6414 } 6415 6416 assert(hex.length); 6417 6418 int power = 1; 6419 int size = 0; 6420 foreach(cc1; retro(hex)) { 6421 dchar cc = cc1; 6422 if(cc >= 'a' && cc <= 'z') 6423 cc -= 0x20; 6424 int val = 0; 6425 if(cc >= '0' && cc <= '9') 6426 val = cc - '0'; 6427 else 6428 val = cc - 'A' + 10; 6429 6430 size += power * val; 6431 power *= 16; 6432 } 6433 6434 chunkSize = size; 6435 assert(size >= 0); 6436 6437 if(loc + 2 > a.length) { 6438 ir.popFront(0, a.length + loc + 2); 6439 a = ir.front(); 6440 } 6441 6442 a = ir.consume(loc + 2); 6443 6444 if(chunkSize == 0) { // we're done with the response 6445 // if we got here, will change must be true.... 6446 more_footers: 6447 loc = locationOf(a, "\r\n"); 6448 if(loc == -1) { 6449 ir.popFront(); 6450 a = ir.front; 6451 goto more_footers; 6452 } else { 6453 assert(loc == 0); 6454 ir.consume(loc + 2); 6455 goto finish; 6456 } 6457 } else { 6458 // if we got here, will change must be true.... 6459 if(a.length < chunkSize + 2) { 6460 ir.popFront(0, chunkSize + 2); 6461 a = ir.front(); 6462 } 6463 6464 ret ~= (a[0..chunkSize]); 6465 6466 if(!(a.length > chunkSize + 2)) { 6467 ir.popFront(0, chunkSize + 2); 6468 a = ir.front(); 6469 } 6470 assert(a[chunkSize] == 13); 6471 assert(a[chunkSize+1] == 10); 6472 a = ir.consume(chunkSize + 2); 6473 chunkSize = 0; 6474 goto another_chunk; 6475 } 6476 6477 finish: 6478 return ret; 6479 } 6480 6481 // I want to be able to get data from multiple sources the same way... 6482 interface ByChunkRange { 6483 bool empty(); 6484 void popFront(); 6485 const(ubyte)[] front(); 6486 } 6487 6488 ByChunkRange byChunk(const(ubyte)[] data) { 6489 return new class ByChunkRange { 6490 override bool empty() { 6491 return !data.length; 6492 } 6493 6494 override void popFront() { 6495 if(data.length > 4096) 6496 data = data[4096 .. $]; 6497 else 6498 data = null; 6499 } 6500 6501 override const(ubyte)[] front() { 6502 return data[0 .. $ > 4096 ? 4096 : $]; 6503 } 6504 }; 6505 } 6506 6507 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6508 const(ubyte)[] f; 6509 6510 f = ir.front; 6511 if(f.length > atMost) 6512 f = f[0 .. atMost]; 6513 6514 return new class ByChunkRange { 6515 override bool empty() { 6516 return atMost == 0; 6517 } 6518 6519 override const(ubyte)[] front() { 6520 return f; 6521 } 6522 6523 override void popFront() { 6524 ir.consume(f.length); 6525 atMost -= f.length; 6526 auto a = ir.front(); 6527 6528 if(a.length <= atMost) { 6529 f = a; 6530 atMost -= a.length; 6531 a = ir.consume(a.length); 6532 if(atMost != 0) 6533 ir.popFront(); 6534 if(f.length == 0) { 6535 f = ir.front(); 6536 } 6537 } else { 6538 // we actually have *more* here than we need.... 6539 f = a[0..atMost]; 6540 atMost = 0; 6541 ir.consume(atMost); 6542 } 6543 } 6544 }; 6545 } 6546 6547 version(cgi_with_websocket) { 6548 // http://tools.ietf.org/html/rfc6455 6549 6550 /++ 6551 WEBSOCKET SUPPORT: 6552 6553 Full example: 6554 --- 6555 import arsd.cgi; 6556 6557 void websocketEcho(Cgi cgi) { 6558 if(cgi.websocketRequested()) { 6559 if(cgi.origin != "http://arsdnet.net") 6560 throw new Exception("bad origin"); 6561 auto websocket = cgi.acceptWebsocket(); 6562 6563 websocket.send("hello"); 6564 websocket.send(" world!"); 6565 6566 auto msg = websocket.recv(); 6567 while(msg.opcode != WebSocketOpcode.close) { 6568 if(msg.opcode == WebSocketOpcode.text) { 6569 websocket.send(msg.textData); 6570 } else if(msg.opcode == WebSocketOpcode.binary) { 6571 websocket.send(msg.data); 6572 } 6573 6574 msg = websocket.recv(); 6575 } 6576 6577 websocket.close(); 6578 } else { 6579 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6580 } 6581 } 6582 6583 mixin GenericMain!websocketEcho; 6584 --- 6585 +/ 6586 6587 class WebSocket { 6588 Cgi cgi; 6589 6590 private this(Cgi cgi) { 6591 this.cgi = cgi; 6592 6593 Socket socket = cgi.idlol.source; 6594 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6595 } 6596 6597 // returns true if data available, false if it timed out 6598 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6599 if(!waitForNextMessageWouldBlock()) 6600 return true; 6601 if(isDataPending(timeout)) 6602 return true; // this is kinda a lie. 6603 6604 return false; 6605 } 6606 6607 public bool lowLevelReceive() { 6608 auto bfr = cgi.idlol; 6609 top: 6610 auto got = bfr.front; 6611 if(got.length) { 6612 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6613 receiveBuffer.length += receiveBufferUsedLength + got.length; 6614 6615 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6616 receiveBufferUsedLength += got.length; 6617 bfr.consume(got.length); 6618 6619 return true; 6620 } 6621 6622 if(bfr.sourceClosed) 6623 return false; 6624 6625 bfr.popFront(0); 6626 if(bfr.sourceClosed) 6627 return false; 6628 goto top; 6629 } 6630 6631 6632 bool isDataPending(Duration timeout = 0.seconds) { 6633 Socket socket = cgi.idlol.source; 6634 6635 auto check = new SocketSet(); 6636 check.add(socket); 6637 6638 auto got = Socket.select(check, null, null, timeout); 6639 if(got > 0) 6640 return true; 6641 return false; 6642 } 6643 6644 // note: this blocks 6645 WebSocketFrame recv() { 6646 return waitForNextMessage(); 6647 } 6648 6649 6650 6651 6652 private void llclose() { 6653 cgi.close(); 6654 } 6655 6656 private void llsend(ubyte[] data) { 6657 cgi.write(data); 6658 cgi.flush(); 6659 } 6660 6661 void unregisterActiveSocket(WebSocket) {} 6662 6663 /* copy/paste section { */ 6664 6665 private int readyState_; 6666 private ubyte[] receiveBuffer; 6667 private size_t receiveBufferUsedLength; 6668 6669 private Config config; 6670 6671 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6672 enum OPEN = 1; /// The connection is open and ready to communicate. 6673 enum CLOSING = 2; /// The connection is in the process of closing. 6674 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6675 6676 /++ 6677 6678 +/ 6679 /// Group: foundational 6680 static struct Config { 6681 /++ 6682 These control the size of the receive buffer. 6683 6684 It starts at the initial size, will temporarily 6685 balloon up to the maximum size, and will reuse 6686 a buffer up to the likely size. 6687 6688 Anything larger than the maximum size will cause 6689 the connection to be aborted and an exception thrown. 6690 This is to protect you against a peer trying to 6691 exhaust your memory, while keeping the user-level 6692 processing simple. 6693 +/ 6694 size_t initialReceiveBufferSize = 4096; 6695 size_t likelyReceiveBufferSize = 4096; /// ditto 6696 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6697 6698 /++ 6699 Maximum combined size of a message. 6700 +/ 6701 size_t maximumMessageSize = 10 * 1024 * 1024; 6702 6703 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6704 string origin; /// Origin URL to send with the handshake, if desired. 6705 string protocol; /// the protocol header, if desired. 6706 6707 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6708 } 6709 6710 /++ 6711 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6712 +/ 6713 int readyState() { 6714 return readyState_; 6715 } 6716 6717 /++ 6718 Closes the connection, sending a graceful teardown message to the other side. 6719 +/ 6720 /// Group: foundational 6721 void close(int code = 0, string reason = null) 6722 //in (reason.length < 123) 6723 in { assert(reason.length < 123); } do 6724 { 6725 if(readyState_ != OPEN) 6726 return; // it cool, we done 6727 WebSocketFrame wss; 6728 wss.fin = true; 6729 wss.opcode = WebSocketOpcode.close; 6730 wss.data = cast(ubyte[]) reason.dup; 6731 wss.send(&llsend); 6732 6733 readyState_ = CLOSING; 6734 6735 llclose(); 6736 } 6737 6738 /++ 6739 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6740 +/ 6741 /// Group: foundational 6742 void ping() { 6743 WebSocketFrame wss; 6744 wss.fin = true; 6745 wss.opcode = WebSocketOpcode.ping; 6746 wss.send(&llsend); 6747 } 6748 6749 // automatically handled.... 6750 void pong() { 6751 WebSocketFrame wss; 6752 wss.fin = true; 6753 wss.opcode = WebSocketOpcode.pong; 6754 wss.send(&llsend); 6755 } 6756 6757 /++ 6758 Sends a text message through the websocket. 6759 +/ 6760 /// Group: foundational 6761 void send(in char[] textData) { 6762 WebSocketFrame wss; 6763 wss.fin = true; 6764 wss.opcode = WebSocketOpcode.text; 6765 wss.data = cast(ubyte[]) textData.dup; 6766 wss.send(&llsend); 6767 } 6768 6769 /++ 6770 Sends a binary message through the websocket. 6771 +/ 6772 /// Group: foundational 6773 void send(in ubyte[] binaryData) { 6774 WebSocketFrame wss; 6775 wss.fin = true; 6776 wss.opcode = WebSocketOpcode.binary; 6777 wss.data = cast(ubyte[]) binaryData.dup; 6778 wss.send(&llsend); 6779 } 6780 6781 /++ 6782 Waits for and returns the next complete message on the socket. 6783 6784 Note that the onmessage function is still called, right before 6785 this returns. 6786 +/ 6787 /// Group: blocking_api 6788 public WebSocketFrame waitForNextMessage() { 6789 do { 6790 auto m = processOnce(); 6791 if(m.populated) 6792 return m; 6793 } while(lowLevelReceive()); 6794 6795 throw new ConnectionClosedException("Websocket receive timed out"); 6796 //return WebSocketFrame.init; // FIXME? maybe. 6797 } 6798 6799 /++ 6800 Tells if [waitForNextMessage] would block. 6801 +/ 6802 /// Group: blocking_api 6803 public bool waitForNextMessageWouldBlock() { 6804 checkAgain: 6805 if(isMessageBuffered()) 6806 return false; 6807 if(!isDataPending()) 6808 return true; 6809 while(isDataPending()) 6810 lowLevelReceive(); 6811 goto checkAgain; 6812 } 6813 6814 /++ 6815 Is there a message in the buffer already? 6816 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6817 If `false`, check [isDataPending] as the next step. 6818 +/ 6819 /// Group: blocking_api 6820 public bool isMessageBuffered() { 6821 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6822 auto s = d; 6823 if(d.length) { 6824 auto orig = d; 6825 auto m = WebSocketFrame.read(d); 6826 // that's how it indicates that it needs more data 6827 if(d !is orig) 6828 return true; 6829 } 6830 6831 return false; 6832 } 6833 6834 private ubyte continuingType; 6835 private ubyte[] continuingData; 6836 //private size_t continuingDataLength; 6837 6838 private WebSocketFrame processOnce() { 6839 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6840 auto s = d; 6841 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6842 WebSocketFrame m; 6843 if(d.length) { 6844 auto orig = d; 6845 m = WebSocketFrame.read(d); 6846 // that's how it indicates that it needs more data 6847 if(d is orig) 6848 return WebSocketFrame.init; 6849 m.unmaskInPlace(); 6850 switch(m.opcode) { 6851 case WebSocketOpcode.continuation: 6852 if(continuingData.length + m.data.length > config.maximumMessageSize) 6853 throw new Exception("message size exceeded"); 6854 6855 continuingData ~= m.data; 6856 if(m.fin) { 6857 if(ontextmessage) 6858 ontextmessage(cast(char[]) continuingData); 6859 if(onbinarymessage) 6860 onbinarymessage(continuingData); 6861 6862 continuingData = null; 6863 } 6864 break; 6865 case WebSocketOpcode.text: 6866 if(m.fin) { 6867 if(ontextmessage) 6868 ontextmessage(m.textData); 6869 } else { 6870 continuingType = m.opcode; 6871 //continuingDataLength = 0; 6872 continuingData = null; 6873 continuingData ~= m.data; 6874 } 6875 break; 6876 case WebSocketOpcode.binary: 6877 if(m.fin) { 6878 if(onbinarymessage) 6879 onbinarymessage(m.data); 6880 } else { 6881 continuingType = m.opcode; 6882 //continuingDataLength = 0; 6883 continuingData = null; 6884 continuingData ~= m.data; 6885 } 6886 break; 6887 case WebSocketOpcode.close: 6888 readyState_ = CLOSED; 6889 if(onclose) 6890 onclose(); 6891 6892 unregisterActiveSocket(this); 6893 break; 6894 case WebSocketOpcode.ping: 6895 pong(); 6896 break; 6897 case WebSocketOpcode.pong: 6898 // just really references it is still alive, nbd. 6899 break; 6900 default: // ignore though i could and perhaps should throw too 6901 } 6902 } 6903 6904 // the recv thing can be invalidated so gotta copy it over ugh 6905 if(d.length) { 6906 m.data = m.data.dup(); 6907 } 6908 6909 import core.stdc.string; 6910 memmove(receiveBuffer.ptr, d.ptr, d.length); 6911 receiveBufferUsedLength = d.length; 6912 6913 return m; 6914 } 6915 6916 private void autoprocess() { 6917 // FIXME 6918 do { 6919 processOnce(); 6920 } while(lowLevelReceive()); 6921 } 6922 6923 6924 void delegate() onclose; /// 6925 void delegate() onerror; /// 6926 void delegate(in char[]) ontextmessage; /// 6927 void delegate(in ubyte[]) onbinarymessage; /// 6928 void delegate() onopen; /// 6929 6930 /++ 6931 6932 +/ 6933 /// Group: browser_api 6934 void onmessage(void delegate(in char[]) dg) { 6935 ontextmessage = dg; 6936 } 6937 6938 /// ditto 6939 void onmessage(void delegate(in ubyte[]) dg) { 6940 onbinarymessage = dg; 6941 } 6942 6943 /* } end copy/paste */ 6944 6945 6946 } 6947 6948 /++ 6949 Returns true if the request headers are asking for a websocket upgrade. 6950 6951 If this returns true, and you want to accept it, call [acceptWebsocket]. 6952 +/ 6953 bool websocketRequested(Cgi cgi) { 6954 return 6955 "sec-websocket-key" in cgi.requestHeaders 6956 && 6957 "connection" in cgi.requestHeaders && 6958 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6959 && 6960 "upgrade" in cgi.requestHeaders && 6961 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6962 ; 6963 } 6964 6965 /++ 6966 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 6967 +/ 6968 WebSocket acceptWebsocket(Cgi cgi) { 6969 assert(!cgi.closed); 6970 assert(!cgi.outputtedResponseData); 6971 cgi.setResponseStatus("101 Switching Protocols"); 6972 cgi.header("Upgrade: WebSocket"); 6973 cgi.header("Connection: upgrade"); 6974 6975 string key = cgi.requestHeaders["sec-websocket-key"]; 6976 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 6977 6978 import std.digest.sha; 6979 auto hash = sha1Of(key); 6980 auto accept = Base64.encode(hash); 6981 6982 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 6983 6984 cgi.websocketMode = true; 6985 cgi.write(""); 6986 6987 cgi.flush(); 6988 6989 return new WebSocket(cgi); 6990 } 6991 6992 // FIXME get websocket to work on other modes, not just embedded_httpd 6993 6994 /* copy/paste in http2.d { */ 6995 enum WebSocketOpcode : ubyte { 6996 continuation = 0, 6997 text = 1, 6998 binary = 2, 6999 // 3, 4, 5, 6, 7 RESERVED 7000 close = 8, 7001 ping = 9, 7002 pong = 10, 7003 // 11,12,13,14,15 RESERVED 7004 } 7005 7006 public struct WebSocketFrame { 7007 private bool populated; 7008 bool fin; 7009 bool rsv1; 7010 bool rsv2; 7011 bool rsv3; 7012 WebSocketOpcode opcode; // 4 bits 7013 bool masked; 7014 ubyte lengthIndicator; // don't set this when building one to send 7015 ulong realLength; // don't use when sending 7016 ubyte[4] maskingKey; // don't set this when sending 7017 ubyte[] data; 7018 7019 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7020 WebSocketFrame msg; 7021 msg.fin = true; 7022 msg.opcode = opcode; 7023 msg.data = cast(ubyte[]) data.dup; 7024 7025 return msg; 7026 } 7027 7028 private void send(scope void delegate(ubyte[]) llsend) { 7029 ubyte[64] headerScratch; 7030 int headerScratchPos = 0; 7031 7032 realLength = data.length; 7033 7034 { 7035 ubyte b1; 7036 b1 |= cast(ubyte) opcode; 7037 b1 |= rsv3 ? (1 << 4) : 0; 7038 b1 |= rsv2 ? (1 << 5) : 0; 7039 b1 |= rsv1 ? (1 << 6) : 0; 7040 b1 |= fin ? (1 << 7) : 0; 7041 7042 headerScratch[0] = b1; 7043 headerScratchPos++; 7044 } 7045 7046 { 7047 headerScratchPos++; // we'll set header[1] at the end of this 7048 auto rlc = realLength; 7049 ubyte b2; 7050 b2 |= masked ? (1 << 7) : 0; 7051 7052 assert(headerScratchPos == 2); 7053 7054 if(realLength > 65535) { 7055 // use 64 bit length 7056 b2 |= 0x7f; 7057 7058 // FIXME: double check endinaness 7059 foreach(i; 0 .. 8) { 7060 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7061 rlc >>>= 8; 7062 } 7063 7064 headerScratchPos += 8; 7065 } else if(realLength > 125) { 7066 // use 16 bit length 7067 b2 |= 0x7e; 7068 7069 // FIXME: double check endinaness 7070 foreach(i; 0 .. 2) { 7071 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7072 rlc >>>= 8; 7073 } 7074 7075 headerScratchPos += 2; 7076 } else { 7077 // use 7 bit length 7078 b2 |= realLength & 0b_0111_1111; 7079 } 7080 7081 headerScratch[1] = b2; 7082 } 7083 7084 //assert(!masked, "masking key not properly implemented"); 7085 if(masked) { 7086 // FIXME: randomize this 7087 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7088 headerScratchPos += 4; 7089 7090 // we'll just mask it in place... 7091 int keyIdx = 0; 7092 foreach(i; 0 .. data.length) { 7093 data[i] = data[i] ^ maskingKey[keyIdx]; 7094 if(keyIdx == 3) 7095 keyIdx = 0; 7096 else 7097 keyIdx++; 7098 } 7099 } 7100 7101 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7102 llsend(headerScratch[0 .. headerScratchPos]); 7103 llsend(data); 7104 } 7105 7106 static WebSocketFrame read(ref ubyte[] d) { 7107 WebSocketFrame msg; 7108 7109 auto orig = d; 7110 7111 WebSocketFrame needsMoreData() { 7112 d = orig; 7113 return WebSocketFrame.init; 7114 } 7115 7116 if(d.length < 2) 7117 return needsMoreData(); 7118 7119 ubyte b = d[0]; 7120 7121 msg.populated = true; 7122 7123 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7124 b >>= 4; 7125 msg.rsv3 = b & 0x01; 7126 b >>= 1; 7127 msg.rsv2 = b & 0x01; 7128 b >>= 1; 7129 msg.rsv1 = b & 0x01; 7130 b >>= 1; 7131 msg.fin = b & 0x01; 7132 7133 b = d[1]; 7134 msg.masked = (b & 0b1000_0000) ? true : false; 7135 msg.lengthIndicator = b & 0b0111_1111; 7136 7137 d = d[2 .. $]; 7138 7139 if(msg.lengthIndicator == 0x7e) { 7140 // 16 bit length 7141 msg.realLength = 0; 7142 7143 if(d.length < 2) return needsMoreData(); 7144 7145 foreach(i; 0 .. 2) { 7146 msg.realLength |= d[0] << ((1-i) * 8); 7147 d = d[1 .. $]; 7148 } 7149 } else if(msg.lengthIndicator == 0x7f) { 7150 // 64 bit length 7151 msg.realLength = 0; 7152 7153 if(d.length < 8) return needsMoreData(); 7154 7155 foreach(i; 0 .. 8) { 7156 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7157 d = d[1 .. $]; 7158 } 7159 } else { 7160 // 7 bit length 7161 msg.realLength = msg.lengthIndicator; 7162 } 7163 7164 if(msg.masked) { 7165 7166 if(d.length < 4) return needsMoreData(); 7167 7168 msg.maskingKey = d[0 .. 4]; 7169 d = d[4 .. $]; 7170 } 7171 7172 if(msg.realLength > d.length) { 7173 return needsMoreData(); 7174 } 7175 7176 msg.data = d[0 .. cast(size_t) msg.realLength]; 7177 d = d[cast(size_t) msg.realLength .. $]; 7178 7179 return msg; 7180 } 7181 7182 void unmaskInPlace() { 7183 if(this.masked) { 7184 int keyIdx = 0; 7185 foreach(i; 0 .. this.data.length) { 7186 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7187 if(keyIdx == 3) 7188 keyIdx = 0; 7189 else 7190 keyIdx++; 7191 } 7192 } 7193 } 7194 7195 char[] textData() { 7196 return cast(char[]) data; 7197 } 7198 } 7199 /* } */ 7200 } 7201 7202 7203 version(Windows) 7204 { 7205 version(CRuntime_DigitalMars) 7206 { 7207 extern(C) int setmode(int, int) nothrow @nogc; 7208 } 7209 else version(CRuntime_Microsoft) 7210 { 7211 extern(C) int _setmode(int, int) nothrow @nogc; 7212 alias setmode = _setmode; 7213 } 7214 else static assert(0); 7215 } 7216 7217 version(Posix) { 7218 import core.sys.posix.unistd; 7219 version(CRuntime_Musl) {} else { 7220 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7221 } 7222 } 7223 7224 7225 // FIXME: these aren't quite public yet. 7226 //private: 7227 7228 // template for laziness 7229 void startAddonServer()(string arg) { 7230 version(OSX) { 7231 assert(0, "Not implemented"); 7232 } else version(linux) { 7233 import core.sys.posix.unistd; 7234 pid_t pid; 7235 const(char)*[16] args; 7236 args[0] = "ARSD_CGI_ADDON_SERVER"; 7237 args[1] = arg.ptr; 7238 posix_spawn(&pid, "/proc/self/exe", 7239 null, 7240 null, 7241 args.ptr, 7242 null // env 7243 ); 7244 } else version(Windows) { 7245 wchar[2048] filename; 7246 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7247 if(len == 0 || len == filename.length) 7248 throw new Exception("could not get process name to start helper server"); 7249 7250 STARTUPINFOW startupInfo; 7251 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7252 PROCESS_INFORMATION processInfo; 7253 7254 import std.utf; 7255 7256 // I *MIGHT* need to run it as a new job or a service... 7257 auto ret = CreateProcessW( 7258 filename.ptr, 7259 toUTF16z(arg), 7260 null, // process attributes 7261 null, // thread attributes 7262 false, // inherit handles 7263 0, // creation flags 7264 null, // environment 7265 null, // working directory 7266 &startupInfo, 7267 &processInfo 7268 ); 7269 7270 if(!ret) 7271 throw new Exception("create process failed"); 7272 7273 // when done with those, if we set them 7274 /* 7275 CloseHandle(hStdInput); 7276 CloseHandle(hStdOutput); 7277 CloseHandle(hStdError); 7278 */ 7279 7280 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7281 } 7282 7283 // template for laziness 7284 /* 7285 The websocket server is a single-process, single-thread, event 7286 I/O thing. It is passed websockets from other CGI processes 7287 and is then responsible for handling their messages and responses. 7288 Note that the CGI process is responsible for websocket setup, 7289 including authentication, etc. 7290 7291 It also gets data sent to it by other processes and is responsible 7292 for distributing that, as necessary. 7293 */ 7294 void runWebsocketServer()() { 7295 assert(0, "not implemented"); 7296 } 7297 7298 void sendToWebsocketServer(WebSocket ws, string group) { 7299 assert(0, "not implemented"); 7300 } 7301 7302 void sendToWebsocketServer(string content, string group) { 7303 assert(0, "not implemented"); 7304 } 7305 7306 7307 void runEventServer()() { 7308 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7309 } 7310 7311 void runTimerServer()() { 7312 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7313 } 7314 7315 version(Posix) { 7316 alias LocalServerConnectionHandle = int; 7317 alias CgiConnectionHandle = int; 7318 alias SocketConnectionHandle = int; 7319 7320 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7321 } else version(Windows) { 7322 alias LocalServerConnectionHandle = HANDLE; 7323 version(embedded_httpd_threads) { 7324 alias CgiConnectionHandle = SOCKET; 7325 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7326 } else version(fastcgi) { 7327 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7328 enum INVALID_CGI_CONNECTION_HANDLE = null; 7329 } else version(scgi) { 7330 alias CgiConnectionHandle = SOCKET; 7331 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7332 } else { /* version(plain_cgi) */ 7333 alias CgiConnectionHandle = HANDLE; 7334 enum INVALID_CGI_CONNECTION_HANDLE = null; 7335 } 7336 alias SocketConnectionHandle = SOCKET; 7337 } 7338 7339 version(with_addon_servers_connections) 7340 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7341 version(Posix) { 7342 import core.sys.posix.unistd; 7343 import core.sys.posix.sys.un; 7344 7345 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7346 if(sock == -1) 7347 throw new Exception("socket " ~ to!string(errno)); 7348 7349 scope(failure) 7350 close(sock); 7351 7352 cloexec(sock); 7353 7354 // add-on server processes are assumed to be local, and thus will 7355 // use unix domain sockets. Besides, I want to pass sockets to them, 7356 // so it basically must be local (except for the session server, but meh). 7357 sockaddr_un addr; 7358 addr.sun_family = AF_UNIX; 7359 version(linux) { 7360 // on linux, we will use the abstract namespace 7361 addr.sun_path[0] = 0; 7362 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7363 } else { 7364 // but otherwise, just use a file cuz we must. 7365 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7366 } 7367 7368 bool alreadyTried; 7369 7370 try_again: 7371 7372 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7373 if(!alreadyTried && errno == ECONNREFUSED) { 7374 // try auto-spawning the server, then attempt connection again 7375 startAddonServer(arg); 7376 import core.thread; 7377 Thread.sleep(50.msecs); 7378 alreadyTried = true; 7379 goto try_again; 7380 } else 7381 throw new Exception("connect " ~ to!string(errno)); 7382 } 7383 7384 return sock; 7385 } else version(Windows) { 7386 return null; // FIXME 7387 } 7388 } 7389 7390 version(with_addon_servers_connections) 7391 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7392 version(Posix) { 7393 import core.sys.posix.unistd; 7394 close(handle); 7395 } else version(Windows) 7396 CloseHandle(handle); 7397 } 7398 7399 void runSessionServer()() { 7400 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7401 } 7402 7403 import core.stdc.errno; 7404 7405 struct IoOp { 7406 @disable this(); 7407 @disable this(this); 7408 7409 /* 7410 So we want to be able to eventually handle generic sockets too. 7411 */ 7412 7413 enum Read = 1; 7414 enum Write = 2; 7415 enum Accept = 3; 7416 enum ReadSocketHandle = 4; 7417 7418 // Your handler may be called in a different thread than the one that initiated the IO request! 7419 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7420 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7421 private void delegate(IoOp*) closeHandler; 7422 private void delegate(IoOp*) completeHandler; 7423 private int internalFd; 7424 private int operation; 7425 private int bufferLengthAllocated; 7426 private int bufferLengthUsed; 7427 private ubyte[1] internalBuffer; // it can be overallocated! 7428 7429 ubyte[] allocatedBuffer() return { 7430 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7431 } 7432 7433 ubyte[] usedBuffer() return { 7434 return allocatedBuffer[0 .. bufferLengthUsed]; 7435 } 7436 7437 void reset() { 7438 bufferLengthUsed = 0; 7439 } 7440 7441 int fd() { 7442 return internalFd; 7443 } 7444 } 7445 7446 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7447 import core.stdc.stdlib; 7448 7449 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7450 if(ptr is null) 7451 assert(0); // out of memory! 7452 7453 auto op = cast(IoOp*) ptr; 7454 7455 op.handler = handler; 7456 op.internalFd = fd; 7457 op.operation = operation; 7458 op.bufferLengthAllocated = bufferSize; 7459 op.bufferLengthUsed = 0; 7460 7461 import core.memory; 7462 7463 GC.addRoot(ptr); 7464 7465 return op; 7466 } 7467 7468 void freeIoOp(ref IoOp* ptr) { 7469 7470 import core.memory; 7471 GC.removeRoot(ptr); 7472 7473 import core.stdc.stdlib; 7474 free(ptr); 7475 ptr = null; 7476 } 7477 7478 version(Posix) 7479 version(with_addon_servers_connections) 7480 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7481 7482 //import std.stdio : writeln; writeln(cast(string) data); 7483 7484 import core.sys.posix.unistd; 7485 7486 auto ret = write(connection, data.ptr, data.length); 7487 if(ret != data.length) { 7488 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7489 // the file is closed, remove it 7490 eis.fileClosed(connection); 7491 } else 7492 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7493 } 7494 } 7495 version(Windows) 7496 version(with_addon_servers_connections) 7497 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7498 // FIXME 7499 } 7500 7501 bool isInvalidHandle(CgiConnectionHandle h) { 7502 return h == INVALID_CGI_CONNECTION_HANDLE; 7503 } 7504 7505 /+ 7506 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7507 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7508 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7509 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7510 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7511 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7512 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7513 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7514 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7515 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7516 7517 +/ 7518 7519 /++ 7520 You can customize your server by subclassing the appropriate server. Then, register your 7521 subclass at compile time with the [registerEventIoServer] template, or implement your own 7522 main function and call it yourself. 7523 7524 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7525 +/ 7526 version(with_addon_servers_connections) 7527 interface EventIoServer { 7528 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7529 void handleLocalConnectionClose(IoOp* op); 7530 void handleLocalConnectionComplete(IoOp* op); 7531 void wait_timeout(); 7532 void fileClosed(int fd); 7533 7534 void epoll_fd(int fd); 7535 } 7536 7537 // the sink should buffer it 7538 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7539 static if(is(T == struct)) { 7540 foreach(member; __traits(allMembers, T)) 7541 serialize(sink, __traits(getMember, t, member)); 7542 } else static if(is(T : int)) { 7543 // no need to think of endianness just because this is only used 7544 // for local, same-machine stuff anyway. thanks private lol 7545 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7546 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7547 // these are common enough to optimize 7548 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7549 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7550 sink(cast(ubyte[]) t[]); 7551 } else static if(is(T : A[], A)) { 7552 // generic array is less optimal but still prolly ok 7553 int len = cast(int) t.length; 7554 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7555 foreach(item; t) 7556 serialize(sink, item); 7557 } else static assert(0, T.stringof); 7558 } 7559 7560 // all may be stack buffers, so use cautio 7561 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7562 static if(is(T == struct)) { 7563 T t; 7564 foreach(member; __traits(allMembers, T)) 7565 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7566 dg(t); 7567 } else static if(is(T : int)) { 7568 // no need to think of endianness just because this is only used 7569 // for local, same-machine stuff anyway. thanks private lol 7570 T t; 7571 auto data = get(t.sizeof); 7572 t = (cast(T[]) data)[0]; 7573 dg(t); 7574 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7575 // these are common enough to optimize 7576 int len; 7577 auto data = get(len.sizeof); 7578 len = (cast(int[]) data)[0]; 7579 7580 /* 7581 typeof(T[0])[2000] stackBuffer; 7582 T buffer; 7583 7584 if(len < stackBuffer.length) 7585 buffer = stackBuffer[0 .. len]; 7586 else 7587 buffer = new T(len); 7588 7589 data = get(len * typeof(T[0]).sizeof); 7590 */ 7591 7592 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7593 7594 dg(t); 7595 } else static if(is(T == E[], E)) { 7596 T t; 7597 int len; 7598 auto data = get(len.sizeof); 7599 len = (cast(int[]) data)[0]; 7600 t.length = len; 7601 foreach(ref e; t) { 7602 deserialize!E(get, (ele) { e = ele; }); 7603 } 7604 dg(t); 7605 } else static assert(0, T.stringof); 7606 } 7607 7608 unittest { 7609 serialize((ubyte[] b) { 7610 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7611 }, 1); 7612 serialize((ubyte[] b) { 7613 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7614 }, 56674); 7615 ubyte[1000] buffer; 7616 int bufferPoint; 7617 void add(scope ubyte[] b) { 7618 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7619 bufferPoint += b.length; 7620 } 7621 ubyte[] get(int sz) { 7622 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7623 bufferPoint += sz; 7624 return b; 7625 } 7626 serialize(&add, "test here"); 7627 bufferPoint = 0; 7628 deserialize!string(&get, (t) { assert(t == "test here"); }); 7629 bufferPoint = 0; 7630 7631 struct Foo { 7632 int a; 7633 ubyte c; 7634 string d; 7635 } 7636 serialize(&add, Foo(403, 37, "amazing")); 7637 bufferPoint = 0; 7638 deserialize!Foo(&get, (t) { 7639 assert(t.a == 403); 7640 assert(t.c == 37); 7641 assert(t.d == "amazing"); 7642 }); 7643 bufferPoint = 0; 7644 } 7645 7646 /* 7647 Here's the way the RPC interface works: 7648 7649 You define the interface that lists the functions you can call on the remote process. 7650 The interface may also have static methods for convenience. These forward to a singleton 7651 instance of an auto-generated class, which actually sends the args over the pipe. 7652 7653 An impl class actually implements it. A receiving server deserializes down the pipe and 7654 calls methods on the class. 7655 7656 I went with the interface to get some nice compiler checking and documentation stuff. 7657 7658 I could have skipped the interface and just implemented it all from the server class definition 7659 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7660 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7661 7662 7663 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7664 7665 Realistically though the bodies would just be 7666 connection.call(this.mangleof, args...) sooooo. 7667 7668 FIXME: overloads aren't supported 7669 */ 7670 7671 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7672 interface SessionObject {} 7673 7674 private immutable void delegate(string[])[string] scheduledJobHandlers; 7675 private immutable void delegate(string[])[string] websocketServers; 7676 7677 version(with_breaking_cgi_features) 7678 mixin(q{ 7679 7680 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7681 static import std.traits; 7682 7683 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7684 static foreach(idx, member; __traits(derivedMembers, T)) { 7685 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7686 mixin( q{ 7687 std.traits.ReturnType!(__traits(getMember, T, member)) 7688 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7689 { 7690 SerializationBuffer buffer; 7691 auto i = cast(ushort) idx; 7692 serialize(&buffer.sink, i); 7693 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7694 foreach(param; params) 7695 serialize(&buffer.sink, param); 7696 7697 auto sendable = buffer.sendable; 7698 7699 version(Posix) {{ 7700 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7701 7702 if(ret == -1) { 7703 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7704 } else if(ret == 0) { 7705 throw new Exception("Connection to addon server lost"); 7706 } if(ret < sendable.length) 7707 throw new Exception("Send failed to send all"); 7708 assert(ret == sendable.length); 7709 }} // FIXME Windows impl 7710 7711 static if(!is(typeof(return) == void)) { 7712 // there is a return value; we need to wait for it too 7713 version(Posix) { 7714 ubyte[3000] revBuffer; 7715 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7716 auto got = revBuffer[0 .. ret]; 7717 7718 int dataLocation; 7719 ubyte[] grab(int sz) { 7720 auto dataLocation1 = dataLocation; 7721 dataLocation += sz; 7722 return got[dataLocation1 .. dataLocation]; 7723 } 7724 7725 typeof(return) retu; 7726 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7727 return retu; 7728 } else { 7729 // FIXME Windows impl 7730 return typeof(return).init; 7731 } 7732 7733 } 7734 }}); 7735 } 7736 7737 private static typeof(this) singletonInstance; 7738 private LocalServerConnectionHandle connectionHandle; 7739 7740 static typeof(this) connection() { 7741 if(singletonInstance is null) { 7742 singletonInstance = new typeof(this)(); 7743 singletonInstance.connect(); 7744 } 7745 return singletonInstance; 7746 } 7747 7748 void connect() { 7749 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7750 } 7751 7752 void disconnect() { 7753 closeLocalServerConnection(connectionHandle); 7754 } 7755 } 7756 7757 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7758 ushort calledIdx; 7759 string calledFunction; 7760 7761 int dataLocation; 7762 ubyte[] grab(int sz) { 7763 if(sz == 0) assert(0); 7764 auto d = data[dataLocation .. dataLocation + sz]; 7765 dataLocation += sz; 7766 return d; 7767 } 7768 7769 again: 7770 7771 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7772 deserialize!string(&grab, (a) { calledFunction = a; }); 7773 7774 import std.traits; 7775 7776 sw: switch(calledIdx) { 7777 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7778 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 7779 case idx: 7780 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7781 7782 Parameters!(__traits(getMember, Interface, memberName)) params; 7783 foreach(ref param; params) 7784 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7785 7786 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7787 __traits(getMember, this_, memberName)(params); 7788 } else { 7789 auto ret = __traits(getMember, this_, memberName)(params); 7790 SerializationBuffer buffer; 7791 serialize(&buffer.sink, ret); 7792 7793 auto sendable = buffer.sendable; 7794 7795 version(Posix) { 7796 auto r = send(fd, sendable.ptr, sendable.length, 0); 7797 if(r == -1) { 7798 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7799 } else if(r == 0) { 7800 throw new Exception("Connection to addon client lost"); 7801 } if(r < sendable.length) 7802 throw new Exception("Send failed to send all"); 7803 7804 } // FIXME Windows impl 7805 } 7806 break sw; 7807 } 7808 default: assert(0); 7809 } 7810 7811 if(dataLocation != data.length) 7812 goto again; 7813 } 7814 7815 7816 private struct SerializationBuffer { 7817 ubyte[2048] bufferBacking; 7818 int bufferLocation; 7819 void sink(scope ubyte[] data) { 7820 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7821 bufferLocation += data.length; 7822 } 7823 7824 ubyte[] sendable() return { 7825 return bufferBacking[0 .. bufferLocation]; 7826 } 7827 } 7828 7829 /* 7830 FIXME: 7831 add a version command line arg 7832 version data in the library 7833 management gui as external program 7834 7835 at server with event_fd for each run 7836 use .mangleof in the at function name 7837 7838 i think the at server will have to: 7839 pipe args to the child 7840 collect child output for logging 7841 get child return value for logging 7842 7843 on windows timers work differently. idk how to best combine with the io stuff. 7844 7845 will have to have dump and restore too, so i can restart without losing stuff. 7846 */ 7847 7848 /++ 7849 A convenience object for talking to the [BasicDataServer] from a higher level. 7850 See: [Cgi.getSessionObject]. 7851 7852 You pass it a `Data` struct describing the data you want saved in the session. 7853 Then, this class will generate getter and setter properties that allow access 7854 to that data. 7855 7856 Note that each load and store will be done as-accessed; it doesn't front-load 7857 mutable data nor does it batch updates out of fear of read-modify-write race 7858 conditions. (In fact, right now it does this for everything, but in the future, 7859 I might batch load `immutable` members of the Data struct.) 7860 7861 At some point in the future, I might also let it do different backends, like 7862 a client-side cookie store too, but idk. 7863 7864 Note that the plain-old-data members of your `Data` struct are wrapped by this 7865 interface via a static foreach to make property functions. 7866 7867 See_Also: [MockSession] 7868 +/ 7869 interface Session(Data) : SessionObject { 7870 @property string sessionId() const; 7871 7872 /++ 7873 Starts a new session. Note that a session is also 7874 implicitly started as soon as you write data to it, 7875 so if you need to alter these parameters from their 7876 defaults, be sure to explicitly call this BEFORE doing 7877 any writes to session data. 7878 7879 Params: 7880 idleLifetime = How long, in seconds, the session 7881 should remain in memory when not being read from 7882 or written to. The default is one day. 7883 7884 NOT IMPLEMENTED 7885 7886 useExtendedLifetimeCookie = The session ID is always 7887 stored in a HTTP cookie, and by default, that cookie 7888 is discarded when the user closes their browser. 7889 7890 But if you set this to true, it will use a non-perishable 7891 cookie for the given idleLifetime. 7892 7893 NOT IMPLEMENTED 7894 +/ 7895 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7896 7897 /++ 7898 Regenerates the session ID and updates the associated 7899 cookie. 7900 7901 This is also your chance to change immutable data 7902 (not yet implemented). 7903 +/ 7904 void regenerateId(); 7905 7906 /++ 7907 Terminates this session, deleting all saved data. 7908 +/ 7909 void terminate(); 7910 7911 /++ 7912 Plain-old-data members of your `Data` struct are wrapped here via 7913 the property getters and setters. 7914 7915 If the member is a non-string array, it returns a magical array proxy 7916 object which allows for atomic appends and replaces via overloaded operators. 7917 You can slice this to get a range representing a $(B const) view of the array. 7918 This is to protect you against read-modify-write race conditions. 7919 +/ 7920 static foreach(memberName; __traits(allMembers, Data)) 7921 static if(is(typeof(__traits(getMember, Data, memberName)))) 7922 mixin(q{ 7923 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7924 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7925 }); 7926 7927 } 7928 7929 /++ 7930 An implementation of [Session] that works on real cgi connections utilizing the 7931 [BasicDataServer]. 7932 7933 As opposed to a [MockSession] which is made for testing purposes. 7934 7935 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7936 +/ 7937 class BasicDataServerSession(Data) : Session!Data { 7938 private Cgi cgi; 7939 private string sessionId_; 7940 7941 public @property string sessionId() const { 7942 return sessionId_; 7943 } 7944 7945 protected @property string sessionId(string s) { 7946 return this.sessionId_ = s; 7947 } 7948 7949 private this(Cgi cgi) { 7950 this.cgi = cgi; 7951 if(auto ptr = "sessionId" in cgi.cookies) 7952 sessionId = (*ptr).length ? *ptr : null; 7953 } 7954 7955 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7956 assert(sessionId is null); 7957 7958 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7959 7960 import std.random, std.conv; 7961 sessionId = to!string(uniform(1, long.max)); 7962 7963 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7964 setCookie(); 7965 } 7966 7967 protected void setCookie() { 7968 cgi.setCookie( 7969 "sessionId", sessionId, 7970 0 /* expiration */, 7971 "/" /* path */, 7972 null /* domain */, 7973 true /* http only */, 7974 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 7975 } 7976 7977 void regenerateId() { 7978 if(sessionId is null) { 7979 start(); 7980 return; 7981 } 7982 import std.random, std.conv; 7983 auto oldSessionId = sessionId; 7984 sessionId = to!string(uniform(1, long.max)); 7985 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 7986 setCookie(); 7987 } 7988 7989 void terminate() { 7990 BasicDataServer.connection.destroySession(sessionId); 7991 sessionId = null; 7992 setCookie(); 7993 } 7994 7995 static foreach(memberName; __traits(allMembers, Data)) 7996 static if(is(typeof(__traits(getMember, Data, memberName)))) 7997 mixin(q{ 7998 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7999 if(sessionId is null) 8000 return typeof(return).init; 8001 8002 import std.traits; 8003 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8004 if(v.length == 0) 8005 return typeof(return).init; 8006 import std.conv; 8007 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8008 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8009 // basically. Assuming the session is POD this should be fine. 8010 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8011 } 8012 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8013 if(sessionId is null) 8014 start(); 8015 import std.conv; 8016 import std.traits; 8017 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8018 return value; 8019 } 8020 }); 8021 } 8022 8023 /++ 8024 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8025 Simply stores the data in its instance members. 8026 +/ 8027 class MockSession(Data) : Session!Data { 8028 pure { 8029 @property string sessionId() const { return "mock"; } 8030 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8031 void regenerateId() {} 8032 void terminate() {} 8033 8034 private Data store_; 8035 8036 static foreach(memberName; __traits(allMembers, Data)) 8037 static if(is(typeof(__traits(getMember, Data, memberName)))) 8038 mixin(q{ 8039 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8040 return __traits(getMember, store_, memberName); 8041 } 8042 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8043 return __traits(getMember, store_, memberName) = value; 8044 } 8045 }); 8046 } 8047 } 8048 8049 /++ 8050 Direct interface to the basic data add-on server. You can 8051 typically use [Cgi.getSessionObject] as a more convenient interface. 8052 +/ 8053 version(with_addon_servers_connections) 8054 interface BasicDataServer { 8055 /// 8056 void createSession(string sessionId, int lifetime); 8057 /// 8058 void renewSession(string sessionId, int lifetime); 8059 /// 8060 void destroySession(string sessionId); 8061 /// 8062 void renameSession(string oldSessionId, string newSessionId); 8063 8064 /// 8065 void setSessionData(string sessionId, string dataKey, string dataValue); 8066 /// 8067 string getSessionData(string sessionId, string dataKey); 8068 8069 /// 8070 static BasicDataServerConnection connection() { 8071 return BasicDataServerConnection.connection(); 8072 } 8073 } 8074 8075 version(with_addon_servers_connections) 8076 class BasicDataServerConnection : BasicDataServer { 8077 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8078 } 8079 8080 version(with_addon_servers) 8081 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8082 8083 void createSession(string sessionId, int lifetime) { 8084 sessions[sessionId.idup] = Session(lifetime); 8085 } 8086 void destroySession(string sessionId) { 8087 sessions.remove(sessionId); 8088 } 8089 void renewSession(string sessionId, int lifetime) { 8090 sessions[sessionId].lifetime = lifetime; 8091 } 8092 void renameSession(string oldSessionId, string newSessionId) { 8093 sessions[newSessionId.idup] = sessions[oldSessionId]; 8094 sessions.remove(oldSessionId); 8095 } 8096 void setSessionData(string sessionId, string dataKey, string dataValue) { 8097 if(sessionId !in sessions) 8098 createSession(sessionId, 3600); // FIXME? 8099 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8100 } 8101 string getSessionData(string sessionId, string dataKey) { 8102 if(auto session = sessionId in sessions) { 8103 if(auto data = dataKey in (*session).values) 8104 return *data; 8105 else 8106 return null; // no such data 8107 8108 } else { 8109 return null; // no session 8110 } 8111 } 8112 8113 8114 protected: 8115 8116 struct Session { 8117 int lifetime; 8118 8119 string[string] values; 8120 } 8121 8122 Session[string] sessions; 8123 8124 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8125 auto data = op.usedBuffer; 8126 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8127 return false; 8128 } 8129 8130 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8131 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8132 void wait_timeout() {} 8133 void fileClosed(int fd) {} // stateless so irrelevant 8134 void epoll_fd(int fd) {} 8135 } 8136 8137 /++ 8138 See [schedule] to make one of these. You then call one of the methods here to set it up: 8139 8140 --- 8141 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8142 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8143 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8144 --- 8145 +/ 8146 version(with_addon_servers_connections) 8147 struct ScheduledJobHelper { 8148 private string func; 8149 private string[] args; 8150 private bool consumed; 8151 8152 private this(string func, string[] args) { 8153 this.func = func; 8154 this.args = args; 8155 } 8156 8157 ~this() { 8158 assert(consumed); 8159 } 8160 8161 /++ 8162 Schedules the job to be run at the given time. 8163 +/ 8164 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8165 consumed = true; 8166 8167 auto conn = ScheduledJobServerConnection.connection; 8168 import std.file; 8169 auto st = SysTime(when, timezone); 8170 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8171 } 8172 8173 /++ 8174 Schedules the job to run at least after the specified delay. 8175 +/ 8176 void delay(Duration delay) { 8177 consumed = true; 8178 8179 auto conn = ScheduledJobServerConnection.connection; 8180 import std.file; 8181 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8182 } 8183 8184 /++ 8185 Runs the job in the background ASAP. 8186 8187 $(NOTE It may run in a background thread. Don't segfault!) 8188 +/ 8189 void asap() { 8190 consumed = true; 8191 8192 auto conn = ScheduledJobServerConnection.connection; 8193 import std.file; 8194 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8195 } 8196 8197 /+ 8198 /++ 8199 Schedules the job to recur on the given pattern. 8200 +/ 8201 void recur(string spec) { 8202 8203 } 8204 +/ 8205 } 8206 8207 /++ 8208 First step to schedule a job on the scheduled job server. 8209 8210 The scheduled job needs to be a top-level function that doesn't read any 8211 variables from outside its arguments because it may be run in a new process, 8212 without any context existing later. 8213 8214 You MUST set details on the returned object to actually do anything! 8215 +/ 8216 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8217 /// 8218 ScheduledJobHelper schedule(T args) { 8219 // this isn't meant to ever be called, but instead just to 8220 // get the compiler to type check the arguments passed for us 8221 auto sample = delegate() { 8222 fn(args); 8223 }; 8224 string[] sargs; 8225 foreach(arg; args) 8226 sargs ~= to!string(arg); 8227 return ScheduledJobHelper(fn.mangleof, sargs); 8228 } 8229 8230 shared static this() { 8231 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8232 import std.traits; 8233 Parameters!fn args; 8234 foreach(idx, ref arg; args) 8235 arg = to!(typeof(arg))(sargs[idx]); 8236 fn(args); 8237 }; 8238 } 8239 } 8240 8241 /// 8242 interface ScheduledJobServer { 8243 /// Use the [schedule] function for a higher-level interface. 8244 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8245 /// 8246 void cancelJob(int jobId); 8247 } 8248 8249 version(with_addon_servers_connections) 8250 class ScheduledJobServerConnection : ScheduledJobServer { 8251 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8252 } 8253 8254 version(with_addon_servers) 8255 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8256 // FIXME: we need to handle SIGCHLD in this somehow 8257 // whenIs is 0 for relative, 1 for absolute 8258 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8259 auto nj = nextJobId; 8260 nextJobId++; 8261 8262 version(linux) { 8263 import core.sys.linux.timerfd; 8264 import core.sys.linux.epoll; 8265 import core.sys.posix.unistd; 8266 8267 8268 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8269 if(fd == -1) 8270 throw new Exception("fd timer create failed"); 8271 8272 foreach(ref arg; args) 8273 arg = arg.idup; 8274 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8275 8276 itimerspec value; 8277 value.it_value.tv_sec = when; 8278 value.it_value.tv_nsec = 0; 8279 8280 value.it_interval.tv_sec = 0; 8281 value.it_interval.tv_nsec = 0; 8282 8283 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8284 throw new Exception("couldn't set fd timer"); 8285 8286 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8287 jobs.remove(nj); 8288 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8289 close(fd); 8290 8291 8292 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8293 8294 return true; 8295 }); 8296 scope(failure) 8297 freeIoOp(op); 8298 8299 epoll_event ev; 8300 ev.events = EPOLLIN | EPOLLET; 8301 ev.data.ptr = op; 8302 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8303 throw new Exception("epoll_ctl " ~ to!string(errno)); 8304 8305 jobs[nj] = job; 8306 return nj; 8307 } else assert(0); 8308 } 8309 8310 protected void cancelJob(int jobId) { 8311 version(linux) { 8312 auto job = jobId in jobs; 8313 if(job is null) 8314 return; 8315 8316 jobs.remove(jobId); 8317 8318 version(linux) { 8319 import core.sys.linux.timerfd; 8320 import core.sys.linux.epoll; 8321 import core.sys.posix.unistd; 8322 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8323 close(job.timerfd); 8324 } 8325 } 8326 jobs.remove(jobId); 8327 } 8328 8329 int nextJobId = 1; 8330 static struct Job { 8331 string executable; 8332 string func; 8333 string[] args; 8334 int timerfd; 8335 int id; 8336 } 8337 Job[int] jobs; 8338 8339 8340 // event io server methods below 8341 8342 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8343 auto data = op.usedBuffer; 8344 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8345 return false; 8346 } 8347 8348 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8349 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8350 void wait_timeout() {} 8351 void fileClosed(int fd) {} // stateless so irrelevant 8352 8353 int epoll_fd_; 8354 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8355 int epoll_fd() { return epoll_fd_; } 8356 } 8357 8358 /// 8359 version(with_addon_servers_connections) 8360 interface EventSourceServer { 8361 /++ 8362 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8363 8364 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8365 8366 See_Also: 8367 [sendEvent] 8368 +/ 8369 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8370 /* 8371 If lastEventId is missing or empty, you just get new events as they come. 8372 8373 If it is set from something else, it sends all since then (that are still alive) 8374 down the pipe immediately. 8375 8376 The reason it can come from the header is that's what the standard defines for 8377 browser reconnects. The reason it can come from a query string is just convenience 8378 in catching up in a user-defined manner. 8379 8380 The reason the header overrides the query string is if the browser tries to reconnect, 8381 it will send the header AND the query (it reconnects to the same url), so we just 8382 want to do the restart thing. 8383 8384 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8385 */ 8386 string lastEventId = cgi.lastEventId; 8387 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8388 lastEventId = cgi.get["lastEventId"]; 8389 8390 cgi.setResponseContentType("text/event-stream"); 8391 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8392 cgi.flush(); 8393 8394 cgi.closed = true; 8395 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8396 scope(exit) 8397 closeLocalServerConnection(s); 8398 8399 version(fastcgi) 8400 throw new Exception("sending fcgi connections not supported"); 8401 else { 8402 auto fd = cgi.getOutputFileHandle(); 8403 if(isInvalidHandle(fd)) 8404 throw new Exception("bad fd from cgi!"); 8405 8406 EventSourceServerImplementation.SendableEventConnection sec; 8407 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8408 8409 version(Posix) { 8410 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8411 assert(res == sec.sizeof); 8412 } else version(Windows) { 8413 // FIXME 8414 } 8415 } 8416 } 8417 8418 /++ 8419 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8420 8421 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8422 8423 Params: 8424 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8425 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8426 data = the event data. Available in JS as `event.data`. 8427 lifetime = the amount of time to keep this event for replaying on the event server. 8428 8429 See_Also: 8430 [sendEventToEventServer] 8431 +/ 8432 public static void sendEvent(string url, string event, string data, int lifetime) { 8433 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8434 scope(exit) 8435 closeLocalServerConnection(s); 8436 8437 EventSourceServerImplementation.SendableEvent sev; 8438 sev.populate(url, event, data, lifetime); 8439 8440 version(Posix) { 8441 auto ret = send(s, &sev, sev.sizeof, 0); 8442 assert(ret == sev.sizeof); 8443 } else version(Windows) { 8444 // FIXME 8445 } 8446 } 8447 8448 /++ 8449 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8450 8451 See_Also: [disconnect] 8452 +/ 8453 void connect(string url, string forwardUrl); 8454 8455 /++ 8456 Disconnects `forwardUrl` from `url` 8457 8458 See_Also: [connect] 8459 +/ 8460 void disconnect(string url, string forwardUrl); 8461 } 8462 8463 /// 8464 version(with_addon_servers) 8465 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8466 8467 protected: 8468 8469 void connect(string url, string forwardUrl) { 8470 pipes[url] ~= forwardUrl; 8471 } 8472 void disconnect(string url, string forwardUrl) { 8473 auto t = url in pipes; 8474 if(t is null) 8475 return; 8476 foreach(idx, n; (*t)) 8477 if(n == forwardUrl) { 8478 (*t)[idx] = (*t)[$-1]; 8479 (*t) = (*t)[0 .. $-1]; 8480 break; 8481 } 8482 } 8483 8484 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8485 if(receivedFd != -1) { 8486 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8487 8488 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8489 8490 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8491 8492 auto url = got.url.idup; 8493 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8494 8495 // FIXME: catch up on past messages here 8496 } else { 8497 auto data = op.usedBuffer; 8498 auto event = cast(SendableEvent*) data.ptr; 8499 8500 if(event.magic == 0xdeadbeef) { 8501 handleInputEvent(event); 8502 8503 if(event.url in pipes) 8504 foreach(pipe; pipes[event.url]) { 8505 event.url = pipe; 8506 handleInputEvent(event); 8507 } 8508 } else { 8509 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8510 } 8511 } 8512 return false; 8513 } 8514 void handleLocalConnectionClose(IoOp* op) { 8515 fileClosed(op.fd); 8516 } 8517 void handleLocalConnectionComplete(IoOp* op) {} 8518 8519 void wait_timeout() { 8520 // just keeping alive 8521 foreach(url, connections; eventConnectionsByUrl) 8522 foreach(connection; connections) 8523 if(connection.needsChunking) 8524 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8525 else 8526 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8527 } 8528 8529 void fileClosed(int fd) { 8530 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8531 foreach(idx, conn; connections) { 8532 if(fd == conn.fd) { 8533 connections[idx] = connections[$-1]; 8534 connections = connections[0 .. $ - 1]; 8535 continue outer; 8536 } 8537 } 8538 } 8539 } 8540 8541 void epoll_fd(int fd) {} 8542 8543 8544 private: 8545 8546 8547 struct SendableEventConnection { 8548 ubyte responseChunked; 8549 8550 int urlLength; 8551 char[256] urlBuffer = 0; 8552 8553 int lastEventIdLength; 8554 char[32] lastEventIdBuffer = 0; 8555 8556 char[] url() return { 8557 return urlBuffer[0 .. urlLength]; 8558 } 8559 void url(in char[] u) { 8560 urlBuffer[0 .. u.length] = u[]; 8561 urlLength = cast(int) u.length; 8562 } 8563 char[] lastEventId() return { 8564 return lastEventIdBuffer[0 .. lastEventIdLength]; 8565 } 8566 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8567 in { 8568 assert(url.length < this.urlBuffer.length); 8569 assert(lastEventId.length < this.lastEventIdBuffer.length); 8570 } 8571 do { 8572 this.responseChunked = responseChunked ? 1 : 0; 8573 this.urlLength = cast(int) url.length; 8574 this.lastEventIdLength = cast(int) lastEventId.length; 8575 8576 this.urlBuffer[0 .. url.length] = url[]; 8577 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8578 } 8579 } 8580 8581 struct SendableEvent { 8582 int magic = 0xdeadbeef; 8583 int urlLength; 8584 char[256] urlBuffer = 0; 8585 int typeLength; 8586 char[32] typeBuffer = 0; 8587 int messageLength; 8588 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8589 int _lifetime; 8590 8591 char[] message() return { 8592 return messageBuffer[0 .. messageLength]; 8593 } 8594 char[] type() return { 8595 return typeBuffer[0 .. typeLength]; 8596 } 8597 char[] url() return { 8598 return urlBuffer[0 .. urlLength]; 8599 } 8600 void url(in char[] u) { 8601 urlBuffer[0 .. u.length] = u[]; 8602 urlLength = cast(int) u.length; 8603 } 8604 int lifetime() { 8605 return _lifetime; 8606 } 8607 8608 /// 8609 void populate(string url, string type, string message, int lifetime) 8610 in { 8611 assert(url.length < this.urlBuffer.length); 8612 assert(type.length < this.typeBuffer.length); 8613 assert(message.length < this.messageBuffer.length); 8614 } 8615 do { 8616 this.urlLength = cast(int) url.length; 8617 this.typeLength = cast(int) type.length; 8618 this.messageLength = cast(int) message.length; 8619 this._lifetime = lifetime; 8620 8621 this.urlBuffer[0 .. url.length] = url[]; 8622 this.typeBuffer[0 .. type.length] = type[]; 8623 this.messageBuffer[0 .. message.length] = message[]; 8624 } 8625 } 8626 8627 struct EventConnection { 8628 int fd; 8629 bool needsChunking; 8630 } 8631 8632 private EventConnection[][string] eventConnectionsByUrl; 8633 private string[][string] pipes; 8634 8635 private void handleInputEvent(scope SendableEvent* event) { 8636 static int eventId; 8637 8638 static struct StoredEvent { 8639 int id; 8640 string type; 8641 string message; 8642 int lifetimeRemaining; 8643 } 8644 8645 StoredEvent[][string] byUrl; 8646 8647 int thisId = ++eventId; 8648 8649 if(event.lifetime) 8650 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8651 8652 auto connectionsPtr = event.url in eventConnectionsByUrl; 8653 EventConnection[] connections; 8654 if(connectionsPtr is null) 8655 return; 8656 else 8657 connections = *connectionsPtr; 8658 8659 char[4096] buffer; 8660 char[] formattedMessage; 8661 8662 void append(const char[] a) { 8663 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8664 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8665 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8666 } 8667 8668 import std.algorithm.iteration; 8669 8670 if(connections.length) { 8671 append("id: "); 8672 append(to!string(thisId)); 8673 append("\n"); 8674 8675 append("event: "); 8676 append(event.type); 8677 append("\n"); 8678 8679 foreach(line; event.message.splitter("\n")) { 8680 append("data: "); 8681 append(line); 8682 append("\n"); 8683 } 8684 8685 append("\n"); 8686 } 8687 8688 // chunk it for HTTP! 8689 auto len = toHex(formattedMessage.length); 8690 buffer[4 .. 6] = "\r\n"[]; 8691 buffer[4 - len.length .. 4] = len[]; 8692 buffer[6 + formattedMessage.length] = '\r'; 8693 buffer[6 + formattedMessage.length + 1] = '\n'; 8694 8695 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8696 // done 8697 8698 // FIXME: send back requests when needed 8699 // FIXME: send a single ":\n" every 15 seconds to keep alive 8700 8701 foreach(connection; connections) { 8702 if(connection.needsChunking) { 8703 nonBlockingWrite(this, connection.fd, chunkedMessage); 8704 } else { 8705 nonBlockingWrite(this, connection.fd, formattedMessage); 8706 } 8707 } 8708 } 8709 } 8710 8711 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8712 version(Posix) { 8713 8714 import core.sys.posix.unistd; 8715 import core.sys.posix.fcntl; 8716 import core.sys.posix.sys.un; 8717 8718 import core.sys.posix.signal; 8719 signal(SIGPIPE, SIG_IGN); 8720 8721 static extern(C) void sigchldhandler(int) { 8722 int status; 8723 import w = core.sys.posix.sys.wait; 8724 w.wait(&status); 8725 } 8726 signal(SIGCHLD, &sigchldhandler); 8727 8728 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8729 if(sock == -1) 8730 throw new Exception("socket " ~ to!string(errno)); 8731 8732 scope(failure) 8733 close(sock); 8734 8735 cloexec(sock); 8736 8737 // add-on server processes are assumed to be local, and thus will 8738 // use unix domain sockets. Besides, I want to pass sockets to them, 8739 // so it basically must be local (except for the session server, but meh). 8740 sockaddr_un addr; 8741 addr.sun_family = AF_UNIX; 8742 version(linux) { 8743 // on linux, we will use the abstract namespace 8744 addr.sun_path[0] = 0; 8745 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8746 } else { 8747 // but otherwise, just use a file cuz we must. 8748 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8749 } 8750 8751 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8752 throw new Exception("bind " ~ to!string(errno)); 8753 8754 if(listen(sock, 128) == -1) 8755 throw new Exception("listen " ~ to!string(errno)); 8756 8757 makeNonBlocking(sock); 8758 8759 version(linux) { 8760 import core.sys.linux.epoll; 8761 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8762 if(epoll_fd == -1) 8763 throw new Exception("epoll_create1 " ~ to!string(errno)); 8764 scope(failure) 8765 close(epoll_fd); 8766 } else { 8767 import core.sys.posix.poll; 8768 } 8769 8770 version(linux) 8771 eis.epoll_fd = epoll_fd; 8772 8773 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8774 scope(exit) 8775 freeIoOp(acceptOp); 8776 8777 version(linux) { 8778 epoll_event ev; 8779 ev.events = EPOLLIN | EPOLLET; 8780 ev.data.ptr = acceptOp; 8781 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8782 throw new Exception("epoll_ctl " ~ to!string(errno)); 8783 8784 epoll_event[64] events; 8785 } else { 8786 pollfd[] pollfds; 8787 IoOp*[int] ioops; 8788 pollfds ~= pollfd(sock, POLLIN); 8789 ioops[sock] = acceptOp; 8790 } 8791 8792 import core.time : MonoTime, seconds; 8793 8794 MonoTime timeout = MonoTime.currTime + 15.seconds; 8795 8796 while(true) { 8797 8798 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8799 8800 int timeout_milliseconds = 0; // -1; // infinite 8801 8802 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8803 if(timeout_milliseconds < 0) 8804 timeout_milliseconds = 0; 8805 8806 //writeln("waiting for ", name); 8807 8808 version(linux) { 8809 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8810 if(nfds == -1) { 8811 if(errno == EINTR) 8812 continue; 8813 throw new Exception("epoll_wait " ~ to!string(errno)); 8814 } 8815 } else { 8816 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8817 size_t lastIdx = 0; 8818 } 8819 8820 if(nfds == 0) { 8821 eis.wait_timeout(); 8822 timeout += 15.seconds; 8823 } 8824 8825 foreach(idx; 0 .. nfds) { 8826 version(linux) { 8827 auto flags = events[idx].events; 8828 auto ioop = cast(IoOp*) events[idx].data.ptr; 8829 } else { 8830 IoOp* ioop; 8831 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8832 if(thing.revents) { 8833 ioop = ioops[thing.fd]; 8834 lastIdx += tidx + 1; 8835 break; 8836 } 8837 } 8838 } 8839 8840 //writeln(flags, " ", ioop.fd); 8841 8842 void newConnection() { 8843 // on edge triggering, it is important that we get it all 8844 while(true) { 8845 version(Android) { 8846 auto size = cast(int) addr.sizeof; 8847 } else { 8848 auto size = cast(uint) addr.sizeof; 8849 } 8850 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8851 if(ns == -1) { 8852 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8853 // all done, got it all 8854 break; 8855 } 8856 throw new Exception("accept " ~ to!string(errno)); 8857 } 8858 cloexec(ns); 8859 8860 makeNonBlocking(ns); 8861 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8862 niop.closeHandler = &eis.handleLocalConnectionClose; 8863 niop.completeHandler = &eis.handleLocalConnectionComplete; 8864 scope(failure) freeIoOp(niop); 8865 8866 version(linux) { 8867 epoll_event nev; 8868 nev.events = EPOLLIN | EPOLLET; 8869 nev.data.ptr = niop; 8870 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8871 throw new Exception("epoll_ctl " ~ to!string(errno)); 8872 } else { 8873 bool found = false; 8874 foreach(ref pfd; pollfds) { 8875 if(pfd.fd < 0) { 8876 pfd.fd = ns; 8877 found = true; 8878 } 8879 } 8880 if(!found) 8881 pollfds ~= pollfd(ns, POLLIN); 8882 ioops[ns] = niop; 8883 } 8884 } 8885 } 8886 8887 bool newConnectionCondition() { 8888 version(linux) 8889 return ioop.fd == sock && (flags & EPOLLIN); 8890 else 8891 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8892 } 8893 8894 if(newConnectionCondition()) { 8895 newConnection(); 8896 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8897 while(true) { 8898 int in_fd; 8899 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8900 if(got == -1) { 8901 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8902 // all done, got it all 8903 if(ioop.completeHandler) 8904 ioop.completeHandler(ioop); 8905 break; 8906 } 8907 throw new Exception("recv " ~ to!string(errno)); 8908 } 8909 8910 if(got == 0) { 8911 if(ioop.closeHandler) { 8912 ioop.closeHandler(ioop); 8913 version(linux) {} // nothing needed 8914 else { 8915 foreach(ref pfd; pollfds) { 8916 if(pfd.fd == ioop.fd) 8917 pfd.fd = -1; 8918 } 8919 } 8920 } 8921 close(ioop.fd); 8922 freeIoOp(ioop); 8923 break; 8924 } 8925 8926 ioop.bufferLengthUsed = cast(int) got; 8927 ioop.handler(ioop, in_fd); 8928 } 8929 } else if(ioop.operation == IoOp.Read) { 8930 while(true) { 8931 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8932 if(got == -1) { 8933 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8934 // all done, got it all 8935 if(ioop.completeHandler) 8936 ioop.completeHandler(ioop); 8937 break; 8938 } 8939 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8940 } 8941 8942 if(got == 0) { 8943 if(ioop.closeHandler) 8944 ioop.closeHandler(ioop); 8945 close(ioop.fd); 8946 freeIoOp(ioop); 8947 break; 8948 } 8949 8950 ioop.bufferLengthUsed = cast(int) got; 8951 if(ioop.handler(ioop, ioop.fd)) { 8952 close(ioop.fd); 8953 freeIoOp(ioop); 8954 break; 8955 } 8956 } 8957 } 8958 8959 // EPOLLHUP? 8960 } 8961 } 8962 } else version(Windows) { 8963 8964 // set up a named pipe 8965 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8966 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8967 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8968 8969 } else static assert(0); 8970 } 8971 8972 8973 version(with_sendfd) 8974 // copied from the web and ported from C 8975 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 8976 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 8977 msghdr msg; 8978 iovec[1] iov; 8979 8980 version(OSX) { 8981 //msg.msg_accrights = cast(cattr_t) &sendfd; 8982 //msg.msg_accrightslen = int.sizeof; 8983 } else version(Android) { 8984 } else { 8985 union ControlUnion { 8986 cmsghdr cm; 8987 char[CMSG_SPACE(int.sizeof)] control; 8988 } 8989 8990 ControlUnion control_un; 8991 cmsghdr* cmptr; 8992 8993 msg.msg_control = control_un.control.ptr; 8994 msg.msg_controllen = control_un.control.length; 8995 8996 cmptr = CMSG_FIRSTHDR(&msg); 8997 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 8998 cmptr.cmsg_level = SOL_SOCKET; 8999 cmptr.cmsg_type = SCM_RIGHTS; 9000 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 9001 } 9002 9003 msg.msg_name = null; 9004 msg.msg_namelen = 0; 9005 9006 iov[0].iov_base = ptr; 9007 iov[0].iov_len = nbytes; 9008 msg.msg_iov = iov.ptr; 9009 msg.msg_iovlen = 1; 9010 9011 return sendmsg(fd, &msg, 0); 9012 } 9013 9014 version(with_sendfd) 9015 // copied from the web and ported from C 9016 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9017 msghdr msg; 9018 iovec[1] iov; 9019 ssize_t n; 9020 int newfd; 9021 9022 version(OSX) { 9023 //msg.msg_accrights = cast(cattr_t) recvfd; 9024 //msg.msg_accrightslen = int.sizeof; 9025 } else version(Android) { 9026 } else { 9027 union ControlUnion { 9028 cmsghdr cm; 9029 char[CMSG_SPACE(int.sizeof)] control; 9030 } 9031 ControlUnion control_un; 9032 cmsghdr* cmptr; 9033 9034 msg.msg_control = control_un.control.ptr; 9035 msg.msg_controllen = control_un.control.length; 9036 } 9037 9038 msg.msg_name = null; 9039 msg.msg_namelen = 0; 9040 9041 iov[0].iov_base = ptr; 9042 iov[0].iov_len = nbytes; 9043 msg.msg_iov = iov.ptr; 9044 msg.msg_iovlen = 1; 9045 9046 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9047 return n; 9048 9049 version(OSX) { 9050 //if(msg.msg_accrightslen != int.sizeof) 9051 //*recvfd = -1; 9052 } else version(Android) { 9053 } else { 9054 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9055 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9056 if (cmptr.cmsg_level != SOL_SOCKET) 9057 throw new Exception("control level != SOL_SOCKET"); 9058 if (cmptr.cmsg_type != SCM_RIGHTS) 9059 throw new Exception("control type != SCM_RIGHTS"); 9060 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9061 } else 9062 *recvfd = -1; /* descriptor was not passed */ 9063 } 9064 9065 return n; 9066 } 9067 /* end read_fd */ 9068 9069 9070 /* 9071 Event source stuff 9072 9073 The api is: 9074 9075 sendEvent(string url, string type, string data, int timeout = 60*10); 9076 9077 attachEventListener(string url, int fd, lastId) 9078 9079 9080 It just sends to all attached listeners, and stores it until the timeout 9081 for replaying via lastEventId. 9082 */ 9083 9084 /* 9085 Session process stuff 9086 9087 it stores it all. the cgi object has a session object that can grab it 9088 9089 session may be done in the same process if possible, there is a version 9090 switch to choose if you want to override. 9091 */ 9092 9093 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9094 alias handler = dispatchHandler; 9095 string urlPrefix; 9096 bool rejectFurther; 9097 immutable(DispatcherDetails) details; 9098 } 9099 9100 private string urlify(string name) pure { 9101 return beautify(name, '-', true); 9102 } 9103 9104 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9105 if(name == "id") 9106 return allLowerCase ? name : "ID"; 9107 9108 char[160] buffer; 9109 int bufferIndex = 0; 9110 bool shouldCap = true; 9111 bool shouldSpace; 9112 bool lastWasCap; 9113 foreach(idx, char ch; name) { 9114 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9115 9116 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9117 if(lastWasCap) { 9118 // two caps in a row, don't change. Prolly acronym. 9119 } else { 9120 if(idx) 9121 shouldSpace = true; // new word, add space 9122 } 9123 9124 lastWasCap = true; 9125 } else { 9126 lastWasCap = false; 9127 } 9128 9129 if(shouldSpace) { 9130 buffer[bufferIndex++] = space; 9131 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9132 shouldSpace = false; 9133 } 9134 if(shouldCap) { 9135 if(ch >= 'a' && ch <= 'z') 9136 ch -= 32; 9137 shouldCap = false; 9138 } 9139 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9140 ch += 32; 9141 buffer[bufferIndex++] = ch; 9142 } 9143 return buffer[0 .. bufferIndex].idup; 9144 } 9145 9146 /* 9147 string urlFor(alias func)() { 9148 return __traits(identifier, func); 9149 } 9150 */ 9151 9152 /++ 9153 UDA: The name displayed to the user in auto-generated HTML. 9154 9155 Default is `beautify(identifier)`. 9156 +/ 9157 struct DisplayName { 9158 string name; 9159 } 9160 9161 /++ 9162 UDA: The name used in the URL or web parameter. 9163 9164 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9165 +/ 9166 struct UrlName { 9167 string name; 9168 } 9169 9170 /++ 9171 UDA: default format to respond for this method 9172 +/ 9173 struct DefaultFormat { string value; } 9174 9175 class MissingArgumentException : Exception { 9176 string functionName; 9177 string argumentName; 9178 string argumentType; 9179 9180 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9181 this.functionName = functionName; 9182 this.argumentName = argumentName; 9183 this.argumentType = argumentType; 9184 9185 super("Missing Argument: " ~ this.argumentName, file, line, next); 9186 } 9187 } 9188 9189 /++ 9190 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9191 9192 History: 9193 Added December 15, 2021 (dub v10.5) 9194 +/ 9195 class ResourceNotFoundException : Exception { 9196 string resourceType; 9197 string resourceId; 9198 9199 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9200 this.resourceType = resourceType; 9201 this.resourceId = resourceId; 9202 9203 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9204 } 9205 9206 } 9207 9208 /++ 9209 This can be attached to any constructor or function called from the cgi system. 9210 9211 If it is present, the function argument can NOT be set from web params, but instead 9212 is set to the return value of the given `func`. 9213 9214 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9215 the current request. Otherwise, it must take zero arguments. 9216 9217 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9218 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9219 the cgi object. 9220 9221 const arguments are also supported. 9222 +/ 9223 struct ifCalledFromWeb(alias func) {} 9224 9225 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9226 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9227 9228 // FIXME: any array of structs should also be settable or gettable from csv as well. 9229 9230 // FIXME: think more about checkboxes and bools. 9231 9232 import std.traits; 9233 9234 Parameters!method params; 9235 alias idents = ParameterIdentifierTuple!method; 9236 alias defaults = ParameterDefaults!method; 9237 9238 const(string)[] names; 9239 const(string)[] values; 9240 9241 // first, check for missing arguments and initialize to defaults if necessary 9242 9243 static if(is(typeof(method) P == __parameters)) 9244 foreach(idx, param; P) {{ 9245 // see: mustNotBeSetFromWebParams 9246 static if(is(param : Cgi)) { 9247 static assert(!is(param == immutable)); 9248 cast() params[idx] = cgi; 9249 } else static if(is(param == Session!D, D)) { 9250 static assert(!is(param == immutable)); 9251 cast() params[idx] = cgi.getSessionObject!D(); 9252 } else { 9253 bool populated; 9254 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9255 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9256 static if(is(typeof(func(cgi)))) 9257 params[idx] = func(cgi); 9258 else 9259 params[idx] = func(); 9260 9261 populated = true; 9262 } 9263 } 9264 9265 if(!populated) { 9266 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9267 params[idx] = param.getAutomaticallyForCgi(cgi); 9268 populated = true; 9269 } 9270 } 9271 9272 if(!populated) { 9273 auto ident = idents[idx]; 9274 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9275 if(ident !in cgi.get) { 9276 static if(is(defaults[idx] == void)) { 9277 static if(is(param == bool)) 9278 params[idx] = false; 9279 else 9280 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9281 } else 9282 params[idx] = defaults[idx]; 9283 } 9284 } else { 9285 if(ident !in cgi.post) { 9286 static if(is(defaults[idx] == void)) { 9287 static if(is(param == bool)) 9288 params[idx] = false; 9289 else 9290 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9291 } else 9292 params[idx] = defaults[idx]; 9293 } 9294 } 9295 } 9296 } 9297 }} 9298 9299 // second, parse the arguments in order to build up arrays, etc. 9300 9301 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9302 static if(is(T == struct)) { 9303 if(name == paramName) { 9304 *what = T.init; 9305 return true; 9306 } else { 9307 // could be a child. gonna allow either obj.field OR obj[field] 9308 9309 string afterName; 9310 9311 if(name[paramName.length] == '[') { 9312 int count = 1; 9313 auto idx = paramName.length + 1; 9314 while(idx < name.length && count > 0) { 9315 if(name[idx] == '[') 9316 count++; 9317 else if(name[idx] == ']') { 9318 count--; 9319 if(count == 0) break; 9320 } 9321 idx++; 9322 } 9323 9324 if(idx == name.length) 9325 return false; // malformed 9326 9327 auto insideBrackets = name[paramName.length + 1 .. idx]; 9328 afterName = name[idx + 1 .. $]; 9329 9330 name = name[0 .. paramName.length]; 9331 9332 paramName = insideBrackets; 9333 9334 } else if(name[paramName.length] == '.') { 9335 paramName = name[paramName.length + 1 .. $]; 9336 name = paramName; 9337 int p = 0; 9338 foreach(ch; paramName) { 9339 if(ch == '.' || ch == '[') 9340 break; 9341 p++; 9342 } 9343 9344 afterName = paramName[p .. $]; 9345 paramName = paramName[0 .. p]; 9346 } else { 9347 return false; 9348 } 9349 9350 if(paramName.length) 9351 // set the child member 9352 switch(paramName) { 9353 foreach(idx, memberName; __traits(allMembers, T)) 9354 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9355 // data member! 9356 case memberName: 9357 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9358 } 9359 default: 9360 // ok, not a member 9361 } 9362 } 9363 9364 return false; 9365 } else static if(is(T == enum)) { 9366 *what = to!T(value); 9367 return true; 9368 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9369 *what = to!T(value); 9370 return true; 9371 } else static if(is(T == bool)) { 9372 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9373 return true; 9374 } else static if(is(T == K[], K)) { 9375 K tmp; 9376 if(name == paramName) { 9377 // direct - set and append 9378 if(setVariable(name, paramName, &tmp, value)) { 9379 (*what) ~= tmp; 9380 return true; 9381 } else { 9382 return false; 9383 } 9384 } else { 9385 // child, append to last element 9386 // FIXME: what about range violations??? 9387 auto ptr = &(*what)[(*what).length - 1]; 9388 return setVariable(name, paramName, ptr, value); 9389 9390 } 9391 } else static if(is(T == V[K], K, V)) { 9392 // assoc array, name[key] is valid 9393 if(name == paramName) { 9394 // no action necessary 9395 return true; 9396 } else if(name[paramName.length] == '[') { 9397 int count = 1; 9398 auto idx = paramName.length + 1; 9399 while(idx < name.length && count > 0) { 9400 if(name[idx] == '[') 9401 count++; 9402 else if(name[idx] == ']') { 9403 count--; 9404 if(count == 0) break; 9405 } 9406 idx++; 9407 } 9408 if(idx == name.length) 9409 return false; // malformed 9410 9411 auto insideBrackets = name[paramName.length + 1 .. idx]; 9412 auto afterName = name[idx + 1 .. $]; 9413 9414 auto k = to!K(insideBrackets); 9415 V v; 9416 if(auto ptr = k in *what) 9417 v = *ptr; 9418 9419 name = name[0 .. paramName.length]; 9420 //writeln(name, afterName, " ", paramName); 9421 9422 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9423 if(ret) { 9424 (*what)[k] = v; 9425 return true; 9426 } 9427 } 9428 9429 return false; 9430 } else { 9431 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9432 } 9433 9434 //return false; 9435 } 9436 9437 void setArgument(string name, string value) { 9438 int p; 9439 foreach(ch; name) { 9440 if(ch == '.' || ch == '[') 9441 break; 9442 p++; 9443 } 9444 9445 auto paramName = name[0 .. p]; 9446 9447 sw: switch(paramName) { 9448 static if(is(typeof(method) P == __parameters)) 9449 foreach(idx, param; P) { 9450 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9451 // cannot be set from the outside 9452 } else { 9453 case idents[idx]: 9454 static if(is(param == Cgi.UploadedFile)) { 9455 params[idx] = cgi.files[name]; 9456 } else static if(is(param : const Cgi.UploadedFile[])) { 9457 (cast() params[idx]) = cgi.filesArray[name]; 9458 } else { 9459 setVariable(name, paramName, ¶ms[idx], value); 9460 } 9461 break sw; 9462 } 9463 } 9464 default: 9465 // ignore; not relevant argument 9466 } 9467 } 9468 9469 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9470 names = cgi.allGetNamesInOrder; 9471 values = cgi.allGetValuesInOrder; 9472 } else { 9473 names = cgi.allPostNamesInOrder; 9474 values = cgi.allPostValuesInOrder; 9475 } 9476 9477 foreach(idx, name; names) { 9478 setArgument(name, values[idx]); 9479 } 9480 9481 static if(is(ReturnType!method == void)) { 9482 typeof(null) ret; 9483 dg(params); 9484 } else { 9485 auto ret = dg(params); 9486 } 9487 9488 // FIXME: format return values 9489 // options are: json, html, csv. 9490 // also may need to wrap in envelope format: none, html, or json. 9491 return ret; 9492 } 9493 9494 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9495 static if(is(T : const(Cgi))) { 9496 return true; 9497 } else static if(is(T : const(Session!D), D)) { 9498 return true; 9499 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9500 return true; 9501 } else { 9502 foreach(uda; attrs) 9503 static if(is(uda == ifCalledFromWeb!func, alias func)) 9504 return true; 9505 return false; 9506 } 9507 } 9508 9509 private bool hasIfCalledFromWeb(attrs...)() { 9510 foreach(uda; attrs) 9511 static if(is(uda == ifCalledFromWeb!func, alias func)) 9512 return true; 9513 return false; 9514 } 9515 9516 /++ 9517 Implies POST path for the thing itself, then GET will get the automatic form. 9518 9519 The given customizer, if present, will be called as a filter on the Form object. 9520 9521 History: 9522 Added December 27, 2020 9523 +/ 9524 template AutomaticForm(alias customizer) { } 9525 9526 /++ 9527 This is meant to be returned by a function that takes a form POST submission. You 9528 want to set the url of the new resource it created, which is set as the http 9529 Location header for a "201 Created" result, and you can also set a separate 9530 destination for browser users, which it sets via a "Refresh" header. 9531 9532 The `resourceRepresentation` should generally be the thing you just created, and 9533 it will be the body of the http response when formatted through the presenter. 9534 The exact thing is up to you - it could just return an id, or the whole object, or 9535 perhaps a partial object. 9536 9537 Examples: 9538 --- 9539 class Test : WebObject { 9540 @(Cgi.RequestMethod.POST) 9541 CreatedResource!int makeThing(string value) { 9542 return CreatedResource!int(value.to!int, "/resources/id"); 9543 } 9544 } 9545 --- 9546 9547 History: 9548 Added December 18, 2021 9549 +/ 9550 struct CreatedResource(T) { 9551 static if(!is(T == void)) 9552 T resourceRepresentation; 9553 string resourceUrl; 9554 string refreshUrl; 9555 } 9556 9557 /+ 9558 /++ 9559 This can be attached as a UDA to a handler to add a http Refresh header on a 9560 successful run. (It will not be attached if the function throws an exception.) 9561 This will refresh the browser the given number of seconds after the page loads, 9562 to the url returned by `urlFunc`, which can be either a static function or a 9563 member method of the current handler object. 9564 9565 You might use this for a POST handler that is normally used from ajax, but you 9566 want it to degrade gracefully to a temporarily flashed message before reloading 9567 the main page. 9568 9569 History: 9570 Added December 18, 2021 9571 +/ 9572 struct Refresh(alias urlFunc) { 9573 int waitInSeconds; 9574 9575 string url() { 9576 static if(__traits(isStaticFunction, urlFunc)) 9577 return urlFunc(); 9578 else static if(is(urlFunc : string)) 9579 return urlFunc; 9580 } 9581 } 9582 +/ 9583 9584 /+ 9585 /++ 9586 Sets a filter to be run before 9587 9588 A before function can do validations of params and log and stop the function from running. 9589 +/ 9590 template Before(alias b) {} 9591 template After(alias b) {} 9592 +/ 9593 9594 /+ 9595 Argument conversions: for the most part, it is to!Thing(string). 9596 9597 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9598 they are passed 9599 9600 arr=foo&arr=bar <-- notice the same name. 9601 9602 Structs are first declared with an empty thing, then have their members set individually, 9603 with dot notation. The members are not required, just the initial declaration. 9604 9605 struct Foo { 9606 int a; 9607 string b; 9608 } 9609 void test(Foo foo){} 9610 9611 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9612 9613 Arrays of structs use this declaration. 9614 9615 void test(Foo[] foo) {} 9616 9617 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9618 9619 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9620 declaration is ignored. 9621 9622 Mind that order matters! The declaration MUST come first in the string. 9623 9624 Arrays of struct members follow this rule recursively. 9625 9626 struct Foo { 9627 int[] a; 9628 } 9629 9630 foo&foo.a=1&foo.a=2&foo&foo.a=1 9631 9632 9633 Associative arrays are formatted with brackets, after a declaration, like structs: 9634 9635 foo&foo[key]=value&foo[other_key]=value 9636 9637 9638 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9639 do not support the strict ordering requirements to work with these struct protocols. 9640 9641 FIXME: also perhaps accept application/json to better work with outside trash. 9642 9643 9644 Return values are also auto-formatted according to user-requested type: 9645 for json, it loops over and converts. 9646 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9647 +/ 9648 9649 /++ 9650 A web presenter is responsible for rendering things to HTML to be usable 9651 in a web browser. 9652 9653 They are passed as template arguments to the base classes of [WebObject] 9654 9655 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9656 and override it. Use forwarding and specialization to customize it. 9657 9658 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9659 class templates and your overridden templates work with each other. 9660 9661 --- 9662 class MyPresenter : WebPresenter!(MyPresenter) { 9663 @Override 9664 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9665 // present the CustomType 9666 } 9667 @Override 9668 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9669 // handle everything else via the super class, which will call 9670 // back to your class when appropriate 9671 super.presentSuccessfulReturnAsHtml(cgi, ret); 9672 } 9673 } 9674 --- 9675 9676 The meta argument in there can be overridden by your own facility. 9677 9678 +/ 9679 class WebPresenter(CRTP) { 9680 9681 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9682 /// If you override a plain method, use `override`. If a template, use `@Override`. 9683 enum Override; 9684 9685 string script() { 9686 return ` 9687 `; 9688 } 9689 9690 string style() { 9691 return ` 9692 :root { 9693 --mild-border: #ccc; 9694 --middle-border: #999; 9695 --accent-color: #f2f2f2; 9696 --sidebar-color: #fefefe; 9697 } 9698 ` ~ genericFormStyling() ~ genericSiteStyling(); 9699 } 9700 9701 string genericFormStyling() { 9702 return 9703 q"css 9704 table.automatic-data-display { 9705 border-collapse: collapse; 9706 border: solid 1px var(--mild-border); 9707 } 9708 9709 table.automatic-data-display td { 9710 vertical-align: top; 9711 border: solid 1px var(--mild-border); 9712 padding: 2px 4px; 9713 } 9714 9715 table.automatic-data-display th { 9716 border: solid 1px var(--mild-border); 9717 border-bottom: solid 1px var(--middle-border); 9718 padding: 2px 4px; 9719 } 9720 9721 ol.automatic-data-display { 9722 margin: 0px; 9723 list-style-position: inside; 9724 padding: 0px; 9725 } 9726 9727 dl.automatic-data-display { 9728 9729 } 9730 9731 .automatic-form { 9732 max-width: 600px; 9733 } 9734 9735 .form-field { 9736 margin: 0.5em; 9737 padding-left: 0.5em; 9738 } 9739 9740 .label-text { 9741 display: block; 9742 font-weight: bold; 9743 margin-left: -0.5em; 9744 } 9745 9746 .submit-button-holder { 9747 padding-left: 2em; 9748 } 9749 9750 .add-array-button { 9751 9752 } 9753 css"; 9754 } 9755 9756 string genericSiteStyling() { 9757 return 9758 q"css 9759 * { box-sizing: border-box; } 9760 html, body { margin: 0px; } 9761 body { 9762 font-family: sans-serif; 9763 } 9764 header { 9765 background: var(--accent-color); 9766 height: 64px; 9767 } 9768 footer { 9769 background: var(--accent-color); 9770 height: 64px; 9771 } 9772 #site-container { 9773 display: flex; 9774 } 9775 main { 9776 flex: 1 1 auto; 9777 order: 2; 9778 min-height: calc(100vh - 64px - 64px); 9779 padding: 4px; 9780 padding-left: 1em; 9781 } 9782 #sidebar { 9783 flex: 0 0 16em; 9784 order: 1; 9785 background: var(--sidebar-color); 9786 } 9787 css"; 9788 } 9789 9790 import arsd.dom; 9791 Element htmlContainer() { 9792 auto document = new Document(q"html 9793 <!DOCTYPE html> 9794 <html class="no-script"> 9795 <head> 9796 <script>document.documentElement.classList.remove("no-script");</script> 9797 <style>.no-script requires-script { display: none; }</style> 9798 <title>D Application</title> 9799 <meta name="viewport" content="initial-scale=1, width=device-width" /> 9800 <link rel="stylesheet" href="style.css" /> 9801 </head> 9802 <body> 9803 <header></header> 9804 <div id="site-container"> 9805 <main></main> 9806 <div id="sidebar"></div> 9807 </div> 9808 <footer></footer> 9809 <script src="script.js"></script> 9810 </body> 9811 </html> 9812 html", true, true); 9813 9814 return document.requireSelector("main"); 9815 } 9816 9817 /// Renders a response as an HTTP error with associated html body 9818 void renderBasicError(Cgi cgi, int httpErrorCode) { 9819 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9820 auto c = htmlContainer(); 9821 c.innerText = getHttpCodeText(httpErrorCode); 9822 cgi.setResponseContentType("text/html; charset=utf-8"); 9823 cgi.write(c.parentDocument.toString(), true); 9824 } 9825 9826 template methodMeta(alias method) { 9827 enum methodMeta = null; 9828 } 9829 9830 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9831 switch(format) { 9832 case "html": 9833 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9834 break; 9835 case "json": 9836 import arsd.jsvar; 9837 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9838 var json; 9839 foreach(index, type; Types) { 9840 if(ret.contains == index) 9841 json = ret.payload[index]; 9842 } 9843 } else { 9844 var json = ret; 9845 } 9846 var envelope = json; // var.emptyObject; 9847 /* 9848 envelope.success = true; 9849 envelope.result = json; 9850 envelope.error = null; 9851 */ 9852 cgi.setResponseContentType("application/json"); 9853 cgi.write(envelope.toJson(), true); 9854 break; 9855 default: 9856 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9857 } 9858 } 9859 9860 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9861 /// in the default presenter - allowing the function to have full low-level control over the 9862 /// response. 9863 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9864 // nothing intentionally! 9865 } 9866 9867 /// Redirections are forwarded to [Cgi.setResponseLocation] 9868 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9869 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9870 } 9871 9872 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9873 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9874 cgi.setResponseStatus(getHttpCodeText(201)); 9875 if(ret.resourceUrl.length) 9876 cgi.header("Location: " ~ ret.resourceUrl); 9877 if(ret.refreshUrl.length) 9878 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9879 static if(!is(R == void)) 9880 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9881 } 9882 9883 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9884 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9885 bool outputted = false; 9886 foreach(index, type; Types) { 9887 if(ret.contains == index) { 9888 assert(!outputted); 9889 outputted = true; 9890 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9891 } 9892 } 9893 if(!outputted) 9894 assert(0); 9895 } 9896 9897 /++ 9898 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9899 +/ 9900 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9901 cgi.setCache(true); // not necessarily true but meh 9902 if(auto fn = ret.filename()) { 9903 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9904 } 9905 cgi.setResponseContentType(ret.contentType); 9906 cgi.write(ret.getData(), true); 9907 } 9908 9909 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9910 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9911 auto container = this.htmlContainer(); 9912 container.appendChild(formatReturnValueAsHtml(ret)); 9913 cgi.write(container.parentDocument.toString(), true); 9914 } 9915 9916 /++ 9917 9918 History: 9919 Added January 23, 2023 (dub v11.0) 9920 +/ 9921 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9922 switch(format) { 9923 case "html": 9924 presentExceptionAsHtml(cgi, t, meta); 9925 break; 9926 case "json": 9927 presentExceptionAsJsonImpl(cgi, t); 9928 break; 9929 default: 9930 } 9931 } 9932 9933 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 9934 cgi.setResponseStatus("500 Internal Server Error"); 9935 cgi.setResponseContentType("application/json"); 9936 import arsd.jsvar; 9937 var v = var.emptyObject; 9938 v.type = typeid(t).toString; 9939 v.msg = t.msg; 9940 v.fullString = t.toString(); 9941 cgi.write(v.toJson(), true); 9942 } 9943 9944 9945 /++ 9946 If you override this, you will need to cast the exception type `t` dynamically, 9947 but can then use the template arguments here to refer back to the function. 9948 9949 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9950 method on the live object. You could, in theory, change arguments and retry, but I 9951 provide that information mostly with the expectation that you will use them to make 9952 useful forms or richer error messages for the user. 9953 9954 History: 9955 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 9956 I removed this in favor of a `Meta` param. 9957 9958 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 9959 9960 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 9961 9962 If you used the func for something, move that something into your `methodMeta` template. 9963 9964 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 9965 enabling an easier implementation of [presentExceptionalReturn]. 9966 +/ 9967 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 9968 Form af; 9969 /+ 9970 foreach(attr; __traits(getAttributes, func)) { 9971 static if(__traits(isSame, attr, AutomaticForm)) { 9972 af = createAutomaticFormForFunction!(func)(dg); 9973 } 9974 } 9975 +/ 9976 presentExceptionAsHtmlImpl(cgi, t, af); 9977 } 9978 9979 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 9980 if(auto e = cast(ResourceNotFoundException) t) { 9981 auto container = this.htmlContainer(); 9982 9983 container.addChild("p", e.msg); 9984 9985 if(!cgi.outputtedResponseData) 9986 cgi.setResponseStatus("404 Not Found"); 9987 cgi.write(container.parentDocument.toString(), true); 9988 } else if(auto mae = cast(MissingArgumentException) t) { 9989 if(automaticForm is null) 9990 goto generic; 9991 auto container = this.htmlContainer(); 9992 if(cgi.requestMethod == Cgi.RequestMethod.POST) 9993 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 9994 container.appendChild(automaticForm); 9995 9996 cgi.write(container.parentDocument.toString(), true); 9997 } else { 9998 generic: 9999 auto container = this.htmlContainer(); 10000 10001 // import std.stdio; writeln(t.toString()); 10002 10003 container.appendChild(exceptionToElement(t)); 10004 10005 container.addChild("h4", "GET"); 10006 foreach(k, v; cgi.get) { 10007 auto deets = container.addChild("details"); 10008 deets.addChild("summary", k); 10009 deets.addChild("div", v); 10010 } 10011 10012 container.addChild("h4", "POST"); 10013 foreach(k, v; cgi.post) { 10014 auto deets = container.addChild("details"); 10015 deets.addChild("summary", k); 10016 deets.addChild("div", v); 10017 } 10018 10019 10020 if(!cgi.outputtedResponseData) 10021 cgi.setResponseStatus("500 Internal Server Error"); 10022 cgi.write(container.parentDocument.toString(), true); 10023 } 10024 } 10025 10026 Element exceptionToElement(Throwable t) { 10027 auto div = Element.make("div"); 10028 div.addClass("exception-display"); 10029 10030 div.addChild("p", t.msg); 10031 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10032 10033 auto pre = div.addChild("pre"); 10034 string s; 10035 s = t.toString(); 10036 Element currentBox; 10037 bool on = false; 10038 foreach(line; s.splitLines) { 10039 if(!on && line.startsWith("-----")) 10040 on = true; 10041 if(!on) continue; 10042 if(line.indexOf("arsd/") != -1) { 10043 if(currentBox is null) { 10044 currentBox = pre.addChild("details"); 10045 currentBox.addChild("summary", "Framework code"); 10046 } 10047 currentBox.addChild("span", line ~ "\n"); 10048 } else { 10049 pre.addChild("span", line ~ "\n"); 10050 currentBox = null; 10051 } 10052 } 10053 10054 return div; 10055 } 10056 10057 /++ 10058 Returns an element for a particular type 10059 +/ 10060 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10061 import std.traits; 10062 10063 auto div = Element.make("div"); 10064 div.addClass("form-field"); 10065 10066 static if(is(T : const Cgi.UploadedFile)) { 10067 Element lbl; 10068 if(displayName !is null) { 10069 lbl = div.addChild("label"); 10070 lbl.addChild("span", displayName, "label-text"); 10071 lbl.appendText(" "); 10072 } else { 10073 lbl = div; 10074 } 10075 auto i = lbl.addChild("input", name); 10076 i.attrs.name = name; 10077 i.attrs.type = "file"; 10078 i.attrs.multiple = "multiple"; 10079 } else static if(is(T == Cgi.UploadedFile)) { 10080 Element lbl; 10081 if(displayName !is null) { 10082 lbl = div.addChild("label"); 10083 lbl.addChild("span", displayName, "label-text"); 10084 lbl.appendText(" "); 10085 } else { 10086 lbl = div; 10087 } 10088 auto i = lbl.addChild("input", name); 10089 i.attrs.name = name; 10090 i.attrs.type = "file"; 10091 } else static if(is(T == enum)) { 10092 Element lbl; 10093 if(displayName !is null) { 10094 lbl = div.addChild("label"); 10095 lbl.addChild("span", displayName, "label-text"); 10096 lbl.appendText(" "); 10097 } else { 10098 lbl = div; 10099 } 10100 auto i = lbl.addChild("select", name); 10101 i.attrs.name = name; 10102 10103 foreach(memberName; __traits(allMembers, T)) 10104 i.addChild("option", memberName); 10105 10106 } else static if(is(T == struct)) { 10107 if(displayName !is null) 10108 div.addChild("span", displayName, "label-text"); 10109 auto fieldset = div.addChild("fieldset"); 10110 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10111 fieldset.addChild("input", name); 10112 foreach(idx, memberName; __traits(allMembers, T)) 10113 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10114 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10115 } 10116 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10117 Element lbl; 10118 if(displayName !is null) { 10119 lbl = div.addChild("label"); 10120 lbl.addChild("span", displayName, "label-text"); 10121 lbl.appendText(" "); 10122 } else { 10123 lbl = div; 10124 } 10125 Element i; 10126 if(udaSuggestion) { 10127 i = udaSuggestion(); 10128 lbl.appendChild(i); 10129 } else { 10130 i = lbl.addChild("input", name); 10131 } 10132 i.attrs.name = name; 10133 static if(isSomeString!T) 10134 i.attrs.type = "text"; 10135 else 10136 i.attrs.type = "number"; 10137 if(i.tagName == "textarea") 10138 i.textContent = to!string(T.init); 10139 else 10140 i.attrs.value = to!string(T.init); 10141 } else static if(is(T == bool)) { 10142 Element lbl; 10143 if(displayName !is null) { 10144 lbl = div.addChild("label"); 10145 lbl.addChild("span", displayName, "label-text"); 10146 lbl.appendText(" "); 10147 } else { 10148 lbl = div; 10149 } 10150 auto i = lbl.addChild("input", name); 10151 i.attrs.type = "checkbox"; 10152 i.attrs.value = "true"; 10153 i.attrs.name = name; 10154 } else static if(is(T == K[], K)) { 10155 auto templ = div.addChild("template"); 10156 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10157 if(displayName !is null) 10158 div.addChild("span", displayName, "label-text"); 10159 auto btn = div.addChild("button"); 10160 btn.addClass("add-array-button"); 10161 btn.attrs.type = "button"; 10162 btn.innerText = "Add"; 10163 btn.attrs.onclick = q{ 10164 var a = document.importNode(this.parentNode.firstChild.content, true); 10165 this.parentNode.insertBefore(a, this); 10166 }; 10167 } else static if(is(T == V[K], K, V)) { 10168 div.innerText = "assoc array not implemented for automatic form at this time"; 10169 } else { 10170 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10171 } 10172 10173 10174 return div; 10175 } 10176 10177 /// creates a form for gathering the function's arguments 10178 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10179 10180 auto form = cast(Form) Element.make("form"); 10181 10182 form.method = "POST"; // FIXME 10183 10184 form.addClass("automatic-form"); 10185 10186 string formDisplayName = beautify(__traits(identifier, method)); 10187 foreach(attr; __traits(getAttributes, method)) 10188 static if(is(typeof(attr) == DisplayName)) 10189 formDisplayName = attr.name; 10190 form.addChild("h3", formDisplayName); 10191 10192 import std.traits; 10193 10194 //Parameters!method params; 10195 //alias idents = ParameterIdentifierTuple!method; 10196 //alias defaults = ParameterDefaults!method; 10197 10198 static if(is(typeof(method) P == __parameters)) 10199 foreach(idx, _; P) {{ 10200 10201 alias param = P[idx .. idx + 1]; 10202 10203 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10204 string displayName = beautify(__traits(identifier, param)); 10205 Element function() element; 10206 foreach(attr; __traits(getAttributes, param)) { 10207 static if(is(typeof(attr) == DisplayName)) 10208 displayName = attr.name; 10209 else static if(is(typeof(attr) : typeof(element))) { 10210 element = attr; 10211 } 10212 } 10213 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10214 if(i.querySelector("input[type=file]") !is null) 10215 form.setAttribute("enctype", "multipart/form-data"); 10216 } 10217 }} 10218 10219 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10220 10221 return form; 10222 } 10223 10224 /// creates a form for gathering object members (for the REST object thing right now) 10225 Form createAutomaticFormForObject(T)(T obj) { 10226 auto form = cast(Form) Element.make("form"); 10227 10228 form.addClass("automatic-form"); 10229 10230 form.addChild("h3", beautify(__traits(identifier, T))); 10231 10232 import std.traits; 10233 10234 //Parameters!method params; 10235 //alias idents = ParameterIdentifierTuple!method; 10236 //alias defaults = ParameterDefaults!method; 10237 10238 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10239 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10240 string displayName = beautify(memberName); 10241 Element function() element; 10242 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10243 static if(is(typeof(attr) == DisplayName)) 10244 displayName = attr.name; 10245 else static if(is(typeof(attr) : typeof(element))) 10246 element = attr; 10247 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10248 10249 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10250 }}} 10251 10252 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10253 10254 return form; 10255 } 10256 10257 /// 10258 Element formatReturnValueAsHtml(T)(T t) { 10259 import std.traits; 10260 10261 static if(is(T == typeof(null))) { 10262 return Element.make("span"); 10263 } else static if(is(T : Element)) { 10264 return t; 10265 } else static if(is(T == MultipleResponses!Types, Types...)) { 10266 foreach(index, type; Types) { 10267 if(t.contains == index) 10268 return formatReturnValueAsHtml(t.payload[index]); 10269 } 10270 assert(0); 10271 } else static if(is(T == Paginated!E, E)) { 10272 auto e = Element.make("div").addClass("paginated-result"); 10273 e.appendChild(formatReturnValueAsHtml(t.items)); 10274 if(t.nextPageUrl.length) 10275 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10276 return e; 10277 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10278 return Element.make("span", to!string(t), "automatic-data-display"); 10279 } else static if(is(T == V[K], K, V)) { 10280 auto dl = Element.make("dl"); 10281 dl.addClass("automatic-data-display associative-array"); 10282 foreach(k, v; t) { 10283 dl.addChild("dt", to!string(k)); 10284 dl.addChild("dd", formatReturnValueAsHtml(v)); 10285 } 10286 return dl; 10287 } else static if(is(T == struct)) { 10288 auto dl = Element.make("dl"); 10289 dl.addClass("automatic-data-display struct"); 10290 10291 foreach(idx, memberName; __traits(allMembers, T)) 10292 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10293 dl.addChild("dt", beautify(memberName)); 10294 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10295 } 10296 10297 return dl; 10298 } else static if(is(T == bool)) { 10299 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10300 } else static if(is(T == E[], E)) { 10301 static if(is(E : RestObject!Proxy, Proxy)) { 10302 // treat RestObject similar to struct 10303 auto table = cast(Table) Element.make("table"); 10304 table.addClass("automatic-data-display"); 10305 string[] names; 10306 foreach(idx, memberName; __traits(derivedMembers, E)) 10307 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10308 names ~= beautify(memberName); 10309 } 10310 table.appendHeaderRow(names); 10311 10312 foreach(l; t) { 10313 auto tr = table.appendRow(); 10314 foreach(idx, memberName; __traits(derivedMembers, E)) 10315 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10316 static if(memberName == "id") { 10317 string val = to!string(__traits(getMember, l, memberName)); 10318 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10319 } else { 10320 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10321 } 10322 } 10323 } 10324 10325 return table; 10326 } else static if(is(E == struct)) { 10327 // an array of structs is kinda special in that I like 10328 // having those formatted as tables. 10329 auto table = cast(Table) Element.make("table"); 10330 table.addClass("automatic-data-display"); 10331 string[] names; 10332 foreach(idx, memberName; __traits(allMembers, E)) 10333 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10334 names ~= beautify(memberName); 10335 } 10336 table.appendHeaderRow(names); 10337 10338 foreach(l; t) { 10339 auto tr = table.appendRow(); 10340 foreach(idx, memberName; __traits(allMembers, E)) 10341 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10342 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10343 } 10344 } 10345 10346 return table; 10347 } else { 10348 // otherwise, I will just make a list. 10349 auto ol = Element.make("ol"); 10350 ol.addClass("automatic-data-display"); 10351 foreach(e; t) 10352 ol.addChild("li", formatReturnValueAsHtml(e)); 10353 return ol; 10354 } 10355 } else static if(is(T : Object)) { 10356 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10357 return Element.make("div", t.toHtml()); 10358 else 10359 return Element.make("div", t.toString()); 10360 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10361 10362 assert(0); 10363 } 10364 10365 } 10366 10367 /++ 10368 The base class for the [dispatcher] function and object support. 10369 +/ 10370 class WebObject { 10371 //protected Cgi cgi; 10372 10373 protected void initialize(Cgi cgi) { 10374 //this.cgi = cgi; 10375 } 10376 } 10377 10378 /++ 10379 Can return one of the given types, decided at runtime. The syntax 10380 is to declare all the possible types in the return value, then you 10381 can `return typeof(return)(...value...)` to construct it. 10382 10383 It has an auto-generated constructor for each value it can hold. 10384 10385 --- 10386 MultipleResponses!(Redirection, string) getData(int how) { 10387 if(how & 1) 10388 return typeof(return)(Redirection("http://dpldocs.info/")); 10389 else 10390 return typeof(return)("hi there!"); 10391 } 10392 --- 10393 10394 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10395 +/ 10396 struct MultipleResponses(T...) { 10397 private size_t contains; 10398 private union { 10399 private T payload; 10400 } 10401 10402 static foreach(index, type; T) 10403 public this(type t) { 10404 contains = index; 10405 payload[index] = t; 10406 } 10407 10408 /++ 10409 This is primarily for testing. It is your way of getting to the response. 10410 10411 Let's say you wanted to test that one holding a Redirection and a string actually 10412 holds a string, by name of "test": 10413 10414 --- 10415 auto valueToTest = your_test_function(); 10416 10417 valueToTest.visit( 10418 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10419 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10420 ); 10421 --- 10422 10423 History: 10424 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10425 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10426 +/ 10427 void visit(Handlers...)(Handlers handlers) { 10428 template findHandler(type, int count, HandlersToCheck...) { 10429 static if(HandlersToCheck.length == 0) 10430 enum findHandler = -1; 10431 else { 10432 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10433 enum findHandler = count; 10434 else 10435 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10436 } 10437 } 10438 foreach(index, type; T) { 10439 enum handlerIndex = findHandler!(type, 0, Handlers); 10440 static if(handlerIndex == -1) 10441 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10442 else { 10443 if(index == this.contains) 10444 handlers[handlerIndex](this.payload[index]); 10445 } 10446 } 10447 } 10448 10449 /+ 10450 auto toArsdJsvar()() { 10451 import arsd.jsvar; 10452 return var(null); 10453 } 10454 +/ 10455 } 10456 10457 // FIXME: implement this somewhere maybe 10458 struct RawResponse { 10459 int code; 10460 string[] headers; 10461 const(ubyte)[] responseBody; 10462 } 10463 10464 /++ 10465 You can return this from [WebObject] subclasses for redirections. 10466 10467 (though note the static types means that class must ALWAYS redirect if 10468 you return this directly. You might want to return [MultipleResponses] if it 10469 can be conditional) 10470 +/ 10471 struct Redirection { 10472 string to; /// The URL to redirect to. 10473 int code = 303; /// The HTTP code to return. 10474 } 10475 10476 /++ 10477 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10478 10479 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10480 the presenter in the dispatcher. 10481 10482 FIXME: explain this better 10483 10484 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10485 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10486 the runtime result of that is undefined. 10487 10488 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10489 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10490 10491 $(WARNING 10492 --- 10493 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10494 // even if you put different URL udas on it, the current code ignores them. 10495 void foo(int a) {} 10496 void foo(string a) {} 10497 --- 10498 ) 10499 10500 See_Also: [serveRestObject], [serveStaticFile] 10501 +/ 10502 auto serveApi(T)(string urlPrefix) { 10503 assert(urlPrefix[$ - 1] == '/'); 10504 return serveApiInternal!T(urlPrefix); 10505 } 10506 10507 private string nextPieceFromSlash(ref string remainingUrl) { 10508 if(remainingUrl.length == 0) 10509 return remainingUrl; 10510 int slash = 0; 10511 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10512 slash++; 10513 10514 // I am specifically passing `null` to differentiate it vs empty string 10515 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10516 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10517 // so if it is the last item, the dot can be used to load an alternative view 10518 // otherwise tho the dot is considered part of the identifier 10519 // FIXME 10520 10521 // again notice "" vs null here! 10522 if(slash == remainingUrl.length) 10523 remainingUrl = null; 10524 else 10525 remainingUrl = remainingUrl[slash + 1 .. $]; 10526 10527 return ident; 10528 } 10529 10530 /++ 10531 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10532 +/ 10533 enum AddTrailingSlash; 10534 /// ditto 10535 enum RemoveTrailingSlash; 10536 10537 private auto serveApiInternal(T)(string urlPrefix) { 10538 10539 import arsd.dom; 10540 import arsd.jsvar; 10541 10542 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10543 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10544 10545 try { 10546 // see duplicated code below by searching subresource_ctor 10547 // also see mustNotBeSetFromWebParams 10548 10549 static if(is(typeof(T.__ctor) P == __parameters)) { 10550 P params; 10551 10552 foreach(pidx, param; P) { 10553 static if(is(param : Cgi)) { 10554 static assert(!is(param == immutable)); 10555 cast() params[pidx] = cgi; 10556 } else static if(is(param == Session!D, D)) { 10557 static assert(!is(param == immutable)); 10558 cast() params[pidx] = cgi.getSessionObject!D(); 10559 10560 } else { 10561 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10562 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10563 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10564 static if(is(typeof(func(cgi)))) 10565 params[pidx] = func(cgi); 10566 else 10567 params[pidx] = func(); 10568 } 10569 } 10570 } else { 10571 10572 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10573 params[pidx] = param.getAutomaticallyForCgi(cgi); 10574 } else static if(is(param == string)) { 10575 auto ident = nextPieceFromSlash(remainingUrl); 10576 params[pidx] = ident; 10577 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10578 } 10579 } 10580 } 10581 10582 auto obj = new T(params); 10583 } else { 10584 auto obj = new T(); 10585 } 10586 10587 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10588 } catch(Throwable t) { 10589 switch(cgi.request("format", "html")) { 10590 case "html": 10591 static void dummy() {} 10592 presenter.presentExceptionAsHtml(cgi, t, null); 10593 return true; 10594 case "json": 10595 var envelope = var.emptyObject; 10596 envelope.success = false; 10597 envelope.result = null; 10598 envelope.error = t.toString(); 10599 cgi.setResponseContentType("application/json"); 10600 cgi.write(envelope.toJson(), true); 10601 return true; 10602 default: 10603 throw t; 10604 // return true; 10605 } 10606 // return true; 10607 } 10608 10609 assert(0); 10610 } 10611 10612 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10613 10614 obj.initialize(cgi); 10615 10616 /+ 10617 Overload rules: 10618 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10619 statically. 10620 10621 Moreover, some args vs no args can be overloaded dynamically. 10622 +/ 10623 10624 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10625 /+ 10626 auto orig = remainingUrl; 10627 assert(0, 10628 (orig is null ? "__null" : orig) 10629 ~ " .. " ~ 10630 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10631 +/ 10632 10633 if(methodNameFromUrl is null) 10634 methodNameFromUrl = "__null"; 10635 10636 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10637 10638 if(remainingUrl.length) 10639 hack ~= "/"; 10640 10641 switch(hack) { 10642 foreach(methodName; __traits(derivedMembers, T)) 10643 static if(methodName != "__ctor") 10644 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10645 static if(is(typeof(overload) P == __parameters)) 10646 static if(is(typeof(overload) R == return)) 10647 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10648 { 10649 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10650 case urlNameForMethod: 10651 10652 static if(is(R : WebObject)) { 10653 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10654 10655 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10656 10657 // subresource_ctor 10658 // also see mustNotBeSetFromWebParams 10659 10660 P params; 10661 10662 string ident; 10663 10664 foreach(pidx, param; P) { 10665 static if(is(param : Cgi)) { 10666 static assert(!is(param == immutable)); 10667 cast() params[pidx] = cgi; 10668 } else static if(is(param == typeof(presenter))) { 10669 cast() param[pidx] = presenter; 10670 } else static if(is(param == Session!D, D)) { 10671 static assert(!is(param == immutable)); 10672 cast() params[pidx] = cgi.getSessionObject!D(); 10673 } else { 10674 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10675 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10676 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10677 static if(is(typeof(func(cgi)))) 10678 params[pidx] = func(cgi); 10679 else 10680 params[pidx] = func(); 10681 } 10682 } 10683 } else { 10684 10685 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10686 params[pidx] = param.getAutomaticallyForCgi(cgi); 10687 } else static if(is(param == string)) { 10688 ident = nextPieceFromSlash(remainingUrl); 10689 if(ident is null) { 10690 // trailing slash mandated on subresources 10691 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10692 return true; 10693 } else { 10694 params[pidx] = ident; 10695 } 10696 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10697 } 10698 } 10699 } 10700 10701 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10702 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10703 } else { 10704 // 404 it if any url left - not a subresource means we don't get to play with that! 10705 if(remainingUrl.length) 10706 return false; 10707 10708 bool automaticForm; 10709 10710 foreach(attr; __traits(getAttributes, overload)) 10711 static if(is(attr == AddTrailingSlash)) { 10712 if(remainingUrl is null) { 10713 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10714 return true; 10715 } 10716 } else static if(is(attr == RemoveTrailingSlash)) { 10717 if(remainingUrl !is null) { 10718 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10719 return true; 10720 } 10721 10722 } else static if(__traits(isSame, AutomaticForm, attr)) { 10723 automaticForm = true; 10724 } 10725 10726 /+ 10727 int zeroArgOverload = -1; 10728 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10729 bool calledWithZeroArgs = true; 10730 foreach(k, v; cgi.get) 10731 if(k != "format") { 10732 calledWithZeroArgs = false; 10733 break; 10734 } 10735 foreach(k, v; cgi.post) 10736 if(k != "format") { 10737 calledWithZeroArgs = false; 10738 break; 10739 } 10740 10741 // first, we need to go through and see if there is an empty one, since that 10742 // changes inside. But otherwise, all the stuff I care about can be done via 10743 // simple looping (other improper overloads might be flagged for runtime semantic check) 10744 // 10745 // an argument of type Cgi is ignored for these purposes 10746 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10747 static if(is(typeof(overload) P == __parameters)) 10748 static if(P.length == 0) 10749 zeroArgOverload = cast(int) idx; 10750 else static if(P.length == 1 && is(P[0] : Cgi)) 10751 zeroArgOverload = cast(int) idx; 10752 }} 10753 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10754 bool overloadHasBeenCalled = false; 10755 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10756 bool callFunction = true; 10757 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10758 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10759 callFunction = false; 10760 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10761 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10762 callFunction = false; 10763 10764 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10765 10766 bool hadAnyMethodRestrictions = false; 10767 bool foundAcceptableMethod = false; 10768 foreach(attr; __traits(getAttributes, overload)) { 10769 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10770 hadAnyMethodRestrictions = true; 10771 if(attr == cgi.requestMethod) 10772 foundAcceptableMethod = true; 10773 } 10774 } 10775 10776 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10777 callFunction = false; 10778 10779 /+ 10780 The overloads we really want to allow are the sane ones 10781 from the web perspective. Which is likely on HTTP verbs, 10782 for the most part, but might also be potentially based on 10783 some args vs zero args, or on argument names. Can't really 10784 do argument types very reliable through the web though; those 10785 should probably be different URLs. 10786 10787 Even names I feel is better done inside the function, so I'm not 10788 going to support that here. But the HTTP verbs and zero vs some 10789 args makes sense - it lets you define custom forms pretty easily. 10790 10791 Moreover, I'm of the opinion that empty overload really only makes 10792 sense on GET for this case. On a POST, it is just a missing argument 10793 exception and that should be handled by the presenter. But meh, I'll 10794 let the user define that, D only allows one empty arg thing anyway 10795 so the method UDAs are irrelevant. 10796 +/ 10797 if(callFunction) 10798 +/ 10799 10800 auto format = cgi.request("format", defaultFormat!overload()); 10801 auto wantsFormFormat = format.startsWith("form-"); 10802 10803 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10804 // Should I still show the form on a json thing? idk... 10805 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10806 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10807 return true; 10808 } 10809 10810 try { 10811 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10812 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10813 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10814 } catch(Throwable t) { 10815 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10816 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10817 } 10818 return true; 10819 //}} 10820 10821 //cgi.header("Accept: POST"); // FIXME list the real thing 10822 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10823 //return true; 10824 } 10825 } 10826 } 10827 case "GET script.js": 10828 cgi.setResponseContentType("text/javascript"); 10829 cgi.gzipResponse = true; 10830 cgi.write(presenter.script(), true); 10831 return true; 10832 case "GET style.css": 10833 cgi.setResponseContentType("text/css"); 10834 cgi.gzipResponse = true; 10835 cgi.write(presenter.style(), true); 10836 return true; 10837 default: 10838 return false; 10839 } 10840 10841 assert(0); 10842 } 10843 return DispatcherDefinition!internalHandler(urlPrefix, false); 10844 } 10845 10846 string defaultFormat(alias method)() { 10847 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10848 foreach(attr; __traits(getAttributes, method)) { 10849 static if(is(typeof(attr) == DefaultFormat)) { 10850 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10851 return attr.value; 10852 } 10853 } 10854 return "html"; 10855 } 10856 10857 struct Paginated(T) { 10858 T[] items; 10859 string nextPageUrl; 10860 } 10861 10862 template urlNamesForMethod(alias method, string default_) { 10863 string[] helper() { 10864 auto verb = Cgi.RequestMethod.GET; 10865 bool foundVerb = false; 10866 bool foundNoun = false; 10867 10868 string def = default_; 10869 10870 bool hasAutomaticForm = false; 10871 10872 foreach(attr; __traits(getAttributes, method)) { 10873 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10874 verb = attr; 10875 if(foundVerb) 10876 assert(0, "Multiple http verbs on one function is not currently supported"); 10877 foundVerb = true; 10878 } 10879 static if(is(typeof(attr) == UrlName)) { 10880 if(foundNoun) 10881 assert(0, "Multiple url names on one function is not currently supported"); 10882 foundNoun = true; 10883 def = attr.name; 10884 } 10885 static if(__traits(isSame, attr, AutomaticForm)) { 10886 hasAutomaticForm = true; 10887 } 10888 } 10889 10890 if(def is null) 10891 def = "__null"; 10892 10893 string[] ret; 10894 10895 static if(is(typeof(method) R == return)) { 10896 static if(is(R : WebObject)) { 10897 def ~= "/"; 10898 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10899 ret ~= v ~ " " ~ def; 10900 } else { 10901 if(hasAutomaticForm) { 10902 ret ~= "GET " ~ def; 10903 ret ~= "POST " ~ def; 10904 } else { 10905 ret ~= to!string(verb) ~ " " ~ def; 10906 } 10907 } 10908 } else static assert(0); 10909 10910 return ret; 10911 } 10912 enum urlNamesForMethod = helper(); 10913 } 10914 10915 10916 enum AccessCheck { 10917 allowed, 10918 denied, 10919 nonExistant, 10920 } 10921 10922 enum Operation { 10923 show, 10924 create, 10925 replace, 10926 remove, 10927 update 10928 } 10929 10930 enum UpdateResult { 10931 accessDenied, 10932 noSuchResource, 10933 success, 10934 failure, 10935 unnecessary 10936 } 10937 10938 enum ValidationResult { 10939 valid, 10940 invalid 10941 } 10942 10943 10944 /++ 10945 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10946 10947 WARNING: this is not stable. 10948 +/ 10949 class RestObject(CRTP) : WebObject { 10950 10951 import arsd.dom; 10952 import arsd.jsvar; 10953 10954 /// Prepare the object to be shown. 10955 void show() {} 10956 /// ditto 10957 void show(string urlId) { 10958 load(urlId); 10959 show(); 10960 } 10961 10962 /// Override this to provide access control to this object. 10963 AccessCheck accessCheck(string urlId, Operation operation) { 10964 return AccessCheck.allowed; 10965 } 10966 10967 ValidationResult validate() { 10968 // FIXME 10969 return ValidationResult.valid; 10970 } 10971 10972 string getUrlSlug() { 10973 import std.conv; 10974 static if(is(typeof(CRTP.id))) 10975 return to!string((cast(CRTP) this).id); 10976 else 10977 return null; 10978 } 10979 10980 // The functions with more arguments are the low-level ones, 10981 // they forward to the ones with fewer arguments by default. 10982 10983 // POST on a parent collection - this is called from a collection class after the members are updated 10984 /++ 10985 Given a populated object, this creates a new entry. Returns the url identifier 10986 of the new object. 10987 +/ 10988 string create(scope void delegate() applyChanges) { 10989 applyChanges(); 10990 save(); 10991 return getUrlSlug(); 10992 } 10993 10994 void replace() { 10995 save(); 10996 } 10997 void replace(string urlId, scope void delegate() applyChanges) { 10998 load(urlId); 10999 applyChanges(); 11000 replace(); 11001 } 11002 11003 void update(string[] fieldList) { 11004 save(); 11005 } 11006 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11007 load(urlId); 11008 applyChanges(); 11009 update(fieldList); 11010 } 11011 11012 void remove() {} 11013 11014 void remove(string urlId) { 11015 load(urlId); 11016 remove(); 11017 } 11018 11019 abstract void load(string urlId); 11020 abstract void save(); 11021 11022 Element toHtml(Presenter)(Presenter presenter) { 11023 import arsd.dom; 11024 import std.conv; 11025 auto obj = cast(CRTP) this; 11026 auto div = Element.make("div"); 11027 div.addClass("Dclass_" ~ CRTP.stringof); 11028 div.dataset.url = getUrlSlug(); 11029 bool first = true; 11030 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11031 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11032 if(!first) div.addChild("br"); else first = false; 11033 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11034 } 11035 return div; 11036 } 11037 11038 var toJson() { 11039 import arsd.jsvar; 11040 var v = var.emptyObject(); 11041 auto obj = cast(CRTP) this; 11042 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11043 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11044 v[memberName] = __traits(getMember, obj, memberName); 11045 } 11046 return v; 11047 } 11048 11049 /+ 11050 auto structOf(this This) { 11051 11052 } 11053 +/ 11054 } 11055 11056 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11057 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11058 11059 /++ 11060 Base class for REST collections. 11061 +/ 11062 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11063 /// You might subclass this and use the cgi object's query params 11064 /// to implement a search filter, for example. 11065 /// 11066 /// FIXME: design a way to auto-generate that form 11067 /// (other than using the WebObject thing above lol 11068 // it'll prolly just be some searchParams UDA or maybe an enum. 11069 // 11070 // pagination too perhaps. 11071 // 11072 // and sorting too 11073 IndexResult index() { return IndexResult.init; } 11074 11075 string[] sortableFields() { return null; } 11076 string[] searchableFields() { return null; } 11077 11078 struct IndexResult { 11079 Obj[] results; 11080 11081 string[] sortableFields; 11082 11083 string previousPageIdentifier; 11084 string nextPageIdentifier; 11085 string firstPageIdentifier; 11086 string lastPageIdentifier; 11087 11088 int numberOfPages; 11089 } 11090 11091 override string create(scope void delegate() applyChanges) { assert(0); } 11092 override void load(string urlId) { assert(0); } 11093 override void save() { assert(0); } 11094 override void show() { 11095 index(); 11096 } 11097 override void show(string urlId) { 11098 show(); 11099 } 11100 11101 /// Proxy POST requests (create calls) to the child collection 11102 alias PostProxy = Obj; 11103 } 11104 11105 /++ 11106 Serves a REST object, similar to a Ruby on Rails resource. 11107 11108 You put data members in your class. cgi.d will automatically make something out of those. 11109 11110 It will call your constructor with the ID from the URL. This may be null. 11111 It will then populate the data members from the request. 11112 It will then call a method, if present, telling what happened. You don't need to write these! 11113 It finally returns a reply. 11114 11115 Your methods are passed a list of fields it actually set. 11116 11117 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11118 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11119 with relative linking. But meh.) 11120 11121 GET /items -> index. all values not set. 11122 GET /items/id -> get. only ID will be set, other params ignored. 11123 POST /items -> create. values set as given 11124 PUT /items/id -> replace. values set as given 11125 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11126 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11127 PATCH /items/id -> update. values set as given, list of changed fields passed 11128 or POST /items/id with cgi.post["_method"] == "PATCH" 11129 DELETE /items/id -> destroy. only ID guaranteed to be set 11130 or POST /items/id with cgi.post["_method"] == "DELETE" 11131 11132 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11133 redirect you away from it. 11134 11135 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11136 11137 I will also let you change the default, if you must. 11138 11139 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11140 11141 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11142 that follow the same thing. They may be individual resources or collections themselves. 11143 11144 Your class is expected to have at least the following methods: 11145 11146 FIXME: i kinda wanna add a routes object to the initialize call 11147 11148 create 11149 Create returns the new address on success, some code on failure. 11150 show 11151 index 11152 update 11153 remove 11154 11155 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11156 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11157 11158 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11159 11160 NOT IMPLEMENTED 11161 11162 11163 Really, a collection is a resource with a bunch of subresources. 11164 11165 GET /items 11166 index because it is GET on the top resource 11167 11168 GET /items/foo 11169 item but different than items? 11170 11171 class Items { 11172 11173 } 11174 11175 ... but meh, a collection can be automated. not worth making it 11176 a separate thing, let's look at a real example. Users has many 11177 items and a virtual one, /users/current. 11178 11179 the individual users have properties and two sub-resources: 11180 session, which is just one, and comments, a collection. 11181 11182 class User : RestObject!() { // no parent 11183 int id; 11184 string name; 11185 11186 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11187 // but you can override them to do it differently. 11188 11189 // any member which is of type RestObject can be linked automatically via href btw. 11190 11191 void show() {} 11192 void show(string urlId) {} // automated! GET of this specific thing 11193 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11194 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11195 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11196 void remove(string urlId) {} // DELETE 11197 11198 void load(string urlId) {} // the default implementation of show() populates the id, then 11199 11200 this() {} 11201 11202 mixin Subresource!Session; 11203 mixin Subresource!Comment; 11204 } 11205 11206 class Session : RestObject!() { 11207 // the parent object may not be fully constructed/loaded 11208 this(User parent) {} 11209 11210 } 11211 11212 class Comment : CollectionOf!Comment { 11213 this(User parent) {} 11214 } 11215 11216 class Users : CollectionOf!User { 11217 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11218 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11219 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11220 } 11221 11222 +/ 11223 auto serveRestObject(T)(string urlPrefix) { 11224 assert(urlPrefix[0] == '/'); 11225 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11226 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11227 string url = cgi.pathInfo[urlPrefix.length .. $]; 11228 11229 if(url.length && url[$ - 1] == '/') { 11230 // remove the final slash... 11231 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11232 return true; 11233 } 11234 11235 return restObjectServeHandler!T(cgi, presenter, url); 11236 } 11237 return DispatcherDefinition!internalHandler(urlPrefix, false); 11238 } 11239 11240 /+ 11241 /// Convenience method for serving a collection. It will be named the same 11242 /// as type T, just with an s at the end. If you need any further, just 11243 /// write the class yourself. 11244 auto serveRestCollectionOf(T)(string urlPrefix) { 11245 assert(urlPrefix[0] == '/'); 11246 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11247 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11248 } 11249 +/ 11250 11251 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11252 string urlId = null; 11253 if(url.length && url[0] == '/') { 11254 // asking for a subobject 11255 urlId = url[1 .. $]; 11256 foreach(idx, ch; urlId) { 11257 if(ch == '/') { 11258 urlId = urlId[0 .. idx]; 11259 break; 11260 } 11261 } 11262 } 11263 11264 // FIXME handle other subresources 11265 11266 static if(is(T : CollectionOf!(C), C)) { 11267 if(urlId !is null) { 11268 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11269 } 11270 } 11271 11272 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11273 11274 auto obj = new T(); 11275 obj.initialize(cgi); 11276 // FIXME: populate reflection info delegates 11277 11278 11279 // FIXME: I am not happy with this. 11280 switch(urlId) { 11281 case "script.js": 11282 cgi.setResponseContentType("text/javascript"); 11283 cgi.gzipResponse = true; 11284 cgi.write(presenter.script(), true); 11285 return true; 11286 case "style.css": 11287 cgi.setResponseContentType("text/css"); 11288 cgi.gzipResponse = true; 11289 cgi.write(presenter.style(), true); 11290 return true; 11291 default: 11292 // intentionally blank 11293 } 11294 11295 11296 11297 11298 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11299 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11300 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11301 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11302 } 11303 } 11304 void applyChanges() { 11305 applyChangesTemplate(cgi, obj); 11306 } 11307 11308 string[] modifiedList; 11309 11310 void writeObject(bool addFormLinks) { 11311 if(cgi.request("format") == "json") { 11312 cgi.setResponseContentType("application/json"); 11313 cgi.write(obj.toJson().toString, true); 11314 } else { 11315 auto container = presenter.htmlContainer(); 11316 if(addFormLinks) { 11317 static if(is(T : CollectionOf!(C), C)) 11318 container.appendHtml(` 11319 <form> 11320 <button type="submit" name="_method" value="POST">Create New</button> 11321 </form> 11322 `); 11323 else 11324 container.appendHtml(` 11325 <a href="..">Back</a> 11326 <form> 11327 <button type="submit" name="_method" value="PATCH">Edit</button> 11328 <button type="submit" name="_method" value="DELETE">Delete</button> 11329 </form> 11330 `); 11331 } 11332 container.appendChild(obj.toHtml(presenter)); 11333 cgi.write(container.parentDocument.toString, true); 11334 } 11335 } 11336 11337 // FIXME: I think I need a set type in here.... 11338 // it will be nice to pass sets of members. 11339 11340 try 11341 switch(cgi.requestMethod) { 11342 case Cgi.RequestMethod.GET: 11343 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11344 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11345 11346 // automatic forms here for usable basic auto site from browser. 11347 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11348 switch(cgi.request("_method", "GET")) { 11349 case "GET": 11350 static if(is(T : CollectionOf!(C), C)) { 11351 auto results = obj.index(); 11352 if(cgi.request("format", "html") == "html") { 11353 auto container = presenter.htmlContainer(); 11354 auto html = presenter.formatReturnValueAsHtml(results.results); 11355 container.appendHtml(` 11356 <form> 11357 <button type="submit" name="_method" value="POST">Create New</button> 11358 </form> 11359 `); 11360 11361 container.appendChild(html); 11362 cgi.write(container.parentDocument.toString, true); 11363 } else { 11364 cgi.setResponseContentType("application/json"); 11365 import arsd.jsvar; 11366 var json = var.emptyArray; 11367 foreach(r; results.results) { 11368 var o = var.emptyObject; 11369 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11370 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11371 o[memberName] = __traits(getMember, r, memberName); 11372 } 11373 11374 json ~= o; 11375 } 11376 cgi.write(json.toJson(), true); 11377 } 11378 } else { 11379 obj.show(urlId); 11380 writeObject(true); 11381 } 11382 break; 11383 case "PATCH": 11384 obj.load(urlId); 11385 goto case; 11386 case "PUT": 11387 case "POST": 11388 // an editing form for the object 11389 auto container = presenter.htmlContainer(); 11390 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11391 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11392 } else { 11393 auto form = presenter.createAutomaticFormForObject(obj); 11394 } 11395 form.attrs.method = "POST"; 11396 form.setValue("_method", cgi.request("_method", "GET")); 11397 container.appendChild(form); 11398 cgi.write(container.parentDocument.toString(), true); 11399 break; 11400 case "DELETE": 11401 // FIXME: a delete form for the object (can be phrased "are you sure?") 11402 auto container = presenter.htmlContainer(); 11403 container.appendHtml(` 11404 <form method="POST"> 11405 Are you sure you want to delete this item? 11406 <input type="hidden" name="_method" value="DELETE" /> 11407 <input type="submit" value="Yes, Delete It" /> 11408 </form> 11409 11410 `); 11411 cgi.write(container.parentDocument.toString(), true); 11412 break; 11413 default: 11414 cgi.write("bad method\n", true); 11415 } 11416 break; 11417 case Cgi.RequestMethod.POST: 11418 // this is to allow compatibility with HTML forms 11419 switch(cgi.request("_method", "POST")) { 11420 case "PUT": 11421 goto PUT; 11422 case "PATCH": 11423 goto PATCH; 11424 case "DELETE": 11425 goto DELETE; 11426 case "POST": 11427 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11428 auto p = new obj.PostProxy(); 11429 void specialApplyChanges() { 11430 applyChangesTemplate(cgi, p); 11431 } 11432 string n = p.create(&specialApplyChanges); 11433 } else { 11434 string n = obj.create(&applyChanges); 11435 } 11436 11437 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11438 cgi.setResponseLocation(newUrl); 11439 cgi.setResponseStatus("201 Created"); 11440 cgi.write(`The object has been created.`); 11441 break; 11442 default: 11443 cgi.write("bad method\n", true); 11444 } 11445 // FIXME this should be valid on the collection, but not the child.... 11446 // 303 See Other 11447 break; 11448 case Cgi.RequestMethod.PUT: 11449 PUT: 11450 obj.replace(urlId, &applyChanges); 11451 writeObject(false); 11452 break; 11453 case Cgi.RequestMethod.PATCH: 11454 PATCH: 11455 obj.update(urlId, &applyChanges, modifiedList); 11456 writeObject(false); 11457 break; 11458 case Cgi.RequestMethod.DELETE: 11459 DELETE: 11460 obj.remove(urlId); 11461 cgi.setResponseStatus("204 No Content"); 11462 break; 11463 default: 11464 // FIXME: OPTIONS, HEAD 11465 } 11466 catch(Throwable t) { 11467 presenter.presentExceptionAsHtml(cgi, t); 11468 } 11469 11470 return true; 11471 } 11472 11473 /+ 11474 struct SetOfFields(T) { 11475 private void[0][string] storage; 11476 void set(string what) { 11477 //storage[what] = 11478 } 11479 void unset(string what) {} 11480 void setAll() {} 11481 void unsetAll() {} 11482 bool isPresent(string what) { return false; } 11483 } 11484 +/ 11485 11486 /+ 11487 enum readonly; 11488 enum hideonindex; 11489 +/ 11490 11491 /++ 11492 Returns true if I recommend gzipping content of this type. You might 11493 want to call it from your Presenter classes before calling cgi.write. 11494 11495 --- 11496 cgi.setResponseContentType(yourContentType); 11497 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11498 cgi.write(yourData, true); 11499 --- 11500 11501 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11502 11503 11504 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11505 11506 History: 11507 Added January 28, 2023 (dub v11.0) 11508 +/ 11509 bool gzipRecommendedForContentType(string contentType) { 11510 if(contentType.startsWith("text/")) 11511 return true; 11512 if(contentType.startsWith("application/javascript")) 11513 return true; 11514 11515 return false; 11516 } 11517 11518 /++ 11519 Serves a static file. To be used with [dispatcher]. 11520 11521 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11522 +/ 11523 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11524 // https://baus.net/on-tcp_cork/ 11525 // man 2 sendfile 11526 assert(urlPrefix[0] == '/'); 11527 if(filename is null) 11528 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11529 if(contentType is null) { 11530 contentType = contentTypeFromFileExtension(filename); 11531 } 11532 11533 static struct DispatcherDetails { 11534 string filename; 11535 string contentType; 11536 } 11537 11538 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11539 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11540 cgi.setCache(true); 11541 cgi.setResponseContentType(details.contentType); 11542 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11543 cgi.write(std.file.read(details.filename), true); 11544 return true; 11545 } 11546 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11547 } 11548 11549 /++ 11550 Serves static data. To be used with [dispatcher]. 11551 11552 History: 11553 Added October 31, 2021 11554 +/ 11555 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11556 assert(urlPrefix[0] == '/'); 11557 if(contentType is null) { 11558 contentType = contentTypeFromFileExtension(urlPrefix); 11559 } 11560 11561 static struct DispatcherDetails { 11562 immutable(void)[] data; 11563 string contentType; 11564 } 11565 11566 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11567 cgi.setCache(true); 11568 cgi.setResponseContentType(details.contentType); 11569 cgi.write(details.data, true); 11570 return true; 11571 } 11572 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11573 } 11574 11575 string contentTypeFromFileExtension(string filename) { 11576 if(filename.endsWith(".png")) 11577 return "image/png"; 11578 if(filename.endsWith(".apng")) 11579 return "image/apng"; 11580 if(filename.endsWith(".svg")) 11581 return "image/svg+xml"; 11582 if(filename.endsWith(".jpg")) 11583 return "image/jpeg"; 11584 if(filename.endsWith(".html")) 11585 return "text/html"; 11586 if(filename.endsWith(".css")) 11587 return "text/css"; 11588 if(filename.endsWith(".js")) 11589 return "application/javascript"; 11590 if(filename.endsWith(".wasm")) 11591 return "application/wasm"; 11592 if(filename.endsWith(".mp3")) 11593 return "audio/mpeg"; 11594 if(filename.endsWith(".pdf")) 11595 return "application/pdf"; 11596 return null; 11597 } 11598 11599 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11600 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11601 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11602 assert(urlPrefix[0] == '/'); 11603 assert(urlPrefix[$-1] == '/'); 11604 11605 static struct DispatcherDetails { 11606 string directory; 11607 bool recursive; 11608 } 11609 11610 if(directory is null) 11611 directory = urlPrefix[1 .. $]; 11612 11613 if(directory.length == 0) 11614 directory = "./"; 11615 11616 assert(directory[$-1] == '/'); 11617 11618 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11619 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11620 11621 if(details.recursive) { 11622 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11623 if(file.indexOf("\\") != -1) 11624 return false; 11625 11626 import std.path; 11627 11628 file = std.path.buildNormalizedPath(file); 11629 enum upOneDir = ".." ~ std.path.dirSeparator; 11630 11631 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11632 if(file == ".." || file.startsWith(upOneDir)) 11633 return false; 11634 if(std.path.isAbsolute(file)) 11635 return false; 11636 11637 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11638 11639 // once it passes these filters it is probably ok. 11640 } else { 11641 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11642 return false; 11643 } 11644 11645 auto contentType = contentTypeFromFileExtension(file); 11646 11647 auto fn = details.directory ~ file; 11648 if(std.file.exists(fn)) { 11649 //if(contentType.indexOf("image/") == 0) 11650 //cgi.setCache(true); 11651 //else if(contentType.indexOf("audio/") == 0) 11652 cgi.setCache(true); 11653 cgi.setResponseContentType(contentType); 11654 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11655 cgi.write(std.file.read(fn), true); 11656 return true; 11657 } else { 11658 return false; 11659 } 11660 } 11661 11662 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11663 } 11664 11665 /++ 11666 Redirects one url to another 11667 11668 See_Also: [dispatcher], [serveStaticFile] 11669 +/ 11670 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11671 assert(urlPrefix[0] == '/'); 11672 static struct DispatcherDetails { 11673 string redirectTo; 11674 string code; 11675 } 11676 11677 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11678 cgi.setResponseLocation(details.redirectTo, true, details.code); 11679 return true; 11680 } 11681 11682 11683 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11684 } 11685 11686 /// Used exclusively with `dispatchTo` 11687 struct DispatcherData(Presenter) { 11688 Cgi cgi; /// You can use this cgi object. 11689 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11690 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11691 } 11692 11693 /++ 11694 Dispatches the URL to a specific function. 11695 +/ 11696 auto handleWith(alias handler)(string urlPrefix) { 11697 // cuz I'm too lazy to do it better right now 11698 static class Hack : WebObject { 11699 static import std.traits; 11700 @UrlName("") 11701 auto handle(std.traits.Parameters!handler args) { 11702 return handler(args); 11703 } 11704 } 11705 11706 return urlPrefix.serveApiInternal!Hack; 11707 } 11708 11709 /++ 11710 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11711 11712 --- 11713 bool other(DD)(DD dd) { 11714 return dd.dispatcher!( 11715 "/whatever".serveRedirect("/success"), 11716 "/api/".serveApi!MyClass 11717 ); 11718 } 11719 --- 11720 11721 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11722 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11723 Or, of course, you could just use the exact type in your own code. 11724 11725 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11726 good job. 11727 11728 11729 +/ 11730 auto dispatchTo(alias handler)(string urlPrefix) { 11731 assert(urlPrefix[0] == '/'); 11732 assert(urlPrefix[$-1] != '/'); 11733 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11734 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11735 } 11736 11737 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11738 } 11739 11740 /++ 11741 See [serveStaticFile] if you want to serve a file off disk. 11742 11743 History: 11744 Added January 28, 2023 (dub v11.0) 11745 +/ 11746 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11747 assert(urlPrefix[0] == '/'); 11748 11749 static struct DispatcherDetails { 11750 immutable(ubyte)[] data; 11751 string contentType; 11752 string filenameToSuggestAsDownload; 11753 } 11754 11755 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11756 cgi.setCache(true); 11757 cgi.setResponseContentType(details.contentType); 11758 if(details.filenameToSuggestAsDownload.length) 11759 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11760 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11761 cgi.write(details.data, true); 11762 return true; 11763 } 11764 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11765 } 11766 11767 /++ 11768 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11769 11770 History: 11771 Added January 28, 2023 (dub v11.0) 11772 +/ 11773 alias KeepExistingPresenter = typeof(null); 11774 11775 /++ 11776 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11777 this issues the given errorCode and stops processing. 11778 11779 --- 11780 bool hasAdminPermissions(Cgi cgi) { 11781 return true; 11782 } 11783 11784 mixin DispatcherMain!( 11785 "/admin".dispatchSubsection!( 11786 passFilterOrIssueError!(hasAdminPermissions, 403), 11787 KeepExistingPresenter, 11788 "/".serveApi!AdminFunctions 11789 ) 11790 ); 11791 --- 11792 11793 History: 11794 Added January 28, 2023 (dub v11.0) 11795 +/ 11796 template passFilterOrIssueError(alias filter, int errorCode) { 11797 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11798 if(filter(dd.cgi)) 11799 return true; 11800 dd.presenter.renderBasicError(dd.cgi, errorCode); 11801 return false; 11802 } 11803 } 11804 11805 /++ 11806 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11807 and then be dispatched to their own handlers. 11808 11809 --- 11810 /+ 11811 // a long-form filter function 11812 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11813 // you are permitted to call mutable methods on the Cgi object 11814 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11815 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11816 11817 if(checkPermissionOnRequest(dd.cgi)) { 11818 return true; // OK, allow processing to continue 11819 } else { 11820 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11821 return false; // and stop further processing into this subsection 11822 } 11823 } 11824 +/ 11825 11826 // but you can also do short-form filters: 11827 11828 bool permissionCheck(Cgi cgi) { 11829 return ("ok" in cgi.get) !is null; 11830 } 11831 11832 // handler for the subsection 11833 class AdminClass : WebObject { 11834 int foo() { return 5; } 11835 } 11836 11837 // handler for the main site 11838 class TheMainSite : WebObject {} 11839 11840 mixin DispatcherMain!( 11841 "/admin".dispatchSubsection!( 11842 // converts our short-form filter into a long-form filter 11843 passFilterOrIssueError!(permissionCheck, 403), 11844 // can use a new presenter if wanted for the subsection 11845 KeepExistingPresenter, 11846 // and then provide child route dispatchers 11847 "/".serveApi!AdminClass 11848 ), 11849 // and back to the top level 11850 "/".serveApi!TheMainSite 11851 ); 11852 --- 11853 11854 Note you can encapsulate sections in files like this: 11855 11856 --- 11857 auto adminDispatcher(string urlPrefix) { 11858 return urlPrefix.dispatchSubsection!( 11859 .... 11860 ); 11861 } 11862 11863 mixin DispatcherMain!( 11864 "/admin".adminDispatcher, 11865 // and so on 11866 ) 11867 --- 11868 11869 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11870 11871 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11872 11873 11874 History: 11875 Added January 28, 2023 (dub v11.0) 11876 +/ 11877 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11878 assert(urlPrefix[0] == '/'); 11879 assert(urlPrefix[$-1] != '/'); 11880 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11881 static if(!is(PreRequestFilter == typeof(null))) { 11882 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11883 return true; // we handled it by rejecting it 11884 } 11885 11886 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11887 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11888 } else { 11889 auto newPresenter = new NewPresenter(); 11890 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11891 } 11892 } 11893 11894 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11895 } 11896 11897 /++ 11898 A URL dispatcher. 11899 11900 --- 11901 if(cgi.dispatcher!( 11902 "/api/".serveApi!MyApiClass, 11903 "/objects/lol".serveRestObject!MyRestObject, 11904 "/file.js".serveStaticFile, 11905 "/admin/".dispatchTo!adminHandler 11906 )) return; 11907 --- 11908 11909 11910 You define a series of url prefixes followed by handlers. 11911 11912 You may want to do different pre- and post- processing there, for example, 11913 an authorization check and different page layout. You can use different 11914 presenters and different function chains. See [dispatchSubsection] for details. 11915 11916 [dispatchTo] will send the request to another function for handling. 11917 +/ 11918 template dispatcher(definitions...) { 11919 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11920 static if(is(Presenter == typeof(null))) { 11921 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11922 auto presenter = new GenericWebPresenter(); 11923 } else 11924 alias presenter = presenterArg; 11925 11926 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11927 } 11928 11929 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11930 // I can prolly make this more efficient later but meh. 11931 foreach(definition; definitions) { 11932 if(definition.rejectFurther) { 11933 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11934 auto ret = definition.handler( 11935 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11936 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11937 if(ret) 11938 return true; 11939 } 11940 } else if( 11941 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11942 // cgi.d dispatcher urls must be complete or have a /; 11943 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11944 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11945 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11946 ) { 11947 auto ret = definition.handler( 11948 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11949 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11950 if(ret) 11951 return true; 11952 } 11953 } 11954 return false; 11955 } 11956 } 11957 11958 }); 11959 11960 private struct StackBuffer { 11961 char[1024] initial = void; 11962 char[] buffer; 11963 size_t position; 11964 11965 this(int a) { 11966 buffer = initial[]; 11967 position = 0; 11968 } 11969 11970 void add(in char[] what) { 11971 if(position + what.length > buffer.length) 11972 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 11973 buffer[position .. position + what.length] = what[]; 11974 position += what.length; 11975 } 11976 11977 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 11978 add(w1); 11979 add(w2); 11980 add(w3); 11981 } 11982 11983 void add(long v) { 11984 char[16] buffer = void; 11985 auto pos = buffer.length; 11986 bool negative; 11987 if(v < 0) { 11988 negative = true; 11989 v = -v; 11990 } 11991 do { 11992 buffer[--pos] = cast(char) (v % 10 + '0'); 11993 v /= 10; 11994 } while(v); 11995 11996 if(negative) 11997 buffer[--pos] = '-'; 11998 11999 auto res = buffer[pos .. $]; 12000 12001 add(res[]); 12002 } 12003 12004 char[] get() @nogc { 12005 return buffer[0 .. position]; 12006 } 12007 } 12008 12009 // duplicated in http2.d 12010 private static string getHttpCodeText(int code) pure nothrow @nogc { 12011 switch(code) { 12012 case 200: return "200 OK"; 12013 case 201: return "201 Created"; 12014 case 202: return "202 Accepted"; 12015 case 203: return "203 Non-Authoritative Information"; 12016 case 204: return "204 No Content"; 12017 case 205: return "205 Reset Content"; 12018 case 206: return "206 Partial Content"; 12019 // 12020 case 300: return "300 Multiple Choices"; 12021 case 301: return "301 Moved Permanently"; 12022 case 302: return "302 Found"; 12023 case 303: return "303 See Other"; 12024 case 304: return "304 Not Modified"; 12025 case 305: return "305 Use Proxy"; 12026 case 307: return "307 Temporary Redirect"; 12027 case 308: return "308 Permanent Redirect"; 12028 12029 // 12030 case 400: return "400 Bad Request"; 12031 case 401: return "401 Unauthorized"; 12032 case 402: return "402 Payment Required"; 12033 case 403: return "403 Forbidden"; 12034 case 404: return "404 Not Found"; 12035 case 405: return "405 Method Not Allowed"; 12036 case 406: return "406 Not Acceptable"; 12037 case 407: return "407 Proxy Authentication Required"; 12038 case 408: return "408 Request Timeout"; 12039 case 409: return "409 Conflict"; 12040 case 410: return "410 Gone"; 12041 case 411: return "411 Length Required"; 12042 case 412: return "412 Precondition Failed"; 12043 case 413: return "413 Payload Too Large"; 12044 case 414: return "414 URI Too Long"; 12045 case 415: return "415 Unsupported Media Type"; 12046 case 416: return "416 Range Not Satisfiable"; 12047 case 417: return "417 Expectation Failed"; 12048 case 418: return "418 I'm a teapot"; 12049 case 421: return "421 Misdirected Request"; 12050 case 422: return "422 Unprocessable Entity (WebDAV)"; 12051 case 423: return "423 Locked (WebDAV)"; 12052 case 424: return "424 Failed Dependency (WebDAV)"; 12053 case 425: return "425 Too Early"; 12054 case 426: return "426 Upgrade Required"; 12055 case 428: return "428 Precondition Required"; 12056 case 431: return "431 Request Header Fields Too Large"; 12057 case 451: return "451 Unavailable For Legal Reasons"; 12058 12059 case 500: return "500 Internal Server Error"; 12060 case 501: return "501 Not Implemented"; 12061 case 502: return "502 Bad Gateway"; 12062 case 503: return "503 Service Unavailable"; 12063 case 504: return "504 Gateway Timeout"; 12064 case 505: return "505 HTTP Version Not Supported"; 12065 case 506: return "506 Variant Also Negotiates"; 12066 case 507: return "507 Insufficient Storage (WebDAV)"; 12067 case 508: return "508 Loop Detected (WebDAV)"; 12068 case 510: return "510 Not Extended"; 12069 case 511: return "511 Network Authentication Required"; 12070 // 12071 default: assert(0, "Unsupported http code"); 12072 } 12073 } 12074 12075 12076 /+ 12077 /++ 12078 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12079 12080 It relies on jsvar.d and dom.d. 12081 12082 12083 You can get javascript out of it to call. The generated functions need to look 12084 like 12085 12086 function name(a,b,c,d,e) { 12087 return _call("name", {"realName":a,"sds":b}); 12088 } 12089 12090 And _call returns an object you can call or set up or whatever. 12091 +/ 12092 bool apiDispatcher()(Cgi cgi) { 12093 import arsd.jsvar; 12094 import arsd.dom; 12095 } 12096 +/ 12097 version(linux) 12098 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12099 /* 12100 Copyright: Adam D. Ruppe, 2008 - 2023 12101 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12102 Authors: Adam D. Ruppe 12103 12104 Copyright Adam D. Ruppe 2008 - 2023. 12105 Distributed under the Boost Software License, Version 1.0. 12106 (See accompanying file LICENSE_1_0.txt or copy at 12107 http://www.boost.org/LICENSE_1_0.txt) 12108 */