1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 static import arsd.core; 490 version(Posix) 491 import arsd.core : makeNonBlocking; 492 493 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 494 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 495 496 /++ 497 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 498 +/ 499 version(Demo) 500 unittest { 501 import arsd.cgi; 502 503 mixin DispatcherMain!( 504 "/".serveStaticFileDirectory(null, true) 505 ); 506 } 507 508 /++ 509 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 510 +/ 511 version(Demo) 512 unittest { 513 import arsd.cgi; 514 515 void requestHandler(Cgi cgi) { 516 cgi.dispatcher!( 517 "/".serveStaticFileDirectory(null, true) 518 ); 519 } 520 521 // mixin GenericMain!requestHandler would add this function: 522 void main(string[] args) { 523 // this is all the content of [cgiMainImpl] which you can also call 524 525 // cgi.d embeds a few add on functions like real time event forwarders 526 // and session servers it can run in other processes. this spawns them, if needed. 527 if(tryAddonServers(args)) 528 return; 529 530 // cgi.d allows you to easily simulate http requests from the command line, 531 // without actually starting a server. this function will do that. 532 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 533 return; 534 535 RequestServer server; 536 // you can change the default port here if you like 537 // server.listeningPort = 9000; 538 539 // then call this to let the command line args override your default 540 server.configureFromCommandLine(args); 541 542 // here is where you could print out the listeningPort to the user if you wanted 543 544 // and serve the request(s) according to the compile configuration 545 server.serve!(requestHandler)(); 546 547 // or you could explicitly choose a serve mode like this: 548 // server.serveEmbeddedHttp!requestHandler(); 549 } 550 } 551 552 /++ 553 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 554 otherwise run through the rest of the internal mechanisms to call your functions without actually 555 spinning up a server. 556 +/ 557 version(Demo) 558 unittest { 559 import arsd.cgi; 560 561 void requestHandler(Cgi cgi) { 562 563 } 564 565 // D doesn't let me embed a unittest inside an example unittest 566 // so this is a function, but you can do it however in your real program 567 /* unittest */ void runTests() { 568 auto tester = new CgiTester(&requestHandler); 569 570 auto response = tester.GET("/"); 571 assert(response.code == 200); 572 } 573 } 574 575 static import std.file; 576 577 // for a single thread, linear request thing, use: 578 // -version=embedded_httpd_threads -version=cgi_no_threads 579 580 version(Posix) { 581 version(CRuntime_Musl) { 582 583 } else version(minimal) { 584 585 } else { 586 version(FreeBSD) { 587 // I never implemented the fancy stuff there either 588 } else { 589 version=with_breaking_cgi_features; 590 version=with_sendfd; 591 version=with_addon_servers; 592 } 593 } 594 } 595 596 version(Windows) { 597 version(minimal) { 598 599 } else { 600 // not too concerned about gdc here since the mingw version is fairly new as well 601 version=with_breaking_cgi_features; 602 } 603 } 604 605 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 606 void cloexec(int fd) { 607 version(Posix) { 608 import core.sys.posix.fcntl; 609 fcntl(fd, F_SETFD, FD_CLOEXEC); 610 } 611 } 612 613 void cloexec(Socket s) { 614 version(Posix) { 615 import core.sys.posix.fcntl; 616 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 617 } 618 } 619 620 version(embedded_httpd_hybrid) { 621 version=embedded_httpd_threads; 622 version(cgi_no_fork) {} else version(Posix) 623 version=cgi_use_fork; 624 version=cgi_use_fiber; 625 } 626 627 version(cgi_use_fork) 628 enum cgi_use_fork_default = true; 629 else 630 enum cgi_use_fork_default = false; 631 632 // the servers must know about the connections to talk to them; the interfaces are vital 633 version(with_addon_servers) 634 version=with_addon_servers_connections; 635 636 version(embedded_httpd) { 637 version(linux) 638 version=embedded_httpd_processes; 639 else { 640 version=embedded_httpd_threads; 641 } 642 643 /* 644 version(with_openssl) { 645 pragma(lib, "crypto"); 646 pragma(lib, "ssl"); 647 } 648 */ 649 } 650 651 version(embedded_httpd_processes) 652 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 653 654 version(embedded_httpd_threads) { 655 // unless the user overrides the default.. 656 version(cgi_session_server_process) 657 {} 658 else 659 version=cgi_embedded_sessions; 660 } 661 version(scgi) { 662 // unless the user overrides the default.. 663 version(cgi_session_server_process) 664 {} 665 else 666 version=cgi_embedded_sessions; 667 } 668 669 // fall back if the other is not defined so we can cleanly version it below 670 version(cgi_embedded_sessions) {} 671 else version=cgi_session_server_process; 672 673 674 version=cgi_with_websocket; 675 676 enum long defaultMaxContentLength = 5_000_000; 677 678 /* 679 680 To do a file download offer in the browser: 681 682 cgi.setResponseContentType("text/csv"); 683 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 684 */ 685 686 // FIXME: the location header is supposed to be an absolute url I guess. 687 688 // FIXME: would be cool to flush part of a dom document before complete 689 // somehow in here and dom.d. 690 691 692 // these are public so you can mixin GenericMain. 693 // FIXME: use a function level import instead! 694 public import std.string; 695 public import std.stdio; 696 public import std.conv; 697 import std.uri; 698 import std.uni; 699 import std.algorithm.comparison; 700 import std.algorithm.searching; 701 import std.exception; 702 import std.base64; 703 static import std.algorithm; 704 import std.datetime; 705 import std.range; 706 707 import std.process; 708 709 import std.zlib; 710 711 712 T[] consume(T)(T[] range, int count) { 713 if(count > range.length) 714 count = range.length; 715 return range[count..$]; 716 } 717 718 int locationOf(T)(T[] data, string item) { 719 const(ubyte[]) d = cast(const(ubyte[])) data; 720 const(ubyte[]) i = cast(const(ubyte[])) item; 721 722 // this is a vague sanity check to ensure we aren't getting insanely 723 // sized input that will infinite loop below. it should never happen; 724 // even huge file uploads ought to come in smaller individual pieces. 725 if(d.length > (int.max/2)) 726 throw new Exception("excessive block of input"); 727 728 for(int a = 0; a < d.length; a++) { 729 if(a + i.length > d.length) 730 return -1; 731 if(d[a..a+i.length] == i) 732 return a; 733 } 734 735 return -1; 736 } 737 738 /// If you are doing a custom cgi class, mixing this in can take care of 739 /// the required constructors for you 740 mixin template ForwardCgiConstructors() { 741 this(long maxContentLength = defaultMaxContentLength, 742 string[string] env = null, 743 const(ubyte)[] delegate() readdata = null, 744 void delegate(const(ubyte)[]) _rawDataOutput = null, 745 void delegate() _flush = null 746 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 747 748 this(string[] args) { super(args); } 749 750 this( 751 BufferedInputRange inputData, 752 string address, ushort _port, 753 int pathInfoStarts = 0, 754 bool _https = false, 755 void delegate(const(ubyte)[]) _rawDataOutput = null, 756 void delegate() _flush = null, 757 // this pointer tells if the connection is supposed to be closed after we handle this 758 bool* closeConnection = null) 759 { 760 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 761 } 762 763 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 764 } 765 766 /// thrown when a connection is closed remotely while we waiting on data from it 767 class ConnectionClosedException : Exception { 768 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 769 super(message, file, line, next); 770 } 771 } 772 773 774 version(Windows) { 775 // FIXME: ugly hack to solve stdin exception problems on Windows: 776 // reading stdin results in StdioException (Bad file descriptor) 777 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 778 private struct stdin { 779 struct ByChunk { // Replicates std.stdio.ByChunk 780 private: 781 ubyte[] chunk_; 782 public: 783 this(size_t size) 784 in { 785 assert(size, "size must be larger than 0"); 786 } 787 do { 788 chunk_ = new ubyte[](size); 789 popFront(); 790 } 791 792 @property bool empty() const { 793 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 794 } 795 @property nothrow ubyte[] front() { return chunk_; } 796 void popFront() { 797 enforce(!empty, "Cannot call popFront on empty range"); 798 chunk_ = stdin.rawRead(chunk_); 799 } 800 } 801 802 import core.sys.windows.windows; 803 static: 804 805 T[] rawRead(T)(T[] buf) { 806 uint bytesRead; 807 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 808 809 if (!result) { 810 auto err = GetLastError(); 811 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 812 return buf[0..0]; 813 // Some other error, throw it 814 815 char* buffer; 816 scope(exit) LocalFree(buffer); 817 818 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 819 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 820 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 821 throw new Exception(to!string(buffer)); 822 } 823 enforce(!(bytesRead % T.sizeof), "I/O error"); 824 return buf[0..bytesRead / T.sizeof]; 825 } 826 827 auto byChunk(size_t sz) { return ByChunk(sz); } 828 829 void close() { 830 std.stdio.stdin.close; 831 } 832 } 833 } 834 835 /// The main interface with the web request 836 class Cgi { 837 public: 838 /// the methods a request can be 839 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 840 // these are defined in the standard, but idk if they are useful for anything 841 OPTIONS, TRACE, CONNECT, 842 // These seem new, I have only recently seen them 843 PATCH, MERGE, 844 // this is an extension for when the method is not specified and you want to assume 845 CommandLine } 846 847 848 /+ 849 /++ 850 Cgi provides a per-request memory pool 851 852 +/ 853 void[] allocateMemory(size_t nBytes) { 854 855 } 856 857 /// ditto 858 void[] reallocateMemory(void[] old, size_t nBytes) { 859 860 } 861 862 /// ditto 863 void freeMemory(void[] memory) { 864 865 } 866 +/ 867 868 869 /* 870 import core.runtime; 871 auto args = Runtime.args(); 872 873 we can call the app a few ways: 874 875 1) set up the environment variables and call the app (manually simulating CGI) 876 2) simulate a call automatically: 877 ./app method 'uri' 878 879 for example: 880 ./app get /path?arg arg2=something 881 882 Anything on the uri is treated as query string etc 883 884 on get method, further args are appended to the query string (encoded automatically) 885 on post method, further args are done as post 886 887 888 @name means import from file "name". if name == -, it uses stdin 889 (so info=@- means set info to the value of stdin) 890 891 892 Other arguments include: 893 --cookie name=value (these are all concated together) 894 --header 'X-Something: cool' 895 --referrer 'something' 896 --port 80 897 --remote-address some.ip.address.here 898 --https yes 899 --user-agent 'something' 900 --userpass 'user:pass' 901 --authorization 'Basic base64encoded_user:pass' 902 --accept 'content' // FIXME: better example 903 --last-event-id 'something' 904 --host 'something.com' 905 906 Non-simulation arguments: 907 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 908 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 909 910 */ 911 912 /** Initializes it with command line arguments (for easy testing) */ 913 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 914 rawDataOutput = _rawDataOutput; 915 // these are all set locally so the loop works 916 // without triggering errors in dmd 2.064 917 // we go ahead and set them at the end of it to the this version 918 int port; 919 string referrer; 920 string remoteAddress; 921 string userAgent; 922 string authorization; 923 string origin; 924 string accept; 925 string lastEventId; 926 bool https; 927 string host; 928 RequestMethod requestMethod; 929 string requestUri; 930 string pathInfo; 931 string queryString; 932 933 bool lookingForMethod; 934 bool lookingForUri; 935 string nextArgIs; 936 937 string _cookie; 938 string _queryString; 939 string[][string] _post; 940 string[string] _headers; 941 942 string[] breakUp(string s) { 943 string k, v; 944 auto idx = s.indexOf("="); 945 if(idx == -1) { 946 k = s; 947 } else { 948 k = s[0 .. idx]; 949 v = s[idx + 1 .. $]; 950 } 951 952 return [k, v]; 953 } 954 955 lookingForMethod = true; 956 957 scriptName = args[0]; 958 scriptFileName = args[0]; 959 960 environmentVariables = cast(const) environment.toAA; 961 962 foreach(arg; args[1 .. $]) { 963 if(arg.startsWith("--")) { 964 nextArgIs = arg[2 .. $]; 965 } else if(nextArgIs.length) { 966 if (nextArgIs == "cookie") { 967 auto info = breakUp(arg); 968 if(_cookie.length) 969 _cookie ~= "; "; 970 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 971 } 972 else if (nextArgIs == "port") { 973 port = to!int(arg); 974 } 975 else if (nextArgIs == "referrer") { 976 referrer = arg; 977 } 978 else if (nextArgIs == "remote-address") { 979 remoteAddress = arg; 980 } 981 else if (nextArgIs == "user-agent") { 982 userAgent = arg; 983 } 984 else if (nextArgIs == "authorization") { 985 authorization = arg; 986 } 987 else if (nextArgIs == "userpass") { 988 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 989 } 990 else if (nextArgIs == "origin") { 991 origin = arg; 992 } 993 else if (nextArgIs == "accept") { 994 accept = arg; 995 } 996 else if (nextArgIs == "last-event-id") { 997 lastEventId = arg; 998 } 999 else if (nextArgIs == "https") { 1000 if(arg == "yes") 1001 https = true; 1002 } 1003 else if (nextArgIs == "header") { 1004 string thing, other; 1005 auto idx = arg.indexOf(":"); 1006 if(idx == -1) 1007 throw new Exception("need a colon in a http header"); 1008 thing = arg[0 .. idx]; 1009 other = arg[idx + 1.. $]; 1010 _headers[thing.strip.toLower()] = other.strip; 1011 } 1012 else if (nextArgIs == "host") { 1013 host = arg; 1014 } 1015 // else 1016 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1017 1018 nextArgIs = null; 1019 } else if(lookingForMethod) { 1020 lookingForMethod = false; 1021 lookingForUri = true; 1022 1023 if(arg.asLowerCase().equal("commandline")) 1024 requestMethod = RequestMethod.CommandLine; 1025 else 1026 requestMethod = to!RequestMethod(arg.toUpper()); 1027 } else if(lookingForUri) { 1028 lookingForUri = false; 1029 1030 requestUri = arg; 1031 1032 auto idx = arg.indexOf("?"); 1033 if(idx == -1) 1034 pathInfo = arg; 1035 else { 1036 pathInfo = arg[0 .. idx]; 1037 _queryString = arg[idx + 1 .. $]; 1038 } 1039 } else { 1040 // it is an argument of some sort 1041 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1042 auto parts = breakUp(arg); 1043 _post[parts[0]] ~= parts[1]; 1044 allPostNamesInOrder ~= parts[0]; 1045 allPostValuesInOrder ~= parts[1]; 1046 } else { 1047 if(_queryString.length) 1048 _queryString ~= "&"; 1049 auto parts = breakUp(arg); 1050 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1051 } 1052 } 1053 } 1054 1055 acceptsGzip = false; 1056 keepAliveRequested = false; 1057 requestHeaders = cast(immutable) _headers; 1058 1059 cookie = _cookie; 1060 cookiesArray = getCookieArray(); 1061 cookies = keepLastOf(cookiesArray); 1062 1063 queryString = _queryString; 1064 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1065 get = keepLastOf(getArray); 1066 1067 postArray = cast(immutable) _post; 1068 post = keepLastOf(_post); 1069 1070 // FIXME 1071 filesArray = null; 1072 files = null; 1073 1074 isCalledWithCommandLineArguments = true; 1075 1076 this.port = port; 1077 this.referrer = referrer; 1078 this.remoteAddress = remoteAddress; 1079 this.userAgent = userAgent; 1080 this.authorization = authorization; 1081 this.origin = origin; 1082 this.accept = accept; 1083 this.lastEventId = lastEventId; 1084 this.https = https; 1085 this.host = host; 1086 this.requestMethod = requestMethod; 1087 this.requestUri = requestUri; 1088 this.pathInfo = pathInfo; 1089 this.queryString = queryString; 1090 this.postBody = null; 1091 } 1092 1093 private { 1094 string[] allPostNamesInOrder; 1095 string[] allPostValuesInOrder; 1096 string[] allGetNamesInOrder; 1097 string[] allGetValuesInOrder; 1098 } 1099 1100 CgiConnectionHandle getOutputFileHandle() { 1101 return _outputFileHandle; 1102 } 1103 1104 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1105 1106 /** Initializes it using a CGI or CGI-like interface */ 1107 this(long maxContentLength = defaultMaxContentLength, 1108 // use this to override the environment variable listing 1109 in string[string] env = null, 1110 // and this should return a chunk of data. return empty when done 1111 const(ubyte)[] delegate() readdata = null, 1112 // finally, use this to do custom output if needed 1113 void delegate(const(ubyte)[]) _rawDataOutput = null, 1114 // to flush teh custom output 1115 void delegate() _flush = null 1116 ) 1117 { 1118 1119 // these are all set locally so the loop works 1120 // without triggering errors in dmd 2.064 1121 // we go ahead and set them at the end of it to the this version 1122 int port; 1123 string referrer; 1124 string remoteAddress; 1125 string userAgent; 1126 string authorization; 1127 string origin; 1128 string accept; 1129 string lastEventId; 1130 bool https; 1131 string host; 1132 RequestMethod requestMethod; 1133 string requestUri; 1134 string pathInfo; 1135 string queryString; 1136 1137 1138 1139 isCalledWithCommandLineArguments = false; 1140 rawDataOutput = _rawDataOutput; 1141 flushDelegate = _flush; 1142 auto getenv = delegate string(string var) { 1143 if(env is null) 1144 return std.process.environment.get(var); 1145 auto e = var in env; 1146 if(e is null) 1147 return null; 1148 return *e; 1149 }; 1150 1151 environmentVariables = env is null ? 1152 cast(const) environment.toAA : 1153 env; 1154 1155 // fetching all the request headers 1156 string[string] requestHeadersHere; 1157 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1158 if(k.startsWith("HTTP_")) { 1159 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1160 } 1161 } 1162 1163 this.requestHeaders = assumeUnique(requestHeadersHere); 1164 1165 requestUri = getenv("REQUEST_URI"); 1166 1167 cookie = getenv("HTTP_COOKIE"); 1168 cookiesArray = getCookieArray(); 1169 cookies = keepLastOf(cookiesArray); 1170 1171 referrer = getenv("HTTP_REFERER"); 1172 userAgent = getenv("HTTP_USER_AGENT"); 1173 remoteAddress = getenv("REMOTE_ADDR"); 1174 host = getenv("HTTP_HOST"); 1175 pathInfo = getenv("PATH_INFO"); 1176 1177 queryString = getenv("QUERY_STRING"); 1178 scriptName = getenv("SCRIPT_NAME"); 1179 { 1180 import core.runtime; 1181 auto sfn = getenv("SCRIPT_FILENAME"); 1182 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1183 } 1184 1185 bool iis = false; 1186 1187 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1188 if(requestUri.length == 0) { 1189 // IIS sometimes includes the script name as part of the path info - we don't want that 1190 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1191 pathInfo = pathInfo[scriptName.length .. $]; 1192 1193 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1194 1195 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1196 1197 // FIXME: this works for apache and iis... but what about others? 1198 } 1199 1200 1201 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1202 getArray = assumeUnique(ugh); 1203 get = keepLastOf(getArray); 1204 1205 1206 // NOTE: on shitpache, you need to specifically forward this 1207 authorization = getenv("HTTP_AUTHORIZATION"); 1208 // this is a hack because Apache is a shitload of fuck and 1209 // refuses to send the real header to us. Compatible 1210 // programs should send both the standard and X- versions 1211 1212 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1213 // unnecessary with mod_rewrite, so it is commented 1214 1215 //if(authorization.length == 0) // if the std is there, use it 1216 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1217 1218 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1219 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1220 port = to!int(getenv("SERVER_PORT")); 1221 else 1222 port = 0; // this was probably called from the command line 1223 1224 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1225 if(ae.length && ae.indexOf("gzip") != -1) 1226 acceptsGzip = true; 1227 1228 accept = getenv("HTTP_ACCEPT"); 1229 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1230 1231 auto ka = getenv("HTTP_CONNECTION"); 1232 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1233 keepAliveRequested = true; 1234 1235 auto or = getenv("HTTP_ORIGIN"); 1236 origin = or; 1237 1238 auto rm = getenv("REQUEST_METHOD"); 1239 if(rm.length) 1240 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1241 else 1242 requestMethod = RequestMethod.CommandLine; 1243 1244 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1245 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1246 1247 // FIXME: DOCUMENT_ROOT? 1248 1249 // FIXME: what about PUT? 1250 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1251 version(preserveData) // a hack to make forwarding simpler 1252 immutable(ubyte)[] data; 1253 size_t amountReceived = 0; 1254 auto contentType = getenv("CONTENT_TYPE"); 1255 1256 // FIXME: is this ever not going to be set? I guess it depends 1257 // on if the server de-chunks and buffers... seems like it has potential 1258 // to be slow if they did that. The spec says it is always there though. 1259 // And it has worked reliably for me all year in the live environment, 1260 // but some servers might be different. 1261 auto cls = getenv("CONTENT_LENGTH"); 1262 auto contentLength = to!size_t(cls.length ? cls : "0"); 1263 1264 immutable originalContentLength = contentLength; 1265 if(contentLength) { 1266 if(maxContentLength > 0 && contentLength > maxContentLength) { 1267 setResponseStatus("413 Request entity too large"); 1268 write("You tried to upload a file that is too large."); 1269 close(); 1270 throw new Exception("POST too large"); 1271 } 1272 prepareForIncomingDataChunks(contentType, contentLength); 1273 1274 1275 int processChunk(in ubyte[] chunk) { 1276 if(chunk.length > contentLength) { 1277 handleIncomingDataChunk(chunk[0..contentLength]); 1278 amountReceived += contentLength; 1279 contentLength = 0; 1280 return 1; 1281 } else { 1282 handleIncomingDataChunk(chunk); 1283 contentLength -= chunk.length; 1284 amountReceived += chunk.length; 1285 } 1286 if(contentLength == 0) 1287 return 1; 1288 1289 onRequestBodyDataReceived(amountReceived, originalContentLength); 1290 return 0; 1291 } 1292 1293 1294 if(readdata is null) { 1295 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1296 if(processChunk(chunk)) 1297 break; 1298 } else { 1299 // we have a custom data source.. 1300 auto chunk = readdata(); 1301 while(chunk.length) { 1302 if(processChunk(chunk)) 1303 break; 1304 chunk = readdata(); 1305 } 1306 } 1307 1308 onRequestBodyDataReceived(amountReceived, originalContentLength); 1309 postArray = assumeUnique(pps._post); 1310 filesArray = assumeUnique(pps._files); 1311 files = keepLastOf(filesArray); 1312 post = keepLastOf(postArray); 1313 this.postBody = pps.postBody; 1314 cleanUpPostDataState(); 1315 } 1316 1317 version(preserveData) 1318 originalPostData = data; 1319 } 1320 // fixme: remote_user script name 1321 1322 1323 this.port = port; 1324 this.referrer = referrer; 1325 this.remoteAddress = remoteAddress; 1326 this.userAgent = userAgent; 1327 this.authorization = authorization; 1328 this.origin = origin; 1329 this.accept = accept; 1330 this.lastEventId = lastEventId; 1331 this.https = https; 1332 this.host = host; 1333 this.requestMethod = requestMethod; 1334 this.requestUri = requestUri; 1335 this.pathInfo = pathInfo; 1336 this.queryString = queryString; 1337 } 1338 1339 /// Cleans up any temporary files. Do not use the object 1340 /// after calling this. 1341 /// 1342 /// NOTE: it is called automatically by GenericMain 1343 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1344 void dispose() { 1345 foreach(file; files) { 1346 if(!file.contentInMemory) 1347 if(std.file.exists(file.contentFilename)) 1348 std.file.remove(file.contentFilename); 1349 } 1350 } 1351 1352 private { 1353 struct PostParserState { 1354 string contentType; 1355 string boundary; 1356 string localBoundary; // the ones used at the end or something lol 1357 bool isMultipart; 1358 bool needsSavedBody; 1359 1360 ulong expectedLength; 1361 ulong contentConsumed; 1362 immutable(ubyte)[] buffer; 1363 1364 // multipart parsing state 1365 int whatDoWeWant; 1366 bool weHaveAPart; 1367 string[] thisOnesHeaders; 1368 immutable(ubyte)[] thisOnesData; 1369 1370 string postBody; 1371 1372 UploadedFile piece; 1373 bool isFile = false; 1374 1375 size_t memoryCommitted; 1376 1377 // do NOT keep mutable references to these anywhere! 1378 // I assume they are unique in the constructor once we're all done getting data. 1379 string[][string] _post; 1380 UploadedFile[][string] _files; 1381 } 1382 1383 PostParserState pps; 1384 } 1385 1386 /// This represents a file the user uploaded via a POST request. 1387 static struct UploadedFile { 1388 /// If you want to create one of these structs for yourself from some data, 1389 /// use this function. 1390 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1391 Cgi.UploadedFile f; 1392 f.filename = name; 1393 f.content = cast(immutable(ubyte)[]) data; 1394 f.contentInMemory = true; 1395 return f; 1396 } 1397 1398 string name; /// The name of the form element. 1399 string filename; /// The filename the user set. 1400 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1401 1402 /** 1403 For small files, cgi.d will buffer the uploaded file in memory, and make it 1404 directly accessible to you through the content member. I find this very convenient 1405 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1406 often want to inspect and modify the file anyway!) 1407 1408 I find the file is very large, it is undesirable to eat that much memory just 1409 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1410 to the constructor so they are accepted, cgi.d will write the content to a temporary 1411 file that you can re-read later. 1412 1413 You can override this behavior by subclassing Cgi and overriding the protected 1414 handlePostChunk method. Note that the object is not initialized when you 1415 write that method - the http headers are available, but the cgi.post method 1416 is not. You may parse the file as it streams in using this method. 1417 1418 1419 Anyway, if the file is small enough to be in memory, contentInMemory will be 1420 set to true, and the content is available in the content member. 1421 1422 If not, contentInMemory will be set to false, and the content saved in a file, 1423 whose name will be available in the contentFilename member. 1424 1425 1426 Tip: if you know you are always dealing with small files, and want the convenience 1427 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1428 a large file comes in, it simply throws an exception (and HTTP error response) 1429 instead of trying to handle it. 1430 1431 The default value of maxContentLength in the constructor is for small files. 1432 */ 1433 bool contentInMemory = true; // the default ought to always be true 1434 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1435 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1436 1437 /// 1438 ulong fileSize() const { 1439 if(contentInMemory) 1440 return content.length; 1441 import std.file; 1442 return std.file.getSize(contentFilename); 1443 1444 } 1445 1446 /// 1447 void writeToFile(string filenameToSaveTo) const { 1448 import std.file; 1449 if(contentInMemory) 1450 std.file.write(filenameToSaveTo, content); 1451 else 1452 std.file.rename(contentFilename, filenameToSaveTo); 1453 } 1454 } 1455 1456 // given a content type and length, decide what we're going to do with the data.. 1457 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1458 pps.expectedLength = contentLength; 1459 1460 auto terminator = contentType.indexOf(";"); 1461 if(terminator == -1) 1462 terminator = contentType.length; 1463 1464 pps.contentType = contentType[0 .. terminator]; 1465 auto b = contentType[terminator .. $]; 1466 if(b.length) { 1467 auto idx = b.indexOf("boundary="); 1468 if(idx != -1) { 1469 pps.boundary = b[idx + "boundary=".length .. $]; 1470 pps.localBoundary = "\r\n--" ~ pps.boundary; 1471 } 1472 } 1473 1474 // while a content type SHOULD be sent according to the RFC, it is 1475 // not required. We're told we SHOULD guess by looking at the content 1476 // but it seems to me that this only happens when it is urlencoded. 1477 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1478 pps.isMultipart = false; 1479 pps.needsSavedBody = false; 1480 } else if(pps.contentType == "multipart/form-data") { 1481 pps.isMultipart = true; 1482 enforce(pps.boundary.length, "no boundary"); 1483 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1484 // save the body so the application can handle it 1485 pps.isMultipart = false; 1486 pps.needsSavedBody = true; 1487 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1488 // save the body so the application can handle it 1489 pps.needsSavedBody = true; 1490 pps.isMultipart = false; 1491 } else { 1492 // the rest is 100% handled by the application. just save the body and send it to them 1493 pps.needsSavedBody = true; 1494 pps.isMultipart = false; 1495 } 1496 } 1497 1498 // handles streaming POST data. If you handle some other content type, you should 1499 // override this. If the data isn't the content type you want, you ought to call 1500 // super.handleIncomingDataChunk so regular forms and files still work. 1501 1502 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1503 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1504 // input anyway, so I'm not going to get too worked up about it right now. 1505 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1506 if(chunk.length == 0) 1507 return; 1508 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1509 // if we're passed big chunks, it might throw unnecessarily. 1510 // just pass it smaller chunks at a time. 1511 if(pps.isMultipart) { 1512 // multipart/form-data 1513 1514 1515 // FIXME: this might want to be factored out and factorized 1516 // need to make sure the stream hooks actually work. 1517 void pieceHasNewContent() { 1518 // we just grew the piece's buffer. Do we have to switch to file backing? 1519 if(pps.piece.contentInMemory) { 1520 if(pps.piece.content.length <= 10 * 1024 * 1024) 1521 // meh, I'm ok with it. 1522 return; 1523 else { 1524 // this is too big. 1525 if(!pps.isFile) 1526 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1527 else { 1528 // a file this large is probably acceptable though... let's use a backing file. 1529 pps.piece.contentInMemory = false; 1530 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1531 1532 int count = 0; 1533 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1534 // odds are this loop will never be entered, but we want it just in case. 1535 while(std.file.exists(pps.piece.contentFilename)) { 1536 count++; 1537 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1538 } 1539 // I hope this creates the file pretty quickly, or the loop might be useless... 1540 // FIXME: maybe I should write some kind of custom transaction here. 1541 std.file.write(pps.piece.contentFilename, pps.piece.content); 1542 1543 pps.piece.content = null; 1544 } 1545 } 1546 } else { 1547 // it's already in a file, so just append it to what we have 1548 if(pps.piece.content.length) { 1549 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1550 std.file.append(pps.piece.contentFilename, pps.piece.content); 1551 pps.piece.content = null; 1552 } 1553 } 1554 } 1555 1556 1557 void commitPart() { 1558 if(!pps.weHaveAPart) 1559 return; 1560 1561 pieceHasNewContent(); // be sure the new content is handled every time 1562 1563 if(pps.isFile) { 1564 // I'm not sure if other environments put files in post or not... 1565 // I used to not do it, but I think I should, since it is there... 1566 pps._post[pps.piece.name] ~= pps.piece.filename; 1567 pps._files[pps.piece.name] ~= pps.piece; 1568 1569 allPostNamesInOrder ~= pps.piece.name; 1570 allPostValuesInOrder ~= pps.piece.filename; 1571 } else { 1572 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1573 1574 allPostNamesInOrder ~= pps.piece.name; 1575 allPostValuesInOrder ~= cast(string) pps.piece.content; 1576 } 1577 1578 /* 1579 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1580 pps.piece.content.length < 1000 1581 ? 1582 to!string(pps.piece.content) 1583 : 1584 "too long"); 1585 */ 1586 1587 // FIXME: the limit here 1588 pps.memoryCommitted += pps.piece.content.length; 1589 1590 pps.weHaveAPart = false; 1591 pps.whatDoWeWant = 1; 1592 pps.thisOnesHeaders = null; 1593 pps.thisOnesData = null; 1594 1595 pps.piece = UploadedFile.init; 1596 pps.isFile = false; 1597 } 1598 1599 void acceptChunk() { 1600 pps.buffer ~= chunk; 1601 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1602 } 1603 1604 immutable(ubyte)[] consume(size_t howMuch) { 1605 pps.contentConsumed += howMuch; 1606 auto ret = pps.buffer[0 .. howMuch]; 1607 pps.buffer = pps.buffer[howMuch .. $]; 1608 return ret; 1609 } 1610 1611 dataConsumptionLoop: do { 1612 switch(pps.whatDoWeWant) { 1613 default: assert(0); 1614 case 0: 1615 acceptChunk(); 1616 // the format begins with two extra leading dashes, then we should be at the boundary 1617 if(pps.buffer.length < 2) 1618 return; 1619 assert(pps.buffer[0] == '-', "no leading dash"); 1620 consume(1); 1621 assert(pps.buffer[0] == '-', "no second leading dash"); 1622 consume(1); 1623 1624 pps.whatDoWeWant = 1; 1625 goto case 1; 1626 /* fallthrough */ 1627 case 1: // looking for headers 1628 // here, we should be lined up right at the boundary, which is followed by a \r\n 1629 1630 // want to keep the buffer under control in case we're under attack 1631 //stderr.writeln("here once"); 1632 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1633 // throw new Exception("wtf is up with the huge mime part headers"); 1634 1635 acceptChunk(); 1636 1637 if(pps.buffer.length < pps.boundary.length) 1638 return; // not enough data, since there should always be a boundary here at least 1639 1640 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1641 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1642 // we *should* be at the end here! 1643 assert(pps.buffer[0] == '-'); 1644 consume(1); 1645 assert(pps.buffer[0] == '-'); 1646 consume(1); 1647 1648 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1649 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1650 "not lined up on boundary " ~ pps.boundary); 1651 consume(pps.boundary.length); 1652 1653 assert(pps.buffer[0] == '-'); 1654 consume(1); 1655 assert(pps.buffer[0] == '-'); 1656 consume(1); 1657 1658 assert(pps.buffer[0] == '\r'); 1659 consume(1); 1660 assert(pps.buffer[0] == '\n'); 1661 consume(1); 1662 1663 assert(pps.buffer.length == 0); 1664 assert(pps.contentConsumed == pps.expectedLength); 1665 break dataConsumptionLoop; // we're done! 1666 } else { 1667 // we're not done yet. We should be lined up on a boundary. 1668 1669 // But, we want to ensure the headers are here before we consume anything! 1670 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1671 if(headerEndLocation == -1) 1672 return; // they *should* all be here, so we can handle them all at once. 1673 1674 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1675 "not lined up on boundary " ~ pps.boundary); 1676 1677 consume(pps.boundary.length); 1678 // the boundary is always followed by a \r\n 1679 assert(pps.buffer[0] == '\r'); 1680 consume(1); 1681 assert(pps.buffer[0] == '\n'); 1682 consume(1); 1683 } 1684 1685 // re-running since by consuming the boundary, we invalidate the old index. 1686 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1687 assert(headerEndLocation >= 0, "no header"); 1688 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1689 1690 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1691 1692 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1693 1694 // now we'll parse the headers 1695 foreach(h; pps.thisOnesHeaders) { 1696 auto p = h.indexOf(":"); 1697 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1698 string hn = h[0..p]; 1699 string hv = h[p+2..$]; 1700 1701 switch(hn.toLower) { 1702 default: assert(0); 1703 case "content-disposition": 1704 auto info = hv.split("; "); 1705 foreach(i; info[1..$]) { // skipping the form-data 1706 auto o = i.split("="); // FIXME 1707 string pn = o[0]; 1708 string pv = o[1][1..$-1]; 1709 1710 if(pn == "name") { 1711 pps.piece.name = pv; 1712 } else if (pn == "filename") { 1713 pps.piece.filename = pv; 1714 pps.isFile = true; 1715 } 1716 } 1717 break; 1718 case "content-type": 1719 pps.piece.contentType = hv; 1720 break; 1721 } 1722 } 1723 1724 pps.whatDoWeWant++; // move to the next step - the data 1725 break; 1726 case 2: 1727 // when we get here, pps.buffer should contain our first chunk of data 1728 1729 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1730 throw new Exception("wtf is up with the huge mime part buffer"); 1731 1732 acceptChunk(); 1733 1734 // so the trick is, we want to process all the data up to the boundary, 1735 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1736 // want to wait for the next chunk. We start by looking for the whole boundary 1737 // in the buffer somewhere. 1738 1739 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1740 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1741 if(boundaryLocation != -1) { 1742 // this is easy - we can see it in it's entirety! 1743 1744 pps.piece.content ~= consume(boundaryLocation); 1745 1746 assert(pps.buffer[0] == '\r'); 1747 consume(1); 1748 assert(pps.buffer[0] == '\n'); 1749 consume(1); 1750 assert(pps.buffer[0] == '-'); 1751 consume(1); 1752 assert(pps.buffer[0] == '-'); 1753 consume(1); 1754 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1755 pps.weHaveAPart = true; 1756 pps.whatDoWeWant = 1; // back to getting headers for the next part 1757 1758 commitPart(); // we're done here 1759 } else { 1760 // we can't see the whole thing, but what if there's a partial boundary? 1761 1762 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1763 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1764 bool potentialBoundaryFound = false; 1765 1766 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1767 // we grow the boundary a bit each time. If we think it looks the 1768 // same, better pull another chunk to be sure it's not the end. 1769 // Starting small because exiting the loop early is desirable, since 1770 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1771 // the best we can do. 1772 if(a > pps.buffer.length) 1773 break; // FIXME: is this right? 1774 assert(a <= pps.buffer.length); 1775 assert(a > 0); 1776 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1777 // ok, there *might* be a boundary here, so let's 1778 // not treat the end as data yet. The rest is good to 1779 // use though, since if there was a boundary there, we'd 1780 // have handled it up above after locationOf. 1781 1782 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1783 consume(pps.buffer.length - a); 1784 pieceHasNewContent(); 1785 potentialBoundaryFound = true; 1786 break boundaryCheck; 1787 } 1788 } 1789 1790 if(!potentialBoundaryFound) { 1791 // we can consume the whole thing 1792 pps.piece.content ~= pps.buffer; 1793 pieceHasNewContent(); 1794 consume(pps.buffer.length); 1795 } else { 1796 // we found a possible boundary, but there was 1797 // insufficient data to be sure. 1798 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1799 1800 return; // wait for the next chunk. 1801 } 1802 } 1803 } 1804 } while(pps.buffer.length); 1805 1806 // btw all boundaries except the first should have a \r\n before them 1807 } else { 1808 // application/x-www-form-urlencoded and application/json 1809 1810 // not using maxContentLength because that might be cranked up to allow 1811 // large file uploads. We can handle them, but a huge post[] isn't any good. 1812 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1813 throw new Exception("wtf is up with such a gigantic form submission????"); 1814 1815 pps.buffer ~= chunk; 1816 1817 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1818 if(pps.buffer.length == pps.expectedLength) { 1819 if(pps.needsSavedBody) 1820 pps.postBody = cast(string) pps.buffer; 1821 else 1822 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1823 version(preserveData) 1824 originalPostData = pps.buffer; 1825 } else { 1826 // just for debugging 1827 } 1828 } 1829 } 1830 1831 protected void cleanUpPostDataState() { 1832 pps = PostParserState.init; 1833 } 1834 1835 /// you can override this function to somehow react 1836 /// to an upload in progress. 1837 /// 1838 /// Take note that parts of the CGI object is not yet 1839 /// initialized! Stuff from HTTP headers, including get[], is usable. 1840 /// But, none of post[] is usable, and you cannot write here. That's 1841 /// why this method is const - mutating the object won't do much anyway. 1842 /// 1843 /// My idea here was so you can output a progress bar or 1844 /// something to a cooperative client (see arsd.rtud for a potential helper) 1845 /// 1846 /// The default is to do nothing. Subclass cgi and use the 1847 /// CustomCgiMain mixin to do something here. 1848 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1849 // This space intentionally left blank. 1850 } 1851 1852 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1853 /// *closeConnection will be set to true if you should close the connection after handling this request 1854 this(BufferedInputRange ir, bool* closeConnection) { 1855 isCalledWithCommandLineArguments = false; 1856 import al = std.algorithm; 1857 1858 immutable(ubyte)[] data; 1859 1860 void rdo(const(ubyte)[] d) { 1861 //import std.stdio; writeln(d); 1862 sendAll(ir.source, d); 1863 } 1864 1865 auto ira = ir.source.remoteAddress(); 1866 auto irLocalAddress = ir.source.localAddress(); 1867 1868 ushort port = 80; 1869 if(auto ia = cast(InternetAddress) irLocalAddress) { 1870 port = ia.port; 1871 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1872 port = ia.port; 1873 } 1874 1875 // that check for UnixAddress is to work around a Phobos bug 1876 // see: https://github.com/dlang/phobos/pull/7383 1877 // but this might be more useful anyway tbh for this case 1878 version(Posix) 1879 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1880 else 1881 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1882 } 1883 1884 /** 1885 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1886 1887 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1888 1889 Params: 1890 inputData = the incoming data, including headers and other raw http data. 1891 When the constructor exits, it will leave this range exactly at the start of 1892 the next request on the connection (if there is one). 1893 1894 address = the IP address of the remote user 1895 _port = the port number of the connection 1896 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1897 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1898 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1899 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1900 closeConnection = if the request asks to close the connection, *closeConnection == true. 1901 */ 1902 this( 1903 BufferedInputRange inputData, 1904 // string[] headers, immutable(ubyte)[] data, 1905 string address, ushort _port, 1906 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1907 bool _https = false, 1908 void delegate(const(ubyte)[]) _rawDataOutput = null, 1909 void delegate() _flush = null, 1910 // this pointer tells if the connection is supposed to be closed after we handle this 1911 bool* closeConnection = null) 1912 { 1913 // these are all set locally so the loop works 1914 // without triggering errors in dmd 2.064 1915 // we go ahead and set them at the end of it to the this version 1916 int port; 1917 string referrer; 1918 string remoteAddress; 1919 string userAgent; 1920 string authorization; 1921 string origin; 1922 string accept; 1923 string lastEventId; 1924 bool https; 1925 string host; 1926 RequestMethod requestMethod; 1927 string requestUri; 1928 string pathInfo; 1929 string queryString; 1930 string scriptName; 1931 string[string] get; 1932 string[][string] getArray; 1933 bool keepAliveRequested; 1934 bool acceptsGzip; 1935 string cookie; 1936 1937 1938 1939 environmentVariables = cast(const) environment.toAA; 1940 1941 idlol = inputData; 1942 1943 isCalledWithCommandLineArguments = false; 1944 1945 https = _https; 1946 port = _port; 1947 1948 rawDataOutput = _rawDataOutput; 1949 flushDelegate = _flush; 1950 nph = true; 1951 1952 remoteAddress = address; 1953 1954 // streaming parser 1955 import al = std.algorithm; 1956 1957 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1958 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1959 while(idx == -1) { 1960 inputData.popFront(0); 1961 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1962 } 1963 1964 assert(idx != -1); 1965 1966 1967 string contentType = ""; 1968 string[string] requestHeadersHere; 1969 1970 size_t contentLength; 1971 1972 bool isChunked; 1973 1974 { 1975 import core.runtime; 1976 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 1977 } 1978 1979 1980 int headerNumber = 0; 1981 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 1982 if(line.length) { 1983 headerNumber++; 1984 auto header = cast(string) line.idup; 1985 if(headerNumber == 1) { 1986 // request line 1987 auto parts = al.splitter(header, " "); 1988 if(parts.front == "PRI") { 1989 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 1990 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 1991 // to bring me benefit) 1992 throw new HttpVersionNotSupportedException(); 1993 } 1994 requestMethod = to!RequestMethod(parts.front); 1995 parts.popFront(); 1996 requestUri = parts.front; 1997 1998 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 1999 scriptName = requestUri[0 .. pathInfoStarts]; 2000 2001 auto question = requestUri.indexOf("?"); 2002 if(question == -1) { 2003 queryString = ""; 2004 // FIXME: double check, this might be wrong since it could be url encoded 2005 pathInfo = requestUri[pathInfoStarts..$]; 2006 } else { 2007 queryString = requestUri[question+1..$]; 2008 pathInfo = requestUri[pathInfoStarts..question]; 2009 } 2010 2011 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2012 getArray = cast(string[][string]) assumeUnique(ugh); 2013 2014 if(header.indexOf("HTTP/1.0") != -1) { 2015 http10 = true; 2016 autoBuffer = true; 2017 if(closeConnection) { 2018 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2019 *closeConnection = true; 2020 } 2021 } 2022 } else { 2023 // other header 2024 auto colon = header.indexOf(":"); 2025 if(colon == -1) 2026 throw new Exception("HTTP headers should have a colon!"); 2027 string name = header[0..colon].toLower; 2028 string value = header[colon+2..$]; // skip the colon and the space 2029 2030 requestHeadersHere[name] = value; 2031 2032 if (name == "accept") { 2033 accept = value; 2034 } 2035 else if (name == "origin") { 2036 origin = value; 2037 } 2038 else if (name == "connection") { 2039 if(value == "close" && closeConnection) 2040 *closeConnection = true; 2041 if(value.asLowerCase().canFind("keep-alive")) { 2042 keepAliveRequested = true; 2043 2044 // on http 1.0, the connection is closed by default, 2045 // but not if they request keep-alive. then we don't close 2046 // anymore - undoing the set above 2047 if(http10 && closeConnection) { 2048 *closeConnection = false; 2049 } 2050 } 2051 } 2052 else if (name == "transfer-encoding") { 2053 if(value == "chunked") 2054 isChunked = true; 2055 } 2056 else if (name == "last-event-id") { 2057 lastEventId = value; 2058 } 2059 else if (name == "authorization") { 2060 authorization = value; 2061 } 2062 else if (name == "content-type") { 2063 contentType = value; 2064 } 2065 else if (name == "content-length") { 2066 contentLength = to!size_t(value); 2067 } 2068 else if (name == "x-forwarded-for") { 2069 remoteAddress = value; 2070 } 2071 else if (name == "x-forwarded-host" || name == "host") { 2072 if(name != "host" || host is null) 2073 host = value; 2074 } 2075 // FIXME: https://tools.ietf.org/html/rfc7239 2076 else if (name == "accept-encoding") { 2077 if(value.indexOf("gzip") != -1) 2078 acceptsGzip = true; 2079 } 2080 else if (name == "user-agent") { 2081 userAgent = value; 2082 } 2083 else if (name == "referer") { 2084 referrer = value; 2085 } 2086 else if (name == "cookie") { 2087 cookie ~= value; 2088 } else if(name == "expect") { 2089 if(value == "100-continue") { 2090 // FIXME we should probably give user code a chance 2091 // to process and reject but that needs to be virtual, 2092 // perhaps part of the CGI redesign. 2093 2094 // FIXME: if size is > max content length it should 2095 // also fail at this point. 2096 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2097 2098 // FIXME: let the user write out 103 early hints too 2099 } 2100 } 2101 // else 2102 // ignore it 2103 2104 } 2105 } 2106 2107 inputData.consume(idx + 4); 2108 // done 2109 2110 requestHeaders = assumeUnique(requestHeadersHere); 2111 2112 ByChunkRange dataByChunk; 2113 2114 // reading Content-Length type data 2115 // We need to read up the data we have, and write it out as a chunk. 2116 if(!isChunked) { 2117 dataByChunk = byChunk(inputData, contentLength); 2118 } else { 2119 // chunked requests happen, but not every day. Since we need to know 2120 // the content length (for now, maybe that should change), we'll buffer 2121 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2122 auto data = dechunk(inputData); 2123 2124 // set the range here 2125 dataByChunk = byChunk(data); 2126 contentLength = data.length; 2127 } 2128 2129 assert(dataByChunk !is null); 2130 2131 if(contentLength) { 2132 prepareForIncomingDataChunks(contentType, contentLength); 2133 foreach(dataChunk; dataByChunk) { 2134 handleIncomingDataChunk(dataChunk); 2135 } 2136 postArray = assumeUnique(pps._post); 2137 filesArray = assumeUnique(pps._files); 2138 files = keepLastOf(filesArray); 2139 post = keepLastOf(postArray); 2140 postBody = pps.postBody; 2141 cleanUpPostDataState(); 2142 } 2143 2144 this.port = port; 2145 this.referrer = referrer; 2146 this.remoteAddress = remoteAddress; 2147 this.userAgent = userAgent; 2148 this.authorization = authorization; 2149 this.origin = origin; 2150 this.accept = accept; 2151 this.lastEventId = lastEventId; 2152 this.https = https; 2153 this.host = host; 2154 this.requestMethod = requestMethod; 2155 this.requestUri = requestUri; 2156 this.pathInfo = pathInfo; 2157 this.queryString = queryString; 2158 2159 this.scriptName = scriptName; 2160 this.get = keepLastOf(getArray); 2161 this.getArray = cast(immutable) getArray; 2162 this.keepAliveRequested = keepAliveRequested; 2163 this.acceptsGzip = acceptsGzip; 2164 this.cookie = cookie; 2165 2166 cookiesArray = getCookieArray(); 2167 cookies = keepLastOf(cookiesArray); 2168 2169 } 2170 BufferedInputRange idlol; 2171 2172 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2173 string[string] ca; 2174 foreach(k, v; arr) 2175 ca[k] = v[$-1]; 2176 2177 return assumeUnique(ca); 2178 } 2179 2180 // FIXME duplication 2181 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2182 UploadedFile[string] ca; 2183 foreach(k, v; arr) 2184 ca[k] = v[$-1]; 2185 2186 return assumeUnique(ca); 2187 } 2188 2189 2190 private immutable(string[][string]) getCookieArray() { 2191 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2192 return assumeUnique(forTheLoveOfGod); 2193 } 2194 2195 /// Very simple method to require a basic auth username and password. 2196 /// If the http request doesn't include the required credentials, it throws a 2197 /// HTTP 401 error, and an exception. 2198 /// 2199 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 2200 /// the user's credentials are sent in plain text on every request. 2201 /// 2202 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2203 /// application. Either use Apache's built in methods for basic authentication, or add 2204 /// something along these lines to your server configuration: 2205 /// 2206 /// RewriteEngine On 2207 /// RewriteCond %{HTTP:Authorization} ^(.*) 2208 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2209 /// 2210 /// To ensure the necessary data is available to cgi.d. 2211 void requireBasicAuth(string user, string pass, string message = null) { 2212 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2213 setResponseStatus("401 Authorization Required"); 2214 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 2215 close(); 2216 throw new Exception("Not authorized; got " ~ authorization); 2217 } 2218 } 2219 2220 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2221 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2222 /// Use setResponseExpires and updateResponseExpires for more control 2223 void setCache(bool allowCaching) { 2224 noCache = !allowCaching; 2225 } 2226 2227 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2228 /// who can accept it 2229 bool gzipResponse; 2230 2231 immutable bool acceptsGzip; 2232 immutable bool keepAliveRequested; 2233 2234 /// Set to true if and only if this was initialized with command line arguments 2235 immutable bool isCalledWithCommandLineArguments; 2236 2237 /// This gets a full url for the current request, including port, protocol, host, path, and query 2238 string getCurrentCompleteUri() const { 2239 ushort defaultPort = https ? 443 : 80; 2240 2241 string uri = "http"; 2242 if(https) 2243 uri ~= "s"; 2244 uri ~= "://"; 2245 uri ~= host; 2246 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2247 version(none) 2248 if(!(!port || port == defaultPort)) { 2249 uri ~= ":"; 2250 uri ~= to!string(port); 2251 } 2252 +/ 2253 uri ~= requestUri; 2254 return uri; 2255 } 2256 2257 /// You can override this if your site base url isn't the same as the script name 2258 string logicalScriptName() const { 2259 return scriptName; 2260 } 2261 2262 /++ 2263 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2264 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2265 Note setResponseStatus() must be called *before* you write() any data to the output. 2266 2267 History: 2268 The `int` overload was added on January 11, 2021. 2269 +/ 2270 void setResponseStatus(string status) { 2271 assert(!outputtedResponseData); 2272 responseStatus = status; 2273 } 2274 /// ditto 2275 void setResponseStatus(int statusCode) { 2276 setResponseStatus(getHttpCodeText(statusCode)); 2277 } 2278 private string responseStatus = null; 2279 2280 /// Returns true if it is still possible to output headers 2281 bool canOutputHeaders() { 2282 return !isClosed && !outputtedResponseData; 2283 } 2284 2285 /// Sets the location header, which the browser will redirect the user to automatically. 2286 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2287 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2288 void setResponseLocation(string uri, bool important = true, string status = null) { 2289 if(!important && isCurrentResponseLocationImportant) 2290 return; // important redirects always override unimportant ones 2291 2292 if(uri is null) { 2293 responseStatus = "200 OK"; 2294 responseLocation = null; 2295 isCurrentResponseLocationImportant = important; 2296 return; // this just cancels the redirect 2297 } 2298 2299 assert(!outputtedResponseData); 2300 if(status is null) 2301 responseStatus = "302 Found"; 2302 else 2303 responseStatus = status; 2304 2305 responseLocation = uri.strip; 2306 isCurrentResponseLocationImportant = important; 2307 } 2308 protected string responseLocation = null; 2309 private bool isCurrentResponseLocationImportant = false; 2310 2311 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2312 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2313 /// Note: the when parameter is different than setCookie's expire parameter. 2314 void setResponseExpires(long when, bool isPublic = false) { 2315 responseExpires = when; 2316 setCache(true); // need to enable caching so the date has meaning 2317 2318 responseIsPublic = isPublic; 2319 responseExpiresRelative = false; 2320 } 2321 2322 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2323 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2324 responseExpires = whenFromNow; 2325 setCache(true); // need to enable caching so the date has meaning 2326 2327 responseIsPublic = isPublic; 2328 responseExpiresRelative = true; 2329 } 2330 private long responseExpires = long.min; 2331 private bool responseIsPublic = false; 2332 private bool responseExpiresRelative = false; 2333 2334 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2335 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2336 /// output as a whole is as cacheable as the least cachable part in the chain. 2337 2338 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2339 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2340 void updateResponseExpires(long when, bool isPublic) { 2341 if(responseExpires == long.min) 2342 setResponseExpires(when, isPublic); 2343 else if(when < responseExpires) 2344 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2345 } 2346 2347 /* 2348 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2349 /// Should generally be false if the user is logged in. It assumes private cache only. 2350 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2351 void setPublicCaching(bool allowPublicCaches) { 2352 publicCaching = allowPublicCaches; 2353 } 2354 private bool publicCaching = false; 2355 */ 2356 2357 /++ 2358 History: 2359 Added January 11, 2021 2360 +/ 2361 enum SameSitePolicy { 2362 Lax, 2363 Strict, 2364 None 2365 } 2366 2367 /++ 2368 Sets an HTTP cookie, automatically encoding the data to the correct string. 2369 expiresIn is how many milliseconds in the future the cookie will expire. 2370 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2371 Note setCookie() must be called *before* you write() any data to the output. 2372 2373 History: 2374 Parameter `sameSitePolicy` was added on January 11, 2021. 2375 +/ 2376 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2377 assert(!outputtedResponseData); 2378 string cookie = std.uri.encodeComponent(name) ~ "="; 2379 cookie ~= std.uri.encodeComponent(data); 2380 if(path !is null) 2381 cookie ~= "; path=" ~ path; 2382 // FIXME: should I just be using max-age here? (also in cache below) 2383 if(expiresIn != 0) 2384 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2385 if(domain !is null) 2386 cookie ~= "; domain=" ~ domain; 2387 if(secure == true) 2388 cookie ~= "; Secure"; 2389 if(httpOnly == true ) 2390 cookie ~= "; HttpOnly"; 2391 final switch(sameSitePolicy) { 2392 case SameSitePolicy.Lax: 2393 cookie ~= "; SameSite=Lax"; 2394 break; 2395 case SameSitePolicy.Strict: 2396 cookie ~= "; SameSite=Strict"; 2397 break; 2398 case SameSitePolicy.None: 2399 cookie ~= "; SameSite=None"; 2400 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2401 break; 2402 } 2403 2404 if(auto idx = name in cookieIndexes) { 2405 responseCookies[*idx] = cookie; 2406 } else { 2407 cookieIndexes[name] = responseCookies.length; 2408 responseCookies ~= cookie; 2409 } 2410 } 2411 private string[] responseCookies; 2412 private size_t[string] cookieIndexes; 2413 2414 /// Clears a previously set cookie with the given name, path, and domain. 2415 void clearCookie(string name, string path = null, string domain = null) { 2416 assert(!outputtedResponseData); 2417 setCookie(name, "", 1, path, domain); 2418 } 2419 2420 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2421 void setResponseContentType(string ct) { 2422 assert(!outputtedResponseData); 2423 responseContentType = ct; 2424 } 2425 private string responseContentType = null; 2426 2427 /// Adds a custom header. It should be the name: value, but without any line terminator. 2428 /// For example: header("X-My-Header: Some value"); 2429 /// Note you should use the specialized functions in this object if possible to avoid 2430 /// duplicates in the output. 2431 void header(string h) { 2432 customHeaders ~= h; 2433 } 2434 2435 /++ 2436 I named the original function `header` after PHP, but this pattern more fits 2437 the rest of the Cgi object. 2438 2439 Either name are allowed. 2440 2441 History: 2442 Alias added June 17, 2022. 2443 +/ 2444 alias setResponseHeader = header; 2445 2446 private string[] customHeaders; 2447 private bool websocketMode; 2448 2449 void flushHeaders(const(void)[] t, bool isAll = false) { 2450 StackBuffer buffer = StackBuffer(0); 2451 2452 prepHeaders(t, isAll, &buffer); 2453 2454 if(rawDataOutput !is null) 2455 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2456 else { 2457 stdout.rawWrite(buffer.get()); 2458 } 2459 } 2460 2461 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2462 string terminator = "\n"; 2463 if(rawDataOutput !is null) 2464 terminator = "\r\n"; 2465 2466 if(responseStatus !is null) { 2467 if(nph) { 2468 if(http10) 2469 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2470 else 2471 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2472 } else 2473 buffer.add("Status: ", responseStatus, terminator); 2474 } else if (nph) { 2475 if(http10) 2476 buffer.add("HTTP/1.0 200 OK", terminator); 2477 else 2478 buffer.add("HTTP/1.1 200 OK", terminator); 2479 } 2480 2481 if(websocketMode) 2482 goto websocket; 2483 2484 if(nph) { // we're responsible for setting the date too according to http 1.1 2485 char[29] db = void; 2486 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2487 buffer.add("Date: ", db[], terminator); 2488 } 2489 2490 // FIXME: what if the user wants to set his own content-length? 2491 // The custom header function can do it, so maybe that's best. 2492 // Or we could reuse the isAll param. 2493 if(responseLocation !is null) { 2494 buffer.add("Location: ", responseLocation, terminator); 2495 } 2496 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2497 if(responseExpiresRelative) { 2498 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2499 buffer.add(responseExpires); 2500 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2501 } else { 2502 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2503 char[29] db = void; 2504 printDateToBuffer(cast(DateTime) expires, db[]); 2505 buffer.add("Expires: ", db[], terminator); 2506 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2507 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2508 buffer.add(terminator); 2509 } 2510 } 2511 if(responseCookies !is null && responseCookies.length > 0) { 2512 foreach(c; responseCookies) 2513 buffer.add("Set-Cookie: ", c, terminator); 2514 } 2515 if(noCache) { // we specifically do not want caching (this is actually the default) 2516 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2517 buffer.add("Expires: 0", terminator); 2518 buffer.add("Pragma: no-cache", terminator); 2519 } else { 2520 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2521 buffer.add("Cache-Control: public", terminator); 2522 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2523 } 2524 } 2525 if(responseContentType !is null) { 2526 buffer.add("Content-Type: ", responseContentType, terminator); 2527 } else 2528 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2529 2530 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2531 buffer.add("Content-Encoding: gzip", terminator); 2532 } 2533 2534 2535 if(!isAll) { 2536 if(nph && !http10) { 2537 buffer.add("Transfer-Encoding: chunked", terminator); 2538 responseChunked = true; 2539 } 2540 } else { 2541 buffer.add("Content-Length: "); 2542 buffer.add(t.length); 2543 buffer.add(terminator); 2544 if(nph && keepAliveRequested) { 2545 buffer.add("Connection: Keep-Alive", terminator); 2546 } 2547 } 2548 2549 websocket: 2550 2551 foreach(hd; customHeaders) 2552 buffer.add(hd, terminator); 2553 2554 // FIXME: what about duplicated headers? 2555 2556 // end of header indicator 2557 buffer.add(terminator); 2558 2559 outputtedResponseData = true; 2560 } 2561 2562 /// Writes the data to the output, flushing headers if they have not yet been sent. 2563 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2564 assert(!closed, "Output has already been closed"); 2565 2566 StackBuffer buffer = StackBuffer(0); 2567 2568 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2569 // actually gzip the data here 2570 2571 auto c = new Compress(HeaderFormat.gzip); // want gzip 2572 2573 auto data = c.compress(t); 2574 data ~= c.flush(); 2575 2576 // std.file.write("/tmp/last-item", data); 2577 2578 t = data; 2579 } 2580 2581 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2582 prepHeaders(t, isAll, &buffer); 2583 } 2584 2585 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2586 if (autoBuffer && !isAll) { 2587 outputBuffer ~= cast(ubyte[]) t; 2588 } 2589 if(!autoBuffer || isAll) { 2590 if(rawDataOutput !is null) 2591 if(nph && responseChunked) { 2592 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2593 // we're making the chunk here instead of in a function 2594 // to avoid unneeded gc pressure 2595 buffer.add(toHex(t.length)); 2596 buffer.add("\r\n"); 2597 buffer.add(cast(char[]) t, "\r\n"); 2598 } else { 2599 buffer.add(cast(char[]) t); 2600 } 2601 else 2602 buffer.add(cast(char[]) t); 2603 } 2604 } 2605 2606 if(rawDataOutput !is null) 2607 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2608 else 2609 stdout.rawWrite(buffer.get()); 2610 2611 if(maybeAutoClose && isAll) 2612 close(); // if you say it is all, that means we're definitely done 2613 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2614 } 2615 2616 /++ 2617 Convenience method to set content type to json and write the string as the complete response. 2618 2619 History: 2620 Added January 16, 2020 2621 +/ 2622 void writeJson(string json) { 2623 this.setResponseContentType("application/json"); 2624 this.write(json, true); 2625 } 2626 2627 /// Flushes the pending buffer, leaving the connection open so you can send more. 2628 void flush() { 2629 if(rawDataOutput is null) 2630 stdout.flush(); 2631 else if(flushDelegate !is null) 2632 flushDelegate(); 2633 } 2634 2635 version(autoBuffer) 2636 bool autoBuffer = true; 2637 else 2638 bool autoBuffer = false; 2639 ubyte[] outputBuffer; 2640 2641 /// Flushes the buffers to the network, signifying that you are done. 2642 /// You should always call this explicitly when you are done outputting data. 2643 void close() { 2644 if(closed) 2645 return; // don't double close 2646 2647 if(!outputtedResponseData) 2648 write("", true, false); 2649 2650 // writing auto buffered data 2651 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2652 if(!nph) 2653 stdout.rawWrite(outputBuffer); 2654 else 2655 write(outputBuffer, true, false); // tell it this is everything 2656 } 2657 2658 // closing the last chunk... 2659 if(nph && rawDataOutput !is null && responseChunked) 2660 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2661 2662 if(flushDelegate) 2663 flushDelegate(); 2664 2665 closed = true; 2666 } 2667 2668 // Closes without doing anything, shouldn't be used often 2669 void rawClose() { 2670 closed = true; 2671 } 2672 2673 /++ 2674 Gets a request variable as a specific type, or the default value of it isn't there 2675 or isn't convertible to the request type. 2676 2677 Checks both GET and POST variables, preferring the POST variable, if available. 2678 2679 A nice trick is using the default value to choose the type: 2680 2681 --- 2682 /* 2683 The return value will match the type of the default. 2684 Here, I gave 10 as a default, so the return value will 2685 be an int. 2686 2687 If the user-supplied value cannot be converted to the 2688 requested type, you will get the default value back. 2689 */ 2690 int a = cgi.request("number", 10); 2691 2692 if(cgi.get["number"] == "11") 2693 assert(a == 11); // conversion succeeds 2694 2695 if("number" !in cgi.get) 2696 assert(a == 10); // no value means you can't convert - give the default 2697 2698 if(cgi.get["number"] == "twelve") 2699 assert(a == 10); // conversion from string to int would fail, so we get the default 2700 --- 2701 2702 You can use an enum as an easy whitelist, too: 2703 2704 --- 2705 enum Operations { 2706 add, remove, query 2707 } 2708 2709 auto op = cgi.request("op", Operations.query); 2710 2711 if(cgi.get["op"] == "add") 2712 assert(op == Operations.add); 2713 if(cgi.get["op"] == "remove") 2714 assert(op == Operations.remove); 2715 if(cgi.get["op"] == "query") 2716 assert(op == Operations.query); 2717 2718 if(cgi.get["op"] == "random string") 2719 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2720 --- 2721 +/ 2722 T request(T = string)(in string name, in T def = T.init) const nothrow { 2723 try { 2724 return 2725 (name in post) ? to!T(post[name]) : 2726 (name in get) ? to!T(get[name]) : 2727 def; 2728 } catch(Exception e) { return def; } 2729 } 2730 2731 /// Is the output already closed? 2732 bool isClosed() const { 2733 return closed; 2734 } 2735 2736 /++ 2737 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2738 +/ 2739 Session!Data getSessionObject(Data)() { 2740 if(testInProcess !is null) { 2741 // test mode 2742 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2743 if(obj !is null) 2744 return cast(typeof(return)) obj; 2745 else { 2746 auto o = new MockSession!Data(); 2747 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2748 return o; 2749 } 2750 } else { 2751 // normal operation 2752 return new BasicDataServerSession!Data(this); 2753 } 2754 } 2755 2756 // if it is in test mode; triggers mock sessions. Used by CgiTester 2757 version(with_breaking_cgi_features) 2758 private CgiTester testInProcess; 2759 2760 /* Hooks for redirecting input and output */ 2761 private void delegate(const(ubyte)[]) rawDataOutput = null; 2762 private void delegate() flushDelegate = null; 2763 2764 /* This info is used when handling a more raw HTTP protocol */ 2765 private bool nph; 2766 private bool http10; 2767 private bool closed; 2768 private bool responseChunked = false; 2769 2770 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2771 immutable(ubyte)[] originalPostData; 2772 2773 /++ 2774 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2775 2776 It is intended to be used for JSON and XML request content types, but also may be used 2777 for other content types your application can handle. But it will NOT be populated 2778 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2779 parsed into the post and postArray members. 2780 2781 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2782 will be discarded to the client with an error. This helps keep this array from being exploded in size 2783 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2784 client in certain build modes.) 2785 2786 History: 2787 Added January 5, 2021 2788 Documented February 21, 2023 (dub v11.0) 2789 +/ 2790 public immutable string postBody; 2791 alias postJson = postBody; // old name 2792 2793 /* Internal state flags */ 2794 private bool outputtedResponseData; 2795 private bool noCache = true; 2796 2797 const(string[string]) environmentVariables; 2798 2799 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2800 partially because it logically is (your code doesn't change what the user requested...) 2801 and partially because I hate how bad programs in PHP change those superglobals to do 2802 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2803 2804 For some of these, you'll want to refer to the http or cgi specs for more details. 2805 */ 2806 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2807 2808 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2809 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2810 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2811 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2812 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2813 immutable(char[]) scriptFileName; /// The physical filename of your script 2814 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2815 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2816 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2817 2818 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2819 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2820 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2821 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2822 2823 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2824 */ 2825 immutable(char[]) referrer; 2826 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2827 2828 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2829 2830 immutable bool https; /// Was the request encrypted via https? 2831 immutable int port; /// On what TCP port number did the server receive the request? 2832 2833 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2834 2835 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2836 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2837 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2838 2839 /** 2840 Represents user uploaded files. 2841 2842 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2843 */ 2844 immutable(UploadedFile[][string]) filesArray; 2845 immutable(UploadedFile[string]) files; 2846 2847 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2848 /// the order of the arrays is the order the data arrives 2849 immutable(string[][string]) getArray; /// like get, but an array of values per name 2850 immutable(string[][string]) postArray; /// ditto for post 2851 immutable(string[][string]) cookiesArray; /// ditto for cookies 2852 2853 // convenience function for appending to a uri without extra ? 2854 // matches the name and effect of javascript's location.search property 2855 string search() const { 2856 if(queryString.length) 2857 return "?" ~ queryString; 2858 return ""; 2859 } 2860 2861 // FIXME: what about multiple files with the same name? 2862 private: 2863 //RequestMethod _requestMethod; 2864 } 2865 2866 /// use this for testing or other isolated things when you want it to be no-ops 2867 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2868 // we want to ignore, not use stdout 2869 if(outputSink is null) 2870 outputSink = delegate void(const(ubyte)[]) { }; 2871 2872 string[string] env; 2873 env["REQUEST_METHOD"] = to!string(method); 2874 env["CONTENT_LENGTH"] = to!string(data.length); 2875 2876 auto cgi = new Cgi( 2877 0, 2878 env, 2879 { return data; }, 2880 outputSink, 2881 null); 2882 2883 return cgi; 2884 } 2885 2886 /++ 2887 A helper test class for request handler unittests. 2888 +/ 2889 version(with_breaking_cgi_features) 2890 class CgiTester { 2891 private { 2892 SessionObject[TypeInfo] mockSessions; 2893 SessionObject getSessionOverride(TypeInfo ti) { 2894 if(auto o = ti in mockSessions) 2895 return *o; 2896 else 2897 return null; 2898 } 2899 void setSessionOverride(TypeInfo ti, SessionObject so) { 2900 mockSessions[ti] = so; 2901 } 2902 } 2903 2904 /++ 2905 Gets (and creates if necessary) a mock session object for this test. Note 2906 it will be the same one used for any test operations through this CgiTester instance. 2907 +/ 2908 Session!Data getSessionObject(Data)() { 2909 auto obj = getSessionOverride(typeid(typeof(return))); 2910 if(obj !is null) 2911 return cast(typeof(return)) obj; 2912 else { 2913 auto o = new MockSession!Data(); 2914 setSessionOverride(typeid(typeof(return)), o); 2915 return o; 2916 } 2917 } 2918 2919 /++ 2920 Pass a reference to your request handler when creating the tester. 2921 +/ 2922 this(void function(Cgi) requestHandler) { 2923 this.requestHandler = requestHandler; 2924 } 2925 2926 /++ 2927 You can check response information with these methods after you call the request handler. 2928 +/ 2929 struct Response { 2930 int code; 2931 string[string] headers; 2932 string responseText; 2933 ubyte[] responseBody; 2934 } 2935 2936 /++ 2937 Executes a test request on your request handler, and returns the response. 2938 2939 Params: 2940 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2941 args = additional arguments. Same format as cgi's command line handler. 2942 +/ 2943 Response GET(string url, string[] args = null) { 2944 return executeTest("GET", url, args); 2945 } 2946 /// ditto 2947 Response POST(string url, string[] args = null) { 2948 return executeTest("POST", url, args); 2949 } 2950 2951 /// ditto 2952 Response executeTest(string method, string url, string[] args) { 2953 ubyte[] outputtedRawData; 2954 void outputSink(const(ubyte)[] data) { 2955 outputtedRawData ~= data; 2956 } 2957 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2958 cgi.testInProcess = this; 2959 scope(exit) cgi.dispose(); 2960 2961 requestHandler(cgi); 2962 2963 cgi.close(); 2964 2965 Response response; 2966 2967 if(outputtedRawData.length) { 2968 enum LINE = "\r\n"; 2969 2970 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2971 assert(idx != -1, to!string(outputtedRawData)); 2972 auto headers = cast(string) outputtedRawData[0 .. idx]; 2973 response.code = 200; 2974 while(headers.length) { 2975 auto i = headers.locationOf(LINE); 2976 if(i == -1) i = cast(int) headers.length; 2977 2978 auto header = headers[0 .. i]; 2979 2980 auto c = header.locationOf(":"); 2981 if(c != -1) { 2982 auto name = header[0 .. c]; 2983 auto value = header[c + 2 ..$]; 2984 2985 if(name == "Status") 2986 response.code = value[0 .. value.locationOf(" ")].to!int; 2987 2988 response.headers[name] = value; 2989 } else { 2990 assert(0); 2991 } 2992 2993 if(i != headers.length) 2994 i += 2; 2995 headers = headers[i .. $]; 2996 } 2997 response.responseBody = outputtedRawData[idx + 4 .. $]; 2998 response.responseText = cast(string) response.responseBody; 2999 } 3000 3001 return response; 3002 } 3003 3004 private void function(Cgi) requestHandler; 3005 } 3006 3007 3008 // should this be a separate module? Probably, but that's a hassle. 3009 3010 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3011 string makeDataUrl(string mimeType, in void[] data) { 3012 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3013 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3014 } 3015 3016 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3017 /// Represents a url that can be broken down or built up through properties 3018 struct Uri { 3019 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3020 3021 // scheme//userinfo@host:port/path?query#fragment 3022 3023 string scheme; /// e.g. "http" in "http://example.com/" 3024 string userinfo; /// the username (and possibly a password) in the uri 3025 string host; /// the domain name 3026 int port; /// port number, if given. Will be zero if a port was not explicitly given 3027 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3028 string query; /// the stuff after the ? in a uri 3029 string fragment; /// the stuff after the # in a uri. 3030 3031 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3032 // the decode ones need to keep different names anyway because we can't overload on return values... 3033 static string encode(string s) { return std.uri.encodeComponent(s); } 3034 static string encode(string[string] s) { return encodeVariables(s); } 3035 static string encode(string[][string] s) { return encodeVariables(s); } 3036 3037 /// Breaks down a uri string to its components 3038 this(string uri) { 3039 reparse(uri); 3040 } 3041 3042 private void reparse(string uri) { 3043 // from RFC 3986 3044 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3045 // it was a nice experiment but just not worth it. 3046 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3047 /* 3048 Captures: 3049 0 = whole url 3050 1 = scheme, with : 3051 2 = scheme, no : 3052 3 = authority, with // 3053 4 = authority, no // 3054 5 = path 3055 6 = query string, with ? 3056 7 = query string, no ? 3057 8 = anchor, with # 3058 9 = anchor, no # 3059 */ 3060 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3061 // instead, I will DIY and cut that down to 0.6s on the same computer. 3062 /* 3063 3064 Note that authority is 3065 user:password@domain:port 3066 where the user:password@ part is optional, and the :port is optional. 3067 3068 Regex translation: 3069 3070 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3071 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3072 Path cannot have any ? or # in it. It is optional. 3073 Query must start with ? and must not have # in it. It is optional. 3074 Anchor must start with # and can have anything else in it to end of string. It is optional. 3075 */ 3076 3077 this = Uri.init; // reset all state 3078 3079 // empty uri = nothing special 3080 if(uri.length == 0) { 3081 return; 3082 } 3083 3084 size_t idx; 3085 3086 scheme_loop: foreach(char c; uri[idx .. $]) { 3087 switch(c) { 3088 case ':': 3089 case '/': 3090 case '?': 3091 case '#': 3092 break scheme_loop; 3093 default: 3094 } 3095 idx++; 3096 } 3097 3098 if(idx == 0 && uri[idx] == ':') { 3099 // this is actually a path! we skip way ahead 3100 goto path_loop; 3101 } 3102 3103 if(idx == uri.length) { 3104 // the whole thing is a path, apparently 3105 path = uri; 3106 return; 3107 } 3108 3109 if(idx > 0 && uri[idx] == ':') { 3110 scheme = uri[0 .. idx]; 3111 idx++; 3112 } else { 3113 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3114 idx = 0; 3115 } 3116 3117 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3118 // we have an authority.... 3119 idx += 2; 3120 3121 auto authority_start = idx; 3122 authority_loop: foreach(char c; uri[idx .. $]) { 3123 switch(c) { 3124 case '/': 3125 case '?': 3126 case '#': 3127 break authority_loop; 3128 default: 3129 } 3130 idx++; 3131 } 3132 3133 auto authority = uri[authority_start .. idx]; 3134 3135 auto idx2 = authority.indexOf("@"); 3136 if(idx2 != -1) { 3137 userinfo = authority[0 .. idx2]; 3138 authority = authority[idx2 + 1 .. $]; 3139 } 3140 3141 if(authority.length && authority[0] == '[') { 3142 // ipv6 address special casing 3143 idx2 = authority.indexOf(']'); 3144 if(idx2 != -1) { 3145 auto end = authority[idx2 + 1 .. $]; 3146 if(end.length && end[0] == ':') 3147 idx2 = idx2 + 1; 3148 else 3149 idx2 = -1; 3150 } 3151 } else { 3152 idx2 = authority.indexOf(":"); 3153 } 3154 3155 if(idx2 == -1) { 3156 port = 0; // 0 means not specified; we should use the default for the scheme 3157 host = authority; 3158 } else { 3159 host = authority[0 .. idx2]; 3160 if(idx2 + 1 < authority.length) 3161 port = to!int(authority[idx2 + 1 .. $]); 3162 else 3163 port = 0; 3164 } 3165 } 3166 3167 path_loop: 3168 auto path_start = idx; 3169 3170 foreach(char c; uri[idx .. $]) { 3171 if(c == '?' || c == '#') 3172 break; 3173 idx++; 3174 } 3175 3176 path = uri[path_start .. idx]; 3177 3178 if(idx == uri.length) 3179 return; // nothing more to examine... 3180 3181 if(uri[idx] == '?') { 3182 idx++; 3183 auto query_start = idx; 3184 foreach(char c; uri[idx .. $]) { 3185 if(c == '#') 3186 break; 3187 idx++; 3188 } 3189 query = uri[query_start .. idx]; 3190 } 3191 3192 if(idx < uri.length && uri[idx] == '#') { 3193 idx++; 3194 fragment = uri[idx .. $]; 3195 } 3196 3197 // uriInvalidated = false; 3198 } 3199 3200 private string rebuildUri() const { 3201 string ret; 3202 if(scheme.length) 3203 ret ~= scheme ~ ":"; 3204 if(userinfo.length || host.length) 3205 ret ~= "//"; 3206 if(userinfo.length) 3207 ret ~= userinfo ~ "@"; 3208 if(host.length) 3209 ret ~= host; 3210 if(port) 3211 ret ~= ":" ~ to!string(port); 3212 3213 ret ~= path; 3214 3215 if(query.length) 3216 ret ~= "?" ~ query; 3217 3218 if(fragment.length) 3219 ret ~= "#" ~ fragment; 3220 3221 // uri = ret; 3222 // uriInvalidated = false; 3223 return ret; 3224 } 3225 3226 /// Converts the broken down parts back into a complete string 3227 string toString() const { 3228 // if(uriInvalidated) 3229 return rebuildUri(); 3230 } 3231 3232 /// Returns a new absolute Uri given a base. It treats this one as 3233 /// relative where possible, but absolute if not. (If protocol, domain, or 3234 /// other info is not set, the new one inherits it from the base.) 3235 /// 3236 /// Browsers use a function like this to figure out links in html. 3237 Uri basedOn(in Uri baseUrl) const { 3238 Uri n = this; // copies 3239 if(n.scheme == "data") 3240 return n; 3241 // n.uriInvalidated = true; // make sure we regenerate... 3242 3243 // userinfo is not inherited... is this wrong? 3244 3245 // if anything is given in the existing url, we don't use the base anymore. 3246 if(n.scheme.empty) { 3247 n.scheme = baseUrl.scheme; 3248 if(n.host.empty) { 3249 n.host = baseUrl.host; 3250 if(n.port == 0) { 3251 n.port = baseUrl.port; 3252 if(n.path.length > 0 && n.path[0] != '/') { 3253 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3254 if(b.length == 0) 3255 b = "/"; 3256 n.path = b ~ n.path; 3257 } else if(n.path.length == 0) { 3258 n.path = baseUrl.path; 3259 } 3260 } 3261 } 3262 } 3263 3264 n.removeDots(); 3265 3266 return n; 3267 } 3268 3269 void removeDots() { 3270 auto parts = this.path.split("/"); 3271 string[] toKeep; 3272 foreach(part; parts) { 3273 if(part == ".") { 3274 continue; 3275 } else if(part == "..") { 3276 //if(toKeep.length > 1) 3277 toKeep = toKeep[0 .. $-1]; 3278 //else 3279 //toKeep = [""]; 3280 continue; 3281 } else { 3282 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3283 //continue; // skip a `//` situation 3284 toKeep ~= part; 3285 } 3286 } 3287 3288 auto path = toKeep.join("/"); 3289 if(path.length && path[0] != '/') 3290 path = "/" ~ path; 3291 3292 this.path = path; 3293 } 3294 3295 unittest { 3296 auto uri = Uri("test.html"); 3297 assert(uri.path == "test.html"); 3298 uri = Uri("path/1/lol"); 3299 assert(uri.path == "path/1/lol"); 3300 uri = Uri("http://me@example.com"); 3301 assert(uri.scheme == "http"); 3302 assert(uri.userinfo == "me"); 3303 assert(uri.host == "example.com"); 3304 uri = Uri("http://example.com/#a"); 3305 assert(uri.scheme == "http"); 3306 assert(uri.host == "example.com"); 3307 assert(uri.fragment == "a"); 3308 uri = Uri("#foo"); 3309 assert(uri.fragment == "foo"); 3310 uri = Uri("?lol"); 3311 assert(uri.query == "lol"); 3312 uri = Uri("#foo?lol"); 3313 assert(uri.fragment == "foo?lol"); 3314 uri = Uri("?lol#foo"); 3315 assert(uri.fragment == "foo"); 3316 assert(uri.query == "lol"); 3317 3318 uri = Uri("http://127.0.0.1/"); 3319 assert(uri.host == "127.0.0.1"); 3320 assert(uri.port == 0); 3321 3322 uri = Uri("http://127.0.0.1:123/"); 3323 assert(uri.host == "127.0.0.1"); 3324 assert(uri.port == 123); 3325 3326 uri = Uri("http://[ff:ff::0]/"); 3327 assert(uri.host == "[ff:ff::0]"); 3328 3329 uri = Uri("http://[ff:ff::0]:123/"); 3330 assert(uri.host == "[ff:ff::0]"); 3331 assert(uri.port == 123); 3332 } 3333 3334 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3335 // the possibilities. 3336 unittest { 3337 auto url = Uri("cool.html"); // checking relative links 3338 3339 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3340 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3341 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3342 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3343 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3344 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3345 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3346 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3347 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3348 3349 url = Uri("/something/cool.html"); // same server, different path 3350 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3351 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3352 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3353 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3354 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3355 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3356 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3357 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3358 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3359 3360 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3361 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3362 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3363 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3364 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3365 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3366 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3367 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3368 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3369 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3370 3371 url = Uri("/test/bar"); 3372 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3373 assert(Uri("../").basedOn(url) == "/"); 3374 3375 url = Uri("http://example.com/"); 3376 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3377 3378 //auto uriBefore = url; 3379 url = Uri("#anchor"); // everything should remain the same except the anchor 3380 //uriBefore.anchor = "anchor"); 3381 //assert(url == uriBefore); 3382 3383 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3384 3385 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3386 3387 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3388 3389 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3390 3391 // FIXME: add something for port too 3392 } 3393 3394 // these are like javascript's location.search and location.hash 3395 string search() const { 3396 return query.length ? ("?" ~ query) : ""; 3397 } 3398 string hash() const { 3399 return fragment.length ? ("#" ~ fragment) : ""; 3400 } 3401 } 3402 3403 3404 /* 3405 for session, see web.d 3406 */ 3407 3408 /// breaks down a url encoded string 3409 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3410 auto vars = data.split(separator); 3411 string[][string] _get; 3412 foreach(var; vars) { 3413 auto equal = var.indexOf("="); 3414 string name; 3415 string value; 3416 if(equal == -1) { 3417 name = decodeComponent(var); 3418 value = ""; 3419 } else { 3420 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3421 // stupid + -> space conversion. 3422 name = decodeComponent(var[0..equal].replace("+", " ")); 3423 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3424 } 3425 3426 _get[name] ~= value; 3427 if(namesInOrder) 3428 (*namesInOrder) ~= name; 3429 if(valuesInOrder) 3430 (*valuesInOrder) ~= value; 3431 } 3432 return _get; 3433 } 3434 3435 /// breaks down a url encoded string, but only returns the last value of any array 3436 string[string] decodeVariablesSingle(string data) { 3437 string[string] va; 3438 auto varArray = decodeVariables(data); 3439 foreach(k, v; varArray) 3440 va[k] = v[$-1]; 3441 3442 return va; 3443 } 3444 3445 /// url encodes the whole string 3446 string encodeVariables(in string[string] data) { 3447 string ret; 3448 3449 bool outputted = false; 3450 foreach(k, v; data) { 3451 if(outputted) 3452 ret ~= "&"; 3453 else 3454 outputted = true; 3455 3456 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3457 } 3458 3459 return ret; 3460 } 3461 3462 /// url encodes a whole string 3463 string encodeVariables(in string[][string] data) { 3464 string ret; 3465 3466 bool outputted = false; 3467 foreach(k, arr; data) { 3468 foreach(v; arr) { 3469 if(outputted) 3470 ret ~= "&"; 3471 else 3472 outputted = true; 3473 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3474 } 3475 } 3476 3477 return ret; 3478 } 3479 3480 /// Encodes all but the explicitly unreserved characters per rfc 3986 3481 /// Alphanumeric and -_.~ are the only ones left unencoded 3482 /// name is borrowed from php 3483 string rawurlencode(in char[] data) { 3484 string ret; 3485 ret.reserve(data.length * 2); 3486 foreach(char c; data) { 3487 if( 3488 (c >= 'a' && c <= 'z') || 3489 (c >= 'A' && c <= 'Z') || 3490 (c >= '0' && c <= '9') || 3491 c == '-' || c == '_' || c == '.' || c == '~') 3492 { 3493 ret ~= c; 3494 } else { 3495 ret ~= '%'; 3496 // since we iterate on char, this should give us the octets of the full utf8 string 3497 ret ~= toHexUpper(c); 3498 } 3499 } 3500 3501 return ret; 3502 } 3503 3504 3505 // http helper functions 3506 3507 // for chunked responses (which embedded http does whenever possible) 3508 version(none) // this is moved up above to avoid making a copy of the data 3509 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3510 const(ubyte)[] ret; 3511 3512 ret = cast(const(ubyte)[]) toHex(data.length); 3513 ret ~= cast(const(ubyte)[]) "\r\n"; 3514 ret ~= data; 3515 ret ~= cast(const(ubyte)[]) "\r\n"; 3516 3517 return ret; 3518 } 3519 3520 string toHex(long num) { 3521 string ret; 3522 while(num) { 3523 int v = num % 16; 3524 num /= 16; 3525 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3526 ret ~= d; 3527 } 3528 3529 return to!string(array(ret.retro)); 3530 } 3531 3532 string toHexUpper(long num) { 3533 string ret; 3534 while(num) { 3535 int v = num % 16; 3536 num /= 16; 3537 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3538 ret ~= d; 3539 } 3540 3541 if(ret.length == 1) 3542 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3543 3544 return to!string(array(ret.retro)); 3545 } 3546 3547 3548 // the generic mixins 3549 3550 /++ 3551 Use this instead of writing your own main 3552 3553 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3554 +/ 3555 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3556 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3557 } 3558 3559 /++ 3560 Boilerplate mixin for a main function that uses the [dispatcher] function. 3561 3562 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3563 3564 History: 3565 Added July 9, 2021 3566 +/ 3567 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3568 /// forwards to [CustomCgiDispatcherMain] with default args 3569 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3570 } 3571 3572 /// ditto 3573 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3574 class GenericPresenter : WebPresenter!GenericPresenter {} 3575 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3576 } 3577 3578 /++ 3579 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3580 3581 History: 3582 Added May 13, 2023 (dub v11.0) 3583 +/ 3584 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3585 /++ 3586 Handler to the generated presenter you can use from your objects, etc. 3587 +/ 3588 Presenter activePresenter; 3589 3590 /++ 3591 Request handler that creates the presenter then forwards to the [dispatcher] function. 3592 Renders 404 if the dispatcher did not handle the request. 3593 3594 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3595 +/ 3596 void handler(Cgi cgi) { 3597 auto presenter = new Presenter; 3598 activePresenter = presenter; 3599 scope(exit) activePresenter = null; 3600 3601 if(cgi.dispatcher!DispatcherArgs(presenter)) 3602 return; 3603 3604 switch(cgi.pathInfo) { 3605 case "/style.css": 3606 cgi.setCache(true); 3607 cgi.setResponseContentType("text/css"); 3608 cgi.write(presenter.style(), true); 3609 break; 3610 case "/script.js": 3611 cgi.setCache(true); 3612 cgi.setResponseContentType("application/javascript"); 3613 cgi.write(presenter.script(), true); 3614 break; 3615 default: 3616 presenter.renderBasicError(cgi, 404); 3617 } 3618 } 3619 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3620 } 3621 3622 /// ditto 3623 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3624 class GenericPresenter : WebPresenter!GenericPresenter {} 3625 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3626 3627 } 3628 3629 private string simpleHtmlEncode(string s) { 3630 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3631 } 3632 3633 string messageFromException(Throwable t) { 3634 string message; 3635 if(t !is null) { 3636 debug message = t.toString(); 3637 else message = "An unexpected error has occurred."; 3638 } else { 3639 message = "Unknown error"; 3640 } 3641 return message; 3642 } 3643 3644 string plainHttpError(bool isCgi, string type, Throwable t) { 3645 auto message = messageFromException(t); 3646 message = simpleHtmlEncode(message); 3647 3648 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3649 isCgi ? "Status:" : "HTTP/1.1", 3650 type, message.length, message); 3651 } 3652 3653 // returns true if we were able to recover reasonably 3654 bool handleException(Cgi cgi, Throwable t) { 3655 if(cgi.isClosed) { 3656 // if the channel has been explicitly closed, we can't handle it here 3657 return true; 3658 } 3659 3660 if(cgi.outputtedResponseData) { 3661 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3662 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3663 } else { 3664 // no headers are sent, we can send a full blown error and recover 3665 cgi.setCache(false); 3666 cgi.setResponseContentType("text/html"); 3667 cgi.setResponseLocation(null); // cancel the redirect 3668 cgi.setResponseStatus("500 Internal Server Error"); 3669 cgi.write(simpleHtmlEncode(messageFromException(t))); 3670 cgi.close(); 3671 return true; 3672 } 3673 } 3674 3675 bool isCgiRequestMethod(string s) { 3676 s = s.toUpper(); 3677 if(s == "COMMANDLINE") 3678 return true; 3679 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3680 if(s == member) 3681 return true; 3682 return false; 3683 } 3684 3685 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3686 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3687 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3688 void main(string[] args) { 3689 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3690 } 3691 } 3692 3693 version(embedded_httpd_processes) 3694 __gshared int processPoolSize = 8; 3695 3696 // Returns true if run. You should exit the program after that. 3697 bool tryAddonServers(string[] args) { 3698 if(args.length > 1) { 3699 // run the special separate processes if needed 3700 switch(args[1]) { 3701 case "--websocket-server": 3702 version(with_addon_servers) 3703 websocketServers[args[2]](args[3 .. $]); 3704 else 3705 printf("Add-on servers not compiled in.\n"); 3706 return true; 3707 case "--websocket-servers": 3708 import core.demangle; 3709 version(with_addon_servers_connections) 3710 foreach(k, v; websocketServers) 3711 writeln(k, "\t", demangle(k)); 3712 return true; 3713 case "--session-server": 3714 version(with_addon_servers) 3715 runSessionServer(); 3716 else 3717 printf("Add-on servers not compiled in.\n"); 3718 return true; 3719 case "--event-server": 3720 version(with_addon_servers) 3721 runEventServer(); 3722 else 3723 printf("Add-on servers not compiled in.\n"); 3724 return true; 3725 case "--timer-server": 3726 version(with_addon_servers) 3727 runTimerServer(); 3728 else 3729 printf("Add-on servers not compiled in.\n"); 3730 return true; 3731 case "--timed-jobs": 3732 import core.demangle; 3733 version(with_addon_servers_connections) 3734 foreach(k, v; scheduledJobHandlers) 3735 writeln(k, "\t", demangle(k)); 3736 return true; 3737 case "--timed-job": 3738 scheduledJobHandlers[args[2]](args[3 .. $]); 3739 return true; 3740 default: 3741 // intentionally blank - do nothing and carry on to run normally 3742 } 3743 } 3744 return false; 3745 } 3746 3747 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3748 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3749 // we support command line thing for easy testing everywhere 3750 // it needs to be called ./app method uri [other args...] 3751 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3752 Cgi cgi = new CustomCgi(args); 3753 scope(exit) cgi.dispose(); 3754 fun(cgi); 3755 cgi.close(); 3756 return true; 3757 } 3758 return false; 3759 } 3760 3761 /++ 3762 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3763 3764 As of version 11 (released August 2023), you can also make things like this: 3765 3766 --- 3767 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3768 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3769 3770 // can also: 3771 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3772 3773 // NOT IMPLEMENTED YET 3774 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3775 3776 foreach(listenSpec; server.listenSpecs) { 3777 // you can check what it actually bound to here and see your assigned ports 3778 } 3779 3780 // NOT IMPLEMENTED YET 3781 // server.start!handler(); // starts and runs in the arsd.core event loop 3782 3783 server.serve!handler(); // blocks the thread until the server exits 3784 --- 3785 3786 History: 3787 Added Sept 26, 2020 (release version 8.5). 3788 3789 The `listenSpec` member was added July 31, 2023. 3790 +/ 3791 struct RequestServer { 3792 /++ 3793 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3794 +/ 3795 string listeningHost = defaultListeningHost(); 3796 /// ditto 3797 ushort listeningPort = defaultListeningPort(); 3798 3799 static struct ListenSpec { 3800 enum Protocol { 3801 http, 3802 https, 3803 scgi 3804 } 3805 Protocol protocol; 3806 3807 enum AddressType { 3808 ip, 3809 unix, 3810 abstract_ 3811 } 3812 AddressType addressType; 3813 3814 string address; 3815 ushort port; 3816 } 3817 3818 /++ 3819 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3820 3821 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3822 3823 `http://localhost:8080` 3824 3825 `http://unix:filename/here` 3826 3827 `scgi://abstract:/name/here` 3828 3829 `http://[::1]:4444` 3830 3831 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3832 3833 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3834 3835 `localhost:8080` serves the default protocol. 3836 3837 `8080` or `:8080` assumes default protocol on localhost. 3838 3839 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3840 3841 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3842 3843 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3844 3845 $(PITFALL 3846 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3847 ) 3848 3849 Bugs: 3850 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3851 3852 History: 3853 Added July 31, 2023 (dub v11.0) 3854 +/ 3855 string[] listenSpec; 3856 3857 /++ 3858 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3859 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3860 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3861 [stop] may not work as well. 3862 3863 History: 3864 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3865 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3866 compatibility. 3867 +/ 3868 bool useFork = cgi_use_fork_default; 3869 3870 /++ 3871 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3872 default based on the number of cpus modified by the server mode. 3873 3874 History: 3875 Added August 12, 2022 (dub v10.9) 3876 +/ 3877 int numberOfThreads = 0; 3878 3879 /++ 3880 Creates a server configured to listen to multiple URLs. 3881 3882 History: 3883 Added July 31, 2023 (dub v11.0) 3884 +/ 3885 this(string[] listenTo) { 3886 this.listenSpec = listenTo; 3887 } 3888 3889 /// Creates a server object configured to listen on a single host and port. 3890 this(string defaultHost, ushort defaultPort) { 3891 this.listeningHost = defaultHost; 3892 this.listeningPort = defaultPort; 3893 } 3894 3895 /// ditto 3896 this(ushort defaultPort) { 3897 listeningPort = defaultPort; 3898 } 3899 3900 /++ 3901 Reads the command line arguments into the values here. 3902 3903 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3904 3905 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 3906 +/ 3907 void configureFromCommandLine(string[] args) { 3908 bool portOrHostFound = false; 3909 3910 bool foundPort = false; 3911 bool foundHost = false; 3912 bool foundUid = false; 3913 bool foundGid = false; 3914 bool foundListen = false; 3915 foreach(arg; args) { 3916 if(foundPort) { 3917 listeningPort = to!ushort(arg); 3918 portOrHostFound = true; 3919 foundPort = false; 3920 continue; 3921 } 3922 if(foundHost) { 3923 listeningHost = arg; 3924 portOrHostFound = true; 3925 foundHost = false; 3926 continue; 3927 } 3928 if(foundUid) { 3929 privilegesDropToUid = to!uid_t(arg); 3930 foundUid = false; 3931 continue; 3932 } 3933 if(foundGid) { 3934 privilegesDropToGid = to!gid_t(arg); 3935 foundGid = false; 3936 continue; 3937 } 3938 if(foundListen) { 3939 this.listenSpec ~= arg; 3940 foundListen = false; 3941 continue; 3942 } 3943 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3944 foundHost = true; 3945 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3946 foundPort = true; 3947 else if(arg == "--uid") 3948 foundUid = true; 3949 else if(arg == "--gid") 3950 foundGid = true; 3951 else if(arg == "--listen") 3952 foundListen = true; 3953 } 3954 3955 if(portOrHostFound && listenSpec.length) { 3956 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 3957 } 3958 } 3959 3960 version(Windows) { 3961 private alias uid_t = int; 3962 private alias gid_t = int; 3963 } 3964 3965 /// user (uid) to drop privileges to 3966 /// 0 … do nothing 3967 uid_t privilegesDropToUid = 0; 3968 /// group (gid) to drop privileges to 3969 /// 0 … do nothing 3970 gid_t privilegesDropToGid = 0; 3971 3972 private void dropPrivileges() { 3973 version(Posix) { 3974 import core.sys.posix.unistd; 3975 3976 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 3977 throw new Exception("Dropping privileges via setgid() failed."); 3978 3979 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 3980 throw new Exception("Dropping privileges via setuid() failed."); 3981 } 3982 else { 3983 // FIXME: Windows? 3984 //pragma(msg, "Dropping privileges is not implemented for this platform"); 3985 } 3986 3987 // done, set zero 3988 privilegesDropToGid = 0; 3989 privilegesDropToUid = 0; 3990 } 3991 3992 /++ 3993 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 3994 3995 History: 3996 Added Oct 10, 2020. 3997 Example: 3998 3999 --- 4000 import arsd.cgi; 4001 void main() { 4002 RequestServer server = RequestServer("127.0.0.1", 6789); 4003 string oauthCode; 4004 string oauthScope; 4005 server.serveHttpOnce!((cgi) { 4006 oauthCode = cgi.request("code"); 4007 oauthScope = cgi.request("scope"); 4008 cgi.write("Thank you, please return to the application."); 4009 }); 4010 // use the code and scope given 4011 } 4012 --- 4013 +/ 4014 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4015 import std.socket; 4016 4017 bool tcp; 4018 void delegate() cleanup; 4019 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4020 auto connection = socket.accept(); 4021 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4022 4023 if(cleanup) 4024 cleanup(); 4025 } 4026 4027 /++ 4028 Starts serving requests according to the current configuration. 4029 +/ 4030 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4031 version(netman_httpd) { 4032 // Obsolete! 4033 4034 import arsd.httpd; 4035 // what about forwarding the other constructor args? 4036 // this probably needs a whole redoing... 4037 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4038 return; 4039 } else 4040 version(embedded_httpd_processes) { 4041 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4042 } else 4043 version(embedded_httpd_threads) { 4044 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4045 } else 4046 version(scgi) { 4047 serveScgi!(fun, CustomCgi, maxContentLength)(); 4048 } else 4049 version(fastcgi) { 4050 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4051 } else 4052 version(stdio_http) { 4053 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4054 } else { 4055 //version=plain_cgi; 4056 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4057 } 4058 } 4059 4060 /++ 4061 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4062 4063 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4064 +/ 4065 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4066 globalStopFlag = false; 4067 static if(__traits(isStaticFunction, fun)) 4068 alias funToUse = fun; 4069 else 4070 void funToUse(CustomCgi cgi) { 4071 static if(__VERSION__ > 2097) 4072 __traits(child, _this, fun)(cgi); 4073 else static assert(0, "Not implemented in your compiler version!"); 4074 } 4075 auto manager = this.listenSpec is null ? 4076 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4077 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4078 manager.listen(); 4079 } 4080 4081 /++ 4082 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4083 +/ 4084 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4085 globalStopFlag = false; 4086 auto manager = this.listenSpec is null ? 4087 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4088 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4089 manager.listen(); 4090 } 4091 4092 /++ 4093 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4094 4095 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4096 4097 History: 4098 Added May 29, 2021 4099 +/ 4100 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4101 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4102 } 4103 4104 /++ 4105 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4106 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4107 +/ 4108 enum ForceStop { 4109 /++ 4110 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4111 +/ 4112 afterQueuedRequestsComplete, 4113 /++ 4114 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4115 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4116 +/ 4117 afterCurrentRequestsComplete, 4118 /++ 4119 Partial response writes will throw an exception, cancelling any streaming response, but complete 4120 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4121 +/ 4122 cancelStreamingRequestsEarly, 4123 /++ 4124 All writes will throw. 4125 +/ 4126 cancelAllRequestsEarly, 4127 /++ 4128 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4129 +/ 4130 forciblyTerminate, 4131 } 4132 4133 version(embedded_httpd_processes) {} else 4134 /++ 4135 Stops serving after the current requests are completed. 4136 4137 Bugs: 4138 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4139 on Windows however). Only partially implemented on non-Linux posix systems. 4140 4141 You might also try SIGINT perhaps. 4142 4143 The stopPriority is not yet fully implemented. 4144 +/ 4145 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4146 globalStopFlag = true; 4147 4148 version(Posix) { 4149 if(cancelfd > 0) { 4150 ulong a = 1; 4151 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4152 } 4153 } 4154 version(Windows) { 4155 if(iocp) { 4156 foreach(i; 0 .. 16) // FIXME 4157 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4158 } 4159 } 4160 } 4161 } 4162 4163 private alias AliasSeq(T...) = T; 4164 4165 version(with_breaking_cgi_features) 4166 mixin(q{ 4167 template ThisFor(alias t) { 4168 static if(__traits(isStaticFunction, t)) { 4169 alias ThisFor = AliasSeq!(); 4170 } else { 4171 alias ThisFor = __traits(parent, t); 4172 } 4173 } 4174 }); 4175 else 4176 alias ThisFor(alias t) = AliasSeq!(); 4177 4178 private __gshared bool globalStopFlag = false; 4179 4180 version(embedded_httpd_processes) 4181 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4182 import core.sys.posix.unistd; 4183 import core.sys.posix.sys.socket; 4184 import core.sys.posix.netinet.in_; 4185 //import std.c.linux.socket; 4186 4187 int sock = socket(AF_INET, SOCK_STREAM, 0); 4188 if(sock == -1) 4189 throw new Exception("socket"); 4190 4191 cloexec(sock); 4192 4193 { 4194 4195 sockaddr_in addr; 4196 addr.sin_family = AF_INET; 4197 addr.sin_port = htons(params.listeningPort); 4198 auto lh = params.listeningHost; 4199 if(lh.length) { 4200 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4201 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4202 } else 4203 addr.sin_addr.s_addr = INADDR_ANY; 4204 4205 // HACKISH 4206 int on = 1; 4207 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4208 // end hack 4209 4210 4211 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4212 close(sock); 4213 throw new Exception("bind"); 4214 } 4215 4216 // FIXME: if this queue is full, it will just ignore it 4217 // and wait for the client to retransmit it. This is an 4218 // obnoxious timeout condition there. 4219 if(sock.listen(128) == -1) { 4220 close(sock); 4221 throw new Exception("listen"); 4222 } 4223 params.dropPrivileges(); 4224 } 4225 4226 version(embedded_httpd_processes_accept_after_fork) {} else { 4227 int pipeReadFd; 4228 int pipeWriteFd; 4229 4230 { 4231 int[2] pipeFd; 4232 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4233 import core.stdc.errno; 4234 throw new Exception("pipe failed " ~ to!string(errno)); 4235 } 4236 4237 pipeReadFd = pipeFd[0]; 4238 pipeWriteFd = pipeFd[1]; 4239 } 4240 } 4241 4242 4243 int processCount; 4244 pid_t newPid; 4245 reopen: 4246 while(processCount < processPoolSize) { 4247 newPid = fork(); 4248 if(newPid == 0) { 4249 // start serving on the socket 4250 //ubyte[4096] backingBuffer; 4251 for(;;) { 4252 bool closeConnection; 4253 uint i; 4254 sockaddr addr; 4255 i = addr.sizeof; 4256 version(embedded_httpd_processes_accept_after_fork) { 4257 int s = accept(sock, &addr, &i); 4258 int opt = 1; 4259 import core.sys.posix.netinet.tcp; 4260 // the Cgi class does internal buffering, so disabling this 4261 // helps with latency in many cases... 4262 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4263 cloexec(s); 4264 } else { 4265 int s; 4266 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4267 if(readret != s.sizeof) { 4268 import core.stdc.errno; 4269 throw new Exception("pipe read failed " ~ to!string(errno)); 4270 } 4271 4272 //writeln("process ", getpid(), " got socket ", s); 4273 } 4274 4275 try { 4276 4277 if(s == -1) 4278 throw new Exception("accept"); 4279 4280 scope(failure) close(s); 4281 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4282 auto ir = new BufferedInputRange(s); 4283 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4284 4285 while(!ir.empty) { 4286 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4287 4288 Cgi cgi; 4289 try { 4290 cgi = new CustomCgi(ir, &closeConnection); 4291 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4292 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4293 if(processPoolSize <= 1) 4294 closeConnection = true; 4295 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4296 } catch(HttpVersionNotSupportedException he) { 4297 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4298 closeConnection = true; 4299 break; 4300 } catch(Throwable t) { 4301 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4302 // anyway let's kill the connection 4303 version(CRuntime_Musl) { 4304 // LockingTextWriter fails here 4305 // so working around it 4306 auto estr = t.toString(); 4307 stderr.rawWrite(estr); 4308 stderr.rawWrite("\n"); 4309 } else 4310 stderr.writeln(t.toString()); 4311 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4312 closeConnection = true; 4313 break; 4314 } 4315 assert(cgi !is null); 4316 scope(exit) 4317 cgi.dispose(); 4318 4319 try { 4320 fun(cgi); 4321 cgi.close(); 4322 if(cgi.websocketMode) 4323 closeConnection = true; 4324 } catch(ConnectionException ce) { 4325 closeConnection = true; 4326 } catch(Throwable t) { 4327 // a processing error can be recovered from 4328 version(CRuntime_Musl) { 4329 // LockingTextWriter fails here 4330 // so working around it 4331 auto estr = t.toString(); 4332 stderr.rawWrite(estr); 4333 } else { 4334 stderr.writeln(t.toString); 4335 } 4336 if(!handleException(cgi, t)) 4337 closeConnection = true; 4338 } 4339 4340 if(closeConnection) { 4341 ir.source.close(); 4342 break; 4343 } else { 4344 if(!ir.empty) 4345 ir.popFront(); // get the next 4346 else if(ir.sourceClosed) { 4347 ir.source.close(); 4348 } 4349 } 4350 } 4351 4352 ir.source.close(); 4353 } catch(Throwable t) { 4354 version(CRuntime_Musl) {} else 4355 debug writeln(t); 4356 // most likely cause is a timeout 4357 } 4358 } 4359 } else if(newPid < 0) { 4360 throw new Exception("fork failed"); 4361 } else { 4362 processCount++; 4363 } 4364 } 4365 4366 // the parent should wait for its children... 4367 if(newPid) { 4368 import core.sys.posix.sys.wait; 4369 4370 version(embedded_httpd_processes_accept_after_fork) {} else { 4371 import core.sys.posix.sys.select; 4372 int[] fdQueue; 4373 while(true) { 4374 // writeln("select call"); 4375 int nfds = pipeWriteFd; 4376 if(sock > pipeWriteFd) 4377 nfds = sock; 4378 nfds += 1; 4379 fd_set read_fds; 4380 fd_set write_fds; 4381 FD_ZERO(&read_fds); 4382 FD_ZERO(&write_fds); 4383 FD_SET(sock, &read_fds); 4384 if(fdQueue.length) 4385 FD_SET(pipeWriteFd, &write_fds); 4386 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4387 if(ret == -1) { 4388 import core.stdc.errno; 4389 if(errno == EINTR) 4390 goto try_wait; 4391 else 4392 throw new Exception("wtf select"); 4393 } 4394 4395 int s = -1; 4396 if(FD_ISSET(sock, &read_fds)) { 4397 uint i; 4398 sockaddr addr; 4399 i = addr.sizeof; 4400 s = accept(sock, &addr, &i); 4401 cloexec(s); 4402 import core.sys.posix.netinet.tcp; 4403 int opt = 1; 4404 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4405 } 4406 4407 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4408 if(s == -1 && fdQueue.length) { 4409 s = fdQueue[0]; 4410 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4411 } 4412 write_fd(pipeWriteFd, &s, s.sizeof, s); 4413 close(s); // we are done with it, let the other process take ownership 4414 } else 4415 fdQueue ~= s; 4416 } 4417 } 4418 4419 try_wait: 4420 4421 int status; 4422 while(-1 != wait(&status)) { 4423 version(CRuntime_Musl) {} else { 4424 import std.stdio; writeln("Process died ", status); 4425 } 4426 processCount--; 4427 goto reopen; 4428 } 4429 close(sock); 4430 } 4431 } 4432 4433 version(fastcgi) 4434 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4435 // SetHandler fcgid-script 4436 FCGX_Stream* input, output, error; 4437 FCGX_ParamArray env; 4438 4439 4440 4441 const(ubyte)[] getFcgiChunk() { 4442 const(ubyte)[] ret; 4443 while(FCGX_HasSeenEOF(input) != -1) 4444 ret ~= cast(ubyte) FCGX_GetChar(input); 4445 return ret; 4446 } 4447 4448 void writeFcgi(const(ubyte)[] data) { 4449 FCGX_PutStr(data.ptr, data.length, output); 4450 } 4451 4452 void doARequest() { 4453 string[string] fcgienv; 4454 4455 for(auto e = env; e !is null && *e !is null; e++) { 4456 string cur = to!string(*e); 4457 auto idx = cur.indexOf("="); 4458 string name, value; 4459 if(idx == -1) 4460 name = cur; 4461 else { 4462 name = cur[0 .. idx]; 4463 value = cur[idx + 1 .. $]; 4464 } 4465 4466 fcgienv[name] = value; 4467 } 4468 4469 void flushFcgi() { 4470 FCGX_FFlush(output); 4471 } 4472 4473 Cgi cgi; 4474 try { 4475 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4476 } catch(Throwable t) { 4477 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4478 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4479 return; //continue; 4480 } 4481 assert(cgi !is null); 4482 scope(exit) cgi.dispose(); 4483 try { 4484 fun(cgi); 4485 cgi.close(); 4486 } catch(Throwable t) { 4487 // log it to the error stream 4488 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4489 // handle it for the user, if we can 4490 if(!handleException(cgi, t)) 4491 return; // continue; 4492 } 4493 } 4494 4495 auto lp = params.listeningPort; 4496 auto host = params.listeningHost; 4497 4498 FCGX_Request request; 4499 if(lp || !host.empty) { 4500 // if a listening port was specified on the command line, we want to spawn ourself 4501 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4502 FCGX_Init(); 4503 4504 int sock; 4505 4506 if(host.startsWith("unix:")) { 4507 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4508 } else if(host.startsWith("abstract:")) { 4509 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4510 } else { 4511 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4512 } 4513 4514 if(sock < 0) 4515 throw new Exception("Couldn't listen on the port"); 4516 FCGX_InitRequest(&request, sock, 0); 4517 while(FCGX_Accept_r(&request) >= 0) { 4518 input = request.inStream; 4519 output = request.outStream; 4520 error = request.errStream; 4521 env = request.envp; 4522 doARequest(); 4523 } 4524 } else { 4525 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4526 // using the version with a global variable since we are separate processes anyway 4527 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4528 doARequest(); 4529 } 4530 } 4531 } 4532 4533 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4534 ushort defaultListeningPort() { 4535 version(netman_httpd) 4536 return 8080; 4537 else version(embedded_httpd_processes) 4538 return 8085; 4539 else version(embedded_httpd_threads) 4540 return 8085; 4541 else version(scgi) 4542 return 4000; 4543 else 4544 return 0; 4545 } 4546 4547 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4548 string defaultListeningHost() { 4549 version(netman_httpd) 4550 return null; 4551 else version(embedded_httpd_processes) 4552 return null; 4553 else version(embedded_httpd_threads) 4554 return null; 4555 else version(scgi) 4556 return "127.0.0.1"; 4557 else 4558 return null; 4559 4560 } 4561 4562 /++ 4563 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4564 4565 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4566 4567 Params: 4568 fun = Your request handler 4569 CustomCgi = a subclass of Cgi, if you wise to customize it further 4570 maxContentLength = max POST size you want to allow 4571 args = command-line arguments 4572 4573 History: 4574 Documented Sept 26, 2020. 4575 +/ 4576 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4577 if(tryAddonServers(args)) 4578 return; 4579 4580 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4581 return; 4582 4583 RequestServer server; 4584 // you can change the port here if you like 4585 // server.listeningPort = 9000; 4586 4587 // then call this to let the command line args override your default 4588 server.configureFromCommandLine(args); 4589 4590 // and serve the request(s). 4591 server.serve!(fun, CustomCgi, maxContentLength)(); 4592 } 4593 4594 //version(plain_cgi) 4595 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4596 // standard CGI is the default version 4597 4598 4599 // Set stdin to binary mode if necessary to avoid mangled newlines 4600 // the fact that stdin is global means this could be trouble but standard cgi request 4601 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4602 version(Windows) { 4603 version(Win64) 4604 _setmode(std.stdio.stdin.fileno(), 0x8000); 4605 else 4606 setmode(std.stdio.stdin.fileno(), 0x8000); 4607 } 4608 4609 Cgi cgi; 4610 try { 4611 cgi = new CustomCgi(maxContentLength); 4612 version(Posix) 4613 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4614 else version(Windows) 4615 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4616 else static assert(0); 4617 } catch(Throwable t) { 4618 version(CRuntime_Musl) { 4619 // LockingTextWriter fails here 4620 // so working around it 4621 auto s = t.toString(); 4622 stderr.rawWrite(s); 4623 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4624 } else { 4625 stderr.writeln(t.msg); 4626 // the real http server will probably handle this; 4627 // most likely, this is a bug in Cgi. But, oh well. 4628 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4629 } 4630 return; 4631 } 4632 assert(cgi !is null); 4633 scope(exit) cgi.dispose(); 4634 4635 try { 4636 fun(cgi); 4637 cgi.close(); 4638 } catch (Throwable t) { 4639 version(CRuntime_Musl) { 4640 // LockingTextWriter fails here 4641 // so working around it 4642 auto s = t.msg; 4643 stderr.rawWrite(s); 4644 } else { 4645 stderr.writeln(t.msg); 4646 } 4647 if(!handleException(cgi, t)) 4648 return; 4649 } 4650 } 4651 4652 private __gshared int cancelfd = -1; 4653 4654 /+ 4655 The event loop for embedded_httpd_threads will prolly fiber dispatch 4656 cgi constructors too, so slow posts will not monopolize a worker thread. 4657 4658 May want to provide the worker task system just need to ensure all the fibers 4659 has a big enough stack for real work... would also ideally like to reuse them. 4660 4661 4662 So prolly bir would switch it to nonblocking. If it would block, it epoll 4663 registers one shot with this existing fiber to take it over. 4664 4665 new connection comes in. it picks a fiber off the free list, 4666 or if there is none, it creates a new one. this fiber handles 4667 this connection the whole time. 4668 4669 epoll triggers the fiber when something comes in. it is called by 4670 a random worker thread, it might change at any time. at least during 4671 the constructor. maybe into the main body it will stay tied to a thread 4672 just so TLS stuff doesn't randomly change in the middle. but I could 4673 specify if you yield all bets are off. 4674 4675 when the request is finished, if there's more data buffered, it just 4676 keeps going. if there is no more data buffered, it epoll ctls to 4677 get triggered when more data comes in. all one shot. 4678 4679 when a connection is closed, the fiber returns and is then reset 4680 and added to the free list. if the free list is full, the fiber is 4681 just freed, this means it will balloon to a certain size but not generally 4682 grow beyond that unless the activity keeps going. 4683 4684 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4685 4686 So the fiber has its own magic methods to read and write. if they would block, it registers 4687 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4688 4689 basically you issue the command and it tells you when it is done 4690 4691 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4692 4693 +/ 4694 4695 /++ 4696 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4697 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4698 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4699 4700 History: 4701 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4702 +/ 4703 version(cgi_use_fiber) 4704 __gshared size_t fiberStackSize = 4096 * 100; 4705 4706 version(cgi_use_fiber) 4707 class CgiFiber : Fiber { 4708 private void function(Socket) f_handler; 4709 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4710 f_handler(s); 4711 } 4712 this(void function(Socket) handler) { 4713 this.f_handler = handler; 4714 this(&f_handler_dg); 4715 } 4716 4717 this(void delegate(Socket) handler) { 4718 this.handler = handler; 4719 super(&run, fiberStackSize); 4720 } 4721 4722 Socket connection; 4723 void delegate(Socket) handler; 4724 4725 void run() { 4726 handler(connection); 4727 } 4728 4729 void delegate() postYield; 4730 4731 private void setPostYield(scope void delegate() py) @nogc { 4732 postYield = cast(void delegate()) py; 4733 } 4734 4735 void proceed() { 4736 try { 4737 call(); 4738 auto py = postYield; 4739 postYield = null; 4740 if(py !is null) 4741 py(); 4742 } catch(Exception e) { 4743 if(connection) 4744 connection.close(); 4745 goto terminate; 4746 } 4747 4748 if(state == State.TERM) { 4749 terminate: 4750 import core.memory; 4751 GC.removeRoot(cast(void*) this); 4752 } 4753 } 4754 } 4755 4756 version(cgi_use_fiber) 4757 version(Windows) { 4758 4759 extern(Windows) private { 4760 4761 import core.sys.windows.mswsock; 4762 4763 alias GROUP=uint; 4764 alias LPWSAPROTOCOL_INFOW = void*; 4765 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4766 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4767 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4768 4769 struct WSABUF { 4770 ULONG len; 4771 CHAR *buf; 4772 } 4773 alias LPWSABUF = WSABUF*; 4774 4775 alias WSAOVERLAPPED = OVERLAPPED; 4776 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4777 /+ 4778 4779 alias LPFN_ACCEPTEX = 4780 BOOL 4781 function( 4782 SOCKET sListenSocket, 4783 SOCKET sAcceptSocket, 4784 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4785 void* lpOutputBuffer, 4786 WORD dwReceiveDataLength, 4787 WORD dwLocalAddressLength, 4788 WORD dwRemoteAddressLength, 4789 LPDWORD lpdwBytesReceived, 4790 LPOVERLAPPED lpOverlapped 4791 ); 4792 4793 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4794 +/ 4795 4796 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4797 } 4798 4799 private class PseudoblockingOverlappedSocket : Socket { 4800 SOCKET handle; 4801 4802 CgiFiber fiber; 4803 4804 this(AddressFamily af, SocketType st) { 4805 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4806 if(!handle) 4807 throw new Exception("WSASocketW"); 4808 this.handle = handle; 4809 4810 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4811 4812 if(iocp is null) { 4813 writeln(GetLastError()); 4814 throw new Exception("CreateIoCompletionPort"); 4815 } 4816 4817 super(cast(socket_t) handle, af); 4818 } 4819 this() pure nothrow @trusted { assert(0); } 4820 4821 override void blocking(bool) {} // meaningless to us, just ignore it. 4822 4823 protected override Socket accepting() pure nothrow { 4824 assert(0); 4825 } 4826 4827 bool addressesParsed; 4828 Address la; 4829 Address ra; 4830 4831 private void populateAddresses() { 4832 if(addressesParsed) 4833 return; 4834 addressesParsed = true; 4835 4836 int lalen, ralen; 4837 4838 sockaddr_in* la; 4839 sockaddr_in* ra; 4840 4841 lpfnGetAcceptExSockaddrs( 4842 scratchBuffer.ptr, 4843 0, // same as in the AcceptEx call! 4844 sockaddr_in.sizeof + 16, 4845 sockaddr_in.sizeof + 16, 4846 cast(sockaddr**) &la, 4847 &lalen, 4848 cast(sockaddr**) &ra, 4849 &ralen 4850 ); 4851 4852 if(la) 4853 this.la = new InternetAddress(*la); 4854 if(ra) 4855 this.ra = new InternetAddress(*ra); 4856 4857 } 4858 4859 override @property @trusted Address localAddress() { 4860 populateAddresses(); 4861 return la; 4862 } 4863 override @property @trusted Address remoteAddress() { 4864 populateAddresses(); 4865 return ra; 4866 } 4867 4868 PseudoblockingOverlappedSocket accepted; 4869 4870 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4871 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4872 4873 override Socket accept() @trusted { 4874 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4875 4876 if(lpfnAcceptEx is null) { 4877 DWORD dwBytes; 4878 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4879 4880 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4881 &GuidAcceptEx, GuidAcceptEx.sizeof, 4882 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4883 &dwBytes, null, null); 4884 4885 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4886 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4887 &GuidAcceptEx, GuidAcceptEx.sizeof, 4888 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4889 &dwBytes, null, null); 4890 4891 } 4892 4893 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4894 accepted = pfa; 4895 4896 SOCKET pendingForAccept = pfa.handle; 4897 DWORD ignored; 4898 4899 auto ret = lpfnAcceptEx(handle, 4900 pendingForAccept, 4901 // buffer to receive up front 4902 pfa.scratchBuffer.ptr, 4903 0, 4904 // size of local and remote addresses. normally + 16. 4905 sockaddr_in.sizeof + 16, 4906 sockaddr_in.sizeof + 16, 4907 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4908 &overlapped 4909 ); 4910 4911 return pfa; 4912 } 4913 4914 override void connect(Address to) { assert(0); } 4915 4916 DWORD lastAnswer; 4917 ubyte[1024] scratchBuffer; 4918 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4919 4920 WSABUF[1] buffer; 4921 OVERLAPPED overlapped; 4922 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4923 overlapped = overlapped.init; 4924 buffer[0].len = cast(DWORD) buf.length; 4925 buffer[0].buf = cast(CHAR*) buf.ptr; 4926 fiber.setPostYield( () { 4927 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4928 if(GetLastError() != 997) { 4929 //throw new Exception("WSASend fail"); 4930 } 4931 } 4932 }); 4933 4934 Fiber.yield(); 4935 return lastAnswer; 4936 } 4937 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4938 overlapped = overlapped.init; 4939 buffer[0].len = cast(DWORD) buf.length; 4940 buffer[0].buf = cast(CHAR*) buf.ptr; 4941 4942 DWORD flags2 = 0; 4943 4944 fiber.setPostYield(() { 4945 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4946 if(GetLastError() != 997) { 4947 //writeln("WSARecv ", WSAGetLastError()); 4948 //throw new Exception("WSARecv fail"); 4949 } 4950 } 4951 }); 4952 4953 Fiber.yield(); 4954 return lastAnswer; 4955 } 4956 4957 // I might go back and implement these for udp things. 4958 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4959 assert(0); 4960 } 4961 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4962 assert(0); 4963 } 4964 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4965 assert(0); 4966 } 4967 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4968 assert(0); 4969 } 4970 4971 // lol overload sets 4972 alias send = typeof(super).send; 4973 alias receive = typeof(super).receive; 4974 alias sendTo = typeof(super).sendTo; 4975 alias receiveFrom = typeof(super).receiveFrom; 4976 4977 } 4978 } 4979 4980 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 4981 assert(connection !is null); 4982 version(cgi_use_fiber) { 4983 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 4984 4985 version(Windows) { 4986 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 4987 } 4988 4989 import core.memory; 4990 GC.addRoot(cast(void*) fiber); 4991 fiber.connection = connection; 4992 fiber.proceed(); 4993 } else { 4994 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 4995 } 4996 } 4997 4998 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 4999 scope(failure) { 5000 // catch all for other errors 5001 try { 5002 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5003 connection.close(); 5004 } catch(Exception e) {} // swallow it, we're aborting anyway. 5005 } 5006 5007 bool closeConnection = alwaysCloseConnection; 5008 5009 /+ 5010 ubyte[4096] inputBuffer = void; 5011 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5012 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5013 5014 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5015 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5016 ir.__ctor(connection, inputBuffer[], true); 5017 +/ 5018 5019 auto ir = new BufferedInputRange(connection); 5020 5021 while(!ir.empty) { 5022 5023 if(ir.view.length == 0) { 5024 ir.popFront(); 5025 if(ir.sourceClosed) { 5026 connection.close(); 5027 closeConnection = true; 5028 break; 5029 } 5030 } 5031 5032 Cgi cgi; 5033 try { 5034 cgi = new CustomCgi(ir, &closeConnection); 5035 // There's a bunch of these casts around because the type matches up with 5036 // the -version=.... specifiers, just you can also create a RequestServer 5037 // and instantiate the things where the types don't match up. It isn't exactly 5038 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5039 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5040 } catch(ConnectionClosedException ce) { 5041 closeConnection = true; 5042 break; 5043 } catch(ConnectionException ce) { 5044 // broken pipe or something, just abort the connection 5045 closeConnection = true; 5046 break; 5047 } catch(HttpVersionNotSupportedException ve) { 5048 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5049 closeConnection = true; 5050 break; 5051 } catch(Throwable t) { 5052 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5053 // anyway let's kill the connection 5054 version(CRuntime_Musl) { 5055 stderr.rawWrite(t.toString()); 5056 stderr.rawWrite("\n"); 5057 } else { 5058 stderr.writeln(t.toString()); 5059 } 5060 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5061 closeConnection = true; 5062 break; 5063 } 5064 assert(cgi !is null); 5065 scope(exit) 5066 cgi.dispose(); 5067 5068 try { 5069 fun(cgi); 5070 cgi.close(); 5071 if(cgi.websocketMode) 5072 closeConnection = true; 5073 } catch(ConnectionException ce) { 5074 // broken pipe or something, just abort the connection 5075 closeConnection = true; 5076 } catch(ConnectionClosedException ce) { 5077 // broken pipe or something, just abort the connection 5078 closeConnection = true; 5079 } catch(Throwable t) { 5080 // a processing error can be recovered from 5081 version(CRuntime_Musl) {} else 5082 stderr.writeln(t.toString); 5083 if(!handleException(cgi, t)) 5084 closeConnection = true; 5085 } 5086 5087 if(globalStopFlag) 5088 closeConnection = true; 5089 5090 if(closeConnection || alwaysCloseConnection) { 5091 connection.shutdown(SocketShutdown.BOTH); 5092 connection.close(); 5093 ir.dispose(); 5094 closeConnection = false; // don't reclose after loop 5095 break; 5096 } else { 5097 if(ir.front.length) { 5098 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5099 } else if(ir.sourceClosed) { 5100 ir.source.shutdown(SocketShutdown.BOTH); 5101 ir.source.close(); 5102 ir.dispose(); 5103 closeConnection = false; 5104 } else { 5105 continue; 5106 // break; // this was for a keepalive experiment 5107 } 5108 } 5109 } 5110 5111 if(closeConnection) { 5112 connection.shutdown(SocketShutdown.BOTH); 5113 connection.close(); 5114 ir.dispose(); 5115 } 5116 5117 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5118 } 5119 5120 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5121 // and now we can buffer 5122 scope(failure) 5123 connection.close(); 5124 5125 import al = std.algorithm; 5126 5127 size_t size; 5128 5129 string[string] headers; 5130 5131 auto range = new BufferedInputRange(connection); 5132 more_data: 5133 auto chunk = range.front(); 5134 // waiting for colon for header length 5135 auto idx = indexOf(cast(string) chunk, ':'); 5136 if(idx == -1) { 5137 try { 5138 range.popFront(); 5139 } catch(Exception e) { 5140 // it is just closed, no big deal 5141 connection.close(); 5142 return; 5143 } 5144 goto more_data; 5145 } 5146 5147 size = to!size_t(cast(string) chunk[0 .. idx]); 5148 chunk = range.consume(idx + 1); 5149 // reading headers 5150 if(chunk.length < size) 5151 range.popFront(0, size + 1); 5152 // we are now guaranteed to have enough 5153 chunk = range.front(); 5154 assert(chunk.length > size); 5155 5156 idx = 0; 5157 string key; 5158 string value; 5159 foreach(part; al.splitter(chunk, '\0')) { 5160 if(idx & 1) { // odd is value 5161 value = cast(string)(part.idup); 5162 headers[key] = value; // commit 5163 } else 5164 key = cast(string)(part.idup); 5165 idx++; 5166 } 5167 5168 enforce(chunk[size] == ','); // the terminator 5169 5170 range.consume(size + 1); 5171 // reading data 5172 // this will be done by Cgi 5173 5174 const(ubyte)[] getScgiChunk() { 5175 // we are already primed 5176 auto data = range.front(); 5177 if(data.length == 0 && !range.sourceClosed) { 5178 range.popFront(0); 5179 data = range.front(); 5180 } else if (range.sourceClosed) 5181 range.source.close(); 5182 5183 return data; 5184 } 5185 5186 void writeScgi(const(ubyte)[] data) { 5187 sendAll(connection, data); 5188 } 5189 5190 void flushScgi() { 5191 // I don't *think* I have to do anything.... 5192 } 5193 5194 Cgi cgi; 5195 try { 5196 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5197 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5198 } catch(Throwable t) { 5199 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5200 connection.close(); 5201 return; // this connection is dead 5202 } 5203 assert(cgi !is null); 5204 scope(exit) cgi.dispose(); 5205 try { 5206 fun(cgi); 5207 cgi.close(); 5208 connection.close(); 5209 } catch(Throwable t) { 5210 // no std err 5211 if(!handleException(cgi, t)) { 5212 connection.close(); 5213 return; 5214 } else { 5215 connection.close(); 5216 return; 5217 } 5218 } 5219 } 5220 5221 string printDate(DateTime date) { 5222 char[29] buffer = void; 5223 printDateToBuffer(date, buffer[]); 5224 return buffer.idup; 5225 } 5226 5227 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5228 assert(buffer.length >= 29); 5229 // 29 static length ? 5230 5231 static immutable daysOfWeek = [ 5232 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5233 ]; 5234 5235 static immutable months = [ 5236 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5237 ]; 5238 5239 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5240 buffer[3 .. 5] = ", "; 5241 buffer[5] = date.day / 10 + '0'; 5242 buffer[6] = date.day % 10 + '0'; 5243 buffer[7] = ' '; 5244 buffer[8 .. 11] = months[date.month]; 5245 buffer[11] = ' '; 5246 auto y = date.year; 5247 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5248 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5249 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5250 buffer[15] = cast(char) (y + '0'); 5251 buffer[16] = ' '; 5252 buffer[17] = date.hour / 10 + '0'; 5253 buffer[18] = date.hour % 10 + '0'; 5254 buffer[19] = ':'; 5255 buffer[20] = date.minute / 10 + '0'; 5256 buffer[21] = date.minute % 10 + '0'; 5257 buffer[22] = ':'; 5258 buffer[23] = date.second / 10 + '0'; 5259 buffer[24] = date.second % 10 + '0'; 5260 buffer[25 .. $] = " GMT"; 5261 5262 return 29; 5263 } 5264 5265 5266 // Referencing this gigantic typeid seems to remind the compiler 5267 // to actually put the symbol in the object file. I guess the immutable 5268 // assoc array array isn't actually included in druntime 5269 void hackAroundLinkerError() { 5270 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5271 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5272 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5273 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5274 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5275 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5276 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5277 // this is getting kinda ridiculous btw. Moving assoc arrays 5278 // to the library is the pain that keeps on coming. 5279 5280 // eh this broke the build on the work server 5281 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5282 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5283 } 5284 5285 5286 5287 5288 5289 version(fastcgi) { 5290 pragma(lib, "fcgi"); 5291 5292 static if(size_t.sizeof == 8) // 64 bit 5293 alias long c_int; 5294 else 5295 alias int c_int; 5296 5297 extern(C) { 5298 struct FCGX_Stream { 5299 ubyte* rdNext; 5300 ubyte* wrNext; 5301 ubyte* stop; 5302 ubyte* stopUnget; 5303 c_int isReader; 5304 c_int isClosed; 5305 c_int wasFCloseCalled; 5306 c_int FCGI_errno; 5307 void* function(FCGX_Stream* stream) fillBuffProc; 5308 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5309 void* data; 5310 } 5311 5312 // note: this is meant to be opaque, so don't access it directly 5313 struct FCGX_Request { 5314 int requestId; 5315 int role; 5316 FCGX_Stream* inStream; 5317 FCGX_Stream* outStream; 5318 FCGX_Stream* errStream; 5319 char** envp; 5320 void* paramsPtr; 5321 int ipcFd; 5322 int isBeginProcessed; 5323 int keepConnection; 5324 int appStatus; 5325 int nWriters; 5326 int flags; 5327 int listen_sock; 5328 } 5329 5330 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5331 void FCGX_Init(); 5332 5333 int FCGX_Accept_r(FCGX_Request *request); 5334 5335 5336 alias char** FCGX_ParamArray; 5337 5338 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5339 c_int FCGX_GetChar(FCGX_Stream* stream); 5340 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5341 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5342 c_int FCGX_FFlush(FCGX_Stream *stream); 5343 5344 int FCGX_OpenSocket(in char*, int); 5345 } 5346 } 5347 5348 5349 /* This might go int a separate module eventually. It is a network input helper class. */ 5350 5351 import std.socket; 5352 5353 version(cgi_use_fiber) { 5354 import core.thread; 5355 5356 version(linux) { 5357 import core.sys.linux.epoll; 5358 5359 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5360 } else version(Windows) { 5361 // declaring the iocp thing below... 5362 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5363 } 5364 5365 version(Windows) 5366 __gshared HANDLE iocp; 5367 5368 version(cgi_use_fiber) { 5369 version(linux) 5370 private enum WakeupEvent { 5371 Read = EPOLLIN, 5372 Write = EPOLLOUT 5373 } 5374 else version(Windows) 5375 private enum WakeupEvent { 5376 Read, Write 5377 } 5378 else static assert(0); 5379 } 5380 5381 version(cgi_use_fiber) 5382 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5383 5384 // static cast since I know what i have in here and don't want to pay for dynamic cast 5385 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5386 5387 version(linux) { 5388 f.setPostYield = () { 5389 if(*registered) { 5390 // rearm 5391 epoll_event evt; 5392 evt.events = e | EPOLLONESHOT; 5393 evt.data.ptr = cast(void*) f; 5394 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5395 throw new Exception("epoll_ctl"); 5396 } else { 5397 // initial registration 5398 *registered = true ; 5399 int fd = source.handle; 5400 epoll_event evt; 5401 evt.events = e | EPOLLONESHOT; 5402 evt.data.ptr = cast(void*) f; 5403 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5404 throw new Exception("epoll_ctl"); 5405 } 5406 }; 5407 5408 Fiber.yield(); 5409 5410 f.setPostYield(null); 5411 } else version(Windows) { 5412 Fiber.yield(); 5413 } 5414 else static assert(0); 5415 } 5416 5417 version(cgi_use_fiber) 5418 void unregisterSource(Socket s) { 5419 version(linux) { 5420 epoll_event evt; 5421 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5422 } else version(Windows) { 5423 // intentionally blank 5424 } 5425 else static assert(0); 5426 } 5427 5428 // it is a class primarily for reference semantics 5429 // I might change this interface 5430 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5431 class BufferedInputRange { 5432 version(Posix) 5433 this(int source, ubyte[] buffer = null) { 5434 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5435 } 5436 5437 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5438 // if they connect but never send stuff to us, we don't want it wasting the process 5439 // so setting a time out 5440 version(cgi_use_fiber) 5441 source.blocking = false; 5442 else 5443 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5444 5445 this.source = source; 5446 if(buffer is null) { 5447 underlyingBuffer = new ubyte[4096]; 5448 this.allowGrowth = true; 5449 } else { 5450 underlyingBuffer = buffer; 5451 this.allowGrowth = allowGrowth; 5452 } 5453 5454 assert(underlyingBuffer.length); 5455 5456 // we assume view.ptr is always inside underlyingBuffer 5457 view = underlyingBuffer[0 .. 0]; 5458 5459 popFront(); // prime 5460 } 5461 5462 version(cgi_use_fiber) { 5463 bool registered; 5464 } 5465 5466 void dispose() { 5467 version(cgi_use_fiber) { 5468 if(registered) 5469 unregisterSource(source); 5470 } 5471 } 5472 5473 /** 5474 A slight difference from regular ranges is you can give it the maximum 5475 number of bytes to consume. 5476 5477 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5478 consume() yourself and use a regular foreach, it will infinitely loop! 5479 5480 The default is to do what a normal range does, and consume the whole buffer 5481 and wait for additional input. 5482 5483 You can also specify 0, to append to the buffer, or any other number 5484 to remove the front n bytes and wait for more. 5485 */ 5486 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5487 if(sourceClosed) 5488 throw new ConnectionClosedException("can't get any more data from a closed source"); 5489 if(!skipConsume) 5490 consume(maxBytesToConsume); 5491 5492 // we might have to grow the buffer 5493 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5494 if(allowGrowth) { 5495 //import std.stdio; writeln("growth"); 5496 auto viewStart = view.ptr - underlyingBuffer.ptr; 5497 size_t growth = 4096; 5498 // make sure we have enough for what we're being asked for 5499 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5500 growth = minBytesToSettleFor - underlyingBuffer.length; 5501 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5502 underlyingBuffer.length += growth; 5503 view = underlyingBuffer[viewStart .. view.length]; 5504 } else 5505 throw new Exception("No room left in the buffer"); 5506 } 5507 5508 do { 5509 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5510 try_again: 5511 auto ret = source.receive(freeSpace); 5512 if(ret == Socket.ERROR) { 5513 if(wouldHaveBlocked()) { 5514 version(cgi_use_fiber) { 5515 registerEventWakeup(®istered, source, WakeupEvent.Read); 5516 goto try_again; 5517 } else { 5518 // gonna treat a timeout here as a close 5519 sourceClosed = true; 5520 return; 5521 } 5522 } 5523 version(Posix) { 5524 import core.stdc.errno; 5525 if(errno == EINTR || errno == EAGAIN) { 5526 goto try_again; 5527 } 5528 if(errno == ECONNRESET) { 5529 sourceClosed = true; 5530 return; 5531 } 5532 } 5533 throw new Exception(lastSocketError); // FIXME 5534 } 5535 if(ret == 0) { 5536 sourceClosed = true; 5537 return; 5538 } 5539 5540 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5541 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5542 //import std.stdio; writeln(cast(string) view); 5543 } while(view.length < minBytesToSettleFor); 5544 } 5545 5546 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5547 /// You might want to idup the data you are consuming if you store it, since it may 5548 /// be overwritten on the new popFront. 5549 /// 5550 /// You do not need to call this if you always want to wait for more data when you 5551 /// consume some. 5552 ubyte[] consume(size_t bytes) { 5553 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5554 view = view[bytes > $ ? $ : bytes .. $]; 5555 if(view.length == 0) { 5556 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5557 /* 5558 writeln("HERE"); 5559 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5560 writeln(cast(string)front); 5561 writeln("DONE"); 5562 */ 5563 } 5564 return front; 5565 } 5566 5567 bool empty() { 5568 return sourceClosed && view.length == 0; 5569 } 5570 5571 ubyte[] front() { 5572 return view; 5573 } 5574 5575 invariant() { 5576 assert(view.ptr >= underlyingBuffer.ptr); 5577 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5578 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5579 } 5580 5581 ubyte[] underlyingBuffer; 5582 bool allowGrowth; 5583 ubyte[] view; 5584 Socket source; 5585 bool sourceClosed; 5586 } 5587 5588 private class FakeSocketForStdin : Socket { 5589 import std.stdio; 5590 5591 this() { 5592 5593 } 5594 5595 private bool closed; 5596 5597 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5598 if(closed) 5599 throw new Exception("Closed"); 5600 return stdin.rawRead(buffer).length; 5601 } 5602 5603 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5604 if(closed) 5605 throw new Exception("Closed"); 5606 stdout.rawWrite(buffer); 5607 return buffer.length; 5608 } 5609 5610 override void close() @trusted scope { 5611 (cast(void delegate() @nogc nothrow) &realClose)(); 5612 } 5613 5614 override void shutdown(SocketShutdown s) { 5615 // FIXME 5616 } 5617 5618 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5619 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5620 5621 override @property @trusted Address remoteAddress() { return null; } 5622 override @property @trusted Address localAddress() { return null; } 5623 5624 void realClose() { 5625 closed = true; 5626 try { 5627 stdin.close(); 5628 stdout.close(); 5629 } catch(Exception e) { 5630 5631 } 5632 } 5633 } 5634 5635 import core.sync.semaphore; 5636 import core.atomic; 5637 5638 /** 5639 To use this thing: 5640 5641 --- 5642 void handler(Socket s) { do something... } 5643 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5644 manager.listen(); 5645 --- 5646 5647 The 4th parameter is optional. 5648 5649 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5650 comes in, you will get control. You can just continue; though to fetch more. 5651 5652 5653 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5654 */ 5655 class ListeningConnectionManager { 5656 Semaphore semaphore; 5657 Socket[256] queue; 5658 shared(ubyte) nextIndexFront; 5659 ubyte nextIndexBack; 5660 shared(int) queueLength; 5661 5662 Socket acceptCancelable() { 5663 version(Posix) { 5664 import core.sys.posix.sys.select; 5665 fd_set read_fds; 5666 FD_ZERO(&read_fds); 5667 int max = 0; 5668 foreach(listener; listeners) { 5669 FD_SET(listener.handle, &read_fds); 5670 if(listener.handle > max) 5671 max = listener.handle; 5672 } 5673 if(cancelfd != -1) { 5674 FD_SET(cancelfd, &read_fds); 5675 if(cancelfd > max) 5676 max = cancelfd; 5677 } 5678 auto ret = select(max + 1, &read_fds, null, null, null); 5679 if(ret == -1) { 5680 import core.stdc.errno; 5681 if(errno == EINTR) 5682 return null; 5683 else 5684 throw new Exception("wtf select"); 5685 } 5686 5687 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5688 return null; 5689 } 5690 5691 foreach(listener; listeners) { 5692 if(FD_ISSET(listener.handle, &read_fds)) 5693 return listener.accept(); 5694 } 5695 5696 return null; 5697 } else { 5698 5699 auto check = new SocketSet(); 5700 5701 keep_looping: 5702 check.reset(); 5703 foreach(listener; listeners) 5704 check.add(listener); 5705 5706 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5707 auto got = Socket.select(check, null, null, 3.seconds); 5708 if(got > 0) 5709 foreach(listener; listeners) 5710 if(check.isSet(listener)) 5711 return listener.accept(); 5712 if(globalStopFlag) 5713 return null; 5714 else 5715 goto keep_looping; 5716 } 5717 } 5718 5719 int defaultNumberOfThreads() { 5720 import std.parallelism; 5721 version(cgi_use_fiber) { 5722 return totalCPUs * 1 + 1; 5723 } else { 5724 // I times 4 here because there's a good chance some will be blocked on i/o. 5725 return totalCPUs * 4; 5726 } 5727 5728 } 5729 5730 void listen() { 5731 shared(int) loopBroken; 5732 5733 version(Posix) { 5734 import core.sys.posix.signal; 5735 signal(SIGPIPE, SIG_IGN); 5736 } 5737 5738 version(linux) { 5739 if(cancelfd == -1) 5740 cancelfd = eventfd(0, 0); 5741 } 5742 5743 version(cgi_no_threads) { 5744 // NEVER USE THIS 5745 // it exists only for debugging and other special occasions 5746 5747 // the thread mode is faster and less likely to stall the whole 5748 // thing when a request is slow 5749 while(!loopBroken && !globalStopFlag) { 5750 auto sn = acceptCancelable(); 5751 if(sn is null) continue; 5752 cloexec(sn); 5753 try { 5754 handler(sn); 5755 } catch(Exception e) { 5756 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5757 sn.close(); 5758 } 5759 } 5760 } else { 5761 5762 if(useFork) { 5763 version(linux) { 5764 //asm { int 3; } 5765 fork(); 5766 } 5767 } 5768 5769 version(cgi_use_fiber) { 5770 5771 version(Windows) { 5772 // please note these are overlapped sockets! so the accept just kicks things off 5773 foreach(listener; listeners) 5774 listener.accept(); 5775 } 5776 5777 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5778 foreach(i, ref thread; threads) { 5779 thread = new WorkerThread(this, handler, cast(int) i); 5780 thread.start(); 5781 } 5782 5783 bool fiber_crash_check() { 5784 bool hasAnyRunning; 5785 foreach(thread; threads) { 5786 if(!thread.isRunning) { 5787 thread.join(); 5788 } else hasAnyRunning = true; 5789 } 5790 5791 return (!hasAnyRunning); 5792 } 5793 5794 5795 while(!globalStopFlag) { 5796 Thread.sleep(1.seconds); 5797 if(fiber_crash_check()) 5798 break; 5799 } 5800 5801 } else { 5802 semaphore = new Semaphore(); 5803 5804 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5805 foreach(i, ref thread; threads) { 5806 thread = new ConnectionThread(this, handler, cast(int) i); 5807 thread.start(); 5808 } 5809 5810 while(!loopBroken && !globalStopFlag) { 5811 Socket sn; 5812 5813 bool crash_check() { 5814 bool hasAnyRunning; 5815 foreach(thread; threads) { 5816 if(!thread.isRunning) { 5817 thread.join(); 5818 } else hasAnyRunning = true; 5819 } 5820 5821 return (!hasAnyRunning); 5822 } 5823 5824 5825 void accept_new_connection() { 5826 sn = acceptCancelable(); 5827 if(sn is null) return; 5828 cloexec(sn); 5829 if(tcp) { 5830 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5831 // on the socket because we do some buffering internally. I think this helps, 5832 // certainly does for small requests, and I think it does for larger ones too 5833 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5834 5835 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5836 } 5837 } 5838 5839 void existing_connection_new_data() { 5840 // wait until a slot opens up 5841 //int waited = 0; 5842 while(queueLength >= queue.length) { 5843 Thread.sleep(1.msecs); 5844 //waited ++; 5845 } 5846 //if(waited) {import std.stdio; writeln(waited);} 5847 synchronized(this) { 5848 queue[nextIndexBack] = sn; 5849 nextIndexBack++; 5850 atomicOp!"+="(queueLength, 1); 5851 } 5852 semaphore.notify(); 5853 } 5854 5855 5856 accept_new_connection(); 5857 if(sn !is null) 5858 existing_connection_new_data(); 5859 else if(sn is null && globalStopFlag) { 5860 foreach(thread; threads) { 5861 semaphore.notify(); 5862 } 5863 Thread.sleep(50.msecs); 5864 } 5865 5866 if(crash_check()) 5867 break; 5868 } 5869 } 5870 5871 // FIXME: i typically stop this with ctrl+c which never 5872 // actually gets here. i need to do a sigint handler. 5873 if(cleanup) 5874 cleanup(); 5875 } 5876 } 5877 5878 //version(linux) 5879 //int epoll_fd; 5880 5881 bool tcp; 5882 void delegate() cleanup; 5883 5884 private void function(Socket) fhandler; 5885 private void dg_handler(Socket s) { 5886 fhandler(s); 5887 } 5888 5889 5890 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5891 fhandler = handler; 5892 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 5893 } 5894 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5895 string[] host; 5896 ushort[] port; 5897 5898 foreach(spec; listenSpec) { 5899 /+ 5900 The format: 5901 5902 protocol:// 5903 address_spec 5904 5905 Protocol is optional. Must be http, https, scgi, or fastcgi. 5906 5907 address_spec is either: 5908 ipv4 address : port 5909 [ipv6 address] : port 5910 unix:filename 5911 abstract:name 5912 port <which is tcp but on any interface> 5913 +/ 5914 5915 string protocol; 5916 string address_spec; 5917 5918 auto protocolIdx = spec.indexOf("://"); 5919 if(protocolIdx != -1) { 5920 protocol = spec[0 .. protocolIdx]; 5921 address_spec = spec[protocolIdx + "://".length .. $]; 5922 } else { 5923 address_spec = spec; 5924 } 5925 5926 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 5927 host ~= address_spec; 5928 port ~= 0; 5929 } else { 5930 auto idx = address_spec.lastIndexOf(":"); 5931 if(idx == -1) { 5932 host ~= null; 5933 } else { 5934 auto as = address_spec[0 .. idx]; 5935 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 5936 as = as[1 .. $-1]; 5937 host ~= as; 5938 } 5939 port ~= address_spec[idx + 1 .. $].to!ushort; 5940 } 5941 5942 } 5943 5944 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 5945 } 5946 5947 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5948 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5949 } 5950 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5951 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5952 } 5953 5954 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5955 fhandler = handler; 5956 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5957 } 5958 5959 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5960 assert(host.length == port.length); 5961 5962 this.handler = handler; 5963 this.useFork = useFork; 5964 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5965 5966 listeners.reserve(host.length); 5967 5968 foreach(i; 0 .. host.length) 5969 if(host[i] == "localhost") { 5970 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 5971 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 5972 } else { 5973 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 5974 } 5975 5976 version(cgi_use_fiber) 5977 if(useFork) { 5978 foreach(listener; listeners) 5979 listener.blocking = false; 5980 } 5981 5982 // this is the UI control thread and thus gets more priority 5983 Thread.getThis.priority = Thread.PRIORITY_MAX; 5984 } 5985 5986 Socket[] listeners; 5987 void delegate(Socket) handler; 5988 5989 immutable bool useFork; 5990 int numberOfThreads; 5991 } 5992 5993 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 5994 Socket listener; 5995 if(host.startsWith("unix:")) { 5996 version(Posix) { 5997 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5998 cloexec(listener); 5999 string filename = host["unix:".length .. $].idup; 6000 listener.bind(new UnixAddress(filename)); 6001 cleanup = delegate() { 6002 listener.close(); 6003 import std.file; 6004 remove(filename); 6005 }; 6006 tcp = false; 6007 } else { 6008 throw new Exception("unix sockets not supported on this system"); 6009 } 6010 } else if(host.startsWith("abstract:")) { 6011 version(linux) { 6012 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6013 cloexec(listener); 6014 string filename = "\0" ~ host["abstract:".length .. $]; 6015 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6016 listener.bind(new UnixAddress(filename)); 6017 tcp = false; 6018 } else { 6019 throw new Exception("abstract unix sockets not supported on this system"); 6020 } 6021 } else { 6022 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6023 version(cgi_use_fiber) { 6024 version(Windows) 6025 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6026 else 6027 listener = new Socket(address.addressFamily, SocketType.STREAM); 6028 } else { 6029 listener = new Socket(address.addressFamily, SocketType.STREAM); 6030 } 6031 cloexec(listener); 6032 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6033 listener.bind(address); 6034 cleanup = delegate() { 6035 listener.close(); 6036 }; 6037 tcp = true; 6038 } 6039 6040 listener.listen(backQueue); 6041 6042 if (dropPrivs !is null) // can be null, backwards compatibility 6043 dropPrivs(); 6044 6045 return listener; 6046 } 6047 6048 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6049 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6050 if(data.length == 0) return; 6051 ptrdiff_t amount; 6052 //import std.stdio; writeln("***",cast(string) data,"///"); 6053 do { 6054 amount = s.send(data); 6055 if(amount == Socket.ERROR) { 6056 version(cgi_use_fiber) { 6057 if(wouldHaveBlocked()) { 6058 bool registered = true; 6059 registerEventWakeup(®istered, s, WakeupEvent.Write); 6060 continue; 6061 } 6062 } 6063 throw new ConnectionException(s, lastSocketError, file, line); 6064 } 6065 assert(amount > 0); 6066 6067 data = data[amount .. $]; 6068 } while(data.length); 6069 } 6070 6071 class ConnectionException : Exception { 6072 Socket socket; 6073 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6074 this.socket = s; 6075 super(msg, file, line); 6076 } 6077 } 6078 6079 class HttpVersionNotSupportedException : Exception { 6080 this(string file = __FILE__, size_t line = __LINE__) { 6081 super("HTTP Version Not Supported", file, line); 6082 } 6083 } 6084 6085 alias void delegate(Socket) CMT; 6086 6087 import core.thread; 6088 /+ 6089 cgi.d now uses a hybrid of event i/o and threads at the top level. 6090 6091 Top level thread is responsible for accepting sockets and selecting on them. 6092 6093 It then indicates to a child that a request is pending, and any random worker 6094 thread that is free handles it. It goes into blocking mode and handles that 6095 http request to completion. 6096 6097 At that point, it goes back into the waiting queue. 6098 6099 6100 This concept is only implemented on Linux. On all other systems, it still 6101 uses the worker threads and semaphores (which is perfectly fine for a lot of 6102 things! Just having a great number of keep-alive connections will break that.) 6103 6104 6105 So the algorithm is: 6106 6107 select(accept, event, pending) 6108 if accept -> send socket to free thread, if any. if not, add socket to queue 6109 if event -> send the signaling thread a socket from the queue, if not, mark it free 6110 - event might block until it can be *written* to. it is a fifo sending socket fds! 6111 6112 A worker only does one http request at a time, then signals its availability back to the boss. 6113 6114 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6115 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6116 actual FD will not be kept out here. 6117 6118 So: 6119 queue = sockets we know are ready to read now, but no worker thread is available 6120 idle list = worker threads not doing anything else. they signal back and forth 6121 6122 the workers all read off the event fd. This is the semaphore wait 6123 6124 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6125 it puts it in the queue and writes to the event fd. 6126 6127 The child could put the socket back in the epoll thing itself. 6128 6129 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6130 +/ 6131 class ConnectionThread : Thread { 6132 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6133 this.lcm = lcm; 6134 this.dg = dg; 6135 this.myThreadNumber = myThreadNumber; 6136 super(&run); 6137 } 6138 6139 void run() { 6140 while(true) { 6141 // so if there's a bunch of idle keep-alive connections, it can 6142 // consume all the worker threads... just sitting there. 6143 lcm.semaphore.wait(); 6144 if(globalStopFlag) 6145 return; 6146 Socket socket; 6147 synchronized(lcm) { 6148 auto idx = lcm.nextIndexFront; 6149 socket = lcm.queue[idx]; 6150 lcm.queue[idx] = null; 6151 atomicOp!"+="(lcm.nextIndexFront, 1); 6152 atomicOp!"-="(lcm.queueLength, 1); 6153 } 6154 try { 6155 //import std.stdio; writeln(myThreadNumber, " taking it"); 6156 dg(socket); 6157 /+ 6158 if(socket.isAlive) { 6159 // process it more later 6160 version(linux) { 6161 import core.sys.linux.epoll; 6162 epoll_event ev; 6163 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6164 ev.data.fd = socket.handle; 6165 import std.stdio; writeln("adding"); 6166 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6167 if(errno == EEXIST) { 6168 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6169 ev.data.fd = socket.handle; 6170 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6171 throw new Exception("epoll_ctl " ~ to!string(errno)); 6172 } else 6173 throw new Exception("epoll_ctl " ~ to!string(errno)); 6174 } 6175 //import std.stdio; writeln("keep alive"); 6176 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6177 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6178 } else { 6179 continue; // hope it times out in a reasonable amount of time... 6180 } 6181 } 6182 +/ 6183 } catch(ConnectionClosedException e) { 6184 // can just ignore this, it is fairly normal 6185 socket.close(); 6186 } catch(Throwable e) { 6187 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6188 socket.close(); 6189 } 6190 } 6191 } 6192 6193 ListeningConnectionManager lcm; 6194 CMT dg; 6195 int myThreadNumber; 6196 } 6197 6198 version(cgi_use_fiber) 6199 class WorkerThread : Thread { 6200 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6201 this.lcm = lcm; 6202 this.dg = dg; 6203 this.myThreadNumber = myThreadNumber; 6204 super(&run); 6205 } 6206 6207 version(Windows) 6208 void run() { 6209 auto timeout = INFINITE; 6210 PseudoblockingOverlappedSocket key; 6211 OVERLAPPED* overlapped; 6212 DWORD bytes; 6213 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6214 if(key is null) 6215 continue; 6216 key.lastAnswer = bytes; 6217 if(key.fiber) { 6218 key.fiber.proceed(); 6219 } else { 6220 // we have a new connection, issue the first receive on it and issue the next accept 6221 6222 auto sn = key.accepted; 6223 6224 key.accept(); 6225 6226 cloexec(sn); 6227 if(lcm.tcp) { 6228 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6229 // on the socket because we do some buffering internally. I think this helps, 6230 // certainly does for small requests, and I think it does for larger ones too 6231 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6232 6233 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6234 } 6235 6236 dg(sn); 6237 } 6238 } 6239 //SleepEx(INFINITE, TRUE); 6240 } 6241 6242 version(linux) 6243 void run() { 6244 6245 import core.sys.linux.epoll; 6246 epfd = epoll_create1(EPOLL_CLOEXEC); 6247 if(epfd == -1) 6248 throw new Exception("epoll_create1 " ~ to!string(errno)); 6249 scope(exit) { 6250 import core.sys.posix.unistd; 6251 close(epfd); 6252 } 6253 6254 { 6255 epoll_event ev; 6256 ev.events = EPOLLIN; 6257 ev.data.fd = cancelfd; 6258 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6259 } 6260 6261 foreach(listener; lcm.listeners) { 6262 epoll_event ev; 6263 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6264 ev.data.fd = listener.handle; 6265 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6266 throw new Exception("epoll_ctl " ~ to!string(errno)); 6267 } 6268 6269 6270 6271 while(!globalStopFlag) { 6272 Socket sn; 6273 6274 epoll_event[64] events; 6275 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6276 if(nfds == -1) { 6277 if(errno == EINTR) 6278 continue; 6279 throw new Exception("epoll_wait " ~ to!string(errno)); 6280 } 6281 6282 outer: foreach(idx; 0 .. nfds) { 6283 auto flags = events[idx].events; 6284 6285 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6286 globalStopFlag = true; 6287 //import std.stdio; writeln("exit heard"); 6288 break; 6289 } else { 6290 foreach(listener; lcm.listeners) { 6291 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6292 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6293 // this try/catch is because it is set to non-blocking mode 6294 // and Phobos' stupid api throws an exception instead of returning 6295 // if it would block. Why would it block? because a forked process 6296 // might have beat us to it, but the wakeup event thundered our herds. 6297 try 6298 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6299 catch(SocketAcceptException e) { continue; } 6300 6301 cloexec(sn); 6302 if(lcm.tcp) { 6303 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6304 // on the socket because we do some buffering internally. I think this helps, 6305 // certainly does for small requests, and I think it does for larger ones too 6306 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6307 6308 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6309 } 6310 6311 dg(sn); 6312 continue outer; 6313 } 6314 } 6315 6316 if(cast(size_t) events[idx].data.ptr < 1024) { 6317 throw new Exception("this doesn't look like a fiber pointer..."); 6318 } 6319 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6320 fiber.proceed(); 6321 } 6322 } 6323 } 6324 } 6325 6326 ListeningConnectionManager lcm; 6327 CMT dg; 6328 int myThreadNumber; 6329 } 6330 6331 6332 /* Done with network helper */ 6333 6334 /* Helpers for doing temporary files. Used both here and in web.d */ 6335 6336 version(Windows) { 6337 import core.sys.windows.windows; 6338 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6339 alias GetTempPathW GetTempPath; 6340 } 6341 6342 version(Posix) { 6343 static import linux = core.sys.posix.unistd; 6344 } 6345 6346 string getTempDirectory() { 6347 string path; 6348 version(Windows) { 6349 wchar[1024] buffer; 6350 auto len = GetTempPath(1024, buffer.ptr); 6351 if(len == 0) 6352 throw new Exception("couldn't find a temporary path"); 6353 6354 auto b = buffer[0 .. len]; 6355 6356 path = to!string(b); 6357 } else 6358 path = "/tmp/"; 6359 6360 return path; 6361 } 6362 6363 6364 // I like std.date. These functions help keep my old code and data working with phobos changing. 6365 6366 long sysTimeToDTime(in SysTime sysTime) { 6367 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6368 } 6369 6370 long dateTimeToDTime(in DateTime dt) { 6371 return sysTimeToDTime(cast(SysTime) dt); 6372 } 6373 6374 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6375 return sysTimeToDTime(Clock.currTime(UTC())); 6376 } 6377 6378 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6379 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6380 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6381 return SysTime(hnsecs, tz); 6382 } 6383 6384 6385 6386 // this is a helper to read HTTP transfer-encoding: chunked responses 6387 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6388 immutable(ubyte)[] ret; 6389 6390 another_chunk: 6391 // If here, we are at the beginning of a chunk. 6392 auto a = ir.front(); 6393 int chunkSize; 6394 int loc = locationOf(a, "\r\n"); 6395 while(loc == -1) { 6396 ir.popFront(); 6397 a = ir.front(); 6398 loc = locationOf(a, "\r\n"); 6399 } 6400 6401 string hex; 6402 hex = ""; 6403 for(int i = 0; i < loc; i++) { 6404 char c = a[i]; 6405 if(c >= 'A' && c <= 'Z') 6406 c += 0x20; 6407 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6408 hex ~= c; 6409 } else { 6410 break; 6411 } 6412 } 6413 6414 assert(hex.length); 6415 6416 int power = 1; 6417 int size = 0; 6418 foreach(cc1; retro(hex)) { 6419 dchar cc = cc1; 6420 if(cc >= 'a' && cc <= 'z') 6421 cc -= 0x20; 6422 int val = 0; 6423 if(cc >= '0' && cc <= '9') 6424 val = cc - '0'; 6425 else 6426 val = cc - 'A' + 10; 6427 6428 size += power * val; 6429 power *= 16; 6430 } 6431 6432 chunkSize = size; 6433 assert(size >= 0); 6434 6435 if(loc + 2 > a.length) { 6436 ir.popFront(0, a.length + loc + 2); 6437 a = ir.front(); 6438 } 6439 6440 a = ir.consume(loc + 2); 6441 6442 if(chunkSize == 0) { // we're done with the response 6443 // if we got here, will change must be true.... 6444 more_footers: 6445 loc = locationOf(a, "\r\n"); 6446 if(loc == -1) { 6447 ir.popFront(); 6448 a = ir.front; 6449 goto more_footers; 6450 } else { 6451 assert(loc == 0); 6452 ir.consume(loc + 2); 6453 goto finish; 6454 } 6455 } else { 6456 // if we got here, will change must be true.... 6457 if(a.length < chunkSize + 2) { 6458 ir.popFront(0, chunkSize + 2); 6459 a = ir.front(); 6460 } 6461 6462 ret ~= (a[0..chunkSize]); 6463 6464 if(!(a.length > chunkSize + 2)) { 6465 ir.popFront(0, chunkSize + 2); 6466 a = ir.front(); 6467 } 6468 assert(a[chunkSize] == 13); 6469 assert(a[chunkSize+1] == 10); 6470 a = ir.consume(chunkSize + 2); 6471 chunkSize = 0; 6472 goto another_chunk; 6473 } 6474 6475 finish: 6476 return ret; 6477 } 6478 6479 // I want to be able to get data from multiple sources the same way... 6480 interface ByChunkRange { 6481 bool empty(); 6482 void popFront(); 6483 const(ubyte)[] front(); 6484 } 6485 6486 ByChunkRange byChunk(const(ubyte)[] data) { 6487 return new class ByChunkRange { 6488 override bool empty() { 6489 return !data.length; 6490 } 6491 6492 override void popFront() { 6493 if(data.length > 4096) 6494 data = data[4096 .. $]; 6495 else 6496 data = null; 6497 } 6498 6499 override const(ubyte)[] front() { 6500 return data[0 .. $ > 4096 ? 4096 : $]; 6501 } 6502 }; 6503 } 6504 6505 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6506 const(ubyte)[] f; 6507 6508 f = ir.front; 6509 if(f.length > atMost) 6510 f = f[0 .. atMost]; 6511 6512 return new class ByChunkRange { 6513 override bool empty() { 6514 return atMost == 0; 6515 } 6516 6517 override const(ubyte)[] front() { 6518 return f; 6519 } 6520 6521 override void popFront() { 6522 ir.consume(f.length); 6523 atMost -= f.length; 6524 auto a = ir.front(); 6525 6526 if(a.length <= atMost) { 6527 f = a; 6528 atMost -= a.length; 6529 a = ir.consume(a.length); 6530 if(atMost != 0) 6531 ir.popFront(); 6532 if(f.length == 0) { 6533 f = ir.front(); 6534 } 6535 } else { 6536 // we actually have *more* here than we need.... 6537 f = a[0..atMost]; 6538 atMost = 0; 6539 ir.consume(atMost); 6540 } 6541 } 6542 }; 6543 } 6544 6545 version(cgi_with_websocket) { 6546 // http://tools.ietf.org/html/rfc6455 6547 6548 /++ 6549 WEBSOCKET SUPPORT: 6550 6551 Full example: 6552 --- 6553 import arsd.cgi; 6554 6555 void websocketEcho(Cgi cgi) { 6556 if(cgi.websocketRequested()) { 6557 if(cgi.origin != "http://arsdnet.net") 6558 throw new Exception("bad origin"); 6559 auto websocket = cgi.acceptWebsocket(); 6560 6561 websocket.send("hello"); 6562 websocket.send(" world!"); 6563 6564 auto msg = websocket.recv(); 6565 while(msg.opcode != WebSocketOpcode.close) { 6566 if(msg.opcode == WebSocketOpcode.text) { 6567 websocket.send(msg.textData); 6568 } else if(msg.opcode == WebSocketOpcode.binary) { 6569 websocket.send(msg.data); 6570 } 6571 6572 msg = websocket.recv(); 6573 } 6574 6575 websocket.close(); 6576 } else { 6577 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6578 } 6579 } 6580 6581 mixin GenericMain!websocketEcho; 6582 --- 6583 +/ 6584 6585 class WebSocket { 6586 Cgi cgi; 6587 6588 private this(Cgi cgi) { 6589 this.cgi = cgi; 6590 6591 Socket socket = cgi.idlol.source; 6592 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6593 } 6594 6595 // returns true if data available, false if it timed out 6596 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6597 if(!waitForNextMessageWouldBlock()) 6598 return true; 6599 if(isDataPending(timeout)) 6600 return true; // this is kinda a lie. 6601 6602 return false; 6603 } 6604 6605 public bool lowLevelReceive() { 6606 auto bfr = cgi.idlol; 6607 top: 6608 auto got = bfr.front; 6609 if(got.length) { 6610 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6611 receiveBuffer.length += receiveBufferUsedLength + got.length; 6612 6613 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6614 receiveBufferUsedLength += got.length; 6615 bfr.consume(got.length); 6616 6617 return true; 6618 } 6619 6620 if(bfr.sourceClosed) 6621 return false; 6622 6623 bfr.popFront(0); 6624 if(bfr.sourceClosed) 6625 return false; 6626 goto top; 6627 } 6628 6629 6630 bool isDataPending(Duration timeout = 0.seconds) { 6631 Socket socket = cgi.idlol.source; 6632 6633 auto check = new SocketSet(); 6634 check.add(socket); 6635 6636 auto got = Socket.select(check, null, null, timeout); 6637 if(got > 0) 6638 return true; 6639 return false; 6640 } 6641 6642 // note: this blocks 6643 WebSocketFrame recv() { 6644 return waitForNextMessage(); 6645 } 6646 6647 6648 6649 6650 private void llclose() { 6651 cgi.close(); 6652 } 6653 6654 private void llsend(ubyte[] data) { 6655 cgi.write(data); 6656 cgi.flush(); 6657 } 6658 6659 void unregisterActiveSocket(WebSocket) {} 6660 6661 /* copy/paste section { */ 6662 6663 private int readyState_; 6664 private ubyte[] receiveBuffer; 6665 private size_t receiveBufferUsedLength; 6666 6667 private Config config; 6668 6669 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6670 enum OPEN = 1; /// The connection is open and ready to communicate. 6671 enum CLOSING = 2; /// The connection is in the process of closing. 6672 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6673 6674 /++ 6675 6676 +/ 6677 /// Group: foundational 6678 static struct Config { 6679 /++ 6680 These control the size of the receive buffer. 6681 6682 It starts at the initial size, will temporarily 6683 balloon up to the maximum size, and will reuse 6684 a buffer up to the likely size. 6685 6686 Anything larger than the maximum size will cause 6687 the connection to be aborted and an exception thrown. 6688 This is to protect you against a peer trying to 6689 exhaust your memory, while keeping the user-level 6690 processing simple. 6691 +/ 6692 size_t initialReceiveBufferSize = 4096; 6693 size_t likelyReceiveBufferSize = 4096; /// ditto 6694 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6695 6696 /++ 6697 Maximum combined size of a message. 6698 +/ 6699 size_t maximumMessageSize = 10 * 1024 * 1024; 6700 6701 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6702 string origin; /// Origin URL to send with the handshake, if desired. 6703 string protocol; /// the protocol header, if desired. 6704 6705 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6706 } 6707 6708 /++ 6709 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6710 +/ 6711 int readyState() { 6712 return readyState_; 6713 } 6714 6715 /++ 6716 Closes the connection, sending a graceful teardown message to the other side. 6717 +/ 6718 /// Group: foundational 6719 void close(int code = 0, string reason = null) 6720 //in (reason.length < 123) 6721 in { assert(reason.length < 123); } do 6722 { 6723 if(readyState_ != OPEN) 6724 return; // it cool, we done 6725 WebSocketFrame wss; 6726 wss.fin = true; 6727 wss.opcode = WebSocketOpcode.close; 6728 wss.data = cast(ubyte[]) reason.dup; 6729 wss.send(&llsend); 6730 6731 readyState_ = CLOSING; 6732 6733 llclose(); 6734 } 6735 6736 /++ 6737 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6738 +/ 6739 /// Group: foundational 6740 void ping() { 6741 WebSocketFrame wss; 6742 wss.fin = true; 6743 wss.opcode = WebSocketOpcode.ping; 6744 wss.send(&llsend); 6745 } 6746 6747 // automatically handled.... 6748 void pong() { 6749 WebSocketFrame wss; 6750 wss.fin = true; 6751 wss.opcode = WebSocketOpcode.pong; 6752 wss.send(&llsend); 6753 } 6754 6755 /++ 6756 Sends a text message through the websocket. 6757 +/ 6758 /// Group: foundational 6759 void send(in char[] textData) { 6760 WebSocketFrame wss; 6761 wss.fin = true; 6762 wss.opcode = WebSocketOpcode.text; 6763 wss.data = cast(ubyte[]) textData.dup; 6764 wss.send(&llsend); 6765 } 6766 6767 /++ 6768 Sends a binary message through the websocket. 6769 +/ 6770 /// Group: foundational 6771 void send(in ubyte[] binaryData) { 6772 WebSocketFrame wss; 6773 wss.fin = true; 6774 wss.opcode = WebSocketOpcode.binary; 6775 wss.data = cast(ubyte[]) binaryData.dup; 6776 wss.send(&llsend); 6777 } 6778 6779 /++ 6780 Waits for and returns the next complete message on the socket. 6781 6782 Note that the onmessage function is still called, right before 6783 this returns. 6784 +/ 6785 /// Group: blocking_api 6786 public WebSocketFrame waitForNextMessage() { 6787 do { 6788 auto m = processOnce(); 6789 if(m.populated) 6790 return m; 6791 } while(lowLevelReceive()); 6792 6793 throw new ConnectionClosedException("Websocket receive timed out"); 6794 //return WebSocketFrame.init; // FIXME? maybe. 6795 } 6796 6797 /++ 6798 Tells if [waitForNextMessage] would block. 6799 +/ 6800 /// Group: blocking_api 6801 public bool waitForNextMessageWouldBlock() { 6802 checkAgain: 6803 if(isMessageBuffered()) 6804 return false; 6805 if(!isDataPending()) 6806 return true; 6807 while(isDataPending()) 6808 lowLevelReceive(); 6809 goto checkAgain; 6810 } 6811 6812 /++ 6813 Is there a message in the buffer already? 6814 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6815 If `false`, check [isDataPending] as the next step. 6816 +/ 6817 /// Group: blocking_api 6818 public bool isMessageBuffered() { 6819 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6820 auto s = d; 6821 if(d.length) { 6822 auto orig = d; 6823 auto m = WebSocketFrame.read(d); 6824 // that's how it indicates that it needs more data 6825 if(d !is orig) 6826 return true; 6827 } 6828 6829 return false; 6830 } 6831 6832 private ubyte continuingType; 6833 private ubyte[] continuingData; 6834 //private size_t continuingDataLength; 6835 6836 private WebSocketFrame processOnce() { 6837 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6838 auto s = d; 6839 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6840 WebSocketFrame m; 6841 if(d.length) { 6842 auto orig = d; 6843 m = WebSocketFrame.read(d); 6844 // that's how it indicates that it needs more data 6845 if(d is orig) 6846 return WebSocketFrame.init; 6847 m.unmaskInPlace(); 6848 switch(m.opcode) { 6849 case WebSocketOpcode.continuation: 6850 if(continuingData.length + m.data.length > config.maximumMessageSize) 6851 throw new Exception("message size exceeded"); 6852 6853 continuingData ~= m.data; 6854 if(m.fin) { 6855 if(ontextmessage) 6856 ontextmessage(cast(char[]) continuingData); 6857 if(onbinarymessage) 6858 onbinarymessage(continuingData); 6859 6860 continuingData = null; 6861 } 6862 break; 6863 case WebSocketOpcode.text: 6864 if(m.fin) { 6865 if(ontextmessage) 6866 ontextmessage(m.textData); 6867 } else { 6868 continuingType = m.opcode; 6869 //continuingDataLength = 0; 6870 continuingData = null; 6871 continuingData ~= m.data; 6872 } 6873 break; 6874 case WebSocketOpcode.binary: 6875 if(m.fin) { 6876 if(onbinarymessage) 6877 onbinarymessage(m.data); 6878 } else { 6879 continuingType = m.opcode; 6880 //continuingDataLength = 0; 6881 continuingData = null; 6882 continuingData ~= m.data; 6883 } 6884 break; 6885 case WebSocketOpcode.close: 6886 readyState_ = CLOSED; 6887 if(onclose) 6888 onclose(); 6889 6890 unregisterActiveSocket(this); 6891 break; 6892 case WebSocketOpcode.ping: 6893 pong(); 6894 break; 6895 case WebSocketOpcode.pong: 6896 // just really references it is still alive, nbd. 6897 break; 6898 default: // ignore though i could and perhaps should throw too 6899 } 6900 } 6901 6902 // the recv thing can be invalidated so gotta copy it over ugh 6903 if(d.length) { 6904 m.data = m.data.dup(); 6905 } 6906 6907 import core.stdc.string; 6908 memmove(receiveBuffer.ptr, d.ptr, d.length); 6909 receiveBufferUsedLength = d.length; 6910 6911 return m; 6912 } 6913 6914 private void autoprocess() { 6915 // FIXME 6916 do { 6917 processOnce(); 6918 } while(lowLevelReceive()); 6919 } 6920 6921 6922 void delegate() onclose; /// 6923 void delegate() onerror; /// 6924 void delegate(in char[]) ontextmessage; /// 6925 void delegate(in ubyte[]) onbinarymessage; /// 6926 void delegate() onopen; /// 6927 6928 /++ 6929 6930 +/ 6931 /// Group: browser_api 6932 void onmessage(void delegate(in char[]) dg) { 6933 ontextmessage = dg; 6934 } 6935 6936 /// ditto 6937 void onmessage(void delegate(in ubyte[]) dg) { 6938 onbinarymessage = dg; 6939 } 6940 6941 /* } end copy/paste */ 6942 6943 6944 } 6945 6946 /++ 6947 Returns true if the request headers are asking for a websocket upgrade. 6948 6949 If this returns true, and you want to accept it, call [acceptWebsocket]. 6950 +/ 6951 bool websocketRequested(Cgi cgi) { 6952 return 6953 "sec-websocket-key" in cgi.requestHeaders 6954 && 6955 "connection" in cgi.requestHeaders && 6956 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6957 && 6958 "upgrade" in cgi.requestHeaders && 6959 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6960 ; 6961 } 6962 6963 /++ 6964 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 6965 +/ 6966 WebSocket acceptWebsocket(Cgi cgi) { 6967 assert(!cgi.closed); 6968 assert(!cgi.outputtedResponseData); 6969 cgi.setResponseStatus("101 Switching Protocols"); 6970 cgi.header("Upgrade: WebSocket"); 6971 cgi.header("Connection: upgrade"); 6972 6973 string key = cgi.requestHeaders["sec-websocket-key"]; 6974 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 6975 6976 import std.digest.sha; 6977 auto hash = sha1Of(key); 6978 auto accept = Base64.encode(hash); 6979 6980 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 6981 6982 cgi.websocketMode = true; 6983 cgi.write(""); 6984 6985 cgi.flush(); 6986 6987 return new WebSocket(cgi); 6988 } 6989 6990 // FIXME get websocket to work on other modes, not just embedded_httpd 6991 6992 /* copy/paste in http2.d { */ 6993 enum WebSocketOpcode : ubyte { 6994 continuation = 0, 6995 text = 1, 6996 binary = 2, 6997 // 3, 4, 5, 6, 7 RESERVED 6998 close = 8, 6999 ping = 9, 7000 pong = 10, 7001 // 11,12,13,14,15 RESERVED 7002 } 7003 7004 public struct WebSocketFrame { 7005 private bool populated; 7006 bool fin; 7007 bool rsv1; 7008 bool rsv2; 7009 bool rsv3; 7010 WebSocketOpcode opcode; // 4 bits 7011 bool masked; 7012 ubyte lengthIndicator; // don't set this when building one to send 7013 ulong realLength; // don't use when sending 7014 ubyte[4] maskingKey; // don't set this when sending 7015 ubyte[] data; 7016 7017 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7018 WebSocketFrame msg; 7019 msg.fin = true; 7020 msg.opcode = opcode; 7021 msg.data = cast(ubyte[]) data.dup; 7022 7023 return msg; 7024 } 7025 7026 private void send(scope void delegate(ubyte[]) llsend) { 7027 ubyte[64] headerScratch; 7028 int headerScratchPos = 0; 7029 7030 realLength = data.length; 7031 7032 { 7033 ubyte b1; 7034 b1 |= cast(ubyte) opcode; 7035 b1 |= rsv3 ? (1 << 4) : 0; 7036 b1 |= rsv2 ? (1 << 5) : 0; 7037 b1 |= rsv1 ? (1 << 6) : 0; 7038 b1 |= fin ? (1 << 7) : 0; 7039 7040 headerScratch[0] = b1; 7041 headerScratchPos++; 7042 } 7043 7044 { 7045 headerScratchPos++; // we'll set header[1] at the end of this 7046 auto rlc = realLength; 7047 ubyte b2; 7048 b2 |= masked ? (1 << 7) : 0; 7049 7050 assert(headerScratchPos == 2); 7051 7052 if(realLength > 65535) { 7053 // use 64 bit length 7054 b2 |= 0x7f; 7055 7056 // FIXME: double check endinaness 7057 foreach(i; 0 .. 8) { 7058 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7059 rlc >>>= 8; 7060 } 7061 7062 headerScratchPos += 8; 7063 } else if(realLength > 125) { 7064 // use 16 bit length 7065 b2 |= 0x7e; 7066 7067 // FIXME: double check endinaness 7068 foreach(i; 0 .. 2) { 7069 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7070 rlc >>>= 8; 7071 } 7072 7073 headerScratchPos += 2; 7074 } else { 7075 // use 7 bit length 7076 b2 |= realLength & 0b_0111_1111; 7077 } 7078 7079 headerScratch[1] = b2; 7080 } 7081 7082 //assert(!masked, "masking key not properly implemented"); 7083 if(masked) { 7084 // FIXME: randomize this 7085 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7086 headerScratchPos += 4; 7087 7088 // we'll just mask it in place... 7089 int keyIdx = 0; 7090 foreach(i; 0 .. data.length) { 7091 data[i] = data[i] ^ maskingKey[keyIdx]; 7092 if(keyIdx == 3) 7093 keyIdx = 0; 7094 else 7095 keyIdx++; 7096 } 7097 } 7098 7099 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7100 llsend(headerScratch[0 .. headerScratchPos]); 7101 llsend(data); 7102 } 7103 7104 static WebSocketFrame read(ref ubyte[] d) { 7105 WebSocketFrame msg; 7106 7107 auto orig = d; 7108 7109 WebSocketFrame needsMoreData() { 7110 d = orig; 7111 return WebSocketFrame.init; 7112 } 7113 7114 if(d.length < 2) 7115 return needsMoreData(); 7116 7117 ubyte b = d[0]; 7118 7119 msg.populated = true; 7120 7121 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7122 b >>= 4; 7123 msg.rsv3 = b & 0x01; 7124 b >>= 1; 7125 msg.rsv2 = b & 0x01; 7126 b >>= 1; 7127 msg.rsv1 = b & 0x01; 7128 b >>= 1; 7129 msg.fin = b & 0x01; 7130 7131 b = d[1]; 7132 msg.masked = (b & 0b1000_0000) ? true : false; 7133 msg.lengthIndicator = b & 0b0111_1111; 7134 7135 d = d[2 .. $]; 7136 7137 if(msg.lengthIndicator == 0x7e) { 7138 // 16 bit length 7139 msg.realLength = 0; 7140 7141 if(d.length < 2) return needsMoreData(); 7142 7143 foreach(i; 0 .. 2) { 7144 msg.realLength |= d[0] << ((1-i) * 8); 7145 d = d[1 .. $]; 7146 } 7147 } else if(msg.lengthIndicator == 0x7f) { 7148 // 64 bit length 7149 msg.realLength = 0; 7150 7151 if(d.length < 8) return needsMoreData(); 7152 7153 foreach(i; 0 .. 8) { 7154 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7155 d = d[1 .. $]; 7156 } 7157 } else { 7158 // 7 bit length 7159 msg.realLength = msg.lengthIndicator; 7160 } 7161 7162 if(msg.masked) { 7163 7164 if(d.length < 4) return needsMoreData(); 7165 7166 msg.maskingKey = d[0 .. 4]; 7167 d = d[4 .. $]; 7168 } 7169 7170 if(msg.realLength > d.length) { 7171 return needsMoreData(); 7172 } 7173 7174 msg.data = d[0 .. cast(size_t) msg.realLength]; 7175 d = d[cast(size_t) msg.realLength .. $]; 7176 7177 return msg; 7178 } 7179 7180 void unmaskInPlace() { 7181 if(this.masked) { 7182 int keyIdx = 0; 7183 foreach(i; 0 .. this.data.length) { 7184 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7185 if(keyIdx == 3) 7186 keyIdx = 0; 7187 else 7188 keyIdx++; 7189 } 7190 } 7191 } 7192 7193 char[] textData() { 7194 return cast(char[]) data; 7195 } 7196 } 7197 /* } */ 7198 } 7199 7200 7201 version(Windows) 7202 { 7203 version(CRuntime_DigitalMars) 7204 { 7205 extern(C) int setmode(int, int) nothrow @nogc; 7206 } 7207 else version(CRuntime_Microsoft) 7208 { 7209 extern(C) int _setmode(int, int) nothrow @nogc; 7210 alias setmode = _setmode; 7211 } 7212 else static assert(0); 7213 } 7214 7215 version(Posix) { 7216 import core.sys.posix.unistd; 7217 version(CRuntime_Musl) {} else { 7218 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7219 } 7220 } 7221 7222 7223 // FIXME: these aren't quite public yet. 7224 //private: 7225 7226 // template for laziness 7227 void startAddonServer()(string arg) { 7228 version(OSX) { 7229 assert(0, "Not implemented"); 7230 } else version(linux) { 7231 import core.sys.posix.unistd; 7232 pid_t pid; 7233 const(char)*[16] args; 7234 args[0] = "ARSD_CGI_ADDON_SERVER"; 7235 args[1] = arg.ptr; 7236 posix_spawn(&pid, "/proc/self/exe", 7237 null, 7238 null, 7239 args.ptr, 7240 null // env 7241 ); 7242 } else version(Windows) { 7243 wchar[2048] filename; 7244 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7245 if(len == 0 || len == filename.length) 7246 throw new Exception("could not get process name to start helper server"); 7247 7248 STARTUPINFOW startupInfo; 7249 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7250 PROCESS_INFORMATION processInfo; 7251 7252 import std.utf; 7253 7254 // I *MIGHT* need to run it as a new job or a service... 7255 auto ret = CreateProcessW( 7256 filename.ptr, 7257 toUTF16z(arg), 7258 null, // process attributes 7259 null, // thread attributes 7260 false, // inherit handles 7261 0, // creation flags 7262 null, // environment 7263 null, // working directory 7264 &startupInfo, 7265 &processInfo 7266 ); 7267 7268 if(!ret) 7269 throw new Exception("create process failed"); 7270 7271 // when done with those, if we set them 7272 /* 7273 CloseHandle(hStdInput); 7274 CloseHandle(hStdOutput); 7275 CloseHandle(hStdError); 7276 */ 7277 7278 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7279 } 7280 7281 // template for laziness 7282 /* 7283 The websocket server is a single-process, single-thread, event 7284 I/O thing. It is passed websockets from other CGI processes 7285 and is then responsible for handling their messages and responses. 7286 Note that the CGI process is responsible for websocket setup, 7287 including authentication, etc. 7288 7289 It also gets data sent to it by other processes and is responsible 7290 for distributing that, as necessary. 7291 */ 7292 void runWebsocketServer()() { 7293 assert(0, "not implemented"); 7294 } 7295 7296 void sendToWebsocketServer(WebSocket ws, string group) { 7297 assert(0, "not implemented"); 7298 } 7299 7300 void sendToWebsocketServer(string content, string group) { 7301 assert(0, "not implemented"); 7302 } 7303 7304 7305 void runEventServer()() { 7306 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7307 } 7308 7309 void runTimerServer()() { 7310 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7311 } 7312 7313 version(Posix) { 7314 alias LocalServerConnectionHandle = int; 7315 alias CgiConnectionHandle = int; 7316 alias SocketConnectionHandle = int; 7317 7318 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7319 } else version(Windows) { 7320 alias LocalServerConnectionHandle = HANDLE; 7321 version(embedded_httpd_threads) { 7322 alias CgiConnectionHandle = SOCKET; 7323 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7324 } else version(fastcgi) { 7325 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7326 enum INVALID_CGI_CONNECTION_HANDLE = null; 7327 } else version(scgi) { 7328 alias CgiConnectionHandle = SOCKET; 7329 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7330 } else { /* version(plain_cgi) */ 7331 alias CgiConnectionHandle = HANDLE; 7332 enum INVALID_CGI_CONNECTION_HANDLE = null; 7333 } 7334 alias SocketConnectionHandle = SOCKET; 7335 } 7336 7337 version(with_addon_servers_connections) 7338 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7339 version(Posix) { 7340 import core.sys.posix.unistd; 7341 import core.sys.posix.sys.un; 7342 7343 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7344 if(sock == -1) 7345 throw new Exception("socket " ~ to!string(errno)); 7346 7347 scope(failure) 7348 close(sock); 7349 7350 cloexec(sock); 7351 7352 // add-on server processes are assumed to be local, and thus will 7353 // use unix domain sockets. Besides, I want to pass sockets to them, 7354 // so it basically must be local (except for the session server, but meh). 7355 sockaddr_un addr; 7356 addr.sun_family = AF_UNIX; 7357 version(linux) { 7358 // on linux, we will use the abstract namespace 7359 addr.sun_path[0] = 0; 7360 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7361 } else { 7362 // but otherwise, just use a file cuz we must. 7363 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7364 } 7365 7366 bool alreadyTried; 7367 7368 try_again: 7369 7370 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7371 if(!alreadyTried && errno == ECONNREFUSED) { 7372 // try auto-spawning the server, then attempt connection again 7373 startAddonServer(arg); 7374 import core.thread; 7375 Thread.sleep(50.msecs); 7376 alreadyTried = true; 7377 goto try_again; 7378 } else 7379 throw new Exception("connect " ~ to!string(errno)); 7380 } 7381 7382 return sock; 7383 } else version(Windows) { 7384 return null; // FIXME 7385 } 7386 } 7387 7388 version(with_addon_servers_connections) 7389 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7390 version(Posix) { 7391 import core.sys.posix.unistd; 7392 close(handle); 7393 } else version(Windows) 7394 CloseHandle(handle); 7395 } 7396 7397 void runSessionServer()() { 7398 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7399 } 7400 7401 import core.stdc.errno; 7402 7403 struct IoOp { 7404 @disable this(); 7405 @disable this(this); 7406 7407 /* 7408 So we want to be able to eventually handle generic sockets too. 7409 */ 7410 7411 enum Read = 1; 7412 enum Write = 2; 7413 enum Accept = 3; 7414 enum ReadSocketHandle = 4; 7415 7416 // Your handler may be called in a different thread than the one that initiated the IO request! 7417 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7418 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7419 private void delegate(IoOp*) closeHandler; 7420 private void delegate(IoOp*) completeHandler; 7421 private int internalFd; 7422 private int operation; 7423 private int bufferLengthAllocated; 7424 private int bufferLengthUsed; 7425 private ubyte[1] internalBuffer; // it can be overallocated! 7426 7427 ubyte[] allocatedBuffer() return { 7428 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7429 } 7430 7431 ubyte[] usedBuffer() return { 7432 return allocatedBuffer[0 .. bufferLengthUsed]; 7433 } 7434 7435 void reset() { 7436 bufferLengthUsed = 0; 7437 } 7438 7439 int fd() { 7440 return internalFd; 7441 } 7442 } 7443 7444 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7445 import core.stdc.stdlib; 7446 7447 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7448 if(ptr is null) 7449 assert(0); // out of memory! 7450 7451 auto op = cast(IoOp*) ptr; 7452 7453 op.handler = handler; 7454 op.internalFd = fd; 7455 op.operation = operation; 7456 op.bufferLengthAllocated = bufferSize; 7457 op.bufferLengthUsed = 0; 7458 7459 import core.memory; 7460 7461 GC.addRoot(ptr); 7462 7463 return op; 7464 } 7465 7466 void freeIoOp(ref IoOp* ptr) { 7467 7468 import core.memory; 7469 GC.removeRoot(ptr); 7470 7471 import core.stdc.stdlib; 7472 free(ptr); 7473 ptr = null; 7474 } 7475 7476 version(Posix) 7477 version(with_addon_servers_connections) 7478 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7479 7480 //import std.stdio : writeln; writeln(cast(string) data); 7481 7482 import core.sys.posix.unistd; 7483 7484 auto ret = write(connection, data.ptr, data.length); 7485 if(ret != data.length) { 7486 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7487 // the file is closed, remove it 7488 eis.fileClosed(connection); 7489 } else 7490 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7491 } 7492 } 7493 version(Windows) 7494 version(with_addon_servers_connections) 7495 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7496 // FIXME 7497 } 7498 7499 bool isInvalidHandle(CgiConnectionHandle h) { 7500 return h == INVALID_CGI_CONNECTION_HANDLE; 7501 } 7502 7503 /+ 7504 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7505 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7506 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7507 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7508 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7509 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7510 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7511 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7512 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7513 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7514 7515 +/ 7516 7517 /++ 7518 You can customize your server by subclassing the appropriate server. Then, register your 7519 subclass at compile time with the [registerEventIoServer] template, or implement your own 7520 main function and call it yourself. 7521 7522 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7523 +/ 7524 version(with_addon_servers_connections) 7525 interface EventIoServer { 7526 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7527 void handleLocalConnectionClose(IoOp* op); 7528 void handleLocalConnectionComplete(IoOp* op); 7529 void wait_timeout(); 7530 void fileClosed(int fd); 7531 7532 void epoll_fd(int fd); 7533 } 7534 7535 // the sink should buffer it 7536 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7537 static if(is(T == struct)) { 7538 foreach(member; __traits(allMembers, T)) 7539 serialize(sink, __traits(getMember, t, member)); 7540 } else static if(is(T : int)) { 7541 // no need to think of endianness just because this is only used 7542 // for local, same-machine stuff anyway. thanks private lol 7543 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7544 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7545 // these are common enough to optimize 7546 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7547 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7548 sink(cast(ubyte[]) t[]); 7549 } else static if(is(T : A[], A)) { 7550 // generic array is less optimal but still prolly ok 7551 int len = cast(int) t.length; 7552 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7553 foreach(item; t) 7554 serialize(sink, item); 7555 } else static assert(0, T.stringof); 7556 } 7557 7558 // all may be stack buffers, so use cautio 7559 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7560 static if(is(T == struct)) { 7561 T t; 7562 foreach(member; __traits(allMembers, T)) 7563 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7564 dg(t); 7565 } else static if(is(T : int)) { 7566 // no need to think of endianness just because this is only used 7567 // for local, same-machine stuff anyway. thanks private lol 7568 T t; 7569 auto data = get(t.sizeof); 7570 t = (cast(T[]) data)[0]; 7571 dg(t); 7572 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7573 // these are common enough to optimize 7574 int len; 7575 auto data = get(len.sizeof); 7576 len = (cast(int[]) data)[0]; 7577 7578 /* 7579 typeof(T[0])[2000] stackBuffer; 7580 T buffer; 7581 7582 if(len < stackBuffer.length) 7583 buffer = stackBuffer[0 .. len]; 7584 else 7585 buffer = new T(len); 7586 7587 data = get(len * typeof(T[0]).sizeof); 7588 */ 7589 7590 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7591 7592 dg(t); 7593 } else static if(is(T == E[], E)) { 7594 T t; 7595 int len; 7596 auto data = get(len.sizeof); 7597 len = (cast(int[]) data)[0]; 7598 t.length = len; 7599 foreach(ref e; t) { 7600 deserialize!E(get, (ele) { e = ele; }); 7601 } 7602 dg(t); 7603 } else static assert(0, T.stringof); 7604 } 7605 7606 unittest { 7607 serialize((ubyte[] b) { 7608 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7609 }, 1); 7610 serialize((ubyte[] b) { 7611 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7612 }, 56674); 7613 ubyte[1000] buffer; 7614 int bufferPoint; 7615 void add(scope ubyte[] b) { 7616 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7617 bufferPoint += b.length; 7618 } 7619 ubyte[] get(int sz) { 7620 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7621 bufferPoint += sz; 7622 return b; 7623 } 7624 serialize(&add, "test here"); 7625 bufferPoint = 0; 7626 deserialize!string(&get, (t) { assert(t == "test here"); }); 7627 bufferPoint = 0; 7628 7629 struct Foo { 7630 int a; 7631 ubyte c; 7632 string d; 7633 } 7634 serialize(&add, Foo(403, 37, "amazing")); 7635 bufferPoint = 0; 7636 deserialize!Foo(&get, (t) { 7637 assert(t.a == 403); 7638 assert(t.c == 37); 7639 assert(t.d == "amazing"); 7640 }); 7641 bufferPoint = 0; 7642 } 7643 7644 /* 7645 Here's the way the RPC interface works: 7646 7647 You define the interface that lists the functions you can call on the remote process. 7648 The interface may also have static methods for convenience. These forward to a singleton 7649 instance of an auto-generated class, which actually sends the args over the pipe. 7650 7651 An impl class actually implements it. A receiving server deserializes down the pipe and 7652 calls methods on the class. 7653 7654 I went with the interface to get some nice compiler checking and documentation stuff. 7655 7656 I could have skipped the interface and just implemented it all from the server class definition 7657 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7658 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7659 7660 7661 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7662 7663 Realistically though the bodies would just be 7664 connection.call(this.mangleof, args...) sooooo. 7665 7666 FIXME: overloads aren't supported 7667 */ 7668 7669 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7670 interface SessionObject {} 7671 7672 private immutable void delegate(string[])[string] scheduledJobHandlers; 7673 private immutable void delegate(string[])[string] websocketServers; 7674 7675 version(with_breaking_cgi_features) 7676 mixin(q{ 7677 7678 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7679 static import std.traits; 7680 7681 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7682 static foreach(idx, member; __traits(derivedMembers, T)) { 7683 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7684 mixin( q{ 7685 std.traits.ReturnType!(__traits(getMember, T, member)) 7686 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7687 { 7688 SerializationBuffer buffer; 7689 auto i = cast(ushort) idx; 7690 serialize(&buffer.sink, i); 7691 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7692 foreach(param; params) 7693 serialize(&buffer.sink, param); 7694 7695 auto sendable = buffer.sendable; 7696 7697 version(Posix) {{ 7698 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7699 7700 if(ret == -1) { 7701 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7702 } else if(ret == 0) { 7703 throw new Exception("Connection to addon server lost"); 7704 } if(ret < sendable.length) 7705 throw new Exception("Send failed to send all"); 7706 assert(ret == sendable.length); 7707 }} // FIXME Windows impl 7708 7709 static if(!is(typeof(return) == void)) { 7710 // there is a return value; we need to wait for it too 7711 version(Posix) { 7712 ubyte[3000] revBuffer; 7713 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7714 auto got = revBuffer[0 .. ret]; 7715 7716 int dataLocation; 7717 ubyte[] grab(int sz) { 7718 auto dataLocation1 = dataLocation; 7719 dataLocation += sz; 7720 return got[dataLocation1 .. dataLocation]; 7721 } 7722 7723 typeof(return) retu; 7724 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7725 return retu; 7726 } else { 7727 // FIXME Windows impl 7728 return typeof(return).init; 7729 } 7730 7731 } 7732 }}); 7733 } 7734 7735 private static typeof(this) singletonInstance; 7736 private LocalServerConnectionHandle connectionHandle; 7737 7738 static typeof(this) connection() { 7739 if(singletonInstance is null) { 7740 singletonInstance = new typeof(this)(); 7741 singletonInstance.connect(); 7742 } 7743 return singletonInstance; 7744 } 7745 7746 void connect() { 7747 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7748 } 7749 7750 void disconnect() { 7751 closeLocalServerConnection(connectionHandle); 7752 } 7753 } 7754 7755 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7756 ushort calledIdx; 7757 string calledFunction; 7758 7759 int dataLocation; 7760 ubyte[] grab(int sz) { 7761 if(sz == 0) assert(0); 7762 auto d = data[dataLocation .. dataLocation + sz]; 7763 dataLocation += sz; 7764 return d; 7765 } 7766 7767 again: 7768 7769 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7770 deserialize!string(&grab, (a) { calledFunction = a; }); 7771 7772 import std.traits; 7773 7774 sw: switch(calledIdx) { 7775 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7776 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 7777 case idx: 7778 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7779 7780 Parameters!(__traits(getMember, Interface, memberName)) params; 7781 foreach(ref param; params) 7782 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7783 7784 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7785 __traits(getMember, this_, memberName)(params); 7786 } else { 7787 auto ret = __traits(getMember, this_, memberName)(params); 7788 SerializationBuffer buffer; 7789 serialize(&buffer.sink, ret); 7790 7791 auto sendable = buffer.sendable; 7792 7793 version(Posix) { 7794 auto r = send(fd, sendable.ptr, sendable.length, 0); 7795 if(r == -1) { 7796 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7797 } else if(r == 0) { 7798 throw new Exception("Connection to addon client lost"); 7799 } if(r < sendable.length) 7800 throw new Exception("Send failed to send all"); 7801 7802 } // FIXME Windows impl 7803 } 7804 break sw; 7805 } 7806 default: assert(0); 7807 } 7808 7809 if(dataLocation != data.length) 7810 goto again; 7811 } 7812 7813 7814 private struct SerializationBuffer { 7815 ubyte[2048] bufferBacking; 7816 int bufferLocation; 7817 void sink(scope ubyte[] data) { 7818 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7819 bufferLocation += data.length; 7820 } 7821 7822 ubyte[] sendable() return { 7823 return bufferBacking[0 .. bufferLocation]; 7824 } 7825 } 7826 7827 /* 7828 FIXME: 7829 add a version command line arg 7830 version data in the library 7831 management gui as external program 7832 7833 at server with event_fd for each run 7834 use .mangleof in the at function name 7835 7836 i think the at server will have to: 7837 pipe args to the child 7838 collect child output for logging 7839 get child return value for logging 7840 7841 on windows timers work differently. idk how to best combine with the io stuff. 7842 7843 will have to have dump and restore too, so i can restart without losing stuff. 7844 */ 7845 7846 /++ 7847 A convenience object for talking to the [BasicDataServer] from a higher level. 7848 See: [Cgi.getSessionObject]. 7849 7850 You pass it a `Data` struct describing the data you want saved in the session. 7851 Then, this class will generate getter and setter properties that allow access 7852 to that data. 7853 7854 Note that each load and store will be done as-accessed; it doesn't front-load 7855 mutable data nor does it batch updates out of fear of read-modify-write race 7856 conditions. (In fact, right now it does this for everything, but in the future, 7857 I might batch load `immutable` members of the Data struct.) 7858 7859 At some point in the future, I might also let it do different backends, like 7860 a client-side cookie store too, but idk. 7861 7862 Note that the plain-old-data members of your `Data` struct are wrapped by this 7863 interface via a static foreach to make property functions. 7864 7865 See_Also: [MockSession] 7866 +/ 7867 interface Session(Data) : SessionObject { 7868 @property string sessionId() const; 7869 7870 /++ 7871 Starts a new session. Note that a session is also 7872 implicitly started as soon as you write data to it, 7873 so if you need to alter these parameters from their 7874 defaults, be sure to explicitly call this BEFORE doing 7875 any writes to session data. 7876 7877 Params: 7878 idleLifetime = How long, in seconds, the session 7879 should remain in memory when not being read from 7880 or written to. The default is one day. 7881 7882 NOT IMPLEMENTED 7883 7884 useExtendedLifetimeCookie = The session ID is always 7885 stored in a HTTP cookie, and by default, that cookie 7886 is discarded when the user closes their browser. 7887 7888 But if you set this to true, it will use a non-perishable 7889 cookie for the given idleLifetime. 7890 7891 NOT IMPLEMENTED 7892 +/ 7893 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7894 7895 /++ 7896 Regenerates the session ID and updates the associated 7897 cookie. 7898 7899 This is also your chance to change immutable data 7900 (not yet implemented). 7901 +/ 7902 void regenerateId(); 7903 7904 /++ 7905 Terminates this session, deleting all saved data. 7906 +/ 7907 void terminate(); 7908 7909 /++ 7910 Plain-old-data members of your `Data` struct are wrapped here via 7911 the property getters and setters. 7912 7913 If the member is a non-string array, it returns a magical array proxy 7914 object which allows for atomic appends and replaces via overloaded operators. 7915 You can slice this to get a range representing a $(B const) view of the array. 7916 This is to protect you against read-modify-write race conditions. 7917 +/ 7918 static foreach(memberName; __traits(allMembers, Data)) 7919 static if(is(typeof(__traits(getMember, Data, memberName)))) 7920 mixin(q{ 7921 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7922 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7923 }); 7924 7925 } 7926 7927 /++ 7928 An implementation of [Session] that works on real cgi connections utilizing the 7929 [BasicDataServer]. 7930 7931 As opposed to a [MockSession] which is made for testing purposes. 7932 7933 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7934 +/ 7935 class BasicDataServerSession(Data) : Session!Data { 7936 private Cgi cgi; 7937 private string sessionId_; 7938 7939 public @property string sessionId() const { 7940 return sessionId_; 7941 } 7942 7943 protected @property string sessionId(string s) { 7944 return this.sessionId_ = s; 7945 } 7946 7947 private this(Cgi cgi) { 7948 this.cgi = cgi; 7949 if(auto ptr = "sessionId" in cgi.cookies) 7950 sessionId = (*ptr).length ? *ptr : null; 7951 } 7952 7953 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7954 assert(sessionId is null); 7955 7956 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7957 7958 import std.random, std.conv; 7959 sessionId = to!string(uniform(1, long.max)); 7960 7961 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7962 setCookie(); 7963 } 7964 7965 protected void setCookie() { 7966 cgi.setCookie( 7967 "sessionId", sessionId, 7968 0 /* expiration */, 7969 "/" /* path */, 7970 null /* domain */, 7971 true /* http only */, 7972 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 7973 } 7974 7975 void regenerateId() { 7976 if(sessionId is null) { 7977 start(); 7978 return; 7979 } 7980 import std.random, std.conv; 7981 auto oldSessionId = sessionId; 7982 sessionId = to!string(uniform(1, long.max)); 7983 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 7984 setCookie(); 7985 } 7986 7987 void terminate() { 7988 BasicDataServer.connection.destroySession(sessionId); 7989 sessionId = null; 7990 setCookie(); 7991 } 7992 7993 static foreach(memberName; __traits(allMembers, Data)) 7994 static if(is(typeof(__traits(getMember, Data, memberName)))) 7995 mixin(q{ 7996 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7997 if(sessionId is null) 7998 return typeof(return).init; 7999 8000 import std.traits; 8001 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8002 if(v.length == 0) 8003 return typeof(return).init; 8004 import std.conv; 8005 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8006 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8007 // basically. Assuming the session is POD this should be fine. 8008 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8009 } 8010 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8011 if(sessionId is null) 8012 start(); 8013 import std.conv; 8014 import std.traits; 8015 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8016 return value; 8017 } 8018 }); 8019 } 8020 8021 /++ 8022 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8023 Simply stores the data in its instance members. 8024 +/ 8025 class MockSession(Data) : Session!Data { 8026 pure { 8027 @property string sessionId() const { return "mock"; } 8028 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8029 void regenerateId() {} 8030 void terminate() {} 8031 8032 private Data store_; 8033 8034 static foreach(memberName; __traits(allMembers, Data)) 8035 static if(is(typeof(__traits(getMember, Data, memberName)))) 8036 mixin(q{ 8037 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8038 return __traits(getMember, store_, memberName); 8039 } 8040 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8041 return __traits(getMember, store_, memberName) = value; 8042 } 8043 }); 8044 } 8045 } 8046 8047 /++ 8048 Direct interface to the basic data add-on server. You can 8049 typically use [Cgi.getSessionObject] as a more convenient interface. 8050 +/ 8051 version(with_addon_servers_connections) 8052 interface BasicDataServer { 8053 /// 8054 void createSession(string sessionId, int lifetime); 8055 /// 8056 void renewSession(string sessionId, int lifetime); 8057 /// 8058 void destroySession(string sessionId); 8059 /// 8060 void renameSession(string oldSessionId, string newSessionId); 8061 8062 /// 8063 void setSessionData(string sessionId, string dataKey, string dataValue); 8064 /// 8065 string getSessionData(string sessionId, string dataKey); 8066 8067 /// 8068 static BasicDataServerConnection connection() { 8069 return BasicDataServerConnection.connection(); 8070 } 8071 } 8072 8073 version(with_addon_servers_connections) 8074 class BasicDataServerConnection : BasicDataServer { 8075 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8076 } 8077 8078 version(with_addon_servers) 8079 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8080 8081 void createSession(string sessionId, int lifetime) { 8082 sessions[sessionId.idup] = Session(lifetime); 8083 } 8084 void destroySession(string sessionId) { 8085 sessions.remove(sessionId); 8086 } 8087 void renewSession(string sessionId, int lifetime) { 8088 sessions[sessionId].lifetime = lifetime; 8089 } 8090 void renameSession(string oldSessionId, string newSessionId) { 8091 sessions[newSessionId.idup] = sessions[oldSessionId]; 8092 sessions.remove(oldSessionId); 8093 } 8094 void setSessionData(string sessionId, string dataKey, string dataValue) { 8095 if(sessionId !in sessions) 8096 createSession(sessionId, 3600); // FIXME? 8097 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8098 } 8099 string getSessionData(string sessionId, string dataKey) { 8100 if(auto session = sessionId in sessions) { 8101 if(auto data = dataKey in (*session).values) 8102 return *data; 8103 else 8104 return null; // no such data 8105 8106 } else { 8107 return null; // no session 8108 } 8109 } 8110 8111 8112 protected: 8113 8114 struct Session { 8115 int lifetime; 8116 8117 string[string] values; 8118 } 8119 8120 Session[string] sessions; 8121 8122 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8123 auto data = op.usedBuffer; 8124 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8125 return false; 8126 } 8127 8128 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8129 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8130 void wait_timeout() {} 8131 void fileClosed(int fd) {} // stateless so irrelevant 8132 void epoll_fd(int fd) {} 8133 } 8134 8135 /++ 8136 See [schedule] to make one of these. You then call one of the methods here to set it up: 8137 8138 --- 8139 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8140 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8141 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8142 --- 8143 +/ 8144 version(with_addon_servers_connections) 8145 struct ScheduledJobHelper { 8146 private string func; 8147 private string[] args; 8148 private bool consumed; 8149 8150 private this(string func, string[] args) { 8151 this.func = func; 8152 this.args = args; 8153 } 8154 8155 ~this() { 8156 assert(consumed); 8157 } 8158 8159 /++ 8160 Schedules the job to be run at the given time. 8161 +/ 8162 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8163 consumed = true; 8164 8165 auto conn = ScheduledJobServerConnection.connection; 8166 import std.file; 8167 auto st = SysTime(when, timezone); 8168 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8169 } 8170 8171 /++ 8172 Schedules the job to run at least after the specified delay. 8173 +/ 8174 void delay(Duration delay) { 8175 consumed = true; 8176 8177 auto conn = ScheduledJobServerConnection.connection; 8178 import std.file; 8179 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8180 } 8181 8182 /++ 8183 Runs the job in the background ASAP. 8184 8185 $(NOTE It may run in a background thread. Don't segfault!) 8186 +/ 8187 void asap() { 8188 consumed = true; 8189 8190 auto conn = ScheduledJobServerConnection.connection; 8191 import std.file; 8192 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8193 } 8194 8195 /+ 8196 /++ 8197 Schedules the job to recur on the given pattern. 8198 +/ 8199 void recur(string spec) { 8200 8201 } 8202 +/ 8203 } 8204 8205 /++ 8206 First step to schedule a job on the scheduled job server. 8207 8208 The scheduled job needs to be a top-level function that doesn't read any 8209 variables from outside its arguments because it may be run in a new process, 8210 without any context existing later. 8211 8212 You MUST set details on the returned object to actually do anything! 8213 +/ 8214 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8215 /// 8216 ScheduledJobHelper schedule(T args) { 8217 // this isn't meant to ever be called, but instead just to 8218 // get the compiler to type check the arguments passed for us 8219 auto sample = delegate() { 8220 fn(args); 8221 }; 8222 string[] sargs; 8223 foreach(arg; args) 8224 sargs ~= to!string(arg); 8225 return ScheduledJobHelper(fn.mangleof, sargs); 8226 } 8227 8228 shared static this() { 8229 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8230 import std.traits; 8231 Parameters!fn args; 8232 foreach(idx, ref arg; args) 8233 arg = to!(typeof(arg))(sargs[idx]); 8234 fn(args); 8235 }; 8236 } 8237 } 8238 8239 /// 8240 interface ScheduledJobServer { 8241 /// Use the [schedule] function for a higher-level interface. 8242 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8243 /// 8244 void cancelJob(int jobId); 8245 } 8246 8247 version(with_addon_servers_connections) 8248 class ScheduledJobServerConnection : ScheduledJobServer { 8249 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8250 } 8251 8252 version(with_addon_servers) 8253 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8254 // FIXME: we need to handle SIGCHLD in this somehow 8255 // whenIs is 0 for relative, 1 for absolute 8256 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8257 auto nj = nextJobId; 8258 nextJobId++; 8259 8260 version(linux) { 8261 import core.sys.linux.timerfd; 8262 import core.sys.linux.epoll; 8263 import core.sys.posix.unistd; 8264 8265 8266 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8267 if(fd == -1) 8268 throw new Exception("fd timer create failed"); 8269 8270 foreach(ref arg; args) 8271 arg = arg.idup; 8272 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8273 8274 itimerspec value; 8275 value.it_value.tv_sec = when; 8276 value.it_value.tv_nsec = 0; 8277 8278 value.it_interval.tv_sec = 0; 8279 value.it_interval.tv_nsec = 0; 8280 8281 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8282 throw new Exception("couldn't set fd timer"); 8283 8284 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8285 jobs.remove(nj); 8286 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8287 close(fd); 8288 8289 8290 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8291 8292 return true; 8293 }); 8294 scope(failure) 8295 freeIoOp(op); 8296 8297 epoll_event ev; 8298 ev.events = EPOLLIN | EPOLLET; 8299 ev.data.ptr = op; 8300 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8301 throw new Exception("epoll_ctl " ~ to!string(errno)); 8302 8303 jobs[nj] = job; 8304 return nj; 8305 } else assert(0); 8306 } 8307 8308 protected void cancelJob(int jobId) { 8309 version(linux) { 8310 auto job = jobId in jobs; 8311 if(job is null) 8312 return; 8313 8314 jobs.remove(jobId); 8315 8316 version(linux) { 8317 import core.sys.linux.timerfd; 8318 import core.sys.linux.epoll; 8319 import core.sys.posix.unistd; 8320 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8321 close(job.timerfd); 8322 } 8323 } 8324 jobs.remove(jobId); 8325 } 8326 8327 int nextJobId = 1; 8328 static struct Job { 8329 string executable; 8330 string func; 8331 string[] args; 8332 int timerfd; 8333 int id; 8334 } 8335 Job[int] jobs; 8336 8337 8338 // event io server methods below 8339 8340 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8341 auto data = op.usedBuffer; 8342 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8343 return false; 8344 } 8345 8346 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8347 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8348 void wait_timeout() {} 8349 void fileClosed(int fd) {} // stateless so irrelevant 8350 8351 int epoll_fd_; 8352 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8353 int epoll_fd() { return epoll_fd_; } 8354 } 8355 8356 /// 8357 version(with_addon_servers_connections) 8358 interface EventSourceServer { 8359 /++ 8360 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8361 8362 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8363 8364 See_Also: 8365 [sendEvent] 8366 +/ 8367 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8368 /* 8369 If lastEventId is missing or empty, you just get new events as they come. 8370 8371 If it is set from something else, it sends all since then (that are still alive) 8372 down the pipe immediately. 8373 8374 The reason it can come from the header is that's what the standard defines for 8375 browser reconnects. The reason it can come from a query string is just convenience 8376 in catching up in a user-defined manner. 8377 8378 The reason the header overrides the query string is if the browser tries to reconnect, 8379 it will send the header AND the query (it reconnects to the same url), so we just 8380 want to do the restart thing. 8381 8382 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8383 */ 8384 string lastEventId = cgi.lastEventId; 8385 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8386 lastEventId = cgi.get["lastEventId"]; 8387 8388 cgi.setResponseContentType("text/event-stream"); 8389 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8390 cgi.flush(); 8391 8392 cgi.closed = true; 8393 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8394 scope(exit) 8395 closeLocalServerConnection(s); 8396 8397 version(fastcgi) 8398 throw new Exception("sending fcgi connections not supported"); 8399 else { 8400 auto fd = cgi.getOutputFileHandle(); 8401 if(isInvalidHandle(fd)) 8402 throw new Exception("bad fd from cgi!"); 8403 8404 EventSourceServerImplementation.SendableEventConnection sec; 8405 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8406 8407 version(Posix) { 8408 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8409 assert(res == sec.sizeof); 8410 } else version(Windows) { 8411 // FIXME 8412 } 8413 } 8414 } 8415 8416 /++ 8417 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8418 8419 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8420 8421 Params: 8422 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8423 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8424 data = the event data. Available in JS as `event.data`. 8425 lifetime = the amount of time to keep this event for replaying on the event server. 8426 8427 See_Also: 8428 [sendEventToEventServer] 8429 +/ 8430 public static void sendEvent(string url, string event, string data, int lifetime) { 8431 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8432 scope(exit) 8433 closeLocalServerConnection(s); 8434 8435 EventSourceServerImplementation.SendableEvent sev; 8436 sev.populate(url, event, data, lifetime); 8437 8438 version(Posix) { 8439 auto ret = send(s, &sev, sev.sizeof, 0); 8440 assert(ret == sev.sizeof); 8441 } else version(Windows) { 8442 // FIXME 8443 } 8444 } 8445 8446 /++ 8447 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8448 8449 See_Also: [disconnect] 8450 +/ 8451 void connect(string url, string forwardUrl); 8452 8453 /++ 8454 Disconnects `forwardUrl` from `url` 8455 8456 See_Also: [connect] 8457 +/ 8458 void disconnect(string url, string forwardUrl); 8459 } 8460 8461 /// 8462 version(with_addon_servers) 8463 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8464 8465 protected: 8466 8467 void connect(string url, string forwardUrl) { 8468 pipes[url] ~= forwardUrl; 8469 } 8470 void disconnect(string url, string forwardUrl) { 8471 auto t = url in pipes; 8472 if(t is null) 8473 return; 8474 foreach(idx, n; (*t)) 8475 if(n == forwardUrl) { 8476 (*t)[idx] = (*t)[$-1]; 8477 (*t) = (*t)[0 .. $-1]; 8478 break; 8479 } 8480 } 8481 8482 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8483 if(receivedFd != -1) { 8484 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8485 8486 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8487 8488 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8489 8490 auto url = got.url.idup; 8491 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8492 8493 // FIXME: catch up on past messages here 8494 } else { 8495 auto data = op.usedBuffer; 8496 auto event = cast(SendableEvent*) data.ptr; 8497 8498 if(event.magic == 0xdeadbeef) { 8499 handleInputEvent(event); 8500 8501 if(event.url in pipes) 8502 foreach(pipe; pipes[event.url]) { 8503 event.url = pipe; 8504 handleInputEvent(event); 8505 } 8506 } else { 8507 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8508 } 8509 } 8510 return false; 8511 } 8512 void handleLocalConnectionClose(IoOp* op) { 8513 fileClosed(op.fd); 8514 } 8515 void handleLocalConnectionComplete(IoOp* op) {} 8516 8517 void wait_timeout() { 8518 // just keeping alive 8519 foreach(url, connections; eventConnectionsByUrl) 8520 foreach(connection; connections) 8521 if(connection.needsChunking) 8522 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8523 else 8524 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8525 } 8526 8527 void fileClosed(int fd) { 8528 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8529 foreach(idx, conn; connections) { 8530 if(fd == conn.fd) { 8531 connections[idx] = connections[$-1]; 8532 connections = connections[0 .. $ - 1]; 8533 continue outer; 8534 } 8535 } 8536 } 8537 } 8538 8539 void epoll_fd(int fd) {} 8540 8541 8542 private: 8543 8544 8545 struct SendableEventConnection { 8546 ubyte responseChunked; 8547 8548 int urlLength; 8549 char[256] urlBuffer = 0; 8550 8551 int lastEventIdLength; 8552 char[32] lastEventIdBuffer = 0; 8553 8554 char[] url() return { 8555 return urlBuffer[0 .. urlLength]; 8556 } 8557 void url(in char[] u) { 8558 urlBuffer[0 .. u.length] = u[]; 8559 urlLength = cast(int) u.length; 8560 } 8561 char[] lastEventId() return { 8562 return lastEventIdBuffer[0 .. lastEventIdLength]; 8563 } 8564 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8565 in { 8566 assert(url.length < this.urlBuffer.length); 8567 assert(lastEventId.length < this.lastEventIdBuffer.length); 8568 } 8569 do { 8570 this.responseChunked = responseChunked ? 1 : 0; 8571 this.urlLength = cast(int) url.length; 8572 this.lastEventIdLength = cast(int) lastEventId.length; 8573 8574 this.urlBuffer[0 .. url.length] = url[]; 8575 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8576 } 8577 } 8578 8579 struct SendableEvent { 8580 int magic = 0xdeadbeef; 8581 int urlLength; 8582 char[256] urlBuffer = 0; 8583 int typeLength; 8584 char[32] typeBuffer = 0; 8585 int messageLength; 8586 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8587 int _lifetime; 8588 8589 char[] message() return { 8590 return messageBuffer[0 .. messageLength]; 8591 } 8592 char[] type() return { 8593 return typeBuffer[0 .. typeLength]; 8594 } 8595 char[] url() return { 8596 return urlBuffer[0 .. urlLength]; 8597 } 8598 void url(in char[] u) { 8599 urlBuffer[0 .. u.length] = u[]; 8600 urlLength = cast(int) u.length; 8601 } 8602 int lifetime() { 8603 return _lifetime; 8604 } 8605 8606 /// 8607 void populate(string url, string type, string message, int lifetime) 8608 in { 8609 assert(url.length < this.urlBuffer.length); 8610 assert(type.length < this.typeBuffer.length); 8611 assert(message.length < this.messageBuffer.length); 8612 } 8613 do { 8614 this.urlLength = cast(int) url.length; 8615 this.typeLength = cast(int) type.length; 8616 this.messageLength = cast(int) message.length; 8617 this._lifetime = lifetime; 8618 8619 this.urlBuffer[0 .. url.length] = url[]; 8620 this.typeBuffer[0 .. type.length] = type[]; 8621 this.messageBuffer[0 .. message.length] = message[]; 8622 } 8623 } 8624 8625 struct EventConnection { 8626 int fd; 8627 bool needsChunking; 8628 } 8629 8630 private EventConnection[][string] eventConnectionsByUrl; 8631 private string[][string] pipes; 8632 8633 private void handleInputEvent(scope SendableEvent* event) { 8634 static int eventId; 8635 8636 static struct StoredEvent { 8637 int id; 8638 string type; 8639 string message; 8640 int lifetimeRemaining; 8641 } 8642 8643 StoredEvent[][string] byUrl; 8644 8645 int thisId = ++eventId; 8646 8647 if(event.lifetime) 8648 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8649 8650 auto connectionsPtr = event.url in eventConnectionsByUrl; 8651 EventConnection[] connections; 8652 if(connectionsPtr is null) 8653 return; 8654 else 8655 connections = *connectionsPtr; 8656 8657 char[4096] buffer; 8658 char[] formattedMessage; 8659 8660 void append(const char[] a) { 8661 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8662 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8663 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8664 } 8665 8666 import std.algorithm.iteration; 8667 8668 if(connections.length) { 8669 append("id: "); 8670 append(to!string(thisId)); 8671 append("\n"); 8672 8673 append("event: "); 8674 append(event.type); 8675 append("\n"); 8676 8677 foreach(line; event.message.splitter("\n")) { 8678 append("data: "); 8679 append(line); 8680 append("\n"); 8681 } 8682 8683 append("\n"); 8684 } 8685 8686 // chunk it for HTTP! 8687 auto len = toHex(formattedMessage.length); 8688 buffer[4 .. 6] = "\r\n"[]; 8689 buffer[4 - len.length .. 4] = len[]; 8690 buffer[6 + formattedMessage.length] = '\r'; 8691 buffer[6 + formattedMessage.length + 1] = '\n'; 8692 8693 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8694 // done 8695 8696 // FIXME: send back requests when needed 8697 // FIXME: send a single ":\n" every 15 seconds to keep alive 8698 8699 foreach(connection; connections) { 8700 if(connection.needsChunking) { 8701 nonBlockingWrite(this, connection.fd, chunkedMessage); 8702 } else { 8703 nonBlockingWrite(this, connection.fd, formattedMessage); 8704 } 8705 } 8706 } 8707 } 8708 8709 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8710 version(Posix) { 8711 8712 import core.sys.posix.unistd; 8713 import core.sys.posix.fcntl; 8714 import core.sys.posix.sys.un; 8715 8716 import core.sys.posix.signal; 8717 signal(SIGPIPE, SIG_IGN); 8718 8719 static extern(C) void sigchldhandler(int) { 8720 int status; 8721 import w = core.sys.posix.sys.wait; 8722 w.wait(&status); 8723 } 8724 signal(SIGCHLD, &sigchldhandler); 8725 8726 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8727 if(sock == -1) 8728 throw new Exception("socket " ~ to!string(errno)); 8729 8730 scope(failure) 8731 close(sock); 8732 8733 cloexec(sock); 8734 8735 // add-on server processes are assumed to be local, and thus will 8736 // use unix domain sockets. Besides, I want to pass sockets to them, 8737 // so it basically must be local (except for the session server, but meh). 8738 sockaddr_un addr; 8739 addr.sun_family = AF_UNIX; 8740 version(linux) { 8741 // on linux, we will use the abstract namespace 8742 addr.sun_path[0] = 0; 8743 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8744 } else { 8745 // but otherwise, just use a file cuz we must. 8746 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8747 } 8748 8749 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8750 throw new Exception("bind " ~ to!string(errno)); 8751 8752 if(listen(sock, 128) == -1) 8753 throw new Exception("listen " ~ to!string(errno)); 8754 8755 makeNonBlocking(sock); 8756 8757 version(linux) { 8758 import core.sys.linux.epoll; 8759 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8760 if(epoll_fd == -1) 8761 throw new Exception("epoll_create1 " ~ to!string(errno)); 8762 scope(failure) 8763 close(epoll_fd); 8764 } else { 8765 import core.sys.posix.poll; 8766 } 8767 8768 version(linux) 8769 eis.epoll_fd = epoll_fd; 8770 8771 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8772 scope(exit) 8773 freeIoOp(acceptOp); 8774 8775 version(linux) { 8776 epoll_event ev; 8777 ev.events = EPOLLIN | EPOLLET; 8778 ev.data.ptr = acceptOp; 8779 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8780 throw new Exception("epoll_ctl " ~ to!string(errno)); 8781 8782 epoll_event[64] events; 8783 } else { 8784 pollfd[] pollfds; 8785 IoOp*[int] ioops; 8786 pollfds ~= pollfd(sock, POLLIN); 8787 ioops[sock] = acceptOp; 8788 } 8789 8790 import core.time : MonoTime, seconds; 8791 8792 MonoTime timeout = MonoTime.currTime + 15.seconds; 8793 8794 while(true) { 8795 8796 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8797 8798 int timeout_milliseconds = 0; // -1; // infinite 8799 8800 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8801 if(timeout_milliseconds < 0) 8802 timeout_milliseconds = 0; 8803 8804 //writeln("waiting for ", name); 8805 8806 version(linux) { 8807 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8808 if(nfds == -1) { 8809 if(errno == EINTR) 8810 continue; 8811 throw new Exception("epoll_wait " ~ to!string(errno)); 8812 } 8813 } else { 8814 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8815 size_t lastIdx = 0; 8816 } 8817 8818 if(nfds == 0) { 8819 eis.wait_timeout(); 8820 timeout += 15.seconds; 8821 } 8822 8823 foreach(idx; 0 .. nfds) { 8824 version(linux) { 8825 auto flags = events[idx].events; 8826 auto ioop = cast(IoOp*) events[idx].data.ptr; 8827 } else { 8828 IoOp* ioop; 8829 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8830 if(thing.revents) { 8831 ioop = ioops[thing.fd]; 8832 lastIdx += tidx + 1; 8833 break; 8834 } 8835 } 8836 } 8837 8838 //writeln(flags, " ", ioop.fd); 8839 8840 void newConnection() { 8841 // on edge triggering, it is important that we get it all 8842 while(true) { 8843 version(Android) { 8844 auto size = cast(int) addr.sizeof; 8845 } else { 8846 auto size = cast(uint) addr.sizeof; 8847 } 8848 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8849 if(ns == -1) { 8850 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8851 // all done, got it all 8852 break; 8853 } 8854 throw new Exception("accept " ~ to!string(errno)); 8855 } 8856 cloexec(ns); 8857 8858 makeNonBlocking(ns); 8859 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8860 niop.closeHandler = &eis.handleLocalConnectionClose; 8861 niop.completeHandler = &eis.handleLocalConnectionComplete; 8862 scope(failure) freeIoOp(niop); 8863 8864 version(linux) { 8865 epoll_event nev; 8866 nev.events = EPOLLIN | EPOLLET; 8867 nev.data.ptr = niop; 8868 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8869 throw new Exception("epoll_ctl " ~ to!string(errno)); 8870 } else { 8871 bool found = false; 8872 foreach(ref pfd; pollfds) { 8873 if(pfd.fd < 0) { 8874 pfd.fd = ns; 8875 found = true; 8876 } 8877 } 8878 if(!found) 8879 pollfds ~= pollfd(ns, POLLIN); 8880 ioops[ns] = niop; 8881 } 8882 } 8883 } 8884 8885 bool newConnectionCondition() { 8886 version(linux) 8887 return ioop.fd == sock && (flags & EPOLLIN); 8888 else 8889 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8890 } 8891 8892 if(newConnectionCondition()) { 8893 newConnection(); 8894 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8895 while(true) { 8896 int in_fd; 8897 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8898 if(got == -1) { 8899 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8900 // all done, got it all 8901 if(ioop.completeHandler) 8902 ioop.completeHandler(ioop); 8903 break; 8904 } 8905 throw new Exception("recv " ~ to!string(errno)); 8906 } 8907 8908 if(got == 0) { 8909 if(ioop.closeHandler) { 8910 ioop.closeHandler(ioop); 8911 version(linux) {} // nothing needed 8912 else { 8913 foreach(ref pfd; pollfds) { 8914 if(pfd.fd == ioop.fd) 8915 pfd.fd = -1; 8916 } 8917 } 8918 } 8919 close(ioop.fd); 8920 freeIoOp(ioop); 8921 break; 8922 } 8923 8924 ioop.bufferLengthUsed = cast(int) got; 8925 ioop.handler(ioop, in_fd); 8926 } 8927 } else if(ioop.operation == IoOp.Read) { 8928 while(true) { 8929 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8930 if(got == -1) { 8931 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8932 // all done, got it all 8933 if(ioop.completeHandler) 8934 ioop.completeHandler(ioop); 8935 break; 8936 } 8937 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8938 } 8939 8940 if(got == 0) { 8941 if(ioop.closeHandler) 8942 ioop.closeHandler(ioop); 8943 close(ioop.fd); 8944 freeIoOp(ioop); 8945 break; 8946 } 8947 8948 ioop.bufferLengthUsed = cast(int) got; 8949 if(ioop.handler(ioop, ioop.fd)) { 8950 close(ioop.fd); 8951 freeIoOp(ioop); 8952 break; 8953 } 8954 } 8955 } 8956 8957 // EPOLLHUP? 8958 } 8959 } 8960 } else version(Windows) { 8961 8962 // set up a named pipe 8963 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8964 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8965 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8966 8967 } else static assert(0); 8968 } 8969 8970 8971 version(with_sendfd) 8972 // copied from the web and ported from C 8973 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 8974 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 8975 msghdr msg; 8976 iovec[1] iov; 8977 8978 version(OSX) { 8979 //msg.msg_accrights = cast(cattr_t) &sendfd; 8980 //msg.msg_accrightslen = int.sizeof; 8981 } else version(Android) { 8982 } else { 8983 union ControlUnion { 8984 cmsghdr cm; 8985 char[CMSG_SPACE(int.sizeof)] control; 8986 } 8987 8988 ControlUnion control_un; 8989 cmsghdr* cmptr; 8990 8991 msg.msg_control = control_un.control.ptr; 8992 msg.msg_controllen = control_un.control.length; 8993 8994 cmptr = CMSG_FIRSTHDR(&msg); 8995 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 8996 cmptr.cmsg_level = SOL_SOCKET; 8997 cmptr.cmsg_type = SCM_RIGHTS; 8998 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 8999 } 9000 9001 msg.msg_name = null; 9002 msg.msg_namelen = 0; 9003 9004 iov[0].iov_base = ptr; 9005 iov[0].iov_len = nbytes; 9006 msg.msg_iov = iov.ptr; 9007 msg.msg_iovlen = 1; 9008 9009 return sendmsg(fd, &msg, 0); 9010 } 9011 9012 version(with_sendfd) 9013 // copied from the web and ported from C 9014 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9015 msghdr msg; 9016 iovec[1] iov; 9017 ssize_t n; 9018 int newfd; 9019 9020 version(OSX) { 9021 //msg.msg_accrights = cast(cattr_t) recvfd; 9022 //msg.msg_accrightslen = int.sizeof; 9023 } else version(Android) { 9024 } else { 9025 union ControlUnion { 9026 cmsghdr cm; 9027 char[CMSG_SPACE(int.sizeof)] control; 9028 } 9029 ControlUnion control_un; 9030 cmsghdr* cmptr; 9031 9032 msg.msg_control = control_un.control.ptr; 9033 msg.msg_controllen = control_un.control.length; 9034 } 9035 9036 msg.msg_name = null; 9037 msg.msg_namelen = 0; 9038 9039 iov[0].iov_base = ptr; 9040 iov[0].iov_len = nbytes; 9041 msg.msg_iov = iov.ptr; 9042 msg.msg_iovlen = 1; 9043 9044 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9045 return n; 9046 9047 version(OSX) { 9048 //if(msg.msg_accrightslen != int.sizeof) 9049 //*recvfd = -1; 9050 } else version(Android) { 9051 } else { 9052 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9053 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9054 if (cmptr.cmsg_level != SOL_SOCKET) 9055 throw new Exception("control level != SOL_SOCKET"); 9056 if (cmptr.cmsg_type != SCM_RIGHTS) 9057 throw new Exception("control type != SCM_RIGHTS"); 9058 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9059 } else 9060 *recvfd = -1; /* descriptor was not passed */ 9061 } 9062 9063 return n; 9064 } 9065 /* end read_fd */ 9066 9067 9068 /* 9069 Event source stuff 9070 9071 The api is: 9072 9073 sendEvent(string url, string type, string data, int timeout = 60*10); 9074 9075 attachEventListener(string url, int fd, lastId) 9076 9077 9078 It just sends to all attached listeners, and stores it until the timeout 9079 for replaying via lastEventId. 9080 */ 9081 9082 /* 9083 Session process stuff 9084 9085 it stores it all. the cgi object has a session object that can grab it 9086 9087 session may be done in the same process if possible, there is a version 9088 switch to choose if you want to override. 9089 */ 9090 9091 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9092 alias handler = dispatchHandler; 9093 string urlPrefix; 9094 bool rejectFurther; 9095 immutable(DispatcherDetails) details; 9096 } 9097 9098 private string urlify(string name) pure { 9099 return beautify(name, '-', true); 9100 } 9101 9102 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9103 if(name == "id") 9104 return allLowerCase ? name : "ID"; 9105 9106 char[160] buffer; 9107 int bufferIndex = 0; 9108 bool shouldCap = true; 9109 bool shouldSpace; 9110 bool lastWasCap; 9111 foreach(idx, char ch; name) { 9112 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9113 9114 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9115 if(lastWasCap) { 9116 // two caps in a row, don't change. Prolly acronym. 9117 } else { 9118 if(idx) 9119 shouldSpace = true; // new word, add space 9120 } 9121 9122 lastWasCap = true; 9123 } else { 9124 lastWasCap = false; 9125 } 9126 9127 if(shouldSpace) { 9128 buffer[bufferIndex++] = space; 9129 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9130 shouldSpace = false; 9131 } 9132 if(shouldCap) { 9133 if(ch >= 'a' && ch <= 'z') 9134 ch -= 32; 9135 shouldCap = false; 9136 } 9137 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9138 ch += 32; 9139 buffer[bufferIndex++] = ch; 9140 } 9141 return buffer[0 .. bufferIndex].idup; 9142 } 9143 9144 /* 9145 string urlFor(alias func)() { 9146 return __traits(identifier, func); 9147 } 9148 */ 9149 9150 /++ 9151 UDA: The name displayed to the user in auto-generated HTML. 9152 9153 Default is `beautify(identifier)`. 9154 +/ 9155 struct DisplayName { 9156 string name; 9157 } 9158 9159 /++ 9160 UDA: The name used in the URL or web parameter. 9161 9162 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9163 +/ 9164 struct UrlName { 9165 string name; 9166 } 9167 9168 /++ 9169 UDA: default format to respond for this method 9170 +/ 9171 struct DefaultFormat { string value; } 9172 9173 class MissingArgumentException : Exception { 9174 string functionName; 9175 string argumentName; 9176 string argumentType; 9177 9178 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9179 this.functionName = functionName; 9180 this.argumentName = argumentName; 9181 this.argumentType = argumentType; 9182 9183 super("Missing Argument: " ~ this.argumentName, file, line, next); 9184 } 9185 } 9186 9187 /++ 9188 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9189 9190 History: 9191 Added December 15, 2021 (dub v10.5) 9192 +/ 9193 class ResourceNotFoundException : Exception { 9194 string resourceType; 9195 string resourceId; 9196 9197 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9198 this.resourceType = resourceType; 9199 this.resourceId = resourceId; 9200 9201 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9202 } 9203 9204 } 9205 9206 /++ 9207 This can be attached to any constructor or function called from the cgi system. 9208 9209 If it is present, the function argument can NOT be set from web params, but instead 9210 is set to the return value of the given `func`. 9211 9212 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9213 the current request. Otherwise, it must take zero arguments. 9214 9215 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9216 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9217 the cgi object. 9218 9219 const arguments are also supported. 9220 +/ 9221 struct ifCalledFromWeb(alias func) {} 9222 9223 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9224 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9225 9226 // FIXME: any array of structs should also be settable or gettable from csv as well. 9227 9228 // FIXME: think more about checkboxes and bools. 9229 9230 import std.traits; 9231 9232 Parameters!method params; 9233 alias idents = ParameterIdentifierTuple!method; 9234 alias defaults = ParameterDefaults!method; 9235 9236 const(string)[] names; 9237 const(string)[] values; 9238 9239 // first, check for missing arguments and initialize to defaults if necessary 9240 9241 static if(is(typeof(method) P == __parameters)) 9242 foreach(idx, param; P) {{ 9243 // see: mustNotBeSetFromWebParams 9244 static if(is(param : Cgi)) { 9245 static assert(!is(param == immutable)); 9246 cast() params[idx] = cgi; 9247 } else static if(is(param == Session!D, D)) { 9248 static assert(!is(param == immutable)); 9249 cast() params[idx] = cgi.getSessionObject!D(); 9250 } else { 9251 bool populated; 9252 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9253 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9254 static if(is(typeof(func(cgi)))) 9255 params[idx] = func(cgi); 9256 else 9257 params[idx] = func(); 9258 9259 populated = true; 9260 } 9261 } 9262 9263 if(!populated) { 9264 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9265 params[idx] = param.getAutomaticallyForCgi(cgi); 9266 populated = true; 9267 } 9268 } 9269 9270 if(!populated) { 9271 auto ident = idents[idx]; 9272 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9273 if(ident !in cgi.get) { 9274 static if(is(defaults[idx] == void)) { 9275 static if(is(param == bool)) 9276 params[idx] = false; 9277 else 9278 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9279 } else 9280 params[idx] = defaults[idx]; 9281 } 9282 } else { 9283 if(ident !in cgi.post) { 9284 static if(is(defaults[idx] == void)) { 9285 static if(is(param == bool)) 9286 params[idx] = false; 9287 else 9288 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9289 } else 9290 params[idx] = defaults[idx]; 9291 } 9292 } 9293 } 9294 } 9295 }} 9296 9297 // second, parse the arguments in order to build up arrays, etc. 9298 9299 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9300 static if(is(T == struct)) { 9301 if(name == paramName) { 9302 *what = T.init; 9303 return true; 9304 } else { 9305 // could be a child. gonna allow either obj.field OR obj[field] 9306 9307 string afterName; 9308 9309 if(name[paramName.length] == '[') { 9310 int count = 1; 9311 auto idx = paramName.length + 1; 9312 while(idx < name.length && count > 0) { 9313 if(name[idx] == '[') 9314 count++; 9315 else if(name[idx] == ']') { 9316 count--; 9317 if(count == 0) break; 9318 } 9319 idx++; 9320 } 9321 9322 if(idx == name.length) 9323 return false; // malformed 9324 9325 auto insideBrackets = name[paramName.length + 1 .. idx]; 9326 afterName = name[idx + 1 .. $]; 9327 9328 name = name[0 .. paramName.length]; 9329 9330 paramName = insideBrackets; 9331 9332 } else if(name[paramName.length] == '.') { 9333 paramName = name[paramName.length + 1 .. $]; 9334 name = paramName; 9335 int p = 0; 9336 foreach(ch; paramName) { 9337 if(ch == '.' || ch == '[') 9338 break; 9339 p++; 9340 } 9341 9342 afterName = paramName[p .. $]; 9343 paramName = paramName[0 .. p]; 9344 } else { 9345 return false; 9346 } 9347 9348 if(paramName.length) 9349 // set the child member 9350 switch(paramName) { 9351 foreach(idx, memberName; __traits(allMembers, T)) 9352 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9353 // data member! 9354 case memberName: 9355 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9356 } 9357 default: 9358 // ok, not a member 9359 } 9360 } 9361 9362 return false; 9363 } else static if(is(T == enum)) { 9364 *what = to!T(value); 9365 return true; 9366 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9367 *what = to!T(value); 9368 return true; 9369 } else static if(is(T == bool)) { 9370 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9371 return true; 9372 } else static if(is(T == K[], K)) { 9373 K tmp; 9374 if(name == paramName) { 9375 // direct - set and append 9376 if(setVariable(name, paramName, &tmp, value)) { 9377 (*what) ~= tmp; 9378 return true; 9379 } else { 9380 return false; 9381 } 9382 } else { 9383 // child, append to last element 9384 // FIXME: what about range violations??? 9385 auto ptr = &(*what)[(*what).length - 1]; 9386 return setVariable(name, paramName, ptr, value); 9387 9388 } 9389 } else static if(is(T == V[K], K, V)) { 9390 // assoc array, name[key] is valid 9391 if(name == paramName) { 9392 // no action necessary 9393 return true; 9394 } else if(name[paramName.length] == '[') { 9395 int count = 1; 9396 auto idx = paramName.length + 1; 9397 while(idx < name.length && count > 0) { 9398 if(name[idx] == '[') 9399 count++; 9400 else if(name[idx] == ']') { 9401 count--; 9402 if(count == 0) break; 9403 } 9404 idx++; 9405 } 9406 if(idx == name.length) 9407 return false; // malformed 9408 9409 auto insideBrackets = name[paramName.length + 1 .. idx]; 9410 auto afterName = name[idx + 1 .. $]; 9411 9412 auto k = to!K(insideBrackets); 9413 V v; 9414 if(auto ptr = k in *what) 9415 v = *ptr; 9416 9417 name = name[0 .. paramName.length]; 9418 //writeln(name, afterName, " ", paramName); 9419 9420 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9421 if(ret) { 9422 (*what)[k] = v; 9423 return true; 9424 } 9425 } 9426 9427 return false; 9428 } else { 9429 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9430 } 9431 9432 //return false; 9433 } 9434 9435 void setArgument(string name, string value) { 9436 int p; 9437 foreach(ch; name) { 9438 if(ch == '.' || ch == '[') 9439 break; 9440 p++; 9441 } 9442 9443 auto paramName = name[0 .. p]; 9444 9445 sw: switch(paramName) { 9446 static if(is(typeof(method) P == __parameters)) 9447 foreach(idx, param; P) { 9448 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9449 // cannot be set from the outside 9450 } else { 9451 case idents[idx]: 9452 static if(is(param == Cgi.UploadedFile)) { 9453 params[idx] = cgi.files[name]; 9454 } else static if(is(param : const Cgi.UploadedFile[])) { 9455 (cast() params[idx]) = cgi.filesArray[name]; 9456 } else { 9457 setVariable(name, paramName, ¶ms[idx], value); 9458 } 9459 break sw; 9460 } 9461 } 9462 default: 9463 // ignore; not relevant argument 9464 } 9465 } 9466 9467 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9468 names = cgi.allGetNamesInOrder; 9469 values = cgi.allGetValuesInOrder; 9470 } else { 9471 names = cgi.allPostNamesInOrder; 9472 values = cgi.allPostValuesInOrder; 9473 } 9474 9475 foreach(idx, name; names) { 9476 setArgument(name, values[idx]); 9477 } 9478 9479 static if(is(ReturnType!method == void)) { 9480 typeof(null) ret; 9481 dg(params); 9482 } else { 9483 auto ret = dg(params); 9484 } 9485 9486 // FIXME: format return values 9487 // options are: json, html, csv. 9488 // also may need to wrap in envelope format: none, html, or json. 9489 return ret; 9490 } 9491 9492 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9493 static if(is(T : const(Cgi))) { 9494 return true; 9495 } else static if(is(T : const(Session!D), D)) { 9496 return true; 9497 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9498 return true; 9499 } else { 9500 foreach(uda; attrs) 9501 static if(is(uda == ifCalledFromWeb!func, alias func)) 9502 return true; 9503 return false; 9504 } 9505 } 9506 9507 private bool hasIfCalledFromWeb(attrs...)() { 9508 foreach(uda; attrs) 9509 static if(is(uda == ifCalledFromWeb!func, alias func)) 9510 return true; 9511 return false; 9512 } 9513 9514 /++ 9515 Implies POST path for the thing itself, then GET will get the automatic form. 9516 9517 The given customizer, if present, will be called as a filter on the Form object. 9518 9519 History: 9520 Added December 27, 2020 9521 +/ 9522 template AutomaticForm(alias customizer) { } 9523 9524 /++ 9525 This is meant to be returned by a function that takes a form POST submission. You 9526 want to set the url of the new resource it created, which is set as the http 9527 Location header for a "201 Created" result, and you can also set a separate 9528 destination for browser users, which it sets via a "Refresh" header. 9529 9530 The `resourceRepresentation` should generally be the thing you just created, and 9531 it will be the body of the http response when formatted through the presenter. 9532 The exact thing is up to you - it could just return an id, or the whole object, or 9533 perhaps a partial object. 9534 9535 Examples: 9536 --- 9537 class Test : WebObject { 9538 @(Cgi.RequestMethod.POST) 9539 CreatedResource!int makeThing(string value) { 9540 return CreatedResource!int(value.to!int, "/resources/id"); 9541 } 9542 } 9543 --- 9544 9545 History: 9546 Added December 18, 2021 9547 +/ 9548 struct CreatedResource(T) { 9549 static if(!is(T == void)) 9550 T resourceRepresentation; 9551 string resourceUrl; 9552 string refreshUrl; 9553 } 9554 9555 /+ 9556 /++ 9557 This can be attached as a UDA to a handler to add a http Refresh header on a 9558 successful run. (It will not be attached if the function throws an exception.) 9559 This will refresh the browser the given number of seconds after the page loads, 9560 to the url returned by `urlFunc`, which can be either a static function or a 9561 member method of the current handler object. 9562 9563 You might use this for a POST handler that is normally used from ajax, but you 9564 want it to degrade gracefully to a temporarily flashed message before reloading 9565 the main page. 9566 9567 History: 9568 Added December 18, 2021 9569 +/ 9570 struct Refresh(alias urlFunc) { 9571 int waitInSeconds; 9572 9573 string url() { 9574 static if(__traits(isStaticFunction, urlFunc)) 9575 return urlFunc(); 9576 else static if(is(urlFunc : string)) 9577 return urlFunc; 9578 } 9579 } 9580 +/ 9581 9582 /+ 9583 /++ 9584 Sets a filter to be run before 9585 9586 A before function can do validations of params and log and stop the function from running. 9587 +/ 9588 template Before(alias b) {} 9589 template After(alias b) {} 9590 +/ 9591 9592 /+ 9593 Argument conversions: for the most part, it is to!Thing(string). 9594 9595 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9596 they are passed 9597 9598 arr=foo&arr=bar <-- notice the same name. 9599 9600 Structs are first declared with an empty thing, then have their members set individually, 9601 with dot notation. The members are not required, just the initial declaration. 9602 9603 struct Foo { 9604 int a; 9605 string b; 9606 } 9607 void test(Foo foo){} 9608 9609 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9610 9611 Arrays of structs use this declaration. 9612 9613 void test(Foo[] foo) {} 9614 9615 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9616 9617 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9618 declaration is ignored. 9619 9620 Mind that order matters! The declaration MUST come first in the string. 9621 9622 Arrays of struct members follow this rule recursively. 9623 9624 struct Foo { 9625 int[] a; 9626 } 9627 9628 foo&foo.a=1&foo.a=2&foo&foo.a=1 9629 9630 9631 Associative arrays are formatted with brackets, after a declaration, like structs: 9632 9633 foo&foo[key]=value&foo[other_key]=value 9634 9635 9636 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9637 do not support the strict ordering requirements to work with these struct protocols. 9638 9639 FIXME: also perhaps accept application/json to better work with outside trash. 9640 9641 9642 Return values are also auto-formatted according to user-requested type: 9643 for json, it loops over and converts. 9644 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9645 +/ 9646 9647 /++ 9648 A web presenter is responsible for rendering things to HTML to be usable 9649 in a web browser. 9650 9651 They are passed as template arguments to the base classes of [WebObject] 9652 9653 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9654 and override it. Use forwarding and specialization to customize it. 9655 9656 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9657 class templates and your overridden templates work with each other. 9658 9659 --- 9660 class MyPresenter : WebPresenter!(MyPresenter) { 9661 @Override 9662 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9663 // present the CustomType 9664 } 9665 @Override 9666 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9667 // handle everything else via the super class, which will call 9668 // back to your class when appropriate 9669 super.presentSuccessfulReturnAsHtml(cgi, ret); 9670 } 9671 } 9672 --- 9673 9674 The meta argument in there can be overridden by your own facility. 9675 9676 +/ 9677 class WebPresenter(CRTP) { 9678 9679 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9680 /// If you override a plain method, use `override`. If a template, use `@Override`. 9681 enum Override; 9682 9683 string script() { 9684 return ` 9685 `; 9686 } 9687 9688 string style() { 9689 return ` 9690 :root { 9691 --mild-border: #ccc; 9692 --middle-border: #999; 9693 --accent-color: #f2f2f2; 9694 --sidebar-color: #fefefe; 9695 } 9696 ` ~ genericFormStyling() ~ genericSiteStyling(); 9697 } 9698 9699 string genericFormStyling() { 9700 return 9701 q"css 9702 table.automatic-data-display { 9703 border-collapse: collapse; 9704 border: solid 1px var(--mild-border); 9705 } 9706 9707 table.automatic-data-display td { 9708 vertical-align: top; 9709 border: solid 1px var(--mild-border); 9710 padding: 2px 4px; 9711 } 9712 9713 table.automatic-data-display th { 9714 border: solid 1px var(--mild-border); 9715 border-bottom: solid 1px var(--middle-border); 9716 padding: 2px 4px; 9717 } 9718 9719 ol.automatic-data-display { 9720 margin: 0px; 9721 list-style-position: inside; 9722 padding: 0px; 9723 } 9724 9725 dl.automatic-data-display { 9726 9727 } 9728 9729 .automatic-form { 9730 max-width: 600px; 9731 } 9732 9733 .form-field { 9734 margin: 0.5em; 9735 padding-left: 0.5em; 9736 } 9737 9738 .label-text { 9739 display: block; 9740 font-weight: bold; 9741 margin-left: -0.5em; 9742 } 9743 9744 .submit-button-holder { 9745 padding-left: 2em; 9746 } 9747 9748 .add-array-button { 9749 9750 } 9751 css"; 9752 } 9753 9754 string genericSiteStyling() { 9755 return 9756 q"css 9757 * { box-sizing: border-box; } 9758 html, body { margin: 0px; } 9759 body { 9760 font-family: sans-serif; 9761 } 9762 header { 9763 background: var(--accent-color); 9764 height: 64px; 9765 } 9766 footer { 9767 background: var(--accent-color); 9768 height: 64px; 9769 } 9770 #site-container { 9771 display: flex; 9772 } 9773 main { 9774 flex: 1 1 auto; 9775 order: 2; 9776 min-height: calc(100vh - 64px - 64px); 9777 padding: 4px; 9778 padding-left: 1em; 9779 } 9780 #sidebar { 9781 flex: 0 0 16em; 9782 order: 1; 9783 background: var(--sidebar-color); 9784 } 9785 css"; 9786 } 9787 9788 import arsd.dom; 9789 Element htmlContainer() { 9790 auto document = new Document(q"html 9791 <!DOCTYPE html> 9792 <html class="no-script"> 9793 <head> 9794 <script>document.documentElement.classList.remove("no-script");</script> 9795 <style>.no-script requires-script { display: none; }</style> 9796 <title>D Application</title> 9797 <meta name="viewport" content="initial-scale=1, width=device-width" /> 9798 <link rel="stylesheet" href="style.css" /> 9799 </head> 9800 <body> 9801 <header></header> 9802 <div id="site-container"> 9803 <main></main> 9804 <div id="sidebar"></div> 9805 </div> 9806 <footer></footer> 9807 <script src="script.js"></script> 9808 </body> 9809 </html> 9810 html", true, true); 9811 9812 return document.requireSelector("main"); 9813 } 9814 9815 /// Renders a response as an HTTP error with associated html body 9816 void renderBasicError(Cgi cgi, int httpErrorCode) { 9817 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9818 auto c = htmlContainer(); 9819 c.innerText = getHttpCodeText(httpErrorCode); 9820 cgi.setResponseContentType("text/html; charset=utf-8"); 9821 cgi.write(c.parentDocument.toString(), true); 9822 } 9823 9824 template methodMeta(alias method) { 9825 enum methodMeta = null; 9826 } 9827 9828 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9829 switch(format) { 9830 case "html": 9831 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9832 break; 9833 case "json": 9834 import arsd.jsvar; 9835 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9836 var json; 9837 foreach(index, type; Types) { 9838 if(ret.contains == index) 9839 json = ret.payload[index]; 9840 } 9841 } else { 9842 var json = ret; 9843 } 9844 var envelope = json; // var.emptyObject; 9845 /* 9846 envelope.success = true; 9847 envelope.result = json; 9848 envelope.error = null; 9849 */ 9850 cgi.setResponseContentType("application/json"); 9851 cgi.write(envelope.toJson(), true); 9852 break; 9853 default: 9854 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9855 } 9856 } 9857 9858 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9859 /// in the default presenter - allowing the function to have full low-level control over the 9860 /// response. 9861 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9862 // nothing intentionally! 9863 } 9864 9865 /// Redirections are forwarded to [Cgi.setResponseLocation] 9866 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9867 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9868 } 9869 9870 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9871 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9872 cgi.setResponseStatus(getHttpCodeText(201)); 9873 if(ret.resourceUrl.length) 9874 cgi.header("Location: " ~ ret.resourceUrl); 9875 if(ret.refreshUrl.length) 9876 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9877 static if(!is(R == void)) 9878 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9879 } 9880 9881 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9882 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9883 bool outputted = false; 9884 foreach(index, type; Types) { 9885 if(ret.contains == index) { 9886 assert(!outputted); 9887 outputted = true; 9888 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9889 } 9890 } 9891 if(!outputted) 9892 assert(0); 9893 } 9894 9895 /++ 9896 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9897 +/ 9898 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9899 cgi.setCache(true); // not necessarily true but meh 9900 if(auto fn = ret.filename()) { 9901 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9902 } 9903 cgi.setResponseContentType(ret.contentType); 9904 cgi.write(ret.getData(), true); 9905 } 9906 9907 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9908 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9909 auto container = this.htmlContainer(); 9910 container.appendChild(formatReturnValueAsHtml(ret)); 9911 cgi.write(container.parentDocument.toString(), true); 9912 } 9913 9914 /++ 9915 9916 History: 9917 Added January 23, 2023 (dub v11.0) 9918 +/ 9919 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9920 switch(format) { 9921 case "html": 9922 presentExceptionAsHtml(cgi, t, meta); 9923 break; 9924 default: 9925 } 9926 } 9927 9928 9929 /++ 9930 If you override this, you will need to cast the exception type `t` dynamically, 9931 but can then use the template arguments here to refer back to the function. 9932 9933 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9934 method on the live object. You could, in theory, change arguments and retry, but I 9935 provide that information mostly with the expectation that you will use them to make 9936 useful forms or richer error messages for the user. 9937 9938 History: 9939 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 9940 I removed this in favor of a `Meta` param. 9941 9942 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 9943 9944 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 9945 9946 If you used the func for something, move that something into your `methodMeta` template. 9947 9948 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 9949 enabling an easier implementation of [presentExceptionalReturn]. 9950 +/ 9951 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 9952 Form af; 9953 /+ 9954 foreach(attr; __traits(getAttributes, func)) { 9955 static if(__traits(isSame, attr, AutomaticForm)) { 9956 af = createAutomaticFormForFunction!(func)(dg); 9957 } 9958 } 9959 +/ 9960 presentExceptionAsHtmlImpl(cgi, t, af); 9961 } 9962 9963 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 9964 if(auto e = cast(ResourceNotFoundException) t) { 9965 auto container = this.htmlContainer(); 9966 9967 container.addChild("p", e.msg); 9968 9969 if(!cgi.outputtedResponseData) 9970 cgi.setResponseStatus("404 Not Found"); 9971 cgi.write(container.parentDocument.toString(), true); 9972 } else if(auto mae = cast(MissingArgumentException) t) { 9973 if(automaticForm is null) 9974 goto generic; 9975 auto container = this.htmlContainer(); 9976 if(cgi.requestMethod == Cgi.RequestMethod.POST) 9977 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 9978 container.appendChild(automaticForm); 9979 9980 cgi.write(container.parentDocument.toString(), true); 9981 } else { 9982 generic: 9983 auto container = this.htmlContainer(); 9984 9985 // import std.stdio; writeln(t.toString()); 9986 9987 container.appendChild(exceptionToElement(t)); 9988 9989 container.addChild("h4", "GET"); 9990 foreach(k, v; cgi.get) { 9991 auto deets = container.addChild("details"); 9992 deets.addChild("summary", k); 9993 deets.addChild("div", v); 9994 } 9995 9996 container.addChild("h4", "POST"); 9997 foreach(k, v; cgi.post) { 9998 auto deets = container.addChild("details"); 9999 deets.addChild("summary", k); 10000 deets.addChild("div", v); 10001 } 10002 10003 10004 if(!cgi.outputtedResponseData) 10005 cgi.setResponseStatus("500 Internal Server Error"); 10006 cgi.write(container.parentDocument.toString(), true); 10007 } 10008 } 10009 10010 Element exceptionToElement(Throwable t) { 10011 auto div = Element.make("div"); 10012 div.addClass("exception-display"); 10013 10014 div.addChild("p", t.msg); 10015 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10016 10017 auto pre = div.addChild("pre"); 10018 string s; 10019 s = t.toString(); 10020 Element currentBox; 10021 bool on = false; 10022 foreach(line; s.splitLines) { 10023 if(!on && line.startsWith("-----")) 10024 on = true; 10025 if(!on) continue; 10026 if(line.indexOf("arsd/") != -1) { 10027 if(currentBox is null) { 10028 currentBox = pre.addChild("details"); 10029 currentBox.addChild("summary", "Framework code"); 10030 } 10031 currentBox.addChild("span", line ~ "\n"); 10032 } else { 10033 pre.addChild("span", line ~ "\n"); 10034 currentBox = null; 10035 } 10036 } 10037 10038 return div; 10039 } 10040 10041 /++ 10042 Returns an element for a particular type 10043 +/ 10044 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10045 import std.traits; 10046 10047 auto div = Element.make("div"); 10048 div.addClass("form-field"); 10049 10050 static if(is(T : const Cgi.UploadedFile)) { 10051 Element lbl; 10052 if(displayName !is null) { 10053 lbl = div.addChild("label"); 10054 lbl.addChild("span", displayName, "label-text"); 10055 lbl.appendText(" "); 10056 } else { 10057 lbl = div; 10058 } 10059 auto i = lbl.addChild("input", name); 10060 i.attrs.name = name; 10061 i.attrs.type = "file"; 10062 i.attrs.multiple = "multiple"; 10063 } else static if(is(T == Cgi.UploadedFile)) { 10064 Element lbl; 10065 if(displayName !is null) { 10066 lbl = div.addChild("label"); 10067 lbl.addChild("span", displayName, "label-text"); 10068 lbl.appendText(" "); 10069 } else { 10070 lbl = div; 10071 } 10072 auto i = lbl.addChild("input", name); 10073 i.attrs.name = name; 10074 i.attrs.type = "file"; 10075 } else static if(is(T == enum)) { 10076 Element lbl; 10077 if(displayName !is null) { 10078 lbl = div.addChild("label"); 10079 lbl.addChild("span", displayName, "label-text"); 10080 lbl.appendText(" "); 10081 } else { 10082 lbl = div; 10083 } 10084 auto i = lbl.addChild("select", name); 10085 i.attrs.name = name; 10086 10087 foreach(memberName; __traits(allMembers, T)) 10088 i.addChild("option", memberName); 10089 10090 } else static if(is(T == struct)) { 10091 if(displayName !is null) 10092 div.addChild("span", displayName, "label-text"); 10093 auto fieldset = div.addChild("fieldset"); 10094 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10095 fieldset.addChild("input", name); 10096 foreach(idx, memberName; __traits(allMembers, T)) 10097 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10098 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10099 } 10100 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10101 Element lbl; 10102 if(displayName !is null) { 10103 lbl = div.addChild("label"); 10104 lbl.addChild("span", displayName, "label-text"); 10105 lbl.appendText(" "); 10106 } else { 10107 lbl = div; 10108 } 10109 Element i; 10110 if(udaSuggestion) { 10111 i = udaSuggestion(); 10112 lbl.appendChild(i); 10113 } else { 10114 i = lbl.addChild("input", name); 10115 } 10116 i.attrs.name = name; 10117 static if(isSomeString!T) 10118 i.attrs.type = "text"; 10119 else 10120 i.attrs.type = "number"; 10121 if(i.tagName == "textarea") 10122 i.textContent = to!string(T.init); 10123 else 10124 i.attrs.value = to!string(T.init); 10125 } else static if(is(T == bool)) { 10126 Element lbl; 10127 if(displayName !is null) { 10128 lbl = div.addChild("label"); 10129 lbl.addChild("span", displayName, "label-text"); 10130 lbl.appendText(" "); 10131 } else { 10132 lbl = div; 10133 } 10134 auto i = lbl.addChild("input", name); 10135 i.attrs.type = "checkbox"; 10136 i.attrs.value = "true"; 10137 i.attrs.name = name; 10138 } else static if(is(T == K[], K)) { 10139 auto templ = div.addChild("template"); 10140 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10141 if(displayName !is null) 10142 div.addChild("span", displayName, "label-text"); 10143 auto btn = div.addChild("button"); 10144 btn.addClass("add-array-button"); 10145 btn.attrs.type = "button"; 10146 btn.innerText = "Add"; 10147 btn.attrs.onclick = q{ 10148 var a = document.importNode(this.parentNode.firstChild.content, true); 10149 this.parentNode.insertBefore(a, this); 10150 }; 10151 } else static if(is(T == V[K], K, V)) { 10152 div.innerText = "assoc array not implemented for automatic form at this time"; 10153 } else { 10154 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10155 } 10156 10157 10158 return div; 10159 } 10160 10161 /// creates a form for gathering the function's arguments 10162 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10163 10164 auto form = cast(Form) Element.make("form"); 10165 10166 form.method = "POST"; // FIXME 10167 10168 form.addClass("automatic-form"); 10169 10170 string formDisplayName = beautify(__traits(identifier, method)); 10171 foreach(attr; __traits(getAttributes, method)) 10172 static if(is(typeof(attr) == DisplayName)) 10173 formDisplayName = attr.name; 10174 form.addChild("h3", formDisplayName); 10175 10176 import std.traits; 10177 10178 //Parameters!method params; 10179 //alias idents = ParameterIdentifierTuple!method; 10180 //alias defaults = ParameterDefaults!method; 10181 10182 static if(is(typeof(method) P == __parameters)) 10183 foreach(idx, _; P) {{ 10184 10185 alias param = P[idx .. idx + 1]; 10186 10187 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10188 string displayName = beautify(__traits(identifier, param)); 10189 Element function() element; 10190 foreach(attr; __traits(getAttributes, param)) { 10191 static if(is(typeof(attr) == DisplayName)) 10192 displayName = attr.name; 10193 else static if(is(typeof(attr) : typeof(element))) { 10194 element = attr; 10195 } 10196 } 10197 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10198 if(i.querySelector("input[type=file]") !is null) 10199 form.setAttribute("enctype", "multipart/form-data"); 10200 } 10201 }} 10202 10203 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10204 10205 return form; 10206 } 10207 10208 /// creates a form for gathering object members (for the REST object thing right now) 10209 Form createAutomaticFormForObject(T)(T obj) { 10210 auto form = cast(Form) Element.make("form"); 10211 10212 form.addClass("automatic-form"); 10213 10214 form.addChild("h3", beautify(__traits(identifier, T))); 10215 10216 import std.traits; 10217 10218 //Parameters!method params; 10219 //alias idents = ParameterIdentifierTuple!method; 10220 //alias defaults = ParameterDefaults!method; 10221 10222 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10223 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10224 string displayName = beautify(memberName); 10225 Element function() element; 10226 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10227 static if(is(typeof(attr) == DisplayName)) 10228 displayName = attr.name; 10229 else static if(is(typeof(attr) : typeof(element))) 10230 element = attr; 10231 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10232 10233 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10234 }}} 10235 10236 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10237 10238 return form; 10239 } 10240 10241 /// 10242 Element formatReturnValueAsHtml(T)(T t) { 10243 import std.traits; 10244 10245 static if(is(T == typeof(null))) { 10246 return Element.make("span"); 10247 } else static if(is(T : Element)) { 10248 return t; 10249 } else static if(is(T == MultipleResponses!Types, Types...)) { 10250 foreach(index, type; Types) { 10251 if(t.contains == index) 10252 return formatReturnValueAsHtml(t.payload[index]); 10253 } 10254 assert(0); 10255 } else static if(is(T == Paginated!E, E)) { 10256 auto e = Element.make("div").addClass("paginated-result"); 10257 e.appendChild(formatReturnValueAsHtml(t.items)); 10258 if(t.nextPageUrl.length) 10259 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10260 return e; 10261 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10262 return Element.make("span", to!string(t), "automatic-data-display"); 10263 } else static if(is(T == V[K], K, V)) { 10264 auto dl = Element.make("dl"); 10265 dl.addClass("automatic-data-display associative-array"); 10266 foreach(k, v; t) { 10267 dl.addChild("dt", to!string(k)); 10268 dl.addChild("dd", formatReturnValueAsHtml(v)); 10269 } 10270 return dl; 10271 } else static if(is(T == struct)) { 10272 auto dl = Element.make("dl"); 10273 dl.addClass("automatic-data-display struct"); 10274 10275 foreach(idx, memberName; __traits(allMembers, T)) 10276 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10277 dl.addChild("dt", beautify(memberName)); 10278 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10279 } 10280 10281 return dl; 10282 } else static if(is(T == bool)) { 10283 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10284 } else static if(is(T == E[], E)) { 10285 static if(is(E : RestObject!Proxy, Proxy)) { 10286 // treat RestObject similar to struct 10287 auto table = cast(Table) Element.make("table"); 10288 table.addClass("automatic-data-display"); 10289 string[] names; 10290 foreach(idx, memberName; __traits(derivedMembers, E)) 10291 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10292 names ~= beautify(memberName); 10293 } 10294 table.appendHeaderRow(names); 10295 10296 foreach(l; t) { 10297 auto tr = table.appendRow(); 10298 foreach(idx, memberName; __traits(derivedMembers, E)) 10299 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10300 static if(memberName == "id") { 10301 string val = to!string(__traits(getMember, l, memberName)); 10302 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10303 } else { 10304 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10305 } 10306 } 10307 } 10308 10309 return table; 10310 } else static if(is(E == struct)) { 10311 // an array of structs is kinda special in that I like 10312 // having those formatted as tables. 10313 auto table = cast(Table) Element.make("table"); 10314 table.addClass("automatic-data-display"); 10315 string[] names; 10316 foreach(idx, memberName; __traits(allMembers, E)) 10317 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10318 names ~= beautify(memberName); 10319 } 10320 table.appendHeaderRow(names); 10321 10322 foreach(l; t) { 10323 auto tr = table.appendRow(); 10324 foreach(idx, memberName; __traits(allMembers, E)) 10325 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10326 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10327 } 10328 } 10329 10330 return table; 10331 } else { 10332 // otherwise, I will just make a list. 10333 auto ol = Element.make("ol"); 10334 ol.addClass("automatic-data-display"); 10335 foreach(e; t) 10336 ol.addChild("li", formatReturnValueAsHtml(e)); 10337 return ol; 10338 } 10339 } else static if(is(T : Object)) { 10340 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10341 return Element.make("div", t.toHtml()); 10342 else 10343 return Element.make("div", t.toString()); 10344 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10345 10346 assert(0); 10347 } 10348 10349 } 10350 10351 /++ 10352 The base class for the [dispatcher] function and object support. 10353 +/ 10354 class WebObject { 10355 //protected Cgi cgi; 10356 10357 protected void initialize(Cgi cgi) { 10358 //this.cgi = cgi; 10359 } 10360 } 10361 10362 /++ 10363 Can return one of the given types, decided at runtime. The syntax 10364 is to declare all the possible types in the return value, then you 10365 can `return typeof(return)(...value...)` to construct it. 10366 10367 It has an auto-generated constructor for each value it can hold. 10368 10369 --- 10370 MultipleResponses!(Redirection, string) getData(int how) { 10371 if(how & 1) 10372 return typeof(return)(Redirection("http://dpldocs.info/")); 10373 else 10374 return typeof(return)("hi there!"); 10375 } 10376 --- 10377 10378 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10379 +/ 10380 struct MultipleResponses(T...) { 10381 private size_t contains; 10382 private union { 10383 private T payload; 10384 } 10385 10386 static foreach(index, type; T) 10387 public this(type t) { 10388 contains = index; 10389 payload[index] = t; 10390 } 10391 10392 /++ 10393 This is primarily for testing. It is your way of getting to the response. 10394 10395 Let's say you wanted to test that one holding a Redirection and a string actually 10396 holds a string, by name of "test": 10397 10398 --- 10399 auto valueToTest = your_test_function(); 10400 10401 valueToTest.visit( 10402 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10403 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10404 ); 10405 --- 10406 10407 History: 10408 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10409 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10410 +/ 10411 void visit(Handlers...)(Handlers handlers) { 10412 template findHandler(type, int count, HandlersToCheck...) { 10413 static if(HandlersToCheck.length == 0) 10414 enum findHandler = -1; 10415 else { 10416 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10417 enum findHandler = count; 10418 else 10419 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10420 } 10421 } 10422 foreach(index, type; T) { 10423 enum handlerIndex = findHandler!(type, 0, Handlers); 10424 static if(handlerIndex == -1) 10425 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10426 else { 10427 if(index == this.contains) 10428 handlers[handlerIndex](this.payload[index]); 10429 } 10430 } 10431 } 10432 10433 /+ 10434 auto toArsdJsvar()() { 10435 import arsd.jsvar; 10436 return var(null); 10437 } 10438 +/ 10439 } 10440 10441 // FIXME: implement this somewhere maybe 10442 struct RawResponse { 10443 int code; 10444 string[] headers; 10445 const(ubyte)[] responseBody; 10446 } 10447 10448 /++ 10449 You can return this from [WebObject] subclasses for redirections. 10450 10451 (though note the static types means that class must ALWAYS redirect if 10452 you return this directly. You might want to return [MultipleResponses] if it 10453 can be conditional) 10454 +/ 10455 struct Redirection { 10456 string to; /// The URL to redirect to. 10457 int code = 303; /// The HTTP code to return. 10458 } 10459 10460 /++ 10461 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10462 10463 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10464 the presenter in the dispatcher. 10465 10466 FIXME: explain this better 10467 10468 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10469 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10470 the runtime result of that is undefined. 10471 10472 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10473 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10474 10475 $(WARNING 10476 --- 10477 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10478 // even if you put different URL udas on it, the current code ignores them. 10479 void foo(int a) {} 10480 void foo(string a) {} 10481 --- 10482 ) 10483 10484 See_Also: [serveRestObject], [serveStaticFile] 10485 +/ 10486 auto serveApi(T)(string urlPrefix) { 10487 assert(urlPrefix[$ - 1] == '/'); 10488 return serveApiInternal!T(urlPrefix); 10489 } 10490 10491 private string nextPieceFromSlash(ref string remainingUrl) { 10492 if(remainingUrl.length == 0) 10493 return remainingUrl; 10494 int slash = 0; 10495 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10496 slash++; 10497 10498 // I am specifically passing `null` to differentiate it vs empty string 10499 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10500 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10501 // so if it is the last item, the dot can be used to load an alternative view 10502 // otherwise tho the dot is considered part of the identifier 10503 // FIXME 10504 10505 // again notice "" vs null here! 10506 if(slash == remainingUrl.length) 10507 remainingUrl = null; 10508 else 10509 remainingUrl = remainingUrl[slash + 1 .. $]; 10510 10511 return ident; 10512 } 10513 10514 /++ 10515 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10516 +/ 10517 enum AddTrailingSlash; 10518 /// ditto 10519 enum RemoveTrailingSlash; 10520 10521 private auto serveApiInternal(T)(string urlPrefix) { 10522 10523 import arsd.dom; 10524 import arsd.jsvar; 10525 10526 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10527 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10528 10529 try { 10530 // see duplicated code below by searching subresource_ctor 10531 // also see mustNotBeSetFromWebParams 10532 10533 static if(is(typeof(T.__ctor) P == __parameters)) { 10534 P params; 10535 10536 foreach(pidx, param; P) { 10537 static if(is(param : Cgi)) { 10538 static assert(!is(param == immutable)); 10539 cast() params[pidx] = cgi; 10540 } else static if(is(param == Session!D, D)) { 10541 static assert(!is(param == immutable)); 10542 cast() params[pidx] = cgi.getSessionObject!D(); 10543 10544 } else { 10545 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10546 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10547 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10548 static if(is(typeof(func(cgi)))) 10549 params[pidx] = func(cgi); 10550 else 10551 params[pidx] = func(); 10552 } 10553 } 10554 } else { 10555 10556 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10557 params[pidx] = param.getAutomaticallyForCgi(cgi); 10558 } else static if(is(param == string)) { 10559 auto ident = nextPieceFromSlash(remainingUrl); 10560 params[pidx] = ident; 10561 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10562 } 10563 } 10564 } 10565 10566 auto obj = new T(params); 10567 } else { 10568 auto obj = new T(); 10569 } 10570 10571 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10572 } catch(Throwable t) { 10573 switch(cgi.request("format", "html")) { 10574 case "html": 10575 static void dummy() {} 10576 presenter.presentExceptionAsHtml(cgi, t, null); 10577 return true; 10578 case "json": 10579 var envelope = var.emptyObject; 10580 envelope.success = false; 10581 envelope.result = null; 10582 envelope.error = t.toString(); 10583 cgi.setResponseContentType("application/json"); 10584 cgi.write(envelope.toJson(), true); 10585 return true; 10586 default: 10587 throw t; 10588 // return true; 10589 } 10590 // return true; 10591 } 10592 10593 assert(0); 10594 } 10595 10596 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10597 10598 obj.initialize(cgi); 10599 10600 /+ 10601 Overload rules: 10602 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10603 statically. 10604 10605 Moreover, some args vs no args can be overloaded dynamically. 10606 +/ 10607 10608 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10609 /+ 10610 auto orig = remainingUrl; 10611 assert(0, 10612 (orig is null ? "__null" : orig) 10613 ~ " .. " ~ 10614 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10615 +/ 10616 10617 if(methodNameFromUrl is null) 10618 methodNameFromUrl = "__null"; 10619 10620 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10621 10622 if(remainingUrl.length) 10623 hack ~= "/"; 10624 10625 switch(hack) { 10626 foreach(methodName; __traits(derivedMembers, T)) 10627 static if(methodName != "__ctor") 10628 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10629 static if(is(typeof(overload) P == __parameters)) 10630 static if(is(typeof(overload) R == return)) 10631 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10632 { 10633 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10634 case urlNameForMethod: 10635 10636 static if(is(R : WebObject)) { 10637 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10638 10639 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10640 10641 // subresource_ctor 10642 // also see mustNotBeSetFromWebParams 10643 10644 P params; 10645 10646 string ident; 10647 10648 foreach(pidx, param; P) { 10649 static if(is(param : Cgi)) { 10650 static assert(!is(param == immutable)); 10651 cast() params[pidx] = cgi; 10652 } else static if(is(param == typeof(presenter))) { 10653 cast() param[pidx] = presenter; 10654 } else static if(is(param == Session!D, D)) { 10655 static assert(!is(param == immutable)); 10656 cast() params[pidx] = cgi.getSessionObject!D(); 10657 } else { 10658 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10659 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10660 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10661 static if(is(typeof(func(cgi)))) 10662 params[pidx] = func(cgi); 10663 else 10664 params[pidx] = func(); 10665 } 10666 } 10667 } else { 10668 10669 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10670 params[pidx] = param.getAutomaticallyForCgi(cgi); 10671 } else static if(is(param == string)) { 10672 ident = nextPieceFromSlash(remainingUrl); 10673 if(ident is null) { 10674 // trailing slash mandated on subresources 10675 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10676 return true; 10677 } else { 10678 params[pidx] = ident; 10679 } 10680 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10681 } 10682 } 10683 } 10684 10685 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10686 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10687 } else { 10688 // 404 it if any url left - not a subresource means we don't get to play with that! 10689 if(remainingUrl.length) 10690 return false; 10691 10692 bool automaticForm; 10693 10694 foreach(attr; __traits(getAttributes, overload)) 10695 static if(is(attr == AddTrailingSlash)) { 10696 if(remainingUrl is null) { 10697 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10698 return true; 10699 } 10700 } else static if(is(attr == RemoveTrailingSlash)) { 10701 if(remainingUrl !is null) { 10702 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10703 return true; 10704 } 10705 10706 } else static if(__traits(isSame, AutomaticForm, attr)) { 10707 automaticForm = true; 10708 } 10709 10710 /+ 10711 int zeroArgOverload = -1; 10712 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10713 bool calledWithZeroArgs = true; 10714 foreach(k, v; cgi.get) 10715 if(k != "format") { 10716 calledWithZeroArgs = false; 10717 break; 10718 } 10719 foreach(k, v; cgi.post) 10720 if(k != "format") { 10721 calledWithZeroArgs = false; 10722 break; 10723 } 10724 10725 // first, we need to go through and see if there is an empty one, since that 10726 // changes inside. But otherwise, all the stuff I care about can be done via 10727 // simple looping (other improper overloads might be flagged for runtime semantic check) 10728 // 10729 // an argument of type Cgi is ignored for these purposes 10730 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10731 static if(is(typeof(overload) P == __parameters)) 10732 static if(P.length == 0) 10733 zeroArgOverload = cast(int) idx; 10734 else static if(P.length == 1 && is(P[0] : Cgi)) 10735 zeroArgOverload = cast(int) idx; 10736 }} 10737 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10738 bool overloadHasBeenCalled = false; 10739 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10740 bool callFunction = true; 10741 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10742 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10743 callFunction = false; 10744 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10745 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10746 callFunction = false; 10747 10748 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10749 10750 bool hadAnyMethodRestrictions = false; 10751 bool foundAcceptableMethod = false; 10752 foreach(attr; __traits(getAttributes, overload)) { 10753 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10754 hadAnyMethodRestrictions = true; 10755 if(attr == cgi.requestMethod) 10756 foundAcceptableMethod = true; 10757 } 10758 } 10759 10760 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10761 callFunction = false; 10762 10763 /+ 10764 The overloads we really want to allow are the sane ones 10765 from the web perspective. Which is likely on HTTP verbs, 10766 for the most part, but might also be potentially based on 10767 some args vs zero args, or on argument names. Can't really 10768 do argument types very reliable through the web though; those 10769 should probably be different URLs. 10770 10771 Even names I feel is better done inside the function, so I'm not 10772 going to support that here. But the HTTP verbs and zero vs some 10773 args makes sense - it lets you define custom forms pretty easily. 10774 10775 Moreover, I'm of the opinion that empty overload really only makes 10776 sense on GET for this case. On a POST, it is just a missing argument 10777 exception and that should be handled by the presenter. But meh, I'll 10778 let the user define that, D only allows one empty arg thing anyway 10779 so the method UDAs are irrelevant. 10780 +/ 10781 if(callFunction) 10782 +/ 10783 10784 auto format = cgi.request("format", defaultFormat!overload()); 10785 auto wantsFormFormat = format.startsWith("form-"); 10786 10787 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10788 // Should I still show the form on a json thing? idk... 10789 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10790 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10791 return true; 10792 } 10793 10794 try { 10795 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10796 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10797 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10798 } catch(Throwable t) { 10799 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10800 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10801 } 10802 return true; 10803 //}} 10804 10805 //cgi.header("Accept: POST"); // FIXME list the real thing 10806 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10807 //return true; 10808 } 10809 } 10810 } 10811 case "GET script.js": 10812 cgi.setResponseContentType("text/javascript"); 10813 cgi.gzipResponse = true; 10814 cgi.write(presenter.script(), true); 10815 return true; 10816 case "GET style.css": 10817 cgi.setResponseContentType("text/css"); 10818 cgi.gzipResponse = true; 10819 cgi.write(presenter.style(), true); 10820 return true; 10821 default: 10822 return false; 10823 } 10824 10825 assert(0); 10826 } 10827 return DispatcherDefinition!internalHandler(urlPrefix, false); 10828 } 10829 10830 string defaultFormat(alias method)() { 10831 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10832 foreach(attr; __traits(getAttributes, method)) { 10833 static if(is(typeof(attr) == DefaultFormat)) { 10834 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10835 return attr.value; 10836 } 10837 } 10838 return "html"; 10839 } 10840 10841 struct Paginated(T) { 10842 T[] items; 10843 string nextPageUrl; 10844 } 10845 10846 template urlNamesForMethod(alias method, string default_) { 10847 string[] helper() { 10848 auto verb = Cgi.RequestMethod.GET; 10849 bool foundVerb = false; 10850 bool foundNoun = false; 10851 10852 string def = default_; 10853 10854 bool hasAutomaticForm = false; 10855 10856 foreach(attr; __traits(getAttributes, method)) { 10857 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10858 verb = attr; 10859 if(foundVerb) 10860 assert(0, "Multiple http verbs on one function is not currently supported"); 10861 foundVerb = true; 10862 } 10863 static if(is(typeof(attr) == UrlName)) { 10864 if(foundNoun) 10865 assert(0, "Multiple url names on one function is not currently supported"); 10866 foundNoun = true; 10867 def = attr.name; 10868 } 10869 static if(__traits(isSame, attr, AutomaticForm)) { 10870 hasAutomaticForm = true; 10871 } 10872 } 10873 10874 if(def is null) 10875 def = "__null"; 10876 10877 string[] ret; 10878 10879 static if(is(typeof(method) R == return)) { 10880 static if(is(R : WebObject)) { 10881 def ~= "/"; 10882 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10883 ret ~= v ~ " " ~ def; 10884 } else { 10885 if(hasAutomaticForm) { 10886 ret ~= "GET " ~ def; 10887 ret ~= "POST " ~ def; 10888 } else { 10889 ret ~= to!string(verb) ~ " " ~ def; 10890 } 10891 } 10892 } else static assert(0); 10893 10894 return ret; 10895 } 10896 enum urlNamesForMethod = helper(); 10897 } 10898 10899 10900 enum AccessCheck { 10901 allowed, 10902 denied, 10903 nonExistant, 10904 } 10905 10906 enum Operation { 10907 show, 10908 create, 10909 replace, 10910 remove, 10911 update 10912 } 10913 10914 enum UpdateResult { 10915 accessDenied, 10916 noSuchResource, 10917 success, 10918 failure, 10919 unnecessary 10920 } 10921 10922 enum ValidationResult { 10923 valid, 10924 invalid 10925 } 10926 10927 10928 /++ 10929 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10930 10931 WARNING: this is not stable. 10932 +/ 10933 class RestObject(CRTP) : WebObject { 10934 10935 import arsd.dom; 10936 import arsd.jsvar; 10937 10938 /// Prepare the object to be shown. 10939 void show() {} 10940 /// ditto 10941 void show(string urlId) { 10942 load(urlId); 10943 show(); 10944 } 10945 10946 /// Override this to provide access control to this object. 10947 AccessCheck accessCheck(string urlId, Operation operation) { 10948 return AccessCheck.allowed; 10949 } 10950 10951 ValidationResult validate() { 10952 // FIXME 10953 return ValidationResult.valid; 10954 } 10955 10956 string getUrlSlug() { 10957 import std.conv; 10958 static if(is(typeof(CRTP.id))) 10959 return to!string((cast(CRTP) this).id); 10960 else 10961 return null; 10962 } 10963 10964 // The functions with more arguments are the low-level ones, 10965 // they forward to the ones with fewer arguments by default. 10966 10967 // POST on a parent collection - this is called from a collection class after the members are updated 10968 /++ 10969 Given a populated object, this creates a new entry. Returns the url identifier 10970 of the new object. 10971 +/ 10972 string create(scope void delegate() applyChanges) { 10973 applyChanges(); 10974 save(); 10975 return getUrlSlug(); 10976 } 10977 10978 void replace() { 10979 save(); 10980 } 10981 void replace(string urlId, scope void delegate() applyChanges) { 10982 load(urlId); 10983 applyChanges(); 10984 replace(); 10985 } 10986 10987 void update(string[] fieldList) { 10988 save(); 10989 } 10990 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 10991 load(urlId); 10992 applyChanges(); 10993 update(fieldList); 10994 } 10995 10996 void remove() {} 10997 10998 void remove(string urlId) { 10999 load(urlId); 11000 remove(); 11001 } 11002 11003 abstract void load(string urlId); 11004 abstract void save(); 11005 11006 Element toHtml(Presenter)(Presenter presenter) { 11007 import arsd.dom; 11008 import std.conv; 11009 auto obj = cast(CRTP) this; 11010 auto div = Element.make("div"); 11011 div.addClass("Dclass_" ~ CRTP.stringof); 11012 div.dataset.url = getUrlSlug(); 11013 bool first = true; 11014 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11015 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11016 if(!first) div.addChild("br"); else first = false; 11017 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11018 } 11019 return div; 11020 } 11021 11022 var toJson() { 11023 import arsd.jsvar; 11024 var v = var.emptyObject(); 11025 auto obj = cast(CRTP) this; 11026 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11027 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11028 v[memberName] = __traits(getMember, obj, memberName); 11029 } 11030 return v; 11031 } 11032 11033 /+ 11034 auto structOf(this This) { 11035 11036 } 11037 +/ 11038 } 11039 11040 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11041 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11042 11043 /++ 11044 Base class for REST collections. 11045 +/ 11046 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11047 /// You might subclass this and use the cgi object's query params 11048 /// to implement a search filter, for example. 11049 /// 11050 /// FIXME: design a way to auto-generate that form 11051 /// (other than using the WebObject thing above lol 11052 // it'll prolly just be some searchParams UDA or maybe an enum. 11053 // 11054 // pagination too perhaps. 11055 // 11056 // and sorting too 11057 IndexResult index() { return IndexResult.init; } 11058 11059 string[] sortableFields() { return null; } 11060 string[] searchableFields() { return null; } 11061 11062 struct IndexResult { 11063 Obj[] results; 11064 11065 string[] sortableFields; 11066 11067 string previousPageIdentifier; 11068 string nextPageIdentifier; 11069 string firstPageIdentifier; 11070 string lastPageIdentifier; 11071 11072 int numberOfPages; 11073 } 11074 11075 override string create(scope void delegate() applyChanges) { assert(0); } 11076 override void load(string urlId) { assert(0); } 11077 override void save() { assert(0); } 11078 override void show() { 11079 index(); 11080 } 11081 override void show(string urlId) { 11082 show(); 11083 } 11084 11085 /// Proxy POST requests (create calls) to the child collection 11086 alias PostProxy = Obj; 11087 } 11088 11089 /++ 11090 Serves a REST object, similar to a Ruby on Rails resource. 11091 11092 You put data members in your class. cgi.d will automatically make something out of those. 11093 11094 It will call your constructor with the ID from the URL. This may be null. 11095 It will then populate the data members from the request. 11096 It will then call a method, if present, telling what happened. You don't need to write these! 11097 It finally returns a reply. 11098 11099 Your methods are passed a list of fields it actually set. 11100 11101 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11102 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11103 with relative linking. But meh.) 11104 11105 GET /items -> index. all values not set. 11106 GET /items/id -> get. only ID will be set, other params ignored. 11107 POST /items -> create. values set as given 11108 PUT /items/id -> replace. values set as given 11109 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11110 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11111 PATCH /items/id -> update. values set as given, list of changed fields passed 11112 or POST /items/id with cgi.post["_method"] == "PATCH" 11113 DELETE /items/id -> destroy. only ID guaranteed to be set 11114 or POST /items/id with cgi.post["_method"] == "DELETE" 11115 11116 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11117 redirect you away from it. 11118 11119 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11120 11121 I will also let you change the default, if you must. 11122 11123 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11124 11125 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11126 that follow the same thing. They may be individual resources or collections themselves. 11127 11128 Your class is expected to have at least the following methods: 11129 11130 FIXME: i kinda wanna add a routes object to the initialize call 11131 11132 create 11133 Create returns the new address on success, some code on failure. 11134 show 11135 index 11136 update 11137 remove 11138 11139 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11140 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11141 11142 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11143 11144 NOT IMPLEMENTED 11145 11146 11147 Really, a collection is a resource with a bunch of subresources. 11148 11149 GET /items 11150 index because it is GET on the top resource 11151 11152 GET /items/foo 11153 item but different than items? 11154 11155 class Items { 11156 11157 } 11158 11159 ... but meh, a collection can be automated. not worth making it 11160 a separate thing, let's look at a real example. Users has many 11161 items and a virtual one, /users/current. 11162 11163 the individual users have properties and two sub-resources: 11164 session, which is just one, and comments, a collection. 11165 11166 class User : RestObject!() { // no parent 11167 int id; 11168 string name; 11169 11170 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11171 // but you can override them to do it differently. 11172 11173 // any member which is of type RestObject can be linked automatically via href btw. 11174 11175 void show() {} 11176 void show(string urlId) {} // automated! GET of this specific thing 11177 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11178 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11179 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11180 void remove(string urlId) {} // DELETE 11181 11182 void load(string urlId) {} // the default implementation of show() populates the id, then 11183 11184 this() {} 11185 11186 mixin Subresource!Session; 11187 mixin Subresource!Comment; 11188 } 11189 11190 class Session : RestObject!() { 11191 // the parent object may not be fully constructed/loaded 11192 this(User parent) {} 11193 11194 } 11195 11196 class Comment : CollectionOf!Comment { 11197 this(User parent) {} 11198 } 11199 11200 class Users : CollectionOf!User { 11201 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11202 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11203 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11204 } 11205 11206 +/ 11207 auto serveRestObject(T)(string urlPrefix) { 11208 assert(urlPrefix[0] == '/'); 11209 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11210 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11211 string url = cgi.pathInfo[urlPrefix.length .. $]; 11212 11213 if(url.length && url[$ - 1] == '/') { 11214 // remove the final slash... 11215 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11216 return true; 11217 } 11218 11219 return restObjectServeHandler!T(cgi, presenter, url); 11220 } 11221 return DispatcherDefinition!internalHandler(urlPrefix, false); 11222 } 11223 11224 /+ 11225 /// Convenience method for serving a collection. It will be named the same 11226 /// as type T, just with an s at the end. If you need any further, just 11227 /// write the class yourself. 11228 auto serveRestCollectionOf(T)(string urlPrefix) { 11229 assert(urlPrefix[0] == '/'); 11230 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11231 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11232 } 11233 +/ 11234 11235 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11236 string urlId = null; 11237 if(url.length && url[0] == '/') { 11238 // asking for a subobject 11239 urlId = url[1 .. $]; 11240 foreach(idx, ch; urlId) { 11241 if(ch == '/') { 11242 urlId = urlId[0 .. idx]; 11243 break; 11244 } 11245 } 11246 } 11247 11248 // FIXME handle other subresources 11249 11250 static if(is(T : CollectionOf!(C), C)) { 11251 if(urlId !is null) { 11252 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11253 } 11254 } 11255 11256 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11257 11258 auto obj = new T(); 11259 obj.initialize(cgi); 11260 // FIXME: populate reflection info delegates 11261 11262 11263 // FIXME: I am not happy with this. 11264 switch(urlId) { 11265 case "script.js": 11266 cgi.setResponseContentType("text/javascript"); 11267 cgi.gzipResponse = true; 11268 cgi.write(presenter.script(), true); 11269 return true; 11270 case "style.css": 11271 cgi.setResponseContentType("text/css"); 11272 cgi.gzipResponse = true; 11273 cgi.write(presenter.style(), true); 11274 return true; 11275 default: 11276 // intentionally blank 11277 } 11278 11279 11280 11281 11282 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11283 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11284 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11285 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11286 } 11287 } 11288 void applyChanges() { 11289 applyChangesTemplate(cgi, obj); 11290 } 11291 11292 string[] modifiedList; 11293 11294 void writeObject(bool addFormLinks) { 11295 if(cgi.request("format") == "json") { 11296 cgi.setResponseContentType("application/json"); 11297 cgi.write(obj.toJson().toString, true); 11298 } else { 11299 auto container = presenter.htmlContainer(); 11300 if(addFormLinks) { 11301 static if(is(T : CollectionOf!(C), C)) 11302 container.appendHtml(` 11303 <form> 11304 <button type="submit" name="_method" value="POST">Create New</button> 11305 </form> 11306 `); 11307 else 11308 container.appendHtml(` 11309 <a href="..">Back</a> 11310 <form> 11311 <button type="submit" name="_method" value="PATCH">Edit</button> 11312 <button type="submit" name="_method" value="DELETE">Delete</button> 11313 </form> 11314 `); 11315 } 11316 container.appendChild(obj.toHtml(presenter)); 11317 cgi.write(container.parentDocument.toString, true); 11318 } 11319 } 11320 11321 // FIXME: I think I need a set type in here.... 11322 // it will be nice to pass sets of members. 11323 11324 try 11325 switch(cgi.requestMethod) { 11326 case Cgi.RequestMethod.GET: 11327 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11328 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11329 11330 // automatic forms here for usable basic auto site from browser. 11331 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11332 switch(cgi.request("_method", "GET")) { 11333 case "GET": 11334 static if(is(T : CollectionOf!(C), C)) { 11335 auto results = obj.index(); 11336 if(cgi.request("format", "html") == "html") { 11337 auto container = presenter.htmlContainer(); 11338 auto html = presenter.formatReturnValueAsHtml(results.results); 11339 container.appendHtml(` 11340 <form> 11341 <button type="submit" name="_method" value="POST">Create New</button> 11342 </form> 11343 `); 11344 11345 container.appendChild(html); 11346 cgi.write(container.parentDocument.toString, true); 11347 } else { 11348 cgi.setResponseContentType("application/json"); 11349 import arsd.jsvar; 11350 var json = var.emptyArray; 11351 foreach(r; results.results) { 11352 var o = var.emptyObject; 11353 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11354 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11355 o[memberName] = __traits(getMember, r, memberName); 11356 } 11357 11358 json ~= o; 11359 } 11360 cgi.write(json.toJson(), true); 11361 } 11362 } else { 11363 obj.show(urlId); 11364 writeObject(true); 11365 } 11366 break; 11367 case "PATCH": 11368 obj.load(urlId); 11369 goto case; 11370 case "PUT": 11371 case "POST": 11372 // an editing form for the object 11373 auto container = presenter.htmlContainer(); 11374 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11375 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11376 } else { 11377 auto form = presenter.createAutomaticFormForObject(obj); 11378 } 11379 form.attrs.method = "POST"; 11380 form.setValue("_method", cgi.request("_method", "GET")); 11381 container.appendChild(form); 11382 cgi.write(container.parentDocument.toString(), true); 11383 break; 11384 case "DELETE": 11385 // FIXME: a delete form for the object (can be phrased "are you sure?") 11386 auto container = presenter.htmlContainer(); 11387 container.appendHtml(` 11388 <form method="POST"> 11389 Are you sure you want to delete this item? 11390 <input type="hidden" name="_method" value="DELETE" /> 11391 <input type="submit" value="Yes, Delete It" /> 11392 </form> 11393 11394 `); 11395 cgi.write(container.parentDocument.toString(), true); 11396 break; 11397 default: 11398 cgi.write("bad method\n", true); 11399 } 11400 break; 11401 case Cgi.RequestMethod.POST: 11402 // this is to allow compatibility with HTML forms 11403 switch(cgi.request("_method", "POST")) { 11404 case "PUT": 11405 goto PUT; 11406 case "PATCH": 11407 goto PATCH; 11408 case "DELETE": 11409 goto DELETE; 11410 case "POST": 11411 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11412 auto p = new obj.PostProxy(); 11413 void specialApplyChanges() { 11414 applyChangesTemplate(cgi, p); 11415 } 11416 string n = p.create(&specialApplyChanges); 11417 } else { 11418 string n = obj.create(&applyChanges); 11419 } 11420 11421 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11422 cgi.setResponseLocation(newUrl); 11423 cgi.setResponseStatus("201 Created"); 11424 cgi.write(`The object has been created.`); 11425 break; 11426 default: 11427 cgi.write("bad method\n", true); 11428 } 11429 // FIXME this should be valid on the collection, but not the child.... 11430 // 303 See Other 11431 break; 11432 case Cgi.RequestMethod.PUT: 11433 PUT: 11434 obj.replace(urlId, &applyChanges); 11435 writeObject(false); 11436 break; 11437 case Cgi.RequestMethod.PATCH: 11438 PATCH: 11439 obj.update(urlId, &applyChanges, modifiedList); 11440 writeObject(false); 11441 break; 11442 case Cgi.RequestMethod.DELETE: 11443 DELETE: 11444 obj.remove(urlId); 11445 cgi.setResponseStatus("204 No Content"); 11446 break; 11447 default: 11448 // FIXME: OPTIONS, HEAD 11449 } 11450 catch(Throwable t) { 11451 presenter.presentExceptionAsHtml(cgi, t); 11452 } 11453 11454 return true; 11455 } 11456 11457 /+ 11458 struct SetOfFields(T) { 11459 private void[0][string] storage; 11460 void set(string what) { 11461 //storage[what] = 11462 } 11463 void unset(string what) {} 11464 void setAll() {} 11465 void unsetAll() {} 11466 bool isPresent(string what) { return false; } 11467 } 11468 +/ 11469 11470 /+ 11471 enum readonly; 11472 enum hideonindex; 11473 +/ 11474 11475 /++ 11476 Returns true if I recommend gzipping content of this type. You might 11477 want to call it from your Presenter classes before calling cgi.write. 11478 11479 --- 11480 cgi.setResponseContentType(yourContentType); 11481 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11482 cgi.write(yourData, true); 11483 --- 11484 11485 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11486 11487 11488 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11489 11490 History: 11491 Added January 28, 2023 (dub v11.0) 11492 +/ 11493 bool gzipRecommendedForContentType(string contentType) { 11494 if(contentType.startsWith("text/")) 11495 return true; 11496 if(contentType.startsWith("application/javascript")) 11497 return true; 11498 11499 return false; 11500 } 11501 11502 /++ 11503 Serves a static file. To be used with [dispatcher]. 11504 11505 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11506 +/ 11507 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11508 // https://baus.net/on-tcp_cork/ 11509 // man 2 sendfile 11510 assert(urlPrefix[0] == '/'); 11511 if(filename is null) 11512 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11513 if(contentType is null) { 11514 contentType = contentTypeFromFileExtension(filename); 11515 } 11516 11517 static struct DispatcherDetails { 11518 string filename; 11519 string contentType; 11520 } 11521 11522 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11523 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11524 cgi.setCache(true); 11525 cgi.setResponseContentType(details.contentType); 11526 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11527 cgi.write(std.file.read(details.filename), true); 11528 return true; 11529 } 11530 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11531 } 11532 11533 /++ 11534 Serves static data. To be used with [dispatcher]. 11535 11536 History: 11537 Added October 31, 2021 11538 +/ 11539 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11540 assert(urlPrefix[0] == '/'); 11541 if(contentType is null) { 11542 contentType = contentTypeFromFileExtension(urlPrefix); 11543 } 11544 11545 static struct DispatcherDetails { 11546 immutable(void)[] data; 11547 string contentType; 11548 } 11549 11550 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11551 cgi.setCache(true); 11552 cgi.setResponseContentType(details.contentType); 11553 cgi.write(details.data, true); 11554 return true; 11555 } 11556 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11557 } 11558 11559 string contentTypeFromFileExtension(string filename) { 11560 if(filename.endsWith(".png")) 11561 return "image/png"; 11562 if(filename.endsWith(".apng")) 11563 return "image/apng"; 11564 if(filename.endsWith(".svg")) 11565 return "image/svg+xml"; 11566 if(filename.endsWith(".jpg")) 11567 return "image/jpeg"; 11568 if(filename.endsWith(".html")) 11569 return "text/html"; 11570 if(filename.endsWith(".css")) 11571 return "text/css"; 11572 if(filename.endsWith(".js")) 11573 return "application/javascript"; 11574 if(filename.endsWith(".wasm")) 11575 return "application/wasm"; 11576 if(filename.endsWith(".mp3")) 11577 return "audio/mpeg"; 11578 if(filename.endsWith(".pdf")) 11579 return "application/pdf"; 11580 return null; 11581 } 11582 11583 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11584 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11585 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11586 assert(urlPrefix[0] == '/'); 11587 assert(urlPrefix[$-1] == '/'); 11588 11589 static struct DispatcherDetails { 11590 string directory; 11591 bool recursive; 11592 } 11593 11594 if(directory is null) 11595 directory = urlPrefix[1 .. $]; 11596 11597 if(directory.length == 0) 11598 directory = "./"; 11599 11600 assert(directory[$-1] == '/'); 11601 11602 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11603 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11604 11605 if(details.recursive) { 11606 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11607 if(file.indexOf("\\") != -1) 11608 return false; 11609 11610 import std.path; 11611 11612 file = std.path.buildNormalizedPath(file); 11613 enum upOneDir = ".." ~ std.path.dirSeparator; 11614 11615 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11616 if(file == ".." || file.startsWith(upOneDir)) 11617 return false; 11618 if(std.path.isAbsolute(file)) 11619 return false; 11620 11621 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11622 11623 // once it passes these filters it is probably ok. 11624 } else { 11625 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11626 return false; 11627 } 11628 11629 auto contentType = contentTypeFromFileExtension(file); 11630 11631 auto fn = details.directory ~ file; 11632 if(std.file.exists(fn)) { 11633 //if(contentType.indexOf("image/") == 0) 11634 //cgi.setCache(true); 11635 //else if(contentType.indexOf("audio/") == 0) 11636 cgi.setCache(true); 11637 cgi.setResponseContentType(contentType); 11638 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11639 cgi.write(std.file.read(fn), true); 11640 return true; 11641 } else { 11642 return false; 11643 } 11644 } 11645 11646 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11647 } 11648 11649 /++ 11650 Redirects one url to another 11651 11652 See_Also: [dispatcher], [serveStaticFile] 11653 +/ 11654 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11655 assert(urlPrefix[0] == '/'); 11656 static struct DispatcherDetails { 11657 string redirectTo; 11658 string code; 11659 } 11660 11661 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11662 cgi.setResponseLocation(details.redirectTo, true, details.code); 11663 return true; 11664 } 11665 11666 11667 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11668 } 11669 11670 /// Used exclusively with `dispatchTo` 11671 struct DispatcherData(Presenter) { 11672 Cgi cgi; /// You can use this cgi object. 11673 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11674 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11675 } 11676 11677 /++ 11678 Dispatches the URL to a specific function. 11679 +/ 11680 auto handleWith(alias handler)(string urlPrefix) { 11681 // cuz I'm too lazy to do it better right now 11682 static class Hack : WebObject { 11683 static import std.traits; 11684 @UrlName("") 11685 auto handle(std.traits.Parameters!handler args) { 11686 return handler(args); 11687 } 11688 } 11689 11690 return urlPrefix.serveApiInternal!Hack; 11691 } 11692 11693 /++ 11694 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11695 11696 --- 11697 bool other(DD)(DD dd) { 11698 return dd.dispatcher!( 11699 "/whatever".serveRedirect("/success"), 11700 "/api/".serveApi!MyClass 11701 ); 11702 } 11703 --- 11704 11705 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11706 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11707 Or, of course, you could just use the exact type in your own code. 11708 11709 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11710 good job. 11711 11712 11713 +/ 11714 auto dispatchTo(alias handler)(string urlPrefix) { 11715 assert(urlPrefix[0] == '/'); 11716 assert(urlPrefix[$-1] != '/'); 11717 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11718 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11719 } 11720 11721 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11722 } 11723 11724 /++ 11725 See [serveStaticFile] if you want to serve a file off disk. 11726 11727 History: 11728 Added January 28, 2023 (dub v11.0) 11729 +/ 11730 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11731 assert(urlPrefix[0] == '/'); 11732 11733 static struct DispatcherDetails { 11734 immutable(ubyte)[] data; 11735 string contentType; 11736 string filenameToSuggestAsDownload; 11737 } 11738 11739 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11740 cgi.setCache(true); 11741 cgi.setResponseContentType(details.contentType); 11742 if(details.filenameToSuggestAsDownload.length) 11743 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11744 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11745 cgi.write(details.data, true); 11746 return true; 11747 } 11748 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11749 } 11750 11751 /++ 11752 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11753 11754 History: 11755 Added January 28, 2023 (dub v11.0) 11756 +/ 11757 alias KeepExistingPresenter = typeof(null); 11758 11759 /++ 11760 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11761 this issues the given errorCode and stops processing. 11762 11763 --- 11764 bool hasAdminPermissions(Cgi cgi) { 11765 return true; 11766 } 11767 11768 mixin DispatcherMain!( 11769 "/admin".dispatchSubsection!( 11770 passFilterOrIssueError!(hasAdminPermissions, 403), 11771 KeepExistingPresenter, 11772 "/".serveApi!AdminFunctions 11773 ) 11774 ); 11775 --- 11776 11777 History: 11778 Added January 28, 2023 (dub v11.0) 11779 +/ 11780 template passFilterOrIssueError(alias filter, int errorCode) { 11781 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11782 if(filter(dd.cgi)) 11783 return true; 11784 dd.presenter.renderBasicError(dd.cgi, errorCode); 11785 return false; 11786 } 11787 } 11788 11789 /++ 11790 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11791 and then be dispatched to their own handlers. 11792 11793 --- 11794 /+ 11795 // a long-form filter function 11796 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11797 // you are permitted to call mutable methods on the Cgi object 11798 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11799 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11800 11801 if(checkPermissionOnRequest(dd.cgi)) { 11802 return true; // OK, allow processing to continue 11803 } else { 11804 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11805 return false; // and stop further processing into this subsection 11806 } 11807 } 11808 +/ 11809 11810 // but you can also do short-form filters: 11811 11812 bool permissionCheck(Cgi cgi) { 11813 return ("ok" in cgi.get) !is null; 11814 } 11815 11816 // handler for the subsection 11817 class AdminClass : WebObject { 11818 int foo() { return 5; } 11819 } 11820 11821 // handler for the main site 11822 class TheMainSite : WebObject {} 11823 11824 mixin DispatcherMain!( 11825 "/admin".dispatchSubsection!( 11826 // converts our short-form filter into a long-form filter 11827 passFilterOrIssueError!(permissionCheck, 403), 11828 // can use a new presenter if wanted for the subsection 11829 KeepExistingPresenter, 11830 // and then provide child route dispatchers 11831 "/".serveApi!AdminClass 11832 ), 11833 // and back to the top level 11834 "/".serveApi!TheMainSite 11835 ); 11836 --- 11837 11838 Note you can encapsulate sections in files like this: 11839 11840 --- 11841 auto adminDispatcher(string urlPrefix) { 11842 return urlPrefix.dispatchSubsection!( 11843 .... 11844 ); 11845 } 11846 11847 mixin DispatcherMain!( 11848 "/admin".adminDispatcher, 11849 // and so on 11850 ) 11851 --- 11852 11853 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11854 11855 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11856 11857 11858 History: 11859 Added January 28, 2023 (dub v11.0) 11860 +/ 11861 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11862 assert(urlPrefix[0] == '/'); 11863 assert(urlPrefix[$-1] != '/'); 11864 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11865 static if(!is(PreRequestFilter == typeof(null))) { 11866 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11867 return true; // we handled it by rejecting it 11868 } 11869 11870 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11871 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11872 } else { 11873 auto newPresenter = new NewPresenter(); 11874 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11875 } 11876 } 11877 11878 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11879 } 11880 11881 /++ 11882 A URL dispatcher. 11883 11884 --- 11885 if(cgi.dispatcher!( 11886 "/api/".serveApi!MyApiClass, 11887 "/objects/lol".serveRestObject!MyRestObject, 11888 "/file.js".serveStaticFile, 11889 "/admin/".dispatchTo!adminHandler 11890 )) return; 11891 --- 11892 11893 11894 You define a series of url prefixes followed by handlers. 11895 11896 You may want to do different pre- and post- processing there, for example, 11897 an authorization check and different page layout. You can use different 11898 presenters and different function chains. See [dispatchSubsection] for details. 11899 11900 [dispatchTo] will send the request to another function for handling. 11901 +/ 11902 template dispatcher(definitions...) { 11903 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11904 static if(is(Presenter == typeof(null))) { 11905 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11906 auto presenter = new GenericWebPresenter(); 11907 } else 11908 alias presenter = presenterArg; 11909 11910 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11911 } 11912 11913 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11914 // I can prolly make this more efficient later but meh. 11915 foreach(definition; definitions) { 11916 if(definition.rejectFurther) { 11917 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11918 auto ret = definition.handler( 11919 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11920 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11921 if(ret) 11922 return true; 11923 } 11924 } else if( 11925 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11926 // cgi.d dispatcher urls must be complete or have a /; 11927 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11928 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11929 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11930 ) { 11931 auto ret = definition.handler( 11932 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11933 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11934 if(ret) 11935 return true; 11936 } 11937 } 11938 return false; 11939 } 11940 } 11941 11942 }); 11943 11944 private struct StackBuffer { 11945 char[1024] initial = void; 11946 char[] buffer; 11947 size_t position; 11948 11949 this(int a) { 11950 buffer = initial[]; 11951 position = 0; 11952 } 11953 11954 void add(in char[] what) { 11955 if(position + what.length > buffer.length) 11956 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 11957 buffer[position .. position + what.length] = what[]; 11958 position += what.length; 11959 } 11960 11961 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 11962 add(w1); 11963 add(w2); 11964 add(w3); 11965 } 11966 11967 void add(long v) { 11968 char[16] buffer = void; 11969 auto pos = buffer.length; 11970 bool negative; 11971 if(v < 0) { 11972 negative = true; 11973 v = -v; 11974 } 11975 do { 11976 buffer[--pos] = cast(char) (v % 10 + '0'); 11977 v /= 10; 11978 } while(v); 11979 11980 if(negative) 11981 buffer[--pos] = '-'; 11982 11983 auto res = buffer[pos .. $]; 11984 11985 add(res[]); 11986 } 11987 11988 char[] get() @nogc { 11989 return buffer[0 .. position]; 11990 } 11991 } 11992 11993 // duplicated in http2.d 11994 private static string getHttpCodeText(int code) pure nothrow @nogc { 11995 switch(code) { 11996 case 200: return "200 OK"; 11997 case 201: return "201 Created"; 11998 case 202: return "202 Accepted"; 11999 case 203: return "203 Non-Authoritative Information"; 12000 case 204: return "204 No Content"; 12001 case 205: return "205 Reset Content"; 12002 case 206: return "206 Partial Content"; 12003 // 12004 case 300: return "300 Multiple Choices"; 12005 case 301: return "301 Moved Permanently"; 12006 case 302: return "302 Found"; 12007 case 303: return "303 See Other"; 12008 case 304: return "304 Not Modified"; 12009 case 305: return "305 Use Proxy"; 12010 case 307: return "307 Temporary Redirect"; 12011 case 308: return "308 Permanent Redirect"; 12012 12013 // 12014 case 400: return "400 Bad Request"; 12015 case 401: return "401 Unauthorized"; 12016 case 402: return "402 Payment Required"; 12017 case 403: return "403 Forbidden"; 12018 case 404: return "404 Not Found"; 12019 case 405: return "405 Method Not Allowed"; 12020 case 406: return "406 Not Acceptable"; 12021 case 407: return "407 Proxy Authentication Required"; 12022 case 408: return "408 Request Timeout"; 12023 case 409: return "409 Conflict"; 12024 case 410: return "410 Gone"; 12025 case 411: return "411 Length Required"; 12026 case 412: return "412 Precondition Failed"; 12027 case 413: return "413 Payload Too Large"; 12028 case 414: return "414 URI Too Long"; 12029 case 415: return "415 Unsupported Media Type"; 12030 case 416: return "416 Range Not Satisfiable"; 12031 case 417: return "417 Expectation Failed"; 12032 case 418: return "418 I'm a teapot"; 12033 case 421: return "421 Misdirected Request"; 12034 case 422: return "422 Unprocessable Entity (WebDAV)"; 12035 case 423: return "423 Locked (WebDAV)"; 12036 case 424: return "424 Failed Dependency (WebDAV)"; 12037 case 425: return "425 Too Early"; 12038 case 426: return "426 Upgrade Required"; 12039 case 428: return "428 Precondition Required"; 12040 case 431: return "431 Request Header Fields Too Large"; 12041 case 451: return "451 Unavailable For Legal Reasons"; 12042 12043 case 500: return "500 Internal Server Error"; 12044 case 501: return "501 Not Implemented"; 12045 case 502: return "502 Bad Gateway"; 12046 case 503: return "503 Service Unavailable"; 12047 case 504: return "504 Gateway Timeout"; 12048 case 505: return "505 HTTP Version Not Supported"; 12049 case 506: return "506 Variant Also Negotiates"; 12050 case 507: return "507 Insufficient Storage (WebDAV)"; 12051 case 508: return "508 Loop Detected (WebDAV)"; 12052 case 510: return "510 Not Extended"; 12053 case 511: return "511 Network Authentication Required"; 12054 // 12055 default: assert(0, "Unsupported http code"); 12056 } 12057 } 12058 12059 12060 /+ 12061 /++ 12062 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12063 12064 It relies on jsvar.d and dom.d. 12065 12066 12067 You can get javascript out of it to call. The generated functions need to look 12068 like 12069 12070 function name(a,b,c,d,e) { 12071 return _call("name", {"realName":a,"sds":b}); 12072 } 12073 12074 And _call returns an object you can call or set up or whatever. 12075 +/ 12076 bool apiDispatcher()(Cgi cgi) { 12077 import arsd.jsvar; 12078 import arsd.dom; 12079 } 12080 +/ 12081 version(linux) 12082 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12083 /* 12084 Copyright: Adam D. Ruppe, 2008 - 2023 12085 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12086 Authors: Adam D. Ruppe 12087 12088 Copyright Adam D. Ruppe 2008 - 2023. 12089 Distributed under the Boost Software License, Version 1.0. 12090 (See accompanying file LICENSE_1_0.txt or copy at 12091 http://www.boost.org/LICENSE_1_0.txt) 12092 */