1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. Note: prior to version 11, this would be embedded_httpd_processes on Linux and embedded_httpd_threads everywhere else. It now means embedded_httpd_hybrid everywhere supported and embedded_httpd_threads everywhere else. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 490 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 491 492 /++ 493 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 494 +/ 495 version(Demo) 496 unittest { 497 import arsd.cgi; 498 499 mixin DispatcherMain!( 500 "/".serveStaticFileDirectory(null, true) 501 ); 502 } 503 504 /++ 505 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 506 +/ 507 version(Demo) 508 unittest { 509 import arsd.cgi; 510 511 void requestHandler(Cgi cgi) { 512 cgi.dispatcher!( 513 "/".serveStaticFileDirectory(null, true) 514 ); 515 } 516 517 // mixin GenericMain!requestHandler would add this function: 518 void main(string[] args) { 519 // this is all the content of [cgiMainImpl] which you can also call 520 521 // cgi.d embeds a few add on functions like real time event forwarders 522 // and session servers it can run in other processes. this spawns them, if needed. 523 if(tryAddonServers(args)) 524 return; 525 526 // cgi.d allows you to easily simulate http requests from the command line, 527 // without actually starting a server. this function will do that. 528 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 529 return; 530 531 RequestServer server; 532 // you can change the default port here if you like 533 // server.listeningPort = 9000; 534 535 // then call this to let the command line args override your default 536 server.configureFromCommandLine(args); 537 538 // here is where you could print out the listeningPort to the user if you wanted 539 540 // and serve the request(s) according to the compile configuration 541 server.serve!(requestHandler)(); 542 543 // or you could explicitly choose a serve mode like this: 544 // server.serveEmbeddedHttp!requestHandler(); 545 } 546 } 547 548 /++ 549 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 550 otherwise run through the rest of the internal mechanisms to call your functions without actually 551 spinning up a server. 552 +/ 553 version(Demo) 554 unittest { 555 import arsd.cgi; 556 557 void requestHandler(Cgi cgi) { 558 559 } 560 561 // D doesn't let me embed a unittest inside an example unittest 562 // so this is a function, but you can do it however in your real program 563 /* unittest */ void runTests() { 564 auto tester = new CgiTester(&requestHandler); 565 566 auto response = tester.GET("/"); 567 assert(response.code == 200); 568 } 569 } 570 571 static import std.file; 572 573 static import arsd.core; 574 version(Posix) 575 import arsd.core : makeNonBlocking; 576 577 578 // for a single thread, linear request thing, use: 579 // -version=embedded_httpd_threads -version=cgi_no_threads 580 581 version(Posix) { 582 version(CRuntime_Musl) { 583 584 } else version(minimal) { 585 586 } else { 587 version(FreeBSD) { 588 // I never implemented the fancy stuff there either 589 } else { 590 version=with_breaking_cgi_features; 591 version=with_sendfd; 592 version=with_addon_servers; 593 } 594 } 595 } 596 597 version(Windows) { 598 version(minimal) { 599 600 } else { 601 // not too concerned about gdc here since the mingw version is fairly new as well 602 version=with_breaking_cgi_features; 603 } 604 } 605 606 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 607 void cloexec(int fd) { 608 version(Posix) { 609 import core.sys.posix.fcntl; 610 fcntl(fd, F_SETFD, FD_CLOEXEC); 611 } 612 } 613 614 void cloexec(Socket s) { 615 version(Posix) { 616 import core.sys.posix.fcntl; 617 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 618 } 619 } 620 621 // the servers must know about the connections to talk to them; the interfaces are vital 622 version(with_addon_servers) 623 version=with_addon_servers_connections; 624 625 version(embedded_httpd) { 626 version=embedded_httpd_hybrid; 627 /* 628 version(with_openssl) { 629 pragma(lib, "crypto"); 630 pragma(lib, "ssl"); 631 } 632 */ 633 } 634 635 version(embedded_httpd_hybrid) { 636 version=embedded_httpd_threads; 637 version(cgi_no_fork) {} else version(Posix) 638 version=cgi_use_fork; 639 version=cgi_use_fiber; 640 } 641 642 version(cgi_use_fork) 643 enum cgi_use_fork_default = true; 644 else 645 enum cgi_use_fork_default = false; 646 647 version(embedded_httpd_processes) 648 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 649 650 version(embedded_httpd_threads) { 651 // unless the user overrides the default.. 652 version(cgi_session_server_process) 653 {} 654 else 655 version=cgi_embedded_sessions; 656 } 657 version(scgi) { 658 // unless the user overrides the default.. 659 version(cgi_session_server_process) 660 {} 661 else 662 version=cgi_embedded_sessions; 663 } 664 665 // fall back if the other is not defined so we can cleanly version it below 666 version(cgi_embedded_sessions) {} 667 else version=cgi_session_server_process; 668 669 670 version=cgi_with_websocket; 671 672 enum long defaultMaxContentLength = 5_000_000; 673 674 /* 675 676 To do a file download offer in the browser: 677 678 cgi.setResponseContentType("text/csv"); 679 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 680 */ 681 682 // FIXME: the location header is supposed to be an absolute url I guess. 683 684 // FIXME: would be cool to flush part of a dom document before complete 685 // somehow in here and dom.d. 686 687 688 // these are public so you can mixin GenericMain. 689 // FIXME: use a function level import instead! 690 public import std.string; 691 public import std.stdio; 692 public import std.conv; 693 import std.uri; 694 import std.uni; 695 import std.algorithm.comparison; 696 import std.algorithm.searching; 697 import std.exception; 698 import std.base64; 699 static import std.algorithm; 700 import std.datetime; 701 import std.range; 702 703 import std.process; 704 705 import std.zlib; 706 707 708 T[] consume(T)(T[] range, int count) { 709 if(count > range.length) 710 count = range.length; 711 return range[count..$]; 712 } 713 714 int locationOf(T)(T[] data, string item) { 715 const(ubyte[]) d = cast(const(ubyte[])) data; 716 const(ubyte[]) i = cast(const(ubyte[])) item; 717 718 // this is a vague sanity check to ensure we aren't getting insanely 719 // sized input that will infinite loop below. it should never happen; 720 // even huge file uploads ought to come in smaller individual pieces. 721 if(d.length > (int.max/2)) 722 throw new Exception("excessive block of input"); 723 724 for(int a = 0; a < d.length; a++) { 725 if(a + i.length > d.length) 726 return -1; 727 if(d[a..a+i.length] == i) 728 return a; 729 } 730 731 return -1; 732 } 733 734 /// If you are doing a custom cgi class, mixing this in can take care of 735 /// the required constructors for you 736 mixin template ForwardCgiConstructors() { 737 this(long maxContentLength = defaultMaxContentLength, 738 string[string] env = null, 739 const(ubyte)[] delegate() readdata = null, 740 void delegate(const(ubyte)[]) _rawDataOutput = null, 741 void delegate() _flush = null 742 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 743 744 this(string[] args) { super(args); } 745 746 this( 747 BufferedInputRange inputData, 748 string address, ushort _port, 749 int pathInfoStarts = 0, 750 bool _https = false, 751 void delegate(const(ubyte)[]) _rawDataOutput = null, 752 void delegate() _flush = null, 753 // this pointer tells if the connection is supposed to be closed after we handle this 754 bool* closeConnection = null) 755 { 756 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 757 } 758 759 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 760 } 761 762 /// thrown when a connection is closed remotely while we waiting on data from it 763 class ConnectionClosedException : Exception { 764 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 765 super(message, file, line, next); 766 } 767 } 768 769 770 version(Windows) { 771 // FIXME: ugly hack to solve stdin exception problems on Windows: 772 // reading stdin results in StdioException (Bad file descriptor) 773 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 774 private struct stdin { 775 struct ByChunk { // Replicates std.stdio.ByChunk 776 private: 777 ubyte[] chunk_; 778 public: 779 this(size_t size) 780 in { 781 assert(size, "size must be larger than 0"); 782 } 783 do { 784 chunk_ = new ubyte[](size); 785 popFront(); 786 } 787 788 @property bool empty() const { 789 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 790 } 791 @property nothrow ubyte[] front() { return chunk_; } 792 void popFront() { 793 enforce(!empty, "Cannot call popFront on empty range"); 794 chunk_ = stdin.rawRead(chunk_); 795 } 796 } 797 798 import core.sys.windows.windows; 799 static: 800 801 T[] rawRead(T)(T[] buf) { 802 uint bytesRead; 803 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 804 805 if (!result) { 806 auto err = GetLastError(); 807 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 808 return buf[0..0]; 809 // Some other error, throw it 810 811 char* buffer; 812 scope(exit) LocalFree(buffer); 813 814 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 815 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 816 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 817 throw new Exception(to!string(buffer)); 818 } 819 enforce(!(bytesRead % T.sizeof), "I/O error"); 820 return buf[0..bytesRead / T.sizeof]; 821 } 822 823 auto byChunk(size_t sz) { return ByChunk(sz); } 824 825 void close() { 826 std.stdio.stdin.close; 827 } 828 } 829 } 830 831 /// The main interface with the web request 832 class Cgi { 833 public: 834 /// the methods a request can be 835 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 836 // these are defined in the standard, but idk if they are useful for anything 837 OPTIONS, TRACE, CONNECT, 838 // These seem new, I have only recently seen them 839 PATCH, MERGE, 840 // this is an extension for when the method is not specified and you want to assume 841 CommandLine } 842 843 844 /+ 845 846 ubyte[] perRequestMemoryPool; 847 void[] perRequestMemoryPoolWithPointers; 848 // might want to just slice the buffer itself too when we happened to have gotten a full request inside it and don't need to decode 849 // then the buffer also can be recycled if it is set. 850 851 // we might also be able to set memory recyclable true by default, but then the property getters set it to false. but not all the things are property getters. but realistically anything except benchmarks are gonna get something lol so meh. 852 853 /+ 854 struct VariableCollection { 855 string[] opIndex(string name) { 856 857 } 858 } 859 860 /++ 861 Call this to indicate that you've not retained any reference to the request-local memory (including all strings returned from the Cgi object) outside the request (you can .idup anything you need to store) and it is thus free to be freed or reused by another request. 862 863 Most handlers should be able to call this; retaining memory is the exception in any cgi program, but since I can't prove it from inside the library, it plays it safe and lets the GC manage it unless you opt into this behavior. All cgi.d functions will duplicate strings if needed (e.g. session ids from cookies) so unless you're doing something yourself, this should be ok. 864 865 History: 866 Added 867 +/ 868 public void recycleMemory() { 869 870 } 871 +/ 872 873 874 /++ 875 Cgi provides a per-request memory pool 876 877 +/ 878 void[] allocateMemory(size_t nBytes) { 879 880 } 881 882 /// ditto 883 void[] reallocateMemory(void[] old, size_t nBytes) { 884 885 } 886 887 /// ditto 888 void freeMemory(void[] memory) { 889 890 } 891 +/ 892 893 894 /* 895 import core.runtime; 896 auto args = Runtime.args(); 897 898 we can call the app a few ways: 899 900 1) set up the environment variables and call the app (manually simulating CGI) 901 2) simulate a call automatically: 902 ./app method 'uri' 903 904 for example: 905 ./app get /path?arg arg2=something 906 907 Anything on the uri is treated as query string etc 908 909 on get method, further args are appended to the query string (encoded automatically) 910 on post method, further args are done as post 911 912 913 @name means import from file "name". if name == -, it uses stdin 914 (so info=@- means set info to the value of stdin) 915 916 917 Other arguments include: 918 --cookie name=value (these are all concated together) 919 --header 'X-Something: cool' 920 --referrer 'something' 921 --port 80 922 --remote-address some.ip.address.here 923 --https yes 924 --user-agent 'something' 925 --userpass 'user:pass' 926 --authorization 'Basic base64encoded_user:pass' 927 --accept 'content' // FIXME: better example 928 --last-event-id 'something' 929 --host 'something.com' 930 931 Non-simulation arguments: 932 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 933 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 934 935 */ 936 937 /** Initializes it with command line arguments (for easy testing) */ 938 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 939 rawDataOutput = _rawDataOutput; 940 // these are all set locally so the loop works 941 // without triggering errors in dmd 2.064 942 // we go ahead and set them at the end of it to the this version 943 int port; 944 string referrer; 945 string remoteAddress; 946 string userAgent; 947 string authorization; 948 string origin; 949 string accept; 950 string lastEventId; 951 bool https; 952 string host; 953 RequestMethod requestMethod; 954 string requestUri; 955 string pathInfo; 956 string queryString; 957 958 bool lookingForMethod; 959 bool lookingForUri; 960 string nextArgIs; 961 962 string _cookie; 963 string _queryString; 964 string[][string] _post; 965 string[string] _headers; 966 967 string[] breakUp(string s) { 968 string k, v; 969 auto idx = s.indexOf("="); 970 if(idx == -1) { 971 k = s; 972 } else { 973 k = s[0 .. idx]; 974 v = s[idx + 1 .. $]; 975 } 976 977 return [k, v]; 978 } 979 980 lookingForMethod = true; 981 982 scriptName = args[0]; 983 scriptFileName = args[0]; 984 985 environmentVariables = cast(const) environment.toAA; 986 987 foreach(arg; args[1 .. $]) { 988 if(arg.startsWith("--")) { 989 nextArgIs = arg[2 .. $]; 990 } else if(nextArgIs.length) { 991 if (nextArgIs == "cookie") { 992 auto info = breakUp(arg); 993 if(_cookie.length) 994 _cookie ~= "; "; 995 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 996 } 997 else if (nextArgIs == "port") { 998 port = to!int(arg); 999 } 1000 else if (nextArgIs == "referrer") { 1001 referrer = arg; 1002 } 1003 else if (nextArgIs == "remote-address") { 1004 remoteAddress = arg; 1005 } 1006 else if (nextArgIs == "user-agent") { 1007 userAgent = arg; 1008 } 1009 else if (nextArgIs == "authorization") { 1010 authorization = arg; 1011 } 1012 else if (nextArgIs == "userpass") { 1013 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 1014 } 1015 else if (nextArgIs == "origin") { 1016 origin = arg; 1017 } 1018 else if (nextArgIs == "accept") { 1019 accept = arg; 1020 } 1021 else if (nextArgIs == "last-event-id") { 1022 lastEventId = arg; 1023 } 1024 else if (nextArgIs == "https") { 1025 if(arg == "yes") 1026 https = true; 1027 } 1028 else if (nextArgIs == "header") { 1029 string thing, other; 1030 auto idx = arg.indexOf(":"); 1031 if(idx == -1) 1032 throw new Exception("need a colon in a http header"); 1033 thing = arg[0 .. idx]; 1034 other = arg[idx + 1.. $]; 1035 _headers[thing.strip.toLower()] = other.strip; 1036 } 1037 else if (nextArgIs == "host") { 1038 host = arg; 1039 } 1040 // else 1041 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1042 1043 nextArgIs = null; 1044 } else if(lookingForMethod) { 1045 lookingForMethod = false; 1046 lookingForUri = true; 1047 1048 if(arg.asLowerCase().equal("commandline")) 1049 requestMethod = RequestMethod.CommandLine; 1050 else 1051 requestMethod = to!RequestMethod(arg.toUpper()); 1052 } else if(lookingForUri) { 1053 lookingForUri = false; 1054 1055 requestUri = arg; 1056 1057 auto idx = arg.indexOf("?"); 1058 if(idx == -1) 1059 pathInfo = arg; 1060 else { 1061 pathInfo = arg[0 .. idx]; 1062 _queryString = arg[idx + 1 .. $]; 1063 } 1064 } else { 1065 // it is an argument of some sort 1066 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1067 auto parts = breakUp(arg); 1068 _post[parts[0]] ~= parts[1]; 1069 allPostNamesInOrder ~= parts[0]; 1070 allPostValuesInOrder ~= parts[1]; 1071 } else { 1072 if(_queryString.length) 1073 _queryString ~= "&"; 1074 auto parts = breakUp(arg); 1075 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1076 } 1077 } 1078 } 1079 1080 acceptsGzip = false; 1081 keepAliveRequested = false; 1082 requestHeaders = cast(immutable) _headers; 1083 1084 cookie = _cookie; 1085 cookiesArray = getCookieArray(); 1086 cookies = keepLastOf(cookiesArray); 1087 1088 queryString = _queryString; 1089 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1090 get = keepLastOf(getArray); 1091 1092 postArray = cast(immutable) _post; 1093 post = keepLastOf(_post); 1094 1095 // FIXME 1096 filesArray = null; 1097 files = null; 1098 1099 isCalledWithCommandLineArguments = true; 1100 1101 this.port = port; 1102 this.referrer = referrer; 1103 this.remoteAddress = remoteAddress; 1104 this.userAgent = userAgent; 1105 this.authorization = authorization; 1106 this.origin = origin; 1107 this.accept = accept; 1108 this.lastEventId = lastEventId; 1109 this.https = https; 1110 this.host = host; 1111 this.requestMethod = requestMethod; 1112 this.requestUri = requestUri; 1113 this.pathInfo = pathInfo; 1114 this.queryString = queryString; 1115 this.postBody = null; 1116 } 1117 1118 private { 1119 string[] allPostNamesInOrder; 1120 string[] allPostValuesInOrder; 1121 string[] allGetNamesInOrder; 1122 string[] allGetValuesInOrder; 1123 } 1124 1125 CgiConnectionHandle getOutputFileHandle() { 1126 return _outputFileHandle; 1127 } 1128 1129 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1130 1131 /** Initializes it using a CGI or CGI-like interface */ 1132 this(long maxContentLength = defaultMaxContentLength, 1133 // use this to override the environment variable listing 1134 in string[string] env = null, 1135 // and this should return a chunk of data. return empty when done 1136 const(ubyte)[] delegate() readdata = null, 1137 // finally, use this to do custom output if needed 1138 void delegate(const(ubyte)[]) _rawDataOutput = null, 1139 // to flush teh custom output 1140 void delegate() _flush = null 1141 ) 1142 { 1143 1144 // these are all set locally so the loop works 1145 // without triggering errors in dmd 2.064 1146 // we go ahead and set them at the end of it to the this version 1147 int port; 1148 string referrer; 1149 string remoteAddress; 1150 string userAgent; 1151 string authorization; 1152 string origin; 1153 string accept; 1154 string lastEventId; 1155 bool https; 1156 string host; 1157 RequestMethod requestMethod; 1158 string requestUri; 1159 string pathInfo; 1160 string queryString; 1161 1162 1163 1164 isCalledWithCommandLineArguments = false; 1165 rawDataOutput = _rawDataOutput; 1166 flushDelegate = _flush; 1167 auto getenv = delegate string(string var) { 1168 if(env is null) 1169 return std.process.environment.get(var); 1170 auto e = var in env; 1171 if(e is null) 1172 return null; 1173 return *e; 1174 }; 1175 1176 environmentVariables = env is null ? 1177 cast(const) environment.toAA : 1178 env; 1179 1180 // fetching all the request headers 1181 string[string] requestHeadersHere; 1182 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1183 if(k.startsWith("HTTP_")) { 1184 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1185 } 1186 } 1187 1188 this.requestHeaders = assumeUnique(requestHeadersHere); 1189 1190 requestUri = getenv("REQUEST_URI"); 1191 1192 cookie = getenv("HTTP_COOKIE"); 1193 cookiesArray = getCookieArray(); 1194 cookies = keepLastOf(cookiesArray); 1195 1196 referrer = getenv("HTTP_REFERER"); 1197 userAgent = getenv("HTTP_USER_AGENT"); 1198 remoteAddress = getenv("REMOTE_ADDR"); 1199 host = getenv("HTTP_HOST"); 1200 pathInfo = getenv("PATH_INFO"); 1201 1202 queryString = getenv("QUERY_STRING"); 1203 scriptName = getenv("SCRIPT_NAME"); 1204 { 1205 import core.runtime; 1206 auto sfn = getenv("SCRIPT_FILENAME"); 1207 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1208 } 1209 1210 bool iis = false; 1211 1212 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1213 if(requestUri.length == 0) { 1214 // IIS sometimes includes the script name as part of the path info - we don't want that 1215 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1216 pathInfo = pathInfo[scriptName.length .. $]; 1217 1218 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1219 1220 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1221 1222 // FIXME: this works for apache and iis... but what about others? 1223 } 1224 1225 1226 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1227 getArray = assumeUnique(ugh); 1228 get = keepLastOf(getArray); 1229 1230 1231 // NOTE: on shitpache, you need to specifically forward this 1232 authorization = getenv("HTTP_AUTHORIZATION"); 1233 // this is a hack because Apache is a shitload of fuck and 1234 // refuses to send the real header to us. Compatible 1235 // programs should send both the standard and X- versions 1236 1237 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1238 // unnecessary with mod_rewrite, so it is commented 1239 1240 //if(authorization.length == 0) // if the std is there, use it 1241 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1242 1243 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1244 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1245 port = to!int(getenv("SERVER_PORT")); 1246 else 1247 port = 0; // this was probably called from the command line 1248 1249 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1250 if(ae.length && ae.indexOf("gzip") != -1) 1251 acceptsGzip = true; 1252 1253 accept = getenv("HTTP_ACCEPT"); 1254 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1255 1256 auto ka = getenv("HTTP_CONNECTION"); 1257 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1258 keepAliveRequested = true; 1259 1260 auto or = getenv("HTTP_ORIGIN"); 1261 origin = or; 1262 1263 auto rm = getenv("REQUEST_METHOD"); 1264 if(rm.length) 1265 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1266 else 1267 requestMethod = RequestMethod.CommandLine; 1268 1269 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1270 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1271 1272 // FIXME: DOCUMENT_ROOT? 1273 1274 // FIXME: what about PUT? 1275 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1276 version(preserveData) // a hack to make forwarding simpler 1277 immutable(ubyte)[] data; 1278 size_t amountReceived = 0; 1279 auto contentType = getenv("CONTENT_TYPE"); 1280 1281 // FIXME: is this ever not going to be set? I guess it depends 1282 // on if the server de-chunks and buffers... seems like it has potential 1283 // to be slow if they did that. The spec says it is always there though. 1284 // And it has worked reliably for me all year in the live environment, 1285 // but some servers might be different. 1286 auto cls = getenv("CONTENT_LENGTH"); 1287 auto contentLength = to!size_t(cls.length ? cls : "0"); 1288 1289 immutable originalContentLength = contentLength; 1290 if(contentLength) { 1291 if(maxContentLength > 0 && contentLength > maxContentLength) { 1292 setResponseStatus("413 Request entity too large"); 1293 write("You tried to upload a file that is too large."); 1294 close(); 1295 throw new Exception("POST too large"); 1296 } 1297 prepareForIncomingDataChunks(contentType, contentLength); 1298 1299 1300 int processChunk(in ubyte[] chunk) { 1301 if(chunk.length > contentLength) { 1302 handleIncomingDataChunk(chunk[0..contentLength]); 1303 amountReceived += contentLength; 1304 contentLength = 0; 1305 return 1; 1306 } else { 1307 handleIncomingDataChunk(chunk); 1308 contentLength -= chunk.length; 1309 amountReceived += chunk.length; 1310 } 1311 if(contentLength == 0) 1312 return 1; 1313 1314 onRequestBodyDataReceived(amountReceived, originalContentLength); 1315 return 0; 1316 } 1317 1318 1319 if(readdata is null) { 1320 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1321 if(processChunk(chunk)) 1322 break; 1323 } else { 1324 // we have a custom data source.. 1325 auto chunk = readdata(); 1326 while(chunk.length) { 1327 if(processChunk(chunk)) 1328 break; 1329 chunk = readdata(); 1330 } 1331 } 1332 1333 onRequestBodyDataReceived(amountReceived, originalContentLength); 1334 postArray = assumeUnique(pps._post); 1335 filesArray = assumeUnique(pps._files); 1336 files = keepLastOf(filesArray); 1337 post = keepLastOf(postArray); 1338 this.postBody = pps.postBody; 1339 cleanUpPostDataState(); 1340 } 1341 1342 version(preserveData) 1343 originalPostData = data; 1344 } 1345 // fixme: remote_user script name 1346 1347 1348 this.port = port; 1349 this.referrer = referrer; 1350 this.remoteAddress = remoteAddress; 1351 this.userAgent = userAgent; 1352 this.authorization = authorization; 1353 this.origin = origin; 1354 this.accept = accept; 1355 this.lastEventId = lastEventId; 1356 this.https = https; 1357 this.host = host; 1358 this.requestMethod = requestMethod; 1359 this.requestUri = requestUri; 1360 this.pathInfo = pathInfo; 1361 this.queryString = queryString; 1362 } 1363 1364 /// Cleans up any temporary files. Do not use the object 1365 /// after calling this. 1366 /// 1367 /// NOTE: it is called automatically by GenericMain 1368 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1369 void dispose() { 1370 foreach(file; files) { 1371 if(!file.contentInMemory) 1372 if(std.file.exists(file.contentFilename)) 1373 std.file.remove(file.contentFilename); 1374 } 1375 } 1376 1377 private { 1378 struct PostParserState { 1379 string contentType; 1380 string boundary; 1381 string localBoundary; // the ones used at the end or something lol 1382 bool isMultipart; 1383 bool needsSavedBody; 1384 1385 ulong expectedLength; 1386 ulong contentConsumed; 1387 immutable(ubyte)[] buffer; 1388 1389 // multipart parsing state 1390 int whatDoWeWant; 1391 bool weHaveAPart; 1392 string[] thisOnesHeaders; 1393 immutable(ubyte)[] thisOnesData; 1394 1395 string postBody; 1396 1397 UploadedFile piece; 1398 bool isFile = false; 1399 1400 size_t memoryCommitted; 1401 1402 // do NOT keep mutable references to these anywhere! 1403 // I assume they are unique in the constructor once we're all done getting data. 1404 string[][string] _post; 1405 UploadedFile[][string] _files; 1406 } 1407 1408 PostParserState pps; 1409 } 1410 1411 /// This represents a file the user uploaded via a POST request. 1412 static struct UploadedFile { 1413 /// If you want to create one of these structs for yourself from some data, 1414 /// use this function. 1415 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1416 Cgi.UploadedFile f; 1417 f.filename = name; 1418 f.content = cast(immutable(ubyte)[]) data; 1419 f.contentInMemory = true; 1420 return f; 1421 } 1422 1423 string name; /// The name of the form element. 1424 string filename; /// The filename the user set. 1425 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1426 1427 /** 1428 For small files, cgi.d will buffer the uploaded file in memory, and make it 1429 directly accessible to you through the content member. I find this very convenient 1430 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1431 often want to inspect and modify the file anyway!) 1432 1433 I find the file is very large, it is undesirable to eat that much memory just 1434 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1435 to the constructor so they are accepted, cgi.d will write the content to a temporary 1436 file that you can re-read later. 1437 1438 You can override this behavior by subclassing Cgi and overriding the protected 1439 handlePostChunk method. Note that the object is not initialized when you 1440 write that method - the http headers are available, but the cgi.post method 1441 is not. You may parse the file as it streams in using this method. 1442 1443 1444 Anyway, if the file is small enough to be in memory, contentInMemory will be 1445 set to true, and the content is available in the content member. 1446 1447 If not, contentInMemory will be set to false, and the content saved in a file, 1448 whose name will be available in the contentFilename member. 1449 1450 1451 Tip: if you know you are always dealing with small files, and want the convenience 1452 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1453 a large file comes in, it simply throws an exception (and HTTP error response) 1454 instead of trying to handle it. 1455 1456 The default value of maxContentLength in the constructor is for small files. 1457 */ 1458 bool contentInMemory = true; // the default ought to always be true 1459 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1460 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1461 1462 /// 1463 ulong fileSize() const { 1464 if(contentInMemory) 1465 return content.length; 1466 import std.file; 1467 return std.file.getSize(contentFilename); 1468 1469 } 1470 1471 /// 1472 void writeToFile(string filenameToSaveTo) const { 1473 import std.file; 1474 if(contentInMemory) 1475 std.file.write(filenameToSaveTo, content); 1476 else 1477 std.file.rename(contentFilename, filenameToSaveTo); 1478 } 1479 } 1480 1481 // given a content type and length, decide what we're going to do with the data.. 1482 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1483 pps.expectedLength = contentLength; 1484 1485 auto terminator = contentType.indexOf(";"); 1486 if(terminator == -1) 1487 terminator = contentType.length; 1488 1489 pps.contentType = contentType[0 .. terminator]; 1490 auto b = contentType[terminator .. $]; 1491 if(b.length) { 1492 auto idx = b.indexOf("boundary="); 1493 if(idx != -1) { 1494 pps.boundary = b[idx + "boundary=".length .. $]; 1495 pps.localBoundary = "\r\n--" ~ pps.boundary; 1496 } 1497 } 1498 1499 // while a content type SHOULD be sent according to the RFC, it is 1500 // not required. We're told we SHOULD guess by looking at the content 1501 // but it seems to me that this only happens when it is urlencoded. 1502 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1503 pps.isMultipart = false; 1504 pps.needsSavedBody = false; 1505 } else if(pps.contentType == "multipart/form-data") { 1506 pps.isMultipart = true; 1507 enforce(pps.boundary.length, "no boundary"); 1508 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1509 // save the body so the application can handle it 1510 pps.isMultipart = false; 1511 pps.needsSavedBody = true; 1512 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1513 // save the body so the application can handle it 1514 pps.needsSavedBody = true; 1515 pps.isMultipart = false; 1516 } else { 1517 // the rest is 100% handled by the application. just save the body and send it to them 1518 pps.needsSavedBody = true; 1519 pps.isMultipart = false; 1520 } 1521 } 1522 1523 // handles streaming POST data. If you handle some other content type, you should 1524 // override this. If the data isn't the content type you want, you ought to call 1525 // super.handleIncomingDataChunk so regular forms and files still work. 1526 1527 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1528 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1529 // input anyway, so I'm not going to get too worked up about it right now. 1530 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1531 if(chunk.length == 0) 1532 return; 1533 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1534 // if we're passed big chunks, it might throw unnecessarily. 1535 // just pass it smaller chunks at a time. 1536 if(pps.isMultipart) { 1537 // multipart/form-data 1538 1539 1540 // FIXME: this might want to be factored out and factorized 1541 // need to make sure the stream hooks actually work. 1542 void pieceHasNewContent() { 1543 // we just grew the piece's buffer. Do we have to switch to file backing? 1544 if(pps.piece.contentInMemory) { 1545 if(pps.piece.content.length <= 10 * 1024 * 1024) 1546 // meh, I'm ok with it. 1547 return; 1548 else { 1549 // this is too big. 1550 if(!pps.isFile) 1551 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1552 else { 1553 // a file this large is probably acceptable though... let's use a backing file. 1554 pps.piece.contentInMemory = false; 1555 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1556 1557 int count = 0; 1558 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1559 // odds are this loop will never be entered, but we want it just in case. 1560 while(std.file.exists(pps.piece.contentFilename)) { 1561 count++; 1562 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1563 } 1564 // I hope this creates the file pretty quickly, or the loop might be useless... 1565 // FIXME: maybe I should write some kind of custom transaction here. 1566 std.file.write(pps.piece.contentFilename, pps.piece.content); 1567 1568 pps.piece.content = null; 1569 } 1570 } 1571 } else { 1572 // it's already in a file, so just append it to what we have 1573 if(pps.piece.content.length) { 1574 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1575 std.file.append(pps.piece.contentFilename, pps.piece.content); 1576 pps.piece.content = null; 1577 } 1578 } 1579 } 1580 1581 1582 void commitPart() { 1583 if(!pps.weHaveAPart) 1584 return; 1585 1586 pieceHasNewContent(); // be sure the new content is handled every time 1587 1588 if(pps.isFile) { 1589 // I'm not sure if other environments put files in post or not... 1590 // I used to not do it, but I think I should, since it is there... 1591 pps._post[pps.piece.name] ~= pps.piece.filename; 1592 pps._files[pps.piece.name] ~= pps.piece; 1593 1594 allPostNamesInOrder ~= pps.piece.name; 1595 allPostValuesInOrder ~= pps.piece.filename; 1596 } else { 1597 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1598 1599 allPostNamesInOrder ~= pps.piece.name; 1600 allPostValuesInOrder ~= cast(string) pps.piece.content; 1601 } 1602 1603 /* 1604 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1605 pps.piece.content.length < 1000 1606 ? 1607 to!string(pps.piece.content) 1608 : 1609 "too long"); 1610 */ 1611 1612 // FIXME: the limit here 1613 pps.memoryCommitted += pps.piece.content.length; 1614 1615 pps.weHaveAPart = false; 1616 pps.whatDoWeWant = 1; 1617 pps.thisOnesHeaders = null; 1618 pps.thisOnesData = null; 1619 1620 pps.piece = UploadedFile.init; 1621 pps.isFile = false; 1622 } 1623 1624 void acceptChunk() { 1625 pps.buffer ~= chunk; 1626 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1627 } 1628 1629 immutable(ubyte)[] consume(size_t howMuch) { 1630 pps.contentConsumed += howMuch; 1631 auto ret = pps.buffer[0 .. howMuch]; 1632 pps.buffer = pps.buffer[howMuch .. $]; 1633 return ret; 1634 } 1635 1636 dataConsumptionLoop: do { 1637 switch(pps.whatDoWeWant) { 1638 default: assert(0); 1639 case 0: 1640 acceptChunk(); 1641 // the format begins with two extra leading dashes, then we should be at the boundary 1642 if(pps.buffer.length < 2) 1643 return; 1644 assert(pps.buffer[0] == '-', "no leading dash"); 1645 consume(1); 1646 assert(pps.buffer[0] == '-', "no second leading dash"); 1647 consume(1); 1648 1649 pps.whatDoWeWant = 1; 1650 goto case 1; 1651 /* fallthrough */ 1652 case 1: // looking for headers 1653 // here, we should be lined up right at the boundary, which is followed by a \r\n 1654 1655 // want to keep the buffer under control in case we're under attack 1656 //stderr.writeln("here once"); 1657 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1658 // throw new Exception("wtf is up with the huge mime part headers"); 1659 1660 acceptChunk(); 1661 1662 if(pps.buffer.length < pps.boundary.length) 1663 return; // not enough data, since there should always be a boundary here at least 1664 1665 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1666 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1667 // we *should* be at the end here! 1668 assert(pps.buffer[0] == '-'); 1669 consume(1); 1670 assert(pps.buffer[0] == '-'); 1671 consume(1); 1672 1673 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1674 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1675 "not lined up on boundary " ~ pps.boundary); 1676 consume(pps.boundary.length); 1677 1678 assert(pps.buffer[0] == '-'); 1679 consume(1); 1680 assert(pps.buffer[0] == '-'); 1681 consume(1); 1682 1683 assert(pps.buffer[0] == '\r'); 1684 consume(1); 1685 assert(pps.buffer[0] == '\n'); 1686 consume(1); 1687 1688 assert(pps.buffer.length == 0); 1689 assert(pps.contentConsumed == pps.expectedLength); 1690 break dataConsumptionLoop; // we're done! 1691 } else { 1692 // we're not done yet. We should be lined up on a boundary. 1693 1694 // But, we want to ensure the headers are here before we consume anything! 1695 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1696 if(headerEndLocation == -1) 1697 return; // they *should* all be here, so we can handle them all at once. 1698 1699 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1700 "not lined up on boundary " ~ pps.boundary); 1701 1702 consume(pps.boundary.length); 1703 // the boundary is always followed by a \r\n 1704 assert(pps.buffer[0] == '\r'); 1705 consume(1); 1706 assert(pps.buffer[0] == '\n'); 1707 consume(1); 1708 } 1709 1710 // re-running since by consuming the boundary, we invalidate the old index. 1711 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1712 assert(headerEndLocation >= 0, "no header"); 1713 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1714 1715 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1716 1717 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1718 1719 // now we'll parse the headers 1720 foreach(h; pps.thisOnesHeaders) { 1721 auto p = h.indexOf(":"); 1722 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1723 string hn = h[0..p]; 1724 string hv = h[p+2..$]; 1725 1726 switch(hn.toLower) { 1727 default: assert(0); 1728 case "content-disposition": 1729 auto info = hv.split("; "); 1730 foreach(i; info[1..$]) { // skipping the form-data 1731 auto o = i.split("="); // FIXME 1732 string pn = o[0]; 1733 string pv = o[1][1..$-1]; 1734 1735 if(pn == "name") { 1736 pps.piece.name = pv; 1737 } else if (pn == "filename") { 1738 pps.piece.filename = pv; 1739 pps.isFile = true; 1740 } 1741 } 1742 break; 1743 case "content-type": 1744 pps.piece.contentType = hv; 1745 break; 1746 } 1747 } 1748 1749 pps.whatDoWeWant++; // move to the next step - the data 1750 break; 1751 case 2: 1752 // when we get here, pps.buffer should contain our first chunk of data 1753 1754 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1755 throw new Exception("wtf is up with the huge mime part buffer"); 1756 1757 acceptChunk(); 1758 1759 // so the trick is, we want to process all the data up to the boundary, 1760 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1761 // want to wait for the next chunk. We start by looking for the whole boundary 1762 // in the buffer somewhere. 1763 1764 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1765 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1766 if(boundaryLocation != -1) { 1767 // this is easy - we can see it in it's entirety! 1768 1769 pps.piece.content ~= consume(boundaryLocation); 1770 1771 assert(pps.buffer[0] == '\r'); 1772 consume(1); 1773 assert(pps.buffer[0] == '\n'); 1774 consume(1); 1775 assert(pps.buffer[0] == '-'); 1776 consume(1); 1777 assert(pps.buffer[0] == '-'); 1778 consume(1); 1779 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1780 pps.weHaveAPart = true; 1781 pps.whatDoWeWant = 1; // back to getting headers for the next part 1782 1783 commitPart(); // we're done here 1784 } else { 1785 // we can't see the whole thing, but what if there's a partial boundary? 1786 1787 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1788 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1789 bool potentialBoundaryFound = false; 1790 1791 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1792 // we grow the boundary a bit each time. If we think it looks the 1793 // same, better pull another chunk to be sure it's not the end. 1794 // Starting small because exiting the loop early is desirable, since 1795 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1796 // the best we can do. 1797 if(a > pps.buffer.length) 1798 break; // FIXME: is this right? 1799 assert(a <= pps.buffer.length); 1800 assert(a > 0); 1801 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1802 // ok, there *might* be a boundary here, so let's 1803 // not treat the end as data yet. The rest is good to 1804 // use though, since if there was a boundary there, we'd 1805 // have handled it up above after locationOf. 1806 1807 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1808 consume(pps.buffer.length - a); 1809 pieceHasNewContent(); 1810 potentialBoundaryFound = true; 1811 break boundaryCheck; 1812 } 1813 } 1814 1815 if(!potentialBoundaryFound) { 1816 // we can consume the whole thing 1817 pps.piece.content ~= pps.buffer; 1818 pieceHasNewContent(); 1819 consume(pps.buffer.length); 1820 } else { 1821 // we found a possible boundary, but there was 1822 // insufficient data to be sure. 1823 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1824 1825 return; // wait for the next chunk. 1826 } 1827 } 1828 } 1829 } while(pps.buffer.length); 1830 1831 // btw all boundaries except the first should have a \r\n before them 1832 } else { 1833 // application/x-www-form-urlencoded and application/json 1834 1835 // not using maxContentLength because that might be cranked up to allow 1836 // large file uploads. We can handle them, but a huge post[] isn't any good. 1837 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1838 throw new Exception("wtf is up with such a gigantic form submission????"); 1839 1840 pps.buffer ~= chunk; 1841 1842 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1843 if(pps.buffer.length == pps.expectedLength) { 1844 if(pps.needsSavedBody) 1845 pps.postBody = cast(string) pps.buffer; 1846 else 1847 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1848 version(preserveData) 1849 originalPostData = pps.buffer; 1850 } else { 1851 // just for debugging 1852 } 1853 } 1854 } 1855 1856 protected void cleanUpPostDataState() { 1857 pps = PostParserState.init; 1858 } 1859 1860 /// you can override this function to somehow react 1861 /// to an upload in progress. 1862 /// 1863 /// Take note that parts of the CGI object is not yet 1864 /// initialized! Stuff from HTTP headers, including get[], is usable. 1865 /// But, none of post[] is usable, and you cannot write here. That's 1866 /// why this method is const - mutating the object won't do much anyway. 1867 /// 1868 /// My idea here was so you can output a progress bar or 1869 /// something to a cooperative client (see arsd.rtud for a potential helper) 1870 /// 1871 /// The default is to do nothing. Subclass cgi and use the 1872 /// CustomCgiMain mixin to do something here. 1873 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1874 // This space intentionally left blank. 1875 } 1876 1877 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1878 /// *closeConnection will be set to true if you should close the connection after handling this request 1879 this(BufferedInputRange ir, bool* closeConnection) { 1880 isCalledWithCommandLineArguments = false; 1881 import al = std.algorithm; 1882 1883 immutable(ubyte)[] data; 1884 1885 void rdo(const(ubyte)[] d) { 1886 //import std.stdio; writeln(d); 1887 sendAll(ir.source, d); 1888 } 1889 1890 auto ira = ir.source.remoteAddress(); 1891 auto irLocalAddress = ir.source.localAddress(); 1892 1893 ushort port = 80; 1894 if(auto ia = cast(InternetAddress) irLocalAddress) { 1895 port = ia.port; 1896 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1897 port = ia.port; 1898 } 1899 1900 // that check for UnixAddress is to work around a Phobos bug 1901 // see: https://github.com/dlang/phobos/pull/7383 1902 // but this might be more useful anyway tbh for this case 1903 version(Posix) 1904 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1905 else 1906 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1907 } 1908 1909 /** 1910 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1911 1912 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1913 1914 Params: 1915 inputData = the incoming data, including headers and other raw http data. 1916 When the constructor exits, it will leave this range exactly at the start of 1917 the next request on the connection (if there is one). 1918 1919 address = the IP address of the remote user 1920 _port = the port number of the connection 1921 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1922 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1923 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1924 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1925 closeConnection = if the request asks to close the connection, *closeConnection == true. 1926 */ 1927 this( 1928 BufferedInputRange inputData, 1929 // string[] headers, immutable(ubyte)[] data, 1930 string address, ushort _port, 1931 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1932 bool _https = false, 1933 void delegate(const(ubyte)[]) _rawDataOutput = null, 1934 void delegate() _flush = null, 1935 // this pointer tells if the connection is supposed to be closed after we handle this 1936 bool* closeConnection = null) 1937 { 1938 // these are all set locally so the loop works 1939 // without triggering errors in dmd 2.064 1940 // we go ahead and set them at the end of it to the this version 1941 int port; 1942 string referrer; 1943 string remoteAddress; 1944 string userAgent; 1945 string authorization; 1946 string origin; 1947 string accept; 1948 string lastEventId; 1949 bool https; 1950 string host; 1951 RequestMethod requestMethod; 1952 string requestUri; 1953 string pathInfo; 1954 string queryString; 1955 string scriptName; 1956 string[string] get; 1957 string[][string] getArray; 1958 bool keepAliveRequested; 1959 bool acceptsGzip; 1960 string cookie; 1961 1962 1963 1964 environmentVariables = cast(const) environment.toAA; 1965 1966 idlol = inputData; 1967 1968 isCalledWithCommandLineArguments = false; 1969 1970 https = _https; 1971 port = _port; 1972 1973 rawDataOutput = _rawDataOutput; 1974 flushDelegate = _flush; 1975 nph = true; 1976 1977 remoteAddress = address; 1978 1979 // streaming parser 1980 import al = std.algorithm; 1981 1982 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1983 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1984 while(idx == -1) { 1985 inputData.popFront(0); 1986 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1987 } 1988 1989 assert(idx != -1); 1990 1991 1992 string contentType = ""; 1993 string[string] requestHeadersHere; 1994 1995 size_t contentLength; 1996 1997 bool isChunked; 1998 1999 { 2000 import core.runtime; 2001 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 2002 } 2003 2004 2005 int headerNumber = 0; 2006 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 2007 if(line.length) { 2008 headerNumber++; 2009 auto header = cast(string) line.idup; 2010 if(headerNumber == 1) { 2011 // request line 2012 auto parts = al.splitter(header, " "); 2013 if(parts.front == "PRI") { 2014 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 2015 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 2016 // to bring me benefit) 2017 throw new HttpVersionNotSupportedException(); 2018 } 2019 requestMethod = to!RequestMethod(parts.front); 2020 parts.popFront(); 2021 requestUri = parts.front; 2022 2023 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 2024 scriptName = requestUri[0 .. pathInfoStarts]; 2025 2026 auto question = requestUri.indexOf("?"); 2027 if(question == -1) { 2028 queryString = ""; 2029 // FIXME: double check, this might be wrong since it could be url encoded 2030 pathInfo = requestUri[pathInfoStarts..$]; 2031 } else { 2032 queryString = requestUri[question+1..$]; 2033 pathInfo = requestUri[pathInfoStarts..question]; 2034 } 2035 2036 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2037 getArray = cast(string[][string]) assumeUnique(ugh); 2038 2039 if(header.indexOf("HTTP/1.0") != -1) { 2040 http10 = true; 2041 autoBuffer = true; 2042 if(closeConnection) { 2043 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2044 *closeConnection = true; 2045 } 2046 } 2047 } else { 2048 // other header 2049 auto colon = header.indexOf(":"); 2050 if(colon == -1) 2051 throw new Exception("HTTP headers should have a colon!"); 2052 string name = header[0..colon].toLower; 2053 string value = header[colon+2..$]; // skip the colon and the space 2054 2055 requestHeadersHere[name] = value; 2056 2057 if (name == "accept") { 2058 accept = value; 2059 } 2060 else if (name == "origin") { 2061 origin = value; 2062 } 2063 else if (name == "connection") { 2064 if(value == "close" && closeConnection) 2065 *closeConnection = true; 2066 if(value.asLowerCase().canFind("keep-alive")) { 2067 keepAliveRequested = true; 2068 2069 // on http 1.0, the connection is closed by default, 2070 // but not if they request keep-alive. then we don't close 2071 // anymore - undoing the set above 2072 if(http10 && closeConnection) { 2073 *closeConnection = false; 2074 } 2075 } 2076 } 2077 else if (name == "transfer-encoding") { 2078 if(value == "chunked") 2079 isChunked = true; 2080 } 2081 else if (name == "last-event-id") { 2082 lastEventId = value; 2083 } 2084 else if (name == "authorization") { 2085 authorization = value; 2086 } 2087 else if (name == "content-type") { 2088 contentType = value; 2089 } 2090 else if (name == "content-length") { 2091 contentLength = to!size_t(value); 2092 } 2093 else if (name == "x-forwarded-for") { 2094 remoteAddress = value; 2095 } 2096 else if (name == "x-forwarded-host" || name == "host") { 2097 if(name != "host" || host is null) 2098 host = value; 2099 } 2100 // FIXME: https://tools.ietf.org/html/rfc7239 2101 else if (name == "accept-encoding") { 2102 if(value.indexOf("gzip") != -1) 2103 acceptsGzip = true; 2104 } 2105 else if (name == "user-agent") { 2106 userAgent = value; 2107 } 2108 else if (name == "referer") { 2109 referrer = value; 2110 } 2111 else if (name == "cookie") { 2112 cookie ~= value; 2113 } else if(name == "expect") { 2114 if(value == "100-continue") { 2115 // FIXME we should probably give user code a chance 2116 // to process and reject but that needs to be virtual, 2117 // perhaps part of the CGI redesign. 2118 2119 // FIXME: if size is > max content length it should 2120 // also fail at this point. 2121 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2122 2123 // FIXME: let the user write out 103 early hints too 2124 } 2125 } 2126 // else 2127 // ignore it 2128 2129 } 2130 } 2131 2132 inputData.consume(idx + 4); 2133 // done 2134 2135 requestHeaders = assumeUnique(requestHeadersHere); 2136 2137 ByChunkRange dataByChunk; 2138 2139 // reading Content-Length type data 2140 // We need to read up the data we have, and write it out as a chunk. 2141 if(!isChunked) { 2142 dataByChunk = byChunk(inputData, contentLength); 2143 } else { 2144 // chunked requests happen, but not every day. Since we need to know 2145 // the content length (for now, maybe that should change), we'll buffer 2146 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2147 auto data = dechunk(inputData); 2148 2149 // set the range here 2150 dataByChunk = byChunk(data); 2151 contentLength = data.length; 2152 } 2153 2154 assert(dataByChunk !is null); 2155 2156 if(contentLength) { 2157 prepareForIncomingDataChunks(contentType, contentLength); 2158 foreach(dataChunk; dataByChunk) { 2159 handleIncomingDataChunk(dataChunk); 2160 } 2161 postArray = assumeUnique(pps._post); 2162 filesArray = assumeUnique(pps._files); 2163 files = keepLastOf(filesArray); 2164 post = keepLastOf(postArray); 2165 postBody = pps.postBody; 2166 cleanUpPostDataState(); 2167 } 2168 2169 this.port = port; 2170 this.referrer = referrer; 2171 this.remoteAddress = remoteAddress; 2172 this.userAgent = userAgent; 2173 this.authorization = authorization; 2174 this.origin = origin; 2175 this.accept = accept; 2176 this.lastEventId = lastEventId; 2177 this.https = https; 2178 this.host = host; 2179 this.requestMethod = requestMethod; 2180 this.requestUri = requestUri; 2181 this.pathInfo = pathInfo; 2182 this.queryString = queryString; 2183 2184 this.scriptName = scriptName; 2185 this.get = keepLastOf(getArray); 2186 this.getArray = cast(immutable) getArray; 2187 this.keepAliveRequested = keepAliveRequested; 2188 this.acceptsGzip = acceptsGzip; 2189 this.cookie = cookie; 2190 2191 cookiesArray = getCookieArray(); 2192 cookies = keepLastOf(cookiesArray); 2193 2194 } 2195 BufferedInputRange idlol; 2196 2197 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2198 string[string] ca; 2199 foreach(k, v; arr) 2200 ca[k] = v[$-1]; 2201 2202 return assumeUnique(ca); 2203 } 2204 2205 // FIXME duplication 2206 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2207 UploadedFile[string] ca; 2208 foreach(k, v; arr) 2209 ca[k] = v[$-1]; 2210 2211 return assumeUnique(ca); 2212 } 2213 2214 2215 private immutable(string[][string]) getCookieArray() { 2216 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2217 return assumeUnique(forTheLoveOfGod); 2218 } 2219 2220 /++ 2221 Very simple method to require a basic auth username and password. 2222 If the http request doesn't include the required credentials, it throws a 2223 HTTP 401 error, and an exception to cancel your handler. Do NOT catch the 2224 `AuthorizationRequiredException` exception thrown by this if you want the 2225 http basic auth prompt to work for the user! 2226 2227 Note: basic auth does not provide great security, especially over unencrypted HTTP; 2228 the user's credentials are sent in plain text on every request. 2229 2230 If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2231 application. Either use Apache's built in methods for basic authentication, or add 2232 something along these lines to your server configuration: 2233 2234 ``` 2235 RewriteEngine On 2236 RewriteCond %{HTTP:Authorization} ^(.*) 2237 RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2238 ``` 2239 2240 To ensure the necessary data is available to cgi.d. 2241 +/ 2242 void requireBasicAuth(string user, string pass, string message = null, string file = __FILE__, size_t line = __LINE__) { 2243 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2244 throw new AuthorizationRequiredException("Basic", message, file, line); 2245 } 2246 } 2247 2248 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2249 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2250 /// Use setResponseExpires and updateResponseExpires for more control 2251 void setCache(bool allowCaching) { 2252 noCache = !allowCaching; 2253 } 2254 2255 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2256 /// who can accept it 2257 bool gzipResponse; 2258 2259 immutable bool acceptsGzip; 2260 immutable bool keepAliveRequested; 2261 2262 /// Set to true if and only if this was initialized with command line arguments 2263 immutable bool isCalledWithCommandLineArguments; 2264 2265 /// This gets a full url for the current request, including port, protocol, host, path, and query 2266 string getCurrentCompleteUri() const { 2267 ushort defaultPort = https ? 443 : 80; 2268 2269 string uri = "http"; 2270 if(https) 2271 uri ~= "s"; 2272 uri ~= "://"; 2273 uri ~= host; 2274 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2275 version(none) 2276 if(!(!port || port == defaultPort)) { 2277 uri ~= ":"; 2278 uri ~= to!string(port); 2279 } 2280 +/ 2281 uri ~= requestUri; 2282 return uri; 2283 } 2284 2285 /// You can override this if your site base url isn't the same as the script name 2286 string logicalScriptName() const { 2287 return scriptName; 2288 } 2289 2290 /++ 2291 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2292 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2293 Note setResponseStatus() must be called *before* you write() any data to the output. 2294 2295 History: 2296 The `int` overload was added on January 11, 2021. 2297 +/ 2298 void setResponseStatus(string status) { 2299 assert(!outputtedResponseData); 2300 responseStatus = status; 2301 } 2302 /// ditto 2303 void setResponseStatus(int statusCode) { 2304 setResponseStatus(getHttpCodeText(statusCode)); 2305 } 2306 private string responseStatus = null; 2307 2308 /// Returns true if it is still possible to output headers 2309 bool canOutputHeaders() { 2310 return !isClosed && !outputtedResponseData; 2311 } 2312 2313 /// Sets the location header, which the browser will redirect the user to automatically. 2314 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2315 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2316 void setResponseLocation(string uri, bool important = true, string status = null) { 2317 if(!important && isCurrentResponseLocationImportant) 2318 return; // important redirects always override unimportant ones 2319 2320 if(uri is null) { 2321 responseStatus = "200 OK"; 2322 responseLocation = null; 2323 isCurrentResponseLocationImportant = important; 2324 return; // this just cancels the redirect 2325 } 2326 2327 assert(!outputtedResponseData); 2328 if(status is null) 2329 responseStatus = "302 Found"; 2330 else 2331 responseStatus = status; 2332 2333 responseLocation = uri.strip; 2334 isCurrentResponseLocationImportant = important; 2335 } 2336 protected string responseLocation = null; 2337 private bool isCurrentResponseLocationImportant = false; 2338 2339 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2340 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2341 /// Note: the when parameter is different than setCookie's expire parameter. 2342 void setResponseExpires(long when, bool isPublic = false) { 2343 responseExpires = when; 2344 setCache(true); // need to enable caching so the date has meaning 2345 2346 responseIsPublic = isPublic; 2347 responseExpiresRelative = false; 2348 } 2349 2350 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2351 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2352 responseExpires = whenFromNow; 2353 setCache(true); // need to enable caching so the date has meaning 2354 2355 responseIsPublic = isPublic; 2356 responseExpiresRelative = true; 2357 } 2358 private long responseExpires = long.min; 2359 private bool responseIsPublic = false; 2360 private bool responseExpiresRelative = false; 2361 2362 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2363 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2364 /// output as a whole is as cacheable as the least cachable part in the chain. 2365 2366 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2367 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2368 void updateResponseExpires(long when, bool isPublic) { 2369 if(responseExpires == long.min) 2370 setResponseExpires(when, isPublic); 2371 else if(when < responseExpires) 2372 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2373 } 2374 2375 /* 2376 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2377 /// Should generally be false if the user is logged in. It assumes private cache only. 2378 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2379 void setPublicCaching(bool allowPublicCaches) { 2380 publicCaching = allowPublicCaches; 2381 } 2382 private bool publicCaching = false; 2383 */ 2384 2385 /++ 2386 History: 2387 Added January 11, 2021 2388 +/ 2389 enum SameSitePolicy { 2390 Lax, 2391 Strict, 2392 None 2393 } 2394 2395 /++ 2396 Sets an HTTP cookie, automatically encoding the data to the correct string. 2397 expiresIn is how many milliseconds in the future the cookie will expire. 2398 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2399 Note setCookie() must be called *before* you write() any data to the output. 2400 2401 History: 2402 Parameter `sameSitePolicy` was added on January 11, 2021. 2403 +/ 2404 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2405 assert(!outputtedResponseData); 2406 string cookie = std.uri.encodeComponent(name) ~ "="; 2407 cookie ~= std.uri.encodeComponent(data); 2408 if(path !is null) 2409 cookie ~= "; path=" ~ path; 2410 // FIXME: should I just be using max-age here? (also in cache below) 2411 if(expiresIn != 0) 2412 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2413 if(domain !is null) 2414 cookie ~= "; domain=" ~ domain; 2415 if(secure == true) 2416 cookie ~= "; Secure"; 2417 if(httpOnly == true ) 2418 cookie ~= "; HttpOnly"; 2419 final switch(sameSitePolicy) { 2420 case SameSitePolicy.Lax: 2421 cookie ~= "; SameSite=Lax"; 2422 break; 2423 case SameSitePolicy.Strict: 2424 cookie ~= "; SameSite=Strict"; 2425 break; 2426 case SameSitePolicy.None: 2427 cookie ~= "; SameSite=None"; 2428 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2429 break; 2430 } 2431 2432 if(auto idx = name in cookieIndexes) { 2433 responseCookies[*idx] = cookie; 2434 } else { 2435 cookieIndexes[name] = responseCookies.length; 2436 responseCookies ~= cookie; 2437 } 2438 } 2439 private string[] responseCookies; 2440 private size_t[string] cookieIndexes; 2441 2442 /// Clears a previously set cookie with the given name, path, and domain. 2443 void clearCookie(string name, string path = null, string domain = null) { 2444 assert(!outputtedResponseData); 2445 setCookie(name, "", 1, path, domain); 2446 } 2447 2448 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2449 void setResponseContentType(string ct) { 2450 assert(!outputtedResponseData); 2451 responseContentType = ct; 2452 } 2453 private string responseContentType = null; 2454 2455 /// Adds a custom header. It should be the name: value, but without any line terminator. 2456 /// For example: header("X-My-Header: Some value"); 2457 /// Note you should use the specialized functions in this object if possible to avoid 2458 /// duplicates in the output. 2459 void header(string h) { 2460 customHeaders ~= h; 2461 } 2462 2463 /++ 2464 I named the original function `header` after PHP, but this pattern more fits 2465 the rest of the Cgi object. 2466 2467 Either name are allowed. 2468 2469 History: 2470 Alias added June 17, 2022. 2471 +/ 2472 alias setResponseHeader = header; 2473 2474 private string[] customHeaders; 2475 private bool websocketMode; 2476 2477 void flushHeaders(const(void)[] t, bool isAll = false) { 2478 StackBuffer buffer = StackBuffer(0); 2479 2480 prepHeaders(t, isAll, &buffer); 2481 2482 if(rawDataOutput !is null) 2483 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2484 else { 2485 stdout.rawWrite(buffer.get()); 2486 } 2487 } 2488 2489 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2490 string terminator = "\n"; 2491 if(rawDataOutput !is null) 2492 terminator = "\r\n"; 2493 2494 if(responseStatus !is null) { 2495 if(nph) { 2496 if(http10) 2497 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2498 else 2499 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2500 } else 2501 buffer.add("Status: ", responseStatus, terminator); 2502 } else if (nph) { 2503 if(http10) 2504 buffer.add("HTTP/1.0 200 OK", terminator); 2505 else 2506 buffer.add("HTTP/1.1 200 OK", terminator); 2507 } 2508 2509 if(websocketMode) 2510 goto websocket; 2511 2512 if(nph) { // we're responsible for setting the date too according to http 1.1 2513 char[29] db = void; 2514 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2515 buffer.add("Date: ", db[], terminator); 2516 } 2517 2518 // FIXME: what if the user wants to set his own content-length? 2519 // The custom header function can do it, so maybe that's best. 2520 // Or we could reuse the isAll param. 2521 if(responseLocation !is null) { 2522 buffer.add("Location: ", responseLocation, terminator); 2523 } 2524 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2525 if(responseExpiresRelative) { 2526 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2527 buffer.add(responseExpires); 2528 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2529 } else { 2530 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2531 char[29] db = void; 2532 printDateToBuffer(cast(DateTime) expires, db[]); 2533 buffer.add("Expires: ", db[], terminator); 2534 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2535 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2536 buffer.add(terminator); 2537 } 2538 } 2539 if(responseCookies !is null && responseCookies.length > 0) { 2540 foreach(c; responseCookies) 2541 buffer.add("Set-Cookie: ", c, terminator); 2542 } 2543 if(noCache) { // we specifically do not want caching (this is actually the default) 2544 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2545 buffer.add("Expires: 0", terminator); 2546 buffer.add("Pragma: no-cache", terminator); 2547 } else { 2548 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2549 buffer.add("Cache-Control: public", terminator); 2550 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2551 } 2552 } 2553 if(responseContentType !is null) { 2554 buffer.add("Content-Type: ", responseContentType, terminator); 2555 } else 2556 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2557 2558 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2559 buffer.add("Content-Encoding: gzip", terminator); 2560 } 2561 2562 2563 if(!isAll) { 2564 if(nph && !http10) { 2565 buffer.add("Transfer-Encoding: chunked", terminator); 2566 responseChunked = true; 2567 } 2568 } else { 2569 buffer.add("Content-Length: "); 2570 buffer.add(t.length); 2571 buffer.add(terminator); 2572 if(nph && keepAliveRequested) { 2573 buffer.add("Connection: Keep-Alive", terminator); 2574 } 2575 } 2576 2577 websocket: 2578 2579 foreach(hd; customHeaders) 2580 buffer.add(hd, terminator); 2581 2582 // FIXME: what about duplicated headers? 2583 2584 // end of header indicator 2585 buffer.add(terminator); 2586 2587 outputtedResponseData = true; 2588 } 2589 2590 /// Writes the data to the output, flushing headers if they have not yet been sent. 2591 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2592 assert(!closed, "Output has already been closed"); 2593 2594 StackBuffer buffer = StackBuffer(0); 2595 2596 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2597 // actually gzip the data here 2598 2599 auto c = new Compress(HeaderFormat.gzip); // want gzip 2600 2601 auto data = c.compress(t); 2602 data ~= c.flush(); 2603 2604 // std.file.write("/tmp/last-item", data); 2605 2606 t = data; 2607 } 2608 2609 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2610 prepHeaders(t, isAll, &buffer); 2611 } 2612 2613 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2614 if (autoBuffer && !isAll) { 2615 outputBuffer ~= cast(ubyte[]) t; 2616 } 2617 if(!autoBuffer || isAll) { 2618 if(rawDataOutput !is null) 2619 if(nph && responseChunked) { 2620 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2621 // we're making the chunk here instead of in a function 2622 // to avoid unneeded gc pressure 2623 buffer.add(toHex(t.length)); 2624 buffer.add("\r\n"); 2625 buffer.add(cast(char[]) t, "\r\n"); 2626 } else { 2627 buffer.add(cast(char[]) t); 2628 } 2629 else 2630 buffer.add(cast(char[]) t); 2631 } 2632 } 2633 2634 if(rawDataOutput !is null) 2635 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2636 else 2637 stdout.rawWrite(buffer.get()); 2638 2639 if(maybeAutoClose && isAll) 2640 close(); // if you say it is all, that means we're definitely done 2641 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2642 } 2643 2644 /++ 2645 Convenience method to set content type to json and write the string as the complete response. 2646 2647 History: 2648 Added January 16, 2020 2649 +/ 2650 void writeJson(string json) { 2651 this.setResponseContentType("application/json"); 2652 this.write(json, true); 2653 } 2654 2655 /// Flushes the pending buffer, leaving the connection open so you can send more. 2656 void flush() { 2657 if(rawDataOutput is null) 2658 stdout.flush(); 2659 else if(flushDelegate !is null) 2660 flushDelegate(); 2661 } 2662 2663 version(autoBuffer) 2664 bool autoBuffer = true; 2665 else 2666 bool autoBuffer = false; 2667 ubyte[] outputBuffer; 2668 2669 /// Flushes the buffers to the network, signifying that you are done. 2670 /// You should always call this explicitly when you are done outputting data. 2671 void close() { 2672 if(closed) 2673 return; // don't double close 2674 2675 if(!outputtedResponseData) 2676 write("", true, false); 2677 2678 // writing auto buffered data 2679 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2680 if(!nph) 2681 stdout.rawWrite(outputBuffer); 2682 else 2683 write(outputBuffer, true, false); // tell it this is everything 2684 } 2685 2686 // closing the last chunk... 2687 if(nph && rawDataOutput !is null && responseChunked) 2688 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2689 2690 if(flushDelegate) 2691 flushDelegate(); 2692 2693 closed = true; 2694 } 2695 2696 // Closes without doing anything, shouldn't be used often 2697 void rawClose() { 2698 closed = true; 2699 } 2700 2701 /++ 2702 Gets a request variable as a specific type, or the default value of it isn't there 2703 or isn't convertible to the request type. 2704 2705 Checks both GET and POST variables, preferring the POST variable, if available. 2706 2707 A nice trick is using the default value to choose the type: 2708 2709 --- 2710 /* 2711 The return value will match the type of the default. 2712 Here, I gave 10 as a default, so the return value will 2713 be an int. 2714 2715 If the user-supplied value cannot be converted to the 2716 requested type, you will get the default value back. 2717 */ 2718 int a = cgi.request("number", 10); 2719 2720 if(cgi.get["number"] == "11") 2721 assert(a == 11); // conversion succeeds 2722 2723 if("number" !in cgi.get) 2724 assert(a == 10); // no value means you can't convert - give the default 2725 2726 if(cgi.get["number"] == "twelve") 2727 assert(a == 10); // conversion from string to int would fail, so we get the default 2728 --- 2729 2730 You can use an enum as an easy whitelist, too: 2731 2732 --- 2733 enum Operations { 2734 add, remove, query 2735 } 2736 2737 auto op = cgi.request("op", Operations.query); 2738 2739 if(cgi.get["op"] == "add") 2740 assert(op == Operations.add); 2741 if(cgi.get["op"] == "remove") 2742 assert(op == Operations.remove); 2743 if(cgi.get["op"] == "query") 2744 assert(op == Operations.query); 2745 2746 if(cgi.get["op"] == "random string") 2747 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2748 --- 2749 +/ 2750 T request(T = string)(in string name, in T def = T.init) const nothrow { 2751 try { 2752 return 2753 (name in post) ? to!T(post[name]) : 2754 (name in get) ? to!T(get[name]) : 2755 def; 2756 } catch(Exception e) { return def; } 2757 } 2758 2759 /// Is the output already closed? 2760 bool isClosed() const { 2761 return closed; 2762 } 2763 2764 /++ 2765 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2766 +/ 2767 Session!Data getSessionObject(Data)() { 2768 if(testInProcess !is null) { 2769 // test mode 2770 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2771 if(obj !is null) 2772 return cast(typeof(return)) obj; 2773 else { 2774 auto o = new MockSession!Data(); 2775 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2776 return o; 2777 } 2778 } else { 2779 // normal operation 2780 return new BasicDataServerSession!Data(this); 2781 } 2782 } 2783 2784 // if it is in test mode; triggers mock sessions. Used by CgiTester 2785 version(with_breaking_cgi_features) 2786 private CgiTester testInProcess; 2787 2788 /* Hooks for redirecting input and output */ 2789 private void delegate(const(ubyte)[]) rawDataOutput = null; 2790 private void delegate() flushDelegate = null; 2791 2792 /* This info is used when handling a more raw HTTP protocol */ 2793 private bool nph; 2794 private bool http10; 2795 private bool closed; 2796 private bool responseChunked = false; 2797 2798 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2799 immutable(ubyte)[] originalPostData; 2800 2801 /++ 2802 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2803 2804 It is intended to be used for JSON and XML request content types, but also may be used 2805 for other content types your application can handle. But it will NOT be populated 2806 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2807 parsed into the post and postArray members. 2808 2809 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2810 will be discarded to the client with an error. This helps keep this array from being exploded in size 2811 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2812 client in certain build modes.) 2813 2814 History: 2815 Added January 5, 2021 2816 Documented February 21, 2023 (dub v11.0) 2817 +/ 2818 public immutable string postBody; 2819 alias postJson = postBody; // old name 2820 2821 /* Internal state flags */ 2822 private bool outputtedResponseData; 2823 private bool noCache = true; 2824 2825 const(string[string]) environmentVariables; 2826 2827 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2828 partially because it logically is (your code doesn't change what the user requested...) 2829 and partially because I hate how bad programs in PHP change those superglobals to do 2830 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2831 2832 For some of these, you'll want to refer to the http or cgi specs for more details. 2833 */ 2834 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2835 2836 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2837 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2838 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2839 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2840 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2841 immutable(char[]) scriptFileName; /// The physical filename of your script 2842 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2843 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2844 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2845 2846 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2847 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2848 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2849 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2850 2851 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2852 */ 2853 immutable(char[]) referrer; 2854 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2855 2856 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2857 2858 immutable bool https; /// Was the request encrypted via https? 2859 immutable int port; /// On what TCP port number did the server receive the request? 2860 2861 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2862 2863 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2864 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2865 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2866 2867 /// added later 2868 alias query = get; 2869 2870 /** 2871 Represents user uploaded files. 2872 2873 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2874 */ 2875 immutable(UploadedFile[][string]) filesArray; 2876 immutable(UploadedFile[string]) files; 2877 2878 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2879 /// the order of the arrays is the order the data arrives 2880 immutable(string[][string]) getArray; /// like get, but an array of values per name 2881 immutable(string[][string]) postArray; /// ditto for post 2882 immutable(string[][string]) cookiesArray; /// ditto for cookies 2883 2884 // convenience function for appending to a uri without extra ? 2885 // matches the name and effect of javascript's location.search property 2886 string search() const { 2887 if(queryString.length) 2888 return "?" ~ queryString; 2889 return ""; 2890 } 2891 2892 // FIXME: what about multiple files with the same name? 2893 private: 2894 //RequestMethod _requestMethod; 2895 } 2896 2897 /// use this for testing or other isolated things when you want it to be no-ops 2898 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2899 // we want to ignore, not use stdout 2900 if(outputSink is null) 2901 outputSink = delegate void(const(ubyte)[]) { }; 2902 2903 string[string] env; 2904 env["REQUEST_METHOD"] = to!string(method); 2905 env["CONTENT_LENGTH"] = to!string(data.length); 2906 2907 auto cgi = new Cgi( 2908 0, 2909 env, 2910 { return data; }, 2911 outputSink, 2912 null); 2913 2914 return cgi; 2915 } 2916 2917 /++ 2918 A helper test class for request handler unittests. 2919 +/ 2920 version(with_breaking_cgi_features) 2921 class CgiTester { 2922 private { 2923 SessionObject[TypeInfo] mockSessions; 2924 SessionObject getSessionOverride(TypeInfo ti) { 2925 if(auto o = ti in mockSessions) 2926 return *o; 2927 else 2928 return null; 2929 } 2930 void setSessionOverride(TypeInfo ti, SessionObject so) { 2931 mockSessions[ti] = so; 2932 } 2933 } 2934 2935 /++ 2936 Gets (and creates if necessary) a mock session object for this test. Note 2937 it will be the same one used for any test operations through this CgiTester instance. 2938 +/ 2939 Session!Data getSessionObject(Data)() { 2940 auto obj = getSessionOverride(typeid(typeof(return))); 2941 if(obj !is null) 2942 return cast(typeof(return)) obj; 2943 else { 2944 auto o = new MockSession!Data(); 2945 setSessionOverride(typeid(typeof(return)), o); 2946 return o; 2947 } 2948 } 2949 2950 /++ 2951 Pass a reference to your request handler when creating the tester. 2952 +/ 2953 this(void function(Cgi) requestHandler) { 2954 this.requestHandler = requestHandler; 2955 } 2956 2957 /++ 2958 You can check response information with these methods after you call the request handler. 2959 +/ 2960 struct Response { 2961 int code; 2962 string[string] headers; 2963 string responseText; 2964 ubyte[] responseBody; 2965 } 2966 2967 /++ 2968 Executes a test request on your request handler, and returns the response. 2969 2970 Params: 2971 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2972 args = additional arguments. Same format as cgi's command line handler. 2973 +/ 2974 Response GET(string url, string[] args = null) { 2975 return executeTest("GET", url, args); 2976 } 2977 /// ditto 2978 Response POST(string url, string[] args = null) { 2979 return executeTest("POST", url, args); 2980 } 2981 2982 /// ditto 2983 Response executeTest(string method, string url, string[] args) { 2984 ubyte[] outputtedRawData; 2985 void outputSink(const(ubyte)[] data) { 2986 outputtedRawData ~= data; 2987 } 2988 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2989 cgi.testInProcess = this; 2990 scope(exit) cgi.dispose(); 2991 2992 requestHandler(cgi); 2993 2994 cgi.close(); 2995 2996 Response response; 2997 2998 if(outputtedRawData.length) { 2999 enum LINE = "\r\n"; 3000 3001 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 3002 assert(idx != -1, to!string(outputtedRawData)); 3003 auto headers = cast(string) outputtedRawData[0 .. idx]; 3004 response.code = 200; 3005 while(headers.length) { 3006 auto i = headers.locationOf(LINE); 3007 if(i == -1) i = cast(int) headers.length; 3008 3009 auto header = headers[0 .. i]; 3010 3011 auto c = header.locationOf(":"); 3012 if(c != -1) { 3013 auto name = header[0 .. c]; 3014 auto value = header[c + 2 ..$]; 3015 3016 if(name == "Status") 3017 response.code = value[0 .. value.locationOf(" ")].to!int; 3018 3019 response.headers[name] = value; 3020 } else { 3021 assert(0); 3022 } 3023 3024 if(i != headers.length) 3025 i += 2; 3026 headers = headers[i .. $]; 3027 } 3028 response.responseBody = outputtedRawData[idx + 4 .. $]; 3029 response.responseText = cast(string) response.responseBody; 3030 } 3031 3032 return response; 3033 } 3034 3035 private void function(Cgi) requestHandler; 3036 } 3037 3038 3039 // should this be a separate module? Probably, but that's a hassle. 3040 3041 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3042 string makeDataUrl(string mimeType, in void[] data) { 3043 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3044 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3045 } 3046 3047 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3048 /// Represents a url that can be broken down or built up through properties 3049 struct Uri { 3050 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3051 3052 // scheme//userinfo@host:port/path?query#fragment 3053 3054 string scheme; /// e.g. "http" in "http://example.com/" 3055 string userinfo; /// the username (and possibly a password) in the uri 3056 string host; /// the domain name 3057 int port; /// port number, if given. Will be zero if a port was not explicitly given 3058 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3059 string query; /// the stuff after the ? in a uri 3060 string fragment; /// the stuff after the # in a uri. 3061 3062 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3063 // the decode ones need to keep different names anyway because we can't overload on return values... 3064 static string encode(string s) { return std.uri.encodeComponent(s); } 3065 static string encode(string[string] s) { return encodeVariables(s); } 3066 static string encode(string[][string] s) { return encodeVariables(s); } 3067 3068 /// Breaks down a uri string to its components 3069 this(string uri) { 3070 reparse(uri); 3071 } 3072 3073 private void reparse(string uri) { 3074 // from RFC 3986 3075 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3076 // it was a nice experiment but just not worth it. 3077 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3078 /* 3079 Captures: 3080 0 = whole url 3081 1 = scheme, with : 3082 2 = scheme, no : 3083 3 = authority, with // 3084 4 = authority, no // 3085 5 = path 3086 6 = query string, with ? 3087 7 = query string, no ? 3088 8 = anchor, with # 3089 9 = anchor, no # 3090 */ 3091 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3092 // instead, I will DIY and cut that down to 0.6s on the same computer. 3093 /* 3094 3095 Note that authority is 3096 user:password@domain:port 3097 where the user:password@ part is optional, and the :port is optional. 3098 3099 Regex translation: 3100 3101 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3102 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3103 Path cannot have any ? or # in it. It is optional. 3104 Query must start with ? and must not have # in it. It is optional. 3105 Anchor must start with # and can have anything else in it to end of string. It is optional. 3106 */ 3107 3108 this = Uri.init; // reset all state 3109 3110 // empty uri = nothing special 3111 if(uri.length == 0) { 3112 return; 3113 } 3114 3115 size_t idx; 3116 3117 scheme_loop: foreach(char c; uri[idx .. $]) { 3118 switch(c) { 3119 case ':': 3120 case '/': 3121 case '?': 3122 case '#': 3123 break scheme_loop; 3124 default: 3125 } 3126 idx++; 3127 } 3128 3129 if(idx == 0 && uri[idx] == ':') { 3130 // this is actually a path! we skip way ahead 3131 goto path_loop; 3132 } 3133 3134 if(idx == uri.length) { 3135 // the whole thing is a path, apparently 3136 path = uri; 3137 return; 3138 } 3139 3140 if(idx > 0 && uri[idx] == ':') { 3141 scheme = uri[0 .. idx]; 3142 idx++; 3143 } else { 3144 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3145 idx = 0; 3146 } 3147 3148 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3149 // we have an authority.... 3150 idx += 2; 3151 3152 auto authority_start = idx; 3153 authority_loop: foreach(char c; uri[idx .. $]) { 3154 switch(c) { 3155 case '/': 3156 case '?': 3157 case '#': 3158 break authority_loop; 3159 default: 3160 } 3161 idx++; 3162 } 3163 3164 auto authority = uri[authority_start .. idx]; 3165 3166 auto idx2 = authority.indexOf("@"); 3167 if(idx2 != -1) { 3168 userinfo = authority[0 .. idx2]; 3169 authority = authority[idx2 + 1 .. $]; 3170 } 3171 3172 if(authority.length && authority[0] == '[') { 3173 // ipv6 address special casing 3174 idx2 = authority.indexOf(']'); 3175 if(idx2 != -1) { 3176 auto end = authority[idx2 + 1 .. $]; 3177 if(end.length && end[0] == ':') 3178 idx2 = idx2 + 1; 3179 else 3180 idx2 = -1; 3181 } 3182 } else { 3183 idx2 = authority.indexOf(":"); 3184 } 3185 3186 if(idx2 == -1) { 3187 port = 0; // 0 means not specified; we should use the default for the scheme 3188 host = authority; 3189 } else { 3190 host = authority[0 .. idx2]; 3191 if(idx2 + 1 < authority.length) 3192 port = to!int(authority[idx2 + 1 .. $]); 3193 else 3194 port = 0; 3195 } 3196 } 3197 3198 path_loop: 3199 auto path_start = idx; 3200 3201 foreach(char c; uri[idx .. $]) { 3202 if(c == '?' || c == '#') 3203 break; 3204 idx++; 3205 } 3206 3207 path = uri[path_start .. idx]; 3208 3209 if(idx == uri.length) 3210 return; // nothing more to examine... 3211 3212 if(uri[idx] == '?') { 3213 idx++; 3214 auto query_start = idx; 3215 foreach(char c; uri[idx .. $]) { 3216 if(c == '#') 3217 break; 3218 idx++; 3219 } 3220 query = uri[query_start .. idx]; 3221 } 3222 3223 if(idx < uri.length && uri[idx] == '#') { 3224 idx++; 3225 fragment = uri[idx .. $]; 3226 } 3227 3228 // uriInvalidated = false; 3229 } 3230 3231 private string rebuildUri() const { 3232 string ret; 3233 if(scheme.length) 3234 ret ~= scheme ~ ":"; 3235 if(userinfo.length || host.length) 3236 ret ~= "//"; 3237 if(userinfo.length) 3238 ret ~= userinfo ~ "@"; 3239 if(host.length) 3240 ret ~= host; 3241 if(port) 3242 ret ~= ":" ~ to!string(port); 3243 3244 ret ~= path; 3245 3246 if(query.length) 3247 ret ~= "?" ~ query; 3248 3249 if(fragment.length) 3250 ret ~= "#" ~ fragment; 3251 3252 // uri = ret; 3253 // uriInvalidated = false; 3254 return ret; 3255 } 3256 3257 /// Converts the broken down parts back into a complete string 3258 string toString() const { 3259 // if(uriInvalidated) 3260 return rebuildUri(); 3261 } 3262 3263 /// Returns a new absolute Uri given a base. It treats this one as 3264 /// relative where possible, but absolute if not. (If protocol, domain, or 3265 /// other info is not set, the new one inherits it from the base.) 3266 /// 3267 /// Browsers use a function like this to figure out links in html. 3268 Uri basedOn(in Uri baseUrl) const { 3269 Uri n = this; // copies 3270 if(n.scheme == "data") 3271 return n; 3272 // n.uriInvalidated = true; // make sure we regenerate... 3273 3274 // userinfo is not inherited... is this wrong? 3275 3276 // if anything is given in the existing url, we don't use the base anymore. 3277 if(n.scheme.empty) { 3278 n.scheme = baseUrl.scheme; 3279 if(n.host.empty) { 3280 n.host = baseUrl.host; 3281 if(n.port == 0) { 3282 n.port = baseUrl.port; 3283 if(n.path.length > 0 && n.path[0] != '/') { 3284 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3285 if(b.length == 0) 3286 b = "/"; 3287 n.path = b ~ n.path; 3288 } else if(n.path.length == 0) { 3289 n.path = baseUrl.path; 3290 } 3291 } 3292 } 3293 } 3294 3295 n.removeDots(); 3296 3297 return n; 3298 } 3299 3300 void removeDots() { 3301 auto parts = this.path.split("/"); 3302 string[] toKeep; 3303 foreach(part; parts) { 3304 if(part == ".") { 3305 continue; 3306 } else if(part == "..") { 3307 //if(toKeep.length > 1) 3308 toKeep = toKeep[0 .. $-1]; 3309 //else 3310 //toKeep = [""]; 3311 continue; 3312 } else { 3313 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3314 //continue; // skip a `//` situation 3315 toKeep ~= part; 3316 } 3317 } 3318 3319 auto path = toKeep.join("/"); 3320 if(path.length && path[0] != '/') 3321 path = "/" ~ path; 3322 3323 this.path = path; 3324 } 3325 3326 unittest { 3327 auto uri = Uri("test.html"); 3328 assert(uri.path == "test.html"); 3329 uri = Uri("path/1/lol"); 3330 assert(uri.path == "path/1/lol"); 3331 uri = Uri("http://me@example.com"); 3332 assert(uri.scheme == "http"); 3333 assert(uri.userinfo == "me"); 3334 assert(uri.host == "example.com"); 3335 uri = Uri("http://example.com/#a"); 3336 assert(uri.scheme == "http"); 3337 assert(uri.host == "example.com"); 3338 assert(uri.fragment == "a"); 3339 uri = Uri("#foo"); 3340 assert(uri.fragment == "foo"); 3341 uri = Uri("?lol"); 3342 assert(uri.query == "lol"); 3343 uri = Uri("#foo?lol"); 3344 assert(uri.fragment == "foo?lol"); 3345 uri = Uri("?lol#foo"); 3346 assert(uri.fragment == "foo"); 3347 assert(uri.query == "lol"); 3348 3349 uri = Uri("http://127.0.0.1/"); 3350 assert(uri.host == "127.0.0.1"); 3351 assert(uri.port == 0); 3352 3353 uri = Uri("http://127.0.0.1:123/"); 3354 assert(uri.host == "127.0.0.1"); 3355 assert(uri.port == 123); 3356 3357 uri = Uri("http://[ff:ff::0]/"); 3358 assert(uri.host == "[ff:ff::0]"); 3359 3360 uri = Uri("http://[ff:ff::0]:123/"); 3361 assert(uri.host == "[ff:ff::0]"); 3362 assert(uri.port == 123); 3363 } 3364 3365 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3366 // the possibilities. 3367 unittest { 3368 auto url = Uri("cool.html"); // checking relative links 3369 3370 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3371 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3372 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3373 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3374 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3375 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3376 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3377 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3378 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3379 3380 url = Uri("/something/cool.html"); // same server, different path 3381 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3382 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3383 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3384 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3385 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3386 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3387 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3388 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3389 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3390 3391 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3392 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3393 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3394 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3395 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3396 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3397 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3398 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3399 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3400 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3401 3402 url = Uri("/test/bar"); 3403 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3404 assert(Uri("../").basedOn(url) == "/"); 3405 3406 url = Uri("http://example.com/"); 3407 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3408 3409 //auto uriBefore = url; 3410 url = Uri("#anchor"); // everything should remain the same except the anchor 3411 //uriBefore.anchor = "anchor"); 3412 //assert(url == uriBefore); 3413 3414 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3415 3416 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3417 3418 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3419 3420 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3421 3422 // FIXME: add something for port too 3423 } 3424 3425 // these are like javascript's location.search and location.hash 3426 string search() const { 3427 return query.length ? ("?" ~ query) : ""; 3428 } 3429 string hash() const { 3430 return fragment.length ? ("#" ~ fragment) : ""; 3431 } 3432 } 3433 3434 3435 /* 3436 for session, see web.d 3437 */ 3438 3439 /// breaks down a url encoded string 3440 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3441 auto vars = data.split(separator); 3442 string[][string] _get; 3443 foreach(var; vars) { 3444 auto equal = var.indexOf("="); 3445 string name; 3446 string value; 3447 if(equal == -1) { 3448 name = decodeComponent(var); 3449 value = ""; 3450 } else { 3451 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3452 // stupid + -> space conversion. 3453 name = decodeComponent(var[0..equal].replace("+", " ")); 3454 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3455 } 3456 3457 _get[name] ~= value; 3458 if(namesInOrder) 3459 (*namesInOrder) ~= name; 3460 if(valuesInOrder) 3461 (*valuesInOrder) ~= value; 3462 } 3463 return _get; 3464 } 3465 3466 /// breaks down a url encoded string, but only returns the last value of any array 3467 string[string] decodeVariablesSingle(string data) { 3468 string[string] va; 3469 auto varArray = decodeVariables(data); 3470 foreach(k, v; varArray) 3471 va[k] = v[$-1]; 3472 3473 return va; 3474 } 3475 3476 /// url encodes the whole string 3477 string encodeVariables(in string[string] data) { 3478 string ret; 3479 3480 bool outputted = false; 3481 foreach(k, v; data) { 3482 if(outputted) 3483 ret ~= "&"; 3484 else 3485 outputted = true; 3486 3487 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3488 } 3489 3490 return ret; 3491 } 3492 3493 /// url encodes a whole string 3494 string encodeVariables(in string[][string] data) { 3495 string ret; 3496 3497 bool outputted = false; 3498 foreach(k, arr; data) { 3499 foreach(v; arr) { 3500 if(outputted) 3501 ret ~= "&"; 3502 else 3503 outputted = true; 3504 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3505 } 3506 } 3507 3508 return ret; 3509 } 3510 3511 /// Encodes all but the explicitly unreserved characters per rfc 3986 3512 /// Alphanumeric and -_.~ are the only ones left unencoded 3513 /// name is borrowed from php 3514 string rawurlencode(in char[] data) { 3515 string ret; 3516 ret.reserve(data.length * 2); 3517 foreach(char c; data) { 3518 if( 3519 (c >= 'a' && c <= 'z') || 3520 (c >= 'A' && c <= 'Z') || 3521 (c >= '0' && c <= '9') || 3522 c == '-' || c == '_' || c == '.' || c == '~') 3523 { 3524 ret ~= c; 3525 } else { 3526 ret ~= '%'; 3527 // since we iterate on char, this should give us the octets of the full utf8 string 3528 ret ~= toHexUpper(c); 3529 } 3530 } 3531 3532 return ret; 3533 } 3534 3535 3536 // http helper functions 3537 3538 // for chunked responses (which embedded http does whenever possible) 3539 version(none) // this is moved up above to avoid making a copy of the data 3540 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3541 const(ubyte)[] ret; 3542 3543 ret = cast(const(ubyte)[]) toHex(data.length); 3544 ret ~= cast(const(ubyte)[]) "\r\n"; 3545 ret ~= data; 3546 ret ~= cast(const(ubyte)[]) "\r\n"; 3547 3548 return ret; 3549 } 3550 3551 string toHex(long num) { 3552 string ret; 3553 while(num) { 3554 int v = num % 16; 3555 num /= 16; 3556 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3557 ret ~= d; 3558 } 3559 3560 return to!string(array(ret.retro)); 3561 } 3562 3563 string toHexUpper(long num) { 3564 string ret; 3565 while(num) { 3566 int v = num % 16; 3567 num /= 16; 3568 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3569 ret ~= d; 3570 } 3571 3572 if(ret.length == 1) 3573 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3574 3575 return to!string(array(ret.retro)); 3576 } 3577 3578 3579 // the generic mixins 3580 3581 /++ 3582 Use this instead of writing your own main 3583 3584 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3585 +/ 3586 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3587 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3588 } 3589 3590 /++ 3591 Boilerplate mixin for a main function that uses the [dispatcher] function. 3592 3593 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3594 3595 History: 3596 Added July 9, 2021 3597 +/ 3598 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3599 /// forwards to [CustomCgiDispatcherMain] with default args 3600 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3601 } 3602 3603 /// ditto 3604 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3605 class GenericPresenter : WebPresenter!GenericPresenter {} 3606 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3607 } 3608 3609 /++ 3610 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3611 3612 History: 3613 Added May 13, 2023 (dub v11.0) 3614 +/ 3615 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3616 /++ 3617 Handler to the generated presenter you can use from your objects, etc. 3618 +/ 3619 Presenter activePresenter; 3620 3621 /++ 3622 Request handler that creates the presenter then forwards to the [dispatcher] function. 3623 Renders 404 if the dispatcher did not handle the request. 3624 3625 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3626 +/ 3627 void handler(Cgi cgi) { 3628 auto presenter = new Presenter; 3629 activePresenter = presenter; 3630 scope(exit) activePresenter = null; 3631 3632 if(cgi.dispatcher!DispatcherArgs(presenter)) 3633 return; 3634 3635 switch(cgi.pathInfo) { 3636 case "/style.css": 3637 cgi.setCache(true); 3638 cgi.setResponseContentType("text/css"); 3639 cgi.write(presenter.style(), true); 3640 break; 3641 case "/script.js": 3642 cgi.setCache(true); 3643 cgi.setResponseContentType("application/javascript"); 3644 cgi.write(presenter.script(), true); 3645 break; 3646 default: 3647 presenter.renderBasicError(cgi, 404); 3648 } 3649 } 3650 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3651 } 3652 3653 /// ditto 3654 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3655 class GenericPresenter : WebPresenter!GenericPresenter {} 3656 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3657 3658 } 3659 3660 private string simpleHtmlEncode(string s) { 3661 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3662 } 3663 3664 string messageFromException(Throwable t) { 3665 string message; 3666 if(t !is null) { 3667 debug message = t.toString(); 3668 else message = "An unexpected error has occurred."; 3669 } else { 3670 message = "Unknown error"; 3671 } 3672 return message; 3673 } 3674 3675 string plainHttpError(bool isCgi, string type, Throwable t) { 3676 auto message = messageFromException(t); 3677 message = simpleHtmlEncode(message); 3678 3679 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3680 isCgi ? "Status:" : "HTTP/1.1", 3681 type, message.length, message); 3682 } 3683 3684 // returns true if we were able to recover reasonably 3685 bool handleException(Cgi cgi, Throwable t) { 3686 if(cgi.isClosed) { 3687 // if the channel has been explicitly closed, we can't handle it here 3688 return true; 3689 } 3690 3691 if(cgi.outputtedResponseData) { 3692 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3693 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3694 } else { 3695 // no headers are sent, we can send a full blown error and recover 3696 cgi.setCache(false); 3697 cgi.setResponseContentType("text/html"); 3698 cgi.setResponseLocation(null); // cancel the redirect 3699 cgi.setResponseStatus("500 Internal Server Error"); 3700 cgi.write(simpleHtmlEncode(messageFromException(t))); 3701 cgi.close(); 3702 return true; 3703 } 3704 } 3705 3706 bool isCgiRequestMethod(string s) { 3707 s = s.toUpper(); 3708 if(s == "COMMANDLINE") 3709 return true; 3710 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3711 if(s == member) 3712 return true; 3713 return false; 3714 } 3715 3716 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3717 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3718 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3719 void main(string[] args) { 3720 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3721 } 3722 } 3723 3724 version(embedded_httpd_processes) 3725 __gshared int processPoolSize = 8; 3726 3727 // Returns true if run. You should exit the program after that. 3728 bool tryAddonServers(string[] args) { 3729 if(args.length > 1) { 3730 // run the special separate processes if needed 3731 switch(args[1]) { 3732 case "--websocket-server": 3733 version(with_addon_servers) 3734 websocketServers[args[2]](args[3 .. $]); 3735 else 3736 printf("Add-on servers not compiled in.\n"); 3737 return true; 3738 case "--websocket-servers": 3739 import core.demangle; 3740 version(with_addon_servers_connections) 3741 foreach(k, v; websocketServers) 3742 writeln(k, "\t", demangle(k)); 3743 return true; 3744 case "--session-server": 3745 version(with_addon_servers) 3746 runSessionServer(); 3747 else 3748 printf("Add-on servers not compiled in.\n"); 3749 return true; 3750 case "--event-server": 3751 version(with_addon_servers) 3752 runEventServer(); 3753 else 3754 printf("Add-on servers not compiled in.\n"); 3755 return true; 3756 case "--timer-server": 3757 version(with_addon_servers) 3758 runTimerServer(); 3759 else 3760 printf("Add-on servers not compiled in.\n"); 3761 return true; 3762 case "--timed-jobs": 3763 import core.demangle; 3764 version(with_addon_servers_connections) 3765 foreach(k, v; scheduledJobHandlers) 3766 writeln(k, "\t", demangle(k)); 3767 return true; 3768 case "--timed-job": 3769 scheduledJobHandlers[args[2]](args[3 .. $]); 3770 return true; 3771 default: 3772 // intentionally blank - do nothing and carry on to run normally 3773 } 3774 } 3775 return false; 3776 } 3777 3778 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3779 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3780 // we support command line thing for easy testing everywhere 3781 // it needs to be called ./app method uri [other args...] 3782 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3783 Cgi cgi = new CustomCgi(args); 3784 scope(exit) cgi.dispose(); 3785 try { 3786 fun(cgi); 3787 cgi.close(); 3788 } catch(AuthorizationRequiredException are) { 3789 cgi.setResponseStatus("401 Authorization Required"); 3790 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 3791 cgi.close(); 3792 } 3793 return true; 3794 } 3795 return false; 3796 } 3797 3798 /++ 3799 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3800 3801 As of version 11 (released August 2023), you can also make things like this: 3802 3803 --- 3804 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3805 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3806 3807 // can also: 3808 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3809 3810 // NOT IMPLEMENTED YET 3811 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3812 3813 foreach(listenSpec; server.listenSpecs) { 3814 // you can check what it actually bound to here and see your assigned ports 3815 } 3816 3817 // NOT IMPLEMENTED YET 3818 // server.start!handler(); // starts and runs in the arsd.core event loop 3819 3820 server.serve!handler(); // blocks the thread until the server exits 3821 --- 3822 3823 History: 3824 Added Sept 26, 2020 (release version 8.5). 3825 3826 The `listenSpec` member was added July 31, 2023. 3827 +/ 3828 struct RequestServer { 3829 /++ 3830 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3831 +/ 3832 string listeningHost = defaultListeningHost(); 3833 /// ditto 3834 ushort listeningPort = defaultListeningPort(); 3835 3836 static struct ListenSpec { 3837 enum Protocol { 3838 http, 3839 https, 3840 scgi 3841 } 3842 Protocol protocol; 3843 3844 enum AddressType { 3845 ip, 3846 unix, 3847 abstract_ 3848 } 3849 AddressType addressType; 3850 3851 string address; 3852 ushort port; 3853 } 3854 3855 /++ 3856 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3857 3858 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3859 3860 `http://localhost:8080` 3861 3862 `http://unix:filename/here` 3863 3864 `scgi://abstract:/name/here` 3865 3866 `http://[::1]:4444` 3867 3868 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3869 3870 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3871 3872 `localhost:8080` serves the default protocol. 3873 3874 `8080` or `:8080` assumes default protocol on localhost. 3875 3876 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3877 3878 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3879 3880 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3881 3882 $(PITFALL 3883 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3884 ) 3885 3886 Bugs: 3887 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3888 3889 History: 3890 Added July 31, 2023 (dub v11.0) 3891 +/ 3892 string[] listenSpec; 3893 3894 /++ 3895 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3896 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3897 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3898 [stop] may not work as well. 3899 3900 History: 3901 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3902 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3903 compatibility. 3904 +/ 3905 bool useFork = cgi_use_fork_default; 3906 3907 /++ 3908 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3909 default based on the number of cpus modified by the server mode. 3910 3911 History: 3912 Added August 12, 2022 (dub v10.9) 3913 +/ 3914 int numberOfThreads = 0; 3915 3916 /++ 3917 Creates a server configured to listen to multiple URLs. 3918 3919 History: 3920 Added July 31, 2023 (dub v11.0) 3921 +/ 3922 this(string[] listenTo) { 3923 this.listenSpec = listenTo; 3924 } 3925 3926 /// Creates a server object configured to listen on a single host and port. 3927 this(string defaultHost, ushort defaultPort) { 3928 this.listeningHost = defaultHost; 3929 this.listeningPort = defaultPort; 3930 } 3931 3932 /// ditto 3933 this(ushort defaultPort) { 3934 listeningPort = defaultPort; 3935 } 3936 3937 /++ 3938 Reads the command line arguments into the values here. 3939 3940 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3941 3942 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 3943 +/ 3944 void configureFromCommandLine(string[] args) { 3945 bool portOrHostFound = false; 3946 3947 bool foundPort = false; 3948 bool foundHost = false; 3949 bool foundUid = false; 3950 bool foundGid = false; 3951 bool foundListen = false; 3952 foreach(arg; args) { 3953 if(foundPort) { 3954 listeningPort = to!ushort(arg); 3955 portOrHostFound = true; 3956 foundPort = false; 3957 continue; 3958 } 3959 if(foundHost) { 3960 listeningHost = arg; 3961 portOrHostFound = true; 3962 foundHost = false; 3963 continue; 3964 } 3965 if(foundUid) { 3966 privilegesDropToUid = to!uid_t(arg); 3967 foundUid = false; 3968 continue; 3969 } 3970 if(foundGid) { 3971 privilegesDropToGid = to!gid_t(arg); 3972 foundGid = false; 3973 continue; 3974 } 3975 if(foundListen) { 3976 this.listenSpec ~= arg; 3977 foundListen = false; 3978 continue; 3979 } 3980 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3981 foundHost = true; 3982 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3983 foundPort = true; 3984 else if(arg == "--uid") 3985 foundUid = true; 3986 else if(arg == "--gid") 3987 foundGid = true; 3988 else if(arg == "--listen") 3989 foundListen = true; 3990 } 3991 3992 if(portOrHostFound && listenSpec.length) { 3993 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 3994 } 3995 } 3996 3997 version(Windows) { 3998 private alias uid_t = int; 3999 private alias gid_t = int; 4000 } 4001 4002 /// user (uid) to drop privileges to 4003 /// 0 … do nothing 4004 uid_t privilegesDropToUid = 0; 4005 /// group (gid) to drop privileges to 4006 /// 0 … do nothing 4007 gid_t privilegesDropToGid = 0; 4008 4009 private void dropPrivileges() { 4010 version(Posix) { 4011 import core.sys.posix.unistd; 4012 4013 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 4014 throw new Exception("Dropping privileges via setgid() failed."); 4015 4016 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 4017 throw new Exception("Dropping privileges via setuid() failed."); 4018 } 4019 else { 4020 // FIXME: Windows? 4021 //pragma(msg, "Dropping privileges is not implemented for this platform"); 4022 } 4023 4024 // done, set zero 4025 privilegesDropToGid = 0; 4026 privilegesDropToUid = 0; 4027 } 4028 4029 /++ 4030 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 4031 4032 History: 4033 Added Oct 10, 2020. 4034 Example: 4035 4036 --- 4037 import arsd.cgi; 4038 void main() { 4039 RequestServer server = RequestServer("127.0.0.1", 6789); 4040 string oauthCode; 4041 string oauthScope; 4042 server.serveHttpOnce!((cgi) { 4043 oauthCode = cgi.request("code"); 4044 oauthScope = cgi.request("scope"); 4045 cgi.write("Thank you, please return to the application."); 4046 }); 4047 // use the code and scope given 4048 } 4049 --- 4050 +/ 4051 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4052 import std.socket; 4053 4054 bool tcp; 4055 void delegate() cleanup; 4056 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4057 auto connection = socket.accept(); 4058 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4059 4060 if(cleanup) 4061 cleanup(); 4062 } 4063 4064 /++ 4065 Starts serving requests according to the current configuration. 4066 +/ 4067 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4068 version(netman_httpd) { 4069 // Obsolete! 4070 4071 import arsd.httpd; 4072 // what about forwarding the other constructor args? 4073 // this probably needs a whole redoing... 4074 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4075 return; 4076 } else 4077 version(embedded_httpd_processes) { 4078 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4079 } else 4080 version(embedded_httpd_threads) { 4081 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4082 } else 4083 version(scgi) { 4084 serveScgi!(fun, CustomCgi, maxContentLength)(); 4085 } else 4086 version(fastcgi) { 4087 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4088 } else 4089 version(stdio_http) { 4090 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4091 } else { 4092 //version=plain_cgi; 4093 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4094 } 4095 } 4096 4097 /++ 4098 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4099 4100 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4101 +/ 4102 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4103 globalStopFlag = false; 4104 static if(__traits(isStaticFunction, fun)) 4105 alias funToUse = fun; 4106 else 4107 void funToUse(CustomCgi cgi) { 4108 static if(__VERSION__ > 2097) 4109 __traits(child, _this, fun)(cgi); 4110 else static assert(0, "Not implemented in your compiler version!"); 4111 } 4112 auto manager = this.listenSpec is null ? 4113 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4114 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4115 manager.listen(); 4116 } 4117 4118 /++ 4119 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4120 +/ 4121 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4122 globalStopFlag = false; 4123 auto manager = this.listenSpec is null ? 4124 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4125 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4126 manager.listen(); 4127 } 4128 4129 /++ 4130 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4131 4132 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4133 4134 History: 4135 Added May 29, 2021 4136 +/ 4137 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4138 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4139 } 4140 4141 /++ 4142 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4143 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4144 +/ 4145 enum ForceStop { 4146 /++ 4147 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4148 +/ 4149 afterQueuedRequestsComplete, 4150 /++ 4151 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4152 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4153 +/ 4154 afterCurrentRequestsComplete, 4155 /++ 4156 Partial response writes will throw an exception, cancelling any streaming response, but complete 4157 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4158 +/ 4159 cancelStreamingRequestsEarly, 4160 /++ 4161 All writes will throw. 4162 +/ 4163 cancelAllRequestsEarly, 4164 /++ 4165 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4166 +/ 4167 forciblyTerminate, 4168 } 4169 4170 version(embedded_httpd_processes) {} else 4171 /++ 4172 Stops serving after the current requests are completed. 4173 4174 Bugs: 4175 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4176 on Windows however). Only partially implemented on non-Linux posix systems. 4177 4178 You might also try SIGINT perhaps. 4179 4180 The stopPriority is not yet fully implemented. 4181 +/ 4182 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4183 globalStopFlag = true; 4184 4185 version(Posix) { 4186 if(cancelfd > 0) { 4187 ulong a = 1; 4188 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4189 } 4190 } 4191 version(Windows) { 4192 if(iocp) { 4193 foreach(i; 0 .. 16) // FIXME 4194 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4195 } 4196 } 4197 } 4198 } 4199 4200 class AuthorizationRequiredException : Exception { 4201 string type; 4202 string realm; 4203 this(string type, string realm, string file, size_t line) { 4204 this.type = type; 4205 this.realm = realm; 4206 4207 super("Authorization Required", file, line); 4208 } 4209 } 4210 4211 private alias AliasSeq(T...) = T; 4212 4213 version(with_breaking_cgi_features) 4214 mixin(q{ 4215 template ThisFor(alias t) { 4216 static if(__traits(isStaticFunction, t)) { 4217 alias ThisFor = AliasSeq!(); 4218 } else { 4219 alias ThisFor = __traits(parent, t); 4220 } 4221 } 4222 }); 4223 else 4224 alias ThisFor(alias t) = AliasSeq!(); 4225 4226 private __gshared bool globalStopFlag = false; 4227 4228 version(embedded_httpd_processes) 4229 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4230 import core.sys.posix.unistd; 4231 import core.sys.posix.sys.socket; 4232 import core.sys.posix.netinet.in_; 4233 //import std.c.linux.socket; 4234 4235 int sock = socket(AF_INET, SOCK_STREAM, 0); 4236 if(sock == -1) 4237 throw new Exception("socket"); 4238 4239 cloexec(sock); 4240 4241 { 4242 4243 sockaddr_in addr; 4244 addr.sin_family = AF_INET; 4245 addr.sin_port = htons(params.listeningPort); 4246 auto lh = params.listeningHost; 4247 if(lh.length) { 4248 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4249 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4250 } else 4251 addr.sin_addr.s_addr = INADDR_ANY; 4252 4253 // HACKISH 4254 int on = 1; 4255 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4256 // end hack 4257 4258 4259 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4260 close(sock); 4261 throw new Exception("bind"); 4262 } 4263 4264 // FIXME: if this queue is full, it will just ignore it 4265 // and wait for the client to retransmit it. This is an 4266 // obnoxious timeout condition there. 4267 if(sock.listen(128) == -1) { 4268 close(sock); 4269 throw new Exception("listen"); 4270 } 4271 params.dropPrivileges(); 4272 } 4273 4274 version(embedded_httpd_processes_accept_after_fork) {} else { 4275 int pipeReadFd; 4276 int pipeWriteFd; 4277 4278 { 4279 int[2] pipeFd; 4280 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4281 import core.stdc.errno; 4282 throw new Exception("pipe failed " ~ to!string(errno)); 4283 } 4284 4285 pipeReadFd = pipeFd[0]; 4286 pipeWriteFd = pipeFd[1]; 4287 } 4288 } 4289 4290 4291 int processCount; 4292 pid_t newPid; 4293 reopen: 4294 while(processCount < processPoolSize) { 4295 newPid = fork(); 4296 if(newPid == 0) { 4297 // start serving on the socket 4298 //ubyte[4096] backingBuffer; 4299 for(;;) { 4300 bool closeConnection; 4301 uint i; 4302 sockaddr addr; 4303 i = addr.sizeof; 4304 version(embedded_httpd_processes_accept_after_fork) { 4305 int s = accept(sock, &addr, &i); 4306 int opt = 1; 4307 import core.sys.posix.netinet.tcp; 4308 // the Cgi class does internal buffering, so disabling this 4309 // helps with latency in many cases... 4310 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4311 cloexec(s); 4312 } else { 4313 int s; 4314 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4315 if(readret != s.sizeof) { 4316 import core.stdc.errno; 4317 throw new Exception("pipe read failed " ~ to!string(errno)); 4318 } 4319 4320 //writeln("process ", getpid(), " got socket ", s); 4321 } 4322 4323 try { 4324 4325 if(s == -1) 4326 throw new Exception("accept"); 4327 4328 scope(failure) close(s); 4329 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4330 auto ir = new BufferedInputRange(s); 4331 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4332 4333 while(!ir.empty) { 4334 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4335 4336 Cgi cgi; 4337 try { 4338 cgi = new CustomCgi(ir, &closeConnection); 4339 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4340 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4341 if(processPoolSize <= 1) 4342 closeConnection = true; 4343 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4344 } catch(HttpVersionNotSupportedException he) { 4345 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4346 closeConnection = true; 4347 break; 4348 } catch(Throwable t) { 4349 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4350 // anyway let's kill the connection 4351 version(CRuntime_Musl) { 4352 // LockingTextWriter fails here 4353 // so working around it 4354 auto estr = t.toString(); 4355 stderr.rawWrite(estr); 4356 stderr.rawWrite("\n"); 4357 } else 4358 stderr.writeln(t.toString()); 4359 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4360 closeConnection = true; 4361 break; 4362 } 4363 assert(cgi !is null); 4364 scope(exit) 4365 cgi.dispose(); 4366 4367 try { 4368 fun(cgi); 4369 cgi.close(); 4370 if(cgi.websocketMode) 4371 closeConnection = true; 4372 4373 } catch(AuthorizationRequiredException are) { 4374 cgi.setResponseStatus("401 Authorization Required"); 4375 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4376 cgi.close(); 4377 } catch(ConnectionException ce) { 4378 closeConnection = true; 4379 } catch(Throwable t) { 4380 // a processing error can be recovered from 4381 version(CRuntime_Musl) { 4382 // LockingTextWriter fails here 4383 // so working around it 4384 auto estr = t.toString(); 4385 stderr.rawWrite(estr); 4386 } else { 4387 stderr.writeln(t.toString); 4388 } 4389 if(!handleException(cgi, t)) 4390 closeConnection = true; 4391 } 4392 4393 if(closeConnection) { 4394 ir.source.close(); 4395 break; 4396 } else { 4397 if(!ir.empty) 4398 ir.popFront(); // get the next 4399 else if(ir.sourceClosed) { 4400 ir.source.close(); 4401 } 4402 } 4403 } 4404 4405 ir.source.close(); 4406 } catch(Throwable t) { 4407 version(CRuntime_Musl) {} else 4408 debug writeln(t); 4409 // most likely cause is a timeout 4410 } 4411 } 4412 } else if(newPid < 0) { 4413 throw new Exception("fork failed"); 4414 } else { 4415 processCount++; 4416 } 4417 } 4418 4419 // the parent should wait for its children... 4420 if(newPid) { 4421 import core.sys.posix.sys.wait; 4422 4423 version(embedded_httpd_processes_accept_after_fork) {} else { 4424 import core.sys.posix.sys.select; 4425 int[] fdQueue; 4426 while(true) { 4427 // writeln("select call"); 4428 int nfds = pipeWriteFd; 4429 if(sock > pipeWriteFd) 4430 nfds = sock; 4431 nfds += 1; 4432 fd_set read_fds; 4433 fd_set write_fds; 4434 FD_ZERO(&read_fds); 4435 FD_ZERO(&write_fds); 4436 FD_SET(sock, &read_fds); 4437 if(fdQueue.length) 4438 FD_SET(pipeWriteFd, &write_fds); 4439 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4440 if(ret == -1) { 4441 import core.stdc.errno; 4442 if(errno == EINTR) 4443 goto try_wait; 4444 else 4445 throw new Exception("wtf select"); 4446 } 4447 4448 int s = -1; 4449 if(FD_ISSET(sock, &read_fds)) { 4450 uint i; 4451 sockaddr addr; 4452 i = addr.sizeof; 4453 s = accept(sock, &addr, &i); 4454 cloexec(s); 4455 import core.sys.posix.netinet.tcp; 4456 int opt = 1; 4457 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4458 } 4459 4460 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4461 if(s == -1 && fdQueue.length) { 4462 s = fdQueue[0]; 4463 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4464 } 4465 write_fd(pipeWriteFd, &s, s.sizeof, s); 4466 close(s); // we are done with it, let the other process take ownership 4467 } else 4468 fdQueue ~= s; 4469 } 4470 } 4471 4472 try_wait: 4473 4474 int status; 4475 while(-1 != wait(&status)) { 4476 version(CRuntime_Musl) {} else { 4477 import std.stdio; writeln("Process died ", status); 4478 } 4479 processCount--; 4480 goto reopen; 4481 } 4482 close(sock); 4483 } 4484 } 4485 4486 version(fastcgi) 4487 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4488 // SetHandler fcgid-script 4489 FCGX_Stream* input, output, error; 4490 FCGX_ParamArray env; 4491 4492 4493 4494 const(ubyte)[] getFcgiChunk() { 4495 const(ubyte)[] ret; 4496 while(FCGX_HasSeenEOF(input) != -1) 4497 ret ~= cast(ubyte) FCGX_GetChar(input); 4498 return ret; 4499 } 4500 4501 void writeFcgi(const(ubyte)[] data) { 4502 FCGX_PutStr(data.ptr, data.length, output); 4503 } 4504 4505 void doARequest() { 4506 string[string] fcgienv; 4507 4508 for(auto e = env; e !is null && *e !is null; e++) { 4509 string cur = to!string(*e); 4510 auto idx = cur.indexOf("="); 4511 string name, value; 4512 if(idx == -1) 4513 name = cur; 4514 else { 4515 name = cur[0 .. idx]; 4516 value = cur[idx + 1 .. $]; 4517 } 4518 4519 fcgienv[name] = value; 4520 } 4521 4522 void flushFcgi() { 4523 FCGX_FFlush(output); 4524 } 4525 4526 Cgi cgi; 4527 try { 4528 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4529 } catch(Throwable t) { 4530 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4531 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4532 return; //continue; 4533 } 4534 assert(cgi !is null); 4535 scope(exit) cgi.dispose(); 4536 try { 4537 fun(cgi); 4538 cgi.close(); 4539 } catch(AuthorizationRequiredException are) { 4540 cgi.setResponseStatus("401 Authorization Required"); 4541 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4542 cgi.close(); 4543 } catch(Throwable t) { 4544 // log it to the error stream 4545 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4546 // handle it for the user, if we can 4547 if(!handleException(cgi, t)) 4548 return; // continue; 4549 } 4550 } 4551 4552 auto lp = params.listeningPort; 4553 auto host = params.listeningHost; 4554 4555 FCGX_Request request; 4556 if(lp || !host.empty) { 4557 // if a listening port was specified on the command line, we want to spawn ourself 4558 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4559 FCGX_Init(); 4560 4561 int sock; 4562 4563 if(host.startsWith("unix:")) { 4564 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4565 } else if(host.startsWith("abstract:")) { 4566 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4567 } else { 4568 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4569 } 4570 4571 if(sock < 0) 4572 throw new Exception("Couldn't listen on the port"); 4573 FCGX_InitRequest(&request, sock, 0); 4574 while(FCGX_Accept_r(&request) >= 0) { 4575 input = request.inStream; 4576 output = request.outStream; 4577 error = request.errStream; 4578 env = request.envp; 4579 doARequest(); 4580 } 4581 } else { 4582 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4583 // using the version with a global variable since we are separate processes anyway 4584 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4585 doARequest(); 4586 } 4587 } 4588 } 4589 4590 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4591 ushort defaultListeningPort() { 4592 version(netman_httpd) 4593 return 8080; 4594 else version(embedded_httpd_processes) 4595 return 8085; 4596 else version(embedded_httpd_threads) 4597 return 8085; 4598 else version(scgi) 4599 return 4000; 4600 else 4601 return 0; 4602 } 4603 4604 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4605 string defaultListeningHost() { 4606 version(netman_httpd) 4607 return null; 4608 else version(embedded_httpd_processes) 4609 return null; 4610 else version(embedded_httpd_threads) 4611 return null; 4612 else version(scgi) 4613 return "127.0.0.1"; 4614 else 4615 return null; 4616 4617 } 4618 4619 /++ 4620 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4621 4622 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4623 4624 Params: 4625 fun = Your request handler 4626 CustomCgi = a subclass of Cgi, if you wise to customize it further 4627 maxContentLength = max POST size you want to allow 4628 args = command-line arguments 4629 4630 History: 4631 Documented Sept 26, 2020. 4632 +/ 4633 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4634 if(tryAddonServers(args)) 4635 return; 4636 4637 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4638 return; 4639 4640 RequestServer server; 4641 // you can change the port here if you like 4642 // server.listeningPort = 9000; 4643 4644 // then call this to let the command line args override your default 4645 server.configureFromCommandLine(args); 4646 4647 // and serve the request(s). 4648 server.serve!(fun, CustomCgi, maxContentLength)(); 4649 } 4650 4651 //version(plain_cgi) 4652 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4653 // standard CGI is the default version 4654 4655 4656 // Set stdin to binary mode if necessary to avoid mangled newlines 4657 // the fact that stdin is global means this could be trouble but standard cgi request 4658 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4659 version(Windows) { 4660 version(Win64) 4661 _setmode(std.stdio.stdin.fileno(), 0x8000); 4662 else 4663 setmode(std.stdio.stdin.fileno(), 0x8000); 4664 } 4665 4666 Cgi cgi; 4667 try { 4668 cgi = new CustomCgi(maxContentLength); 4669 version(Posix) 4670 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4671 else version(Windows) 4672 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4673 else static assert(0); 4674 } catch(Throwable t) { 4675 version(CRuntime_Musl) { 4676 // LockingTextWriter fails here 4677 // so working around it 4678 auto s = t.toString(); 4679 stderr.rawWrite(s); 4680 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4681 } else { 4682 stderr.writeln(t.msg); 4683 // the real http server will probably handle this; 4684 // most likely, this is a bug in Cgi. But, oh well. 4685 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4686 } 4687 return; 4688 } 4689 assert(cgi !is null); 4690 scope(exit) cgi.dispose(); 4691 4692 try { 4693 fun(cgi); 4694 cgi.close(); 4695 } catch(AuthorizationRequiredException are) { 4696 cgi.setResponseStatus("401 Authorization Required"); 4697 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4698 cgi.close(); 4699 } catch (Throwable t) { 4700 version(CRuntime_Musl) { 4701 // LockingTextWriter fails here 4702 // so working around it 4703 auto s = t.msg; 4704 stderr.rawWrite(s); 4705 } else { 4706 stderr.writeln(t.msg); 4707 } 4708 if(!handleException(cgi, t)) 4709 return; 4710 } 4711 } 4712 4713 private __gshared int cancelfd = -1; 4714 4715 /+ 4716 The event loop for embedded_httpd_threads will prolly fiber dispatch 4717 cgi constructors too, so slow posts will not monopolize a worker thread. 4718 4719 May want to provide the worker task system just need to ensure all the fibers 4720 has a big enough stack for real work... would also ideally like to reuse them. 4721 4722 4723 So prolly bir would switch it to nonblocking. If it would block, it epoll 4724 registers one shot with this existing fiber to take it over. 4725 4726 new connection comes in. it picks a fiber off the free list, 4727 or if there is none, it creates a new one. this fiber handles 4728 this connection the whole time. 4729 4730 epoll triggers the fiber when something comes in. it is called by 4731 a random worker thread, it might change at any time. at least during 4732 the constructor. maybe into the main body it will stay tied to a thread 4733 just so TLS stuff doesn't randomly change in the middle. but I could 4734 specify if you yield all bets are off. 4735 4736 when the request is finished, if there's more data buffered, it just 4737 keeps going. if there is no more data buffered, it epoll ctls to 4738 get triggered when more data comes in. all one shot. 4739 4740 when a connection is closed, the fiber returns and is then reset 4741 and added to the free list. if the free list is full, the fiber is 4742 just freed, this means it will balloon to a certain size but not generally 4743 grow beyond that unless the activity keeps going. 4744 4745 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4746 4747 So the fiber has its own magic methods to read and write. if they would block, it registers 4748 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4749 4750 basically you issue the command and it tells you when it is done 4751 4752 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4753 4754 +/ 4755 4756 /++ 4757 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4758 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4759 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4760 4761 History: 4762 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4763 +/ 4764 version(cgi_use_fiber) 4765 __gshared size_t fiberStackSize = 4096 * 100; 4766 4767 version(cgi_use_fiber) 4768 class CgiFiber : Fiber { 4769 private void function(Socket) f_handler; 4770 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4771 f_handler(s); 4772 } 4773 this(void function(Socket) handler) { 4774 this.f_handler = handler; 4775 this(&f_handler_dg); 4776 } 4777 4778 this(void delegate(Socket) handler) { 4779 this.handler = handler; 4780 super(&run, fiberStackSize); 4781 } 4782 4783 Socket connection; 4784 void delegate(Socket) handler; 4785 4786 void run() { 4787 handler(connection); 4788 } 4789 4790 void delegate() postYield; 4791 4792 private void setPostYield(scope void delegate() py) @nogc { 4793 postYield = cast(void delegate()) py; 4794 } 4795 4796 void proceed() { 4797 try { 4798 call(); 4799 auto py = postYield; 4800 postYield = null; 4801 if(py !is null) 4802 py(); 4803 } catch(Exception e) { 4804 if(connection) 4805 connection.close(); 4806 goto terminate; 4807 } 4808 4809 if(state == State.TERM) { 4810 terminate: 4811 import core.memory; 4812 GC.removeRoot(cast(void*) this); 4813 } 4814 } 4815 } 4816 4817 version(cgi_use_fiber) 4818 version(Windows) { 4819 4820 extern(Windows) private { 4821 4822 import core.sys.windows.mswsock; 4823 4824 alias GROUP=uint; 4825 alias LPWSAPROTOCOL_INFOW = void*; 4826 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4827 alias WSASend = arsd.core.WSASend; 4828 alias WSARecv = arsd.core.WSARecv; 4829 alias WSABUF = arsd.core.WSABUF; 4830 4831 /+ 4832 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4833 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4834 4835 struct WSABUF { 4836 ULONG len; 4837 CHAR *buf; 4838 } 4839 +/ 4840 alias LPWSABUF = WSABUF*; 4841 4842 alias WSAOVERLAPPED = OVERLAPPED; 4843 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4844 /+ 4845 4846 alias LPFN_ACCEPTEX = 4847 BOOL 4848 function( 4849 SOCKET sListenSocket, 4850 SOCKET sAcceptSocket, 4851 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4852 void* lpOutputBuffer, 4853 WORD dwReceiveDataLength, 4854 WORD dwLocalAddressLength, 4855 WORD dwRemoteAddressLength, 4856 LPDWORD lpdwBytesReceived, 4857 LPOVERLAPPED lpOverlapped 4858 ); 4859 4860 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4861 +/ 4862 4863 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4864 } 4865 4866 private class PseudoblockingOverlappedSocket : Socket { 4867 SOCKET handle; 4868 4869 CgiFiber fiber; 4870 4871 this(AddressFamily af, SocketType st) { 4872 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4873 if(!handle) 4874 throw new Exception("WSASocketW"); 4875 this.handle = handle; 4876 4877 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4878 4879 if(iocp is null) { 4880 writeln(GetLastError()); 4881 throw new Exception("CreateIoCompletionPort"); 4882 } 4883 4884 super(cast(socket_t) handle, af); 4885 } 4886 this() pure nothrow @trusted { assert(0); } 4887 4888 override void blocking(bool) {} // meaningless to us, just ignore it. 4889 4890 protected override Socket accepting() pure nothrow { 4891 assert(0); 4892 } 4893 4894 bool addressesParsed; 4895 Address la; 4896 Address ra; 4897 4898 private void populateAddresses() { 4899 if(addressesParsed) 4900 return; 4901 addressesParsed = true; 4902 4903 int lalen, ralen; 4904 4905 sockaddr_in* la; 4906 sockaddr_in* ra; 4907 4908 lpfnGetAcceptExSockaddrs( 4909 scratchBuffer.ptr, 4910 0, // same as in the AcceptEx call! 4911 sockaddr_in.sizeof + 16, 4912 sockaddr_in.sizeof + 16, 4913 cast(sockaddr**) &la, 4914 &lalen, 4915 cast(sockaddr**) &ra, 4916 &ralen 4917 ); 4918 4919 if(la) 4920 this.la = new InternetAddress(*la); 4921 if(ra) 4922 this.ra = new InternetAddress(*ra); 4923 4924 } 4925 4926 override @property @trusted Address localAddress() { 4927 populateAddresses(); 4928 return la; 4929 } 4930 override @property @trusted Address remoteAddress() { 4931 populateAddresses(); 4932 return ra; 4933 } 4934 4935 PseudoblockingOverlappedSocket accepted; 4936 4937 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4938 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4939 4940 override Socket accept() @trusted { 4941 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4942 4943 if(lpfnAcceptEx is null) { 4944 DWORD dwBytes; 4945 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4946 4947 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4948 &GuidAcceptEx, GuidAcceptEx.sizeof, 4949 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4950 &dwBytes, null, null); 4951 4952 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4953 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4954 &GuidAcceptEx, GuidAcceptEx.sizeof, 4955 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4956 &dwBytes, null, null); 4957 4958 } 4959 4960 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4961 accepted = pfa; 4962 4963 SOCKET pendingForAccept = pfa.handle; 4964 DWORD ignored; 4965 4966 auto ret = lpfnAcceptEx(handle, 4967 pendingForAccept, 4968 // buffer to receive up front 4969 pfa.scratchBuffer.ptr, 4970 0, 4971 // size of local and remote addresses. normally + 16. 4972 sockaddr_in.sizeof + 16, 4973 sockaddr_in.sizeof + 16, 4974 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4975 &overlapped 4976 ); 4977 4978 return pfa; 4979 } 4980 4981 override void connect(Address to) { assert(0); } 4982 4983 DWORD lastAnswer; 4984 ubyte[1024] scratchBuffer; 4985 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4986 4987 WSABUF[1] buffer; 4988 OVERLAPPED overlapped; 4989 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4990 overlapped = overlapped.init; 4991 buffer[0].len = cast(DWORD) buf.length; 4992 buffer[0].buf = cast(ubyte*) buf.ptr; 4993 fiber.setPostYield( () { 4994 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4995 if(GetLastError() != 997) { 4996 //throw new Exception("WSASend fail"); 4997 } 4998 } 4999 }); 5000 5001 Fiber.yield(); 5002 return lastAnswer; 5003 } 5004 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 5005 overlapped = overlapped.init; 5006 buffer[0].len = cast(DWORD) buf.length; 5007 buffer[0].buf = cast(ubyte*) buf.ptr; 5008 5009 DWORD flags2 = 0; 5010 5011 fiber.setPostYield(() { 5012 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 5013 if(GetLastError() != 997) { 5014 //writeln("WSARecv ", WSAGetLastError()); 5015 //throw new Exception("WSARecv fail"); 5016 } 5017 } 5018 }); 5019 5020 Fiber.yield(); 5021 return lastAnswer; 5022 } 5023 5024 // I might go back and implement these for udp things. 5025 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 5026 assert(0); 5027 } 5028 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 5029 assert(0); 5030 } 5031 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 5032 assert(0); 5033 } 5034 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 5035 assert(0); 5036 } 5037 5038 // lol overload sets 5039 alias send = typeof(super).send; 5040 alias receive = typeof(super).receive; 5041 alias sendTo = typeof(super).sendTo; 5042 alias receiveFrom = typeof(super).receiveFrom; 5043 5044 } 5045 } 5046 5047 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 5048 assert(connection !is null); 5049 version(cgi_use_fiber) { 5050 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 5051 5052 version(Windows) { 5053 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 5054 } 5055 5056 import core.memory; 5057 GC.addRoot(cast(void*) fiber); 5058 fiber.connection = connection; 5059 fiber.proceed(); 5060 } else { 5061 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 5062 } 5063 } 5064 5065 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 5066 scope(failure) { 5067 // catch all for other errors 5068 try { 5069 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5070 connection.close(); 5071 } catch(Exception e) {} // swallow it, we're aborting anyway. 5072 } 5073 5074 bool closeConnection = alwaysCloseConnection; 5075 5076 /+ 5077 ubyte[4096] inputBuffer = void; 5078 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5079 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5080 5081 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5082 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5083 ir.__ctor(connection, inputBuffer[], true); 5084 +/ 5085 5086 auto ir = new BufferedInputRange(connection); 5087 5088 while(!ir.empty) { 5089 5090 if(ir.view.length == 0) { 5091 ir.popFront(); 5092 if(ir.sourceClosed) { 5093 connection.close(); 5094 closeConnection = true; 5095 break; 5096 } 5097 } 5098 5099 Cgi cgi; 5100 try { 5101 cgi = new CustomCgi(ir, &closeConnection); 5102 // There's a bunch of these casts around because the type matches up with 5103 // the -version=.... specifiers, just you can also create a RequestServer 5104 // and instantiate the things where the types don't match up. It isn't exactly 5105 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5106 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5107 } catch(ConnectionClosedException ce) { 5108 closeConnection = true; 5109 break; 5110 } catch(ConnectionException ce) { 5111 // broken pipe or something, just abort the connection 5112 closeConnection = true; 5113 break; 5114 } catch(HttpVersionNotSupportedException ve) { 5115 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5116 closeConnection = true; 5117 break; 5118 } catch(Throwable t) { 5119 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5120 // anyway let's kill the connection 5121 version(CRuntime_Musl) { 5122 stderr.rawWrite(t.toString()); 5123 stderr.rawWrite("\n"); 5124 } else { 5125 stderr.writeln(t.toString()); 5126 } 5127 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5128 closeConnection = true; 5129 break; 5130 } 5131 assert(cgi !is null); 5132 scope(exit) 5133 cgi.dispose(); 5134 5135 try { 5136 fun(cgi); 5137 cgi.close(); 5138 if(cgi.websocketMode) 5139 closeConnection = true; 5140 } catch(AuthorizationRequiredException are) { 5141 cgi.setResponseStatus("401 Authorization Required"); 5142 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5143 cgi.close(); 5144 } catch(ConnectionException ce) { 5145 // broken pipe or something, just abort the connection 5146 closeConnection = true; 5147 } catch(ConnectionClosedException ce) { 5148 // broken pipe or something, just abort the connection 5149 closeConnection = true; 5150 } catch(Throwable t) { 5151 // a processing error can be recovered from 5152 version(CRuntime_Musl) {} else 5153 stderr.writeln(t.toString); 5154 if(!handleException(cgi, t)) 5155 closeConnection = true; 5156 } 5157 5158 if(globalStopFlag) 5159 closeConnection = true; 5160 5161 if(closeConnection || alwaysCloseConnection) { 5162 connection.shutdown(SocketShutdown.BOTH); 5163 connection.close(); 5164 ir.dispose(); 5165 closeConnection = false; // don't reclose after loop 5166 break; 5167 } else { 5168 if(ir.front.length) { 5169 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5170 } else if(ir.sourceClosed) { 5171 ir.source.shutdown(SocketShutdown.BOTH); 5172 ir.source.close(); 5173 ir.dispose(); 5174 closeConnection = false; 5175 } else { 5176 continue; 5177 // break; // this was for a keepalive experiment 5178 } 5179 } 5180 } 5181 5182 if(closeConnection) { 5183 connection.shutdown(SocketShutdown.BOTH); 5184 connection.close(); 5185 ir.dispose(); 5186 } 5187 5188 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5189 } 5190 5191 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5192 // and now we can buffer 5193 scope(failure) 5194 connection.close(); 5195 5196 import al = std.algorithm; 5197 5198 size_t size; 5199 5200 string[string] headers; 5201 5202 auto range = new BufferedInputRange(connection); 5203 more_data: 5204 auto chunk = range.front(); 5205 // waiting for colon for header length 5206 auto idx = indexOf(cast(string) chunk, ':'); 5207 if(idx == -1) { 5208 try { 5209 range.popFront(); 5210 } catch(Exception e) { 5211 // it is just closed, no big deal 5212 connection.close(); 5213 return; 5214 } 5215 goto more_data; 5216 } 5217 5218 size = to!size_t(cast(string) chunk[0 .. idx]); 5219 chunk = range.consume(idx + 1); 5220 // reading headers 5221 if(chunk.length < size) 5222 range.popFront(0, size + 1); 5223 // we are now guaranteed to have enough 5224 chunk = range.front(); 5225 assert(chunk.length > size); 5226 5227 idx = 0; 5228 string key; 5229 string value; 5230 foreach(part; al.splitter(chunk, '\0')) { 5231 if(idx & 1) { // odd is value 5232 value = cast(string)(part.idup); 5233 headers[key] = value; // commit 5234 } else 5235 key = cast(string)(part.idup); 5236 idx++; 5237 } 5238 5239 enforce(chunk[size] == ','); // the terminator 5240 5241 range.consume(size + 1); 5242 // reading data 5243 // this will be done by Cgi 5244 5245 const(ubyte)[] getScgiChunk() { 5246 // we are already primed 5247 auto data = range.front(); 5248 if(data.length == 0 && !range.sourceClosed) { 5249 range.popFront(0); 5250 data = range.front(); 5251 } else if (range.sourceClosed) 5252 range.source.close(); 5253 5254 return data; 5255 } 5256 5257 void writeScgi(const(ubyte)[] data) { 5258 sendAll(connection, data); 5259 } 5260 5261 void flushScgi() { 5262 // I don't *think* I have to do anything.... 5263 } 5264 5265 Cgi cgi; 5266 try { 5267 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5268 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5269 } catch(Throwable t) { 5270 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5271 connection.close(); 5272 return; // this connection is dead 5273 } 5274 assert(cgi !is null); 5275 scope(exit) cgi.dispose(); 5276 try { 5277 fun(cgi); 5278 cgi.close(); 5279 connection.close(); 5280 5281 } catch(AuthorizationRequiredException are) { 5282 cgi.setResponseStatus("401 Authorization Required"); 5283 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5284 cgi.close(); 5285 } catch(Throwable t) { 5286 // no std err 5287 if(!handleException(cgi, t)) { 5288 connection.close(); 5289 return; 5290 } else { 5291 connection.close(); 5292 return; 5293 } 5294 } 5295 } 5296 5297 string printDate(DateTime date) { 5298 char[29] buffer = void; 5299 printDateToBuffer(date, buffer[]); 5300 return buffer.idup; 5301 } 5302 5303 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5304 assert(buffer.length >= 29); 5305 // 29 static length ? 5306 5307 static immutable daysOfWeek = [ 5308 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5309 ]; 5310 5311 static immutable months = [ 5312 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5313 ]; 5314 5315 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5316 buffer[3 .. 5] = ", "; 5317 buffer[5] = date.day / 10 + '0'; 5318 buffer[6] = date.day % 10 + '0'; 5319 buffer[7] = ' '; 5320 buffer[8 .. 11] = months[date.month]; 5321 buffer[11] = ' '; 5322 auto y = date.year; 5323 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5324 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5325 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5326 buffer[15] = cast(char) (y + '0'); 5327 buffer[16] = ' '; 5328 buffer[17] = date.hour / 10 + '0'; 5329 buffer[18] = date.hour % 10 + '0'; 5330 buffer[19] = ':'; 5331 buffer[20] = date.minute / 10 + '0'; 5332 buffer[21] = date.minute % 10 + '0'; 5333 buffer[22] = ':'; 5334 buffer[23] = date.second / 10 + '0'; 5335 buffer[24] = date.second % 10 + '0'; 5336 buffer[25 .. $] = " GMT"; 5337 5338 return 29; 5339 } 5340 5341 5342 // Referencing this gigantic typeid seems to remind the compiler 5343 // to actually put the symbol in the object file. I guess the immutable 5344 // assoc array array isn't actually included in druntime 5345 void hackAroundLinkerError() { 5346 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5347 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5348 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5349 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5350 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5351 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5352 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5353 // this is getting kinda ridiculous btw. Moving assoc arrays 5354 // to the library is the pain that keeps on coming. 5355 5356 // eh this broke the build on the work server 5357 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5358 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5359 } 5360 5361 5362 5363 5364 5365 version(fastcgi) { 5366 pragma(lib, "fcgi"); 5367 5368 static if(size_t.sizeof == 8) // 64 bit 5369 alias long c_int; 5370 else 5371 alias int c_int; 5372 5373 extern(C) { 5374 struct FCGX_Stream { 5375 ubyte* rdNext; 5376 ubyte* wrNext; 5377 ubyte* stop; 5378 ubyte* stopUnget; 5379 c_int isReader; 5380 c_int isClosed; 5381 c_int wasFCloseCalled; 5382 c_int FCGI_errno; 5383 void* function(FCGX_Stream* stream) fillBuffProc; 5384 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5385 void* data; 5386 } 5387 5388 // note: this is meant to be opaque, so don't access it directly 5389 struct FCGX_Request { 5390 int requestId; 5391 int role; 5392 FCGX_Stream* inStream; 5393 FCGX_Stream* outStream; 5394 FCGX_Stream* errStream; 5395 char** envp; 5396 void* paramsPtr; 5397 int ipcFd; 5398 int isBeginProcessed; 5399 int keepConnection; 5400 int appStatus; 5401 int nWriters; 5402 int flags; 5403 int listen_sock; 5404 } 5405 5406 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5407 void FCGX_Init(); 5408 5409 int FCGX_Accept_r(FCGX_Request *request); 5410 5411 5412 alias char** FCGX_ParamArray; 5413 5414 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5415 c_int FCGX_GetChar(FCGX_Stream* stream); 5416 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5417 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5418 c_int FCGX_FFlush(FCGX_Stream *stream); 5419 5420 int FCGX_OpenSocket(in char*, int); 5421 } 5422 } 5423 5424 5425 /* This might go int a separate module eventually. It is a network input helper class. */ 5426 5427 import std.socket; 5428 5429 version(cgi_use_fiber) { 5430 import core.thread; 5431 5432 version(linux) { 5433 import core.sys.linux.epoll; 5434 5435 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5436 } else version(Windows) { 5437 // declaring the iocp thing below... 5438 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5439 } 5440 5441 version(Windows) 5442 __gshared HANDLE iocp; 5443 5444 version(cgi_use_fiber) { 5445 version(linux) 5446 private enum WakeupEvent { 5447 Read = EPOLLIN, 5448 Write = EPOLLOUT 5449 } 5450 else version(Windows) 5451 private enum WakeupEvent { 5452 Read, Write 5453 } 5454 else static assert(0); 5455 } 5456 5457 version(cgi_use_fiber) 5458 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5459 5460 // static cast since I know what i have in here and don't want to pay for dynamic cast 5461 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5462 5463 version(linux) { 5464 f.setPostYield = () { 5465 if(*registered) { 5466 // rearm 5467 epoll_event evt; 5468 evt.events = e | EPOLLONESHOT; 5469 evt.data.ptr = cast(void*) f; 5470 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5471 throw new Exception("epoll_ctl"); 5472 } else { 5473 // initial registration 5474 *registered = true ; 5475 int fd = source.handle; 5476 epoll_event evt; 5477 evt.events = e | EPOLLONESHOT; 5478 evt.data.ptr = cast(void*) f; 5479 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5480 throw new Exception("epoll_ctl"); 5481 } 5482 }; 5483 5484 Fiber.yield(); 5485 5486 f.setPostYield(null); 5487 } else version(Windows) { 5488 Fiber.yield(); 5489 } 5490 else static assert(0); 5491 } 5492 5493 version(cgi_use_fiber) 5494 void unregisterSource(Socket s) { 5495 version(linux) { 5496 epoll_event evt; 5497 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5498 } else version(Windows) { 5499 // intentionally blank 5500 } 5501 else static assert(0); 5502 } 5503 5504 // it is a class primarily for reference semantics 5505 // I might change this interface 5506 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5507 class BufferedInputRange { 5508 version(Posix) 5509 this(int source, ubyte[] buffer = null) { 5510 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5511 } 5512 5513 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5514 // if they connect but never send stuff to us, we don't want it wasting the process 5515 // so setting a time out 5516 version(cgi_use_fiber) 5517 source.blocking = false; 5518 else 5519 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5520 5521 this.source = source; 5522 if(buffer is null) { 5523 underlyingBuffer = new ubyte[4096]; 5524 this.allowGrowth = true; 5525 } else { 5526 underlyingBuffer = buffer; 5527 this.allowGrowth = allowGrowth; 5528 } 5529 5530 assert(underlyingBuffer.length); 5531 5532 // we assume view.ptr is always inside underlyingBuffer 5533 view = underlyingBuffer[0 .. 0]; 5534 5535 popFront(); // prime 5536 } 5537 5538 version(cgi_use_fiber) { 5539 bool registered; 5540 } 5541 5542 void dispose() { 5543 version(cgi_use_fiber) { 5544 if(registered) 5545 unregisterSource(source); 5546 } 5547 } 5548 5549 /** 5550 A slight difference from regular ranges is you can give it the maximum 5551 number of bytes to consume. 5552 5553 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5554 consume() yourself and use a regular foreach, it will infinitely loop! 5555 5556 The default is to do what a normal range does, and consume the whole buffer 5557 and wait for additional input. 5558 5559 You can also specify 0, to append to the buffer, or any other number 5560 to remove the front n bytes and wait for more. 5561 */ 5562 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5563 if(sourceClosed) 5564 throw new ConnectionClosedException("can't get any more data from a closed source"); 5565 if(!skipConsume) 5566 consume(maxBytesToConsume); 5567 5568 // we might have to grow the buffer 5569 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5570 if(allowGrowth) { 5571 //import std.stdio; writeln("growth"); 5572 auto viewStart = view.ptr - underlyingBuffer.ptr; 5573 size_t growth = 4096; 5574 // make sure we have enough for what we're being asked for 5575 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5576 growth = minBytesToSettleFor - underlyingBuffer.length; 5577 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5578 underlyingBuffer.length += growth; 5579 view = underlyingBuffer[viewStart .. view.length]; 5580 } else 5581 throw new Exception("No room left in the buffer"); 5582 } 5583 5584 do { 5585 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5586 try_again: 5587 auto ret = source.receive(freeSpace); 5588 if(ret == Socket.ERROR) { 5589 if(wouldHaveBlocked()) { 5590 version(cgi_use_fiber) { 5591 registerEventWakeup(®istered, source, WakeupEvent.Read); 5592 goto try_again; 5593 } else { 5594 // gonna treat a timeout here as a close 5595 sourceClosed = true; 5596 return; 5597 } 5598 } 5599 version(Posix) { 5600 import core.stdc.errno; 5601 if(errno == EINTR || errno == EAGAIN) { 5602 goto try_again; 5603 } 5604 if(errno == ECONNRESET) { 5605 sourceClosed = true; 5606 return; 5607 } 5608 } 5609 throw new Exception(lastSocketError); // FIXME 5610 } 5611 if(ret == 0) { 5612 sourceClosed = true; 5613 return; 5614 } 5615 5616 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5617 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5618 //import std.stdio; writeln(cast(string) view); 5619 } while(view.length < minBytesToSettleFor); 5620 } 5621 5622 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5623 /// You might want to idup the data you are consuming if you store it, since it may 5624 /// be overwritten on the new popFront. 5625 /// 5626 /// You do not need to call this if you always want to wait for more data when you 5627 /// consume some. 5628 ubyte[] consume(size_t bytes) { 5629 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5630 view = view[bytes > $ ? $ : bytes .. $]; 5631 if(view.length == 0) { 5632 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5633 /* 5634 writeln("HERE"); 5635 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5636 writeln(cast(string)front); 5637 writeln("DONE"); 5638 */ 5639 } 5640 return front; 5641 } 5642 5643 bool empty() { 5644 return sourceClosed && view.length == 0; 5645 } 5646 5647 ubyte[] front() { 5648 return view; 5649 } 5650 5651 invariant() { 5652 assert(view.ptr >= underlyingBuffer.ptr); 5653 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5654 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5655 } 5656 5657 ubyte[] underlyingBuffer; 5658 bool allowGrowth; 5659 ubyte[] view; 5660 Socket source; 5661 bool sourceClosed; 5662 } 5663 5664 private class FakeSocketForStdin : Socket { 5665 import std.stdio; 5666 5667 this() { 5668 5669 } 5670 5671 private bool closed; 5672 5673 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5674 if(closed) 5675 throw new Exception("Closed"); 5676 return stdin.rawRead(buffer).length; 5677 } 5678 5679 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5680 if(closed) 5681 throw new Exception("Closed"); 5682 stdout.rawWrite(buffer); 5683 return buffer.length; 5684 } 5685 5686 override void close() @trusted scope { 5687 (cast(void delegate() @nogc nothrow) &realClose)(); 5688 } 5689 5690 override void shutdown(SocketShutdown s) { 5691 // FIXME 5692 } 5693 5694 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5695 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5696 5697 override @property @trusted Address remoteAddress() { return null; } 5698 override @property @trusted Address localAddress() { return null; } 5699 5700 void realClose() { 5701 closed = true; 5702 try { 5703 stdin.close(); 5704 stdout.close(); 5705 } catch(Exception e) { 5706 5707 } 5708 } 5709 } 5710 5711 import core.sync.semaphore; 5712 import core.atomic; 5713 5714 /** 5715 To use this thing: 5716 5717 --- 5718 void handler(Socket s) { do something... } 5719 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5720 manager.listen(); 5721 --- 5722 5723 The 4th parameter is optional. 5724 5725 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5726 comes in, you will get control. You can just continue; though to fetch more. 5727 5728 5729 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5730 */ 5731 class ListeningConnectionManager { 5732 Semaphore semaphore; 5733 Socket[256] queue; 5734 shared(ubyte) nextIndexFront; 5735 ubyte nextIndexBack; 5736 shared(int) queueLength; 5737 5738 Socket acceptCancelable() { 5739 version(Posix) { 5740 import core.sys.posix.sys.select; 5741 fd_set read_fds; 5742 FD_ZERO(&read_fds); 5743 int max = 0; 5744 foreach(listener; listeners) { 5745 FD_SET(listener.handle, &read_fds); 5746 if(listener.handle > max) 5747 max = listener.handle; 5748 } 5749 if(cancelfd != -1) { 5750 FD_SET(cancelfd, &read_fds); 5751 if(cancelfd > max) 5752 max = cancelfd; 5753 } 5754 auto ret = select(max + 1, &read_fds, null, null, null); 5755 if(ret == -1) { 5756 import core.stdc.errno; 5757 if(errno == EINTR) 5758 return null; 5759 else 5760 throw new Exception("wtf select"); 5761 } 5762 5763 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5764 return null; 5765 } 5766 5767 foreach(listener; listeners) { 5768 if(FD_ISSET(listener.handle, &read_fds)) 5769 return listener.accept(); 5770 } 5771 5772 return null; 5773 } else { 5774 5775 auto check = new SocketSet(); 5776 5777 keep_looping: 5778 check.reset(); 5779 foreach(listener; listeners) 5780 check.add(listener); 5781 5782 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5783 auto got = Socket.select(check, null, null, 3.seconds); 5784 if(got > 0) 5785 foreach(listener; listeners) 5786 if(check.isSet(listener)) 5787 return listener.accept(); 5788 if(globalStopFlag) 5789 return null; 5790 else 5791 goto keep_looping; 5792 } 5793 } 5794 5795 int defaultNumberOfThreads() { 5796 import std.parallelism; 5797 version(cgi_use_fiber) { 5798 return totalCPUs * 2 + 1; // still chance some will be pointlessly blocked anyway 5799 } else { 5800 // I times 4 here because there's a good chance some will be blocked on i/o. 5801 return totalCPUs * 4; 5802 } 5803 5804 } 5805 5806 void listen() { 5807 shared(int) loopBroken; 5808 5809 version(Posix) { 5810 import core.sys.posix.signal; 5811 signal(SIGPIPE, SIG_IGN); 5812 } 5813 5814 version(linux) { 5815 if(cancelfd == -1) 5816 cancelfd = eventfd(0, 0); 5817 } 5818 5819 version(cgi_no_threads) { 5820 // NEVER USE THIS 5821 // it exists only for debugging and other special occasions 5822 5823 // the thread mode is faster and less likely to stall the whole 5824 // thing when a request is slow 5825 while(!loopBroken && !globalStopFlag) { 5826 auto sn = acceptCancelable(); 5827 if(sn is null) continue; 5828 cloexec(sn); 5829 try { 5830 handler(sn); 5831 } catch(Exception e) { 5832 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5833 sn.close(); 5834 } 5835 } 5836 } else { 5837 5838 if(useFork) { 5839 version(linux) { 5840 //asm { int 3; } 5841 fork(); 5842 } 5843 } 5844 5845 version(cgi_use_fiber) { 5846 5847 version(Windows) { 5848 // please note these are overlapped sockets! so the accept just kicks things off 5849 foreach(listener; listeners) 5850 listener.accept(); 5851 } 5852 5853 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5854 foreach(i, ref thread; threads) { 5855 thread = new WorkerThread(this, handler, cast(int) i); 5856 thread.start(); 5857 } 5858 5859 bool fiber_crash_check() { 5860 bool hasAnyRunning; 5861 foreach(thread; threads) { 5862 if(!thread.isRunning) { 5863 thread.join(); 5864 } else hasAnyRunning = true; 5865 } 5866 5867 return (!hasAnyRunning); 5868 } 5869 5870 5871 while(!globalStopFlag) { 5872 Thread.sleep(1.seconds); 5873 if(fiber_crash_check()) 5874 break; 5875 } 5876 5877 } else { 5878 semaphore = new Semaphore(); 5879 5880 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5881 foreach(i, ref thread; threads) { 5882 thread = new ConnectionThread(this, handler, cast(int) i); 5883 thread.start(); 5884 } 5885 5886 while(!loopBroken && !globalStopFlag) { 5887 Socket sn; 5888 5889 bool crash_check() { 5890 bool hasAnyRunning; 5891 foreach(thread; threads) { 5892 if(!thread.isRunning) { 5893 thread.join(); 5894 } else hasAnyRunning = true; 5895 } 5896 5897 return (!hasAnyRunning); 5898 } 5899 5900 5901 void accept_new_connection() { 5902 sn = acceptCancelable(); 5903 if(sn is null) return; 5904 cloexec(sn); 5905 if(tcp) { 5906 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5907 // on the socket because we do some buffering internally. I think this helps, 5908 // certainly does for small requests, and I think it does for larger ones too 5909 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5910 5911 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5912 } 5913 } 5914 5915 void existing_connection_new_data() { 5916 // wait until a slot opens up 5917 // int waited = 0; 5918 while(queueLength >= queue.length) { 5919 Thread.sleep(1.msecs); 5920 // waited ++; 5921 } 5922 // if(waited) {import std.stdio; writeln(waited);} 5923 synchronized(this) { 5924 queue[nextIndexBack] = sn; 5925 nextIndexBack++; 5926 atomicOp!"+="(queueLength, 1); 5927 } 5928 semaphore.notify(); 5929 } 5930 5931 5932 accept_new_connection(); 5933 if(sn !is null) 5934 existing_connection_new_data(); 5935 else if(sn is null && globalStopFlag) { 5936 foreach(thread; threads) { 5937 semaphore.notify(); 5938 } 5939 Thread.sleep(50.msecs); 5940 } 5941 5942 if(crash_check()) 5943 break; 5944 } 5945 } 5946 5947 // FIXME: i typically stop this with ctrl+c which never 5948 // actually gets here. i need to do a sigint handler. 5949 if(cleanup) 5950 cleanup(); 5951 } 5952 } 5953 5954 //version(linux) 5955 //int epoll_fd; 5956 5957 bool tcp; 5958 void delegate() cleanup; 5959 5960 private void function(Socket) fhandler; 5961 private void dg_handler(Socket s) { 5962 fhandler(s); 5963 } 5964 5965 5966 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5967 fhandler = handler; 5968 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 5969 } 5970 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5971 string[] host; 5972 ushort[] port; 5973 5974 foreach(spec; listenSpec) { 5975 /+ 5976 The format: 5977 5978 protocol:// 5979 address_spec 5980 5981 Protocol is optional. Must be http, https, scgi, or fastcgi. 5982 5983 address_spec is either: 5984 ipv4 address : port 5985 [ipv6 address] : port 5986 unix:filename 5987 abstract:name 5988 port <which is tcp but on any interface> 5989 +/ 5990 5991 string protocol; 5992 string address_spec; 5993 5994 auto protocolIdx = spec.indexOf("://"); 5995 if(protocolIdx != -1) { 5996 protocol = spec[0 .. protocolIdx]; 5997 address_spec = spec[protocolIdx + "://".length .. $]; 5998 } else { 5999 address_spec = spec; 6000 } 6001 6002 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 6003 host ~= address_spec; 6004 port ~= 0; 6005 } else { 6006 auto idx = address_spec.lastIndexOf(":"); 6007 if(idx == -1) { 6008 host ~= null; 6009 } else { 6010 auto as = address_spec[0 .. idx]; 6011 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 6012 as = as[1 .. $-1]; 6013 host ~= as; 6014 } 6015 port ~= address_spec[idx + 1 .. $].to!ushort; 6016 } 6017 6018 } 6019 6020 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 6021 } 6022 6023 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6024 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6025 } 6026 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6027 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6028 } 6029 6030 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6031 fhandler = handler; 6032 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 6033 } 6034 6035 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6036 assert(host.length == port.length); 6037 6038 this.handler = handler; 6039 this.useFork = useFork; 6040 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 6041 6042 listeners.reserve(host.length); 6043 6044 foreach(i; 0 .. host.length) 6045 if(host[i] == "localhost") { 6046 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 6047 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 6048 } else { 6049 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 6050 } 6051 6052 version(cgi_use_fiber) 6053 if(useFork) { 6054 foreach(listener; listeners) 6055 listener.blocking = false; 6056 } 6057 6058 // this is the UI control thread and thus gets more priority 6059 Thread.getThis.priority = Thread.PRIORITY_MAX; 6060 } 6061 6062 Socket[] listeners; 6063 void delegate(Socket) handler; 6064 6065 immutable bool useFork; 6066 int numberOfThreads; 6067 } 6068 6069 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 6070 Socket listener; 6071 if(host.startsWith("unix:")) { 6072 version(Posix) { 6073 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6074 cloexec(listener); 6075 string filename = host["unix:".length .. $].idup; 6076 listener.bind(new UnixAddress(filename)); 6077 cleanup = delegate() { 6078 listener.close(); 6079 import std.file; 6080 remove(filename); 6081 }; 6082 tcp = false; 6083 } else { 6084 throw new Exception("unix sockets not supported on this system"); 6085 } 6086 } else if(host.startsWith("abstract:")) { 6087 version(linux) { 6088 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6089 cloexec(listener); 6090 string filename = "\0" ~ host["abstract:".length .. $]; 6091 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6092 listener.bind(new UnixAddress(filename)); 6093 tcp = false; 6094 } else { 6095 throw new Exception("abstract unix sockets not supported on this system"); 6096 } 6097 } else { 6098 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6099 version(cgi_use_fiber) { 6100 version(Windows) 6101 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6102 else 6103 listener = new Socket(address.addressFamily, SocketType.STREAM); 6104 } else { 6105 listener = new Socket(address.addressFamily, SocketType.STREAM); 6106 } 6107 cloexec(listener); 6108 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6109 if(address.addressFamily == AddressFamily.INET6) 6110 listener.setOption(SocketOptionLevel.IPV6, SocketOption.IPV6_V6ONLY, true); 6111 listener.bind(address); 6112 cleanup = delegate() { 6113 listener.close(); 6114 }; 6115 tcp = true; 6116 } 6117 6118 listener.listen(backQueue); 6119 6120 if (dropPrivs !is null) // can be null, backwards compatibility 6121 dropPrivs(); 6122 6123 return listener; 6124 } 6125 6126 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6127 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6128 if(data.length == 0) return; 6129 ptrdiff_t amount; 6130 //import std.stdio; writeln("***",cast(string) data,"///"); 6131 do { 6132 amount = s.send(data); 6133 if(amount == Socket.ERROR) { 6134 version(cgi_use_fiber) { 6135 if(wouldHaveBlocked()) { 6136 bool registered = true; 6137 registerEventWakeup(®istered, s, WakeupEvent.Write); 6138 continue; 6139 } 6140 } 6141 throw new ConnectionException(s, lastSocketError, file, line); 6142 } 6143 assert(amount > 0); 6144 6145 data = data[amount .. $]; 6146 } while(data.length); 6147 } 6148 6149 class ConnectionException : Exception { 6150 Socket socket; 6151 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6152 this.socket = s; 6153 super(msg, file, line); 6154 } 6155 } 6156 6157 class HttpVersionNotSupportedException : Exception { 6158 this(string file = __FILE__, size_t line = __LINE__) { 6159 super("HTTP Version Not Supported", file, line); 6160 } 6161 } 6162 6163 alias void delegate(Socket) CMT; 6164 6165 import core.thread; 6166 /+ 6167 cgi.d now uses a hybrid of event i/o and threads at the top level. 6168 6169 Top level thread is responsible for accepting sockets and selecting on them. 6170 6171 It then indicates to a child that a request is pending, and any random worker 6172 thread that is free handles it. It goes into blocking mode and handles that 6173 http request to completion. 6174 6175 At that point, it goes back into the waiting queue. 6176 6177 6178 This concept is only implemented on Linux. On all other systems, it still 6179 uses the worker threads and semaphores (which is perfectly fine for a lot of 6180 things! Just having a great number of keep-alive connections will break that.) 6181 6182 6183 So the algorithm is: 6184 6185 select(accept, event, pending) 6186 if accept -> send socket to free thread, if any. if not, add socket to queue 6187 if event -> send the signaling thread a socket from the queue, if not, mark it free 6188 - event might block until it can be *written* to. it is a fifo sending socket fds! 6189 6190 A worker only does one http request at a time, then signals its availability back to the boss. 6191 6192 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6193 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6194 actual FD will not be kept out here. 6195 6196 So: 6197 queue = sockets we know are ready to read now, but no worker thread is available 6198 idle list = worker threads not doing anything else. they signal back and forth 6199 6200 the workers all read off the event fd. This is the semaphore wait 6201 6202 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6203 it puts it in the queue and writes to the event fd. 6204 6205 The child could put the socket back in the epoll thing itself. 6206 6207 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6208 +/ 6209 class ConnectionThread : Thread { 6210 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6211 this.lcm = lcm; 6212 this.dg = dg; 6213 this.myThreadNumber = myThreadNumber; 6214 super(&run); 6215 } 6216 6217 void run() { 6218 while(true) { 6219 // so if there's a bunch of idle keep-alive connections, it can 6220 // consume all the worker threads... just sitting there. 6221 lcm.semaphore.wait(); 6222 if(globalStopFlag) 6223 return; 6224 Socket socket; 6225 synchronized(lcm) { 6226 auto idx = lcm.nextIndexFront; 6227 socket = lcm.queue[idx]; 6228 lcm.queue[idx] = null; 6229 atomicOp!"+="(lcm.nextIndexFront, 1); 6230 atomicOp!"-="(lcm.queueLength, 1); 6231 } 6232 try { 6233 //import std.stdio; writeln(myThreadNumber, " taking it"); 6234 dg(socket); 6235 /+ 6236 if(socket.isAlive) { 6237 // process it more later 6238 version(linux) { 6239 import core.sys.linux.epoll; 6240 epoll_event ev; 6241 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6242 ev.data.fd = socket.handle; 6243 import std.stdio; writeln("adding"); 6244 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6245 if(errno == EEXIST) { 6246 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6247 ev.data.fd = socket.handle; 6248 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6249 throw new Exception("epoll_ctl " ~ to!string(errno)); 6250 } else 6251 throw new Exception("epoll_ctl " ~ to!string(errno)); 6252 } 6253 //import std.stdio; writeln("keep alive"); 6254 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6255 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6256 } else { 6257 continue; // hope it times out in a reasonable amount of time... 6258 } 6259 } 6260 +/ 6261 } catch(ConnectionClosedException e) { 6262 // can just ignore this, it is fairly normal 6263 socket.close(); 6264 } catch(Throwable e) { 6265 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6266 socket.close(); 6267 } 6268 } 6269 } 6270 6271 ListeningConnectionManager lcm; 6272 CMT dg; 6273 int myThreadNumber; 6274 } 6275 6276 version(cgi_use_fiber) 6277 class WorkerThread : Thread { 6278 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6279 this.lcm = lcm; 6280 this.dg = dg; 6281 this.myThreadNumber = myThreadNumber; 6282 super(&run); 6283 } 6284 6285 version(Windows) 6286 void run() { 6287 auto timeout = INFINITE; 6288 PseudoblockingOverlappedSocket key; 6289 OVERLAPPED* overlapped; 6290 DWORD bytes; 6291 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6292 if(key is null) 6293 continue; 6294 key.lastAnswer = bytes; 6295 if(key.fiber) { 6296 key.fiber.proceed(); 6297 } else { 6298 // we have a new connection, issue the first receive on it and issue the next accept 6299 6300 auto sn = key.accepted; 6301 6302 key.accept(); 6303 6304 cloexec(sn); 6305 if(lcm.tcp) { 6306 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6307 // on the socket because we do some buffering internally. I think this helps, 6308 // certainly does for small requests, and I think it does for larger ones too 6309 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6310 6311 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6312 } 6313 6314 dg(sn); 6315 } 6316 } 6317 //SleepEx(INFINITE, TRUE); 6318 } 6319 6320 version(linux) 6321 void run() { 6322 6323 import core.sys.linux.epoll; 6324 epfd = epoll_create1(EPOLL_CLOEXEC); 6325 if(epfd == -1) 6326 throw new Exception("epoll_create1 " ~ to!string(errno)); 6327 scope(exit) { 6328 import core.sys.posix.unistd; 6329 close(epfd); 6330 } 6331 6332 { 6333 epoll_event ev; 6334 ev.events = EPOLLIN; 6335 ev.data.fd = cancelfd; 6336 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6337 } 6338 6339 foreach(listener; lcm.listeners) { 6340 epoll_event ev; 6341 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6342 ev.data.fd = listener.handle; 6343 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6344 throw new Exception("epoll_ctl " ~ to!string(errno)); 6345 } 6346 6347 6348 6349 while(!globalStopFlag) { 6350 Socket sn; 6351 6352 epoll_event[64] events; 6353 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6354 if(nfds == -1) { 6355 if(errno == EINTR) 6356 continue; 6357 throw new Exception("epoll_wait " ~ to!string(errno)); 6358 } 6359 6360 outer: foreach(idx; 0 .. nfds) { 6361 auto flags = events[idx].events; 6362 6363 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6364 globalStopFlag = true; 6365 //import std.stdio; writeln("exit heard"); 6366 break; 6367 } else { 6368 foreach(listener; lcm.listeners) { 6369 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6370 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6371 // this try/catch is because it is set to non-blocking mode 6372 // and Phobos' stupid api throws an exception instead of returning 6373 // if it would block. Why would it block? because a forked process 6374 // might have beat us to it, but the wakeup event thundered our herds. 6375 try 6376 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6377 catch(SocketAcceptException e) { continue outer; } 6378 6379 cloexec(sn); 6380 if(lcm.tcp) { 6381 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6382 // on the socket because we do some buffering internally. I think this helps, 6383 // certainly does for small requests, and I think it does for larger ones too 6384 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6385 6386 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6387 } 6388 6389 dg(sn); 6390 continue outer; 6391 } else { 6392 // writeln(events[idx].data.ptr); 6393 } 6394 } 6395 6396 if(cast(size_t) events[idx].data.ptr < 1024) { 6397 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6398 } 6399 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6400 fiber.proceed(); 6401 } 6402 } 6403 } 6404 } 6405 6406 ListeningConnectionManager lcm; 6407 CMT dg; 6408 int myThreadNumber; 6409 } 6410 6411 6412 /* Done with network helper */ 6413 6414 /* Helpers for doing temporary files. Used both here and in web.d */ 6415 6416 version(Windows) { 6417 import core.sys.windows.windows; 6418 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6419 alias GetTempPathW GetTempPath; 6420 } 6421 6422 version(Posix) { 6423 static import linux = core.sys.posix.unistd; 6424 } 6425 6426 string getTempDirectory() { 6427 string path; 6428 version(Windows) { 6429 wchar[1024] buffer; 6430 auto len = GetTempPath(1024, buffer.ptr); 6431 if(len == 0) 6432 throw new Exception("couldn't find a temporary path"); 6433 6434 auto b = buffer[0 .. len]; 6435 6436 path = to!string(b); 6437 } else 6438 path = "/tmp/"; 6439 6440 return path; 6441 } 6442 6443 6444 // I like std.date. These functions help keep my old code and data working with phobos changing. 6445 6446 long sysTimeToDTime(in SysTime sysTime) { 6447 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6448 } 6449 6450 long dateTimeToDTime(in DateTime dt) { 6451 return sysTimeToDTime(cast(SysTime) dt); 6452 } 6453 6454 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6455 return sysTimeToDTime(Clock.currTime(UTC())); 6456 } 6457 6458 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6459 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6460 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6461 return SysTime(hnsecs, tz); 6462 } 6463 6464 6465 6466 // this is a helper to read HTTP transfer-encoding: chunked responses 6467 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6468 immutable(ubyte)[] ret; 6469 6470 another_chunk: 6471 // If here, we are at the beginning of a chunk. 6472 auto a = ir.front(); 6473 int chunkSize; 6474 int loc = locationOf(a, "\r\n"); 6475 while(loc == -1) { 6476 ir.popFront(); 6477 a = ir.front(); 6478 loc = locationOf(a, "\r\n"); 6479 } 6480 6481 string hex; 6482 hex = ""; 6483 for(int i = 0; i < loc; i++) { 6484 char c = a[i]; 6485 if(c >= 'A' && c <= 'Z') 6486 c += 0x20; 6487 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6488 hex ~= c; 6489 } else { 6490 break; 6491 } 6492 } 6493 6494 assert(hex.length); 6495 6496 int power = 1; 6497 int size = 0; 6498 foreach(cc1; retro(hex)) { 6499 dchar cc = cc1; 6500 if(cc >= 'a' && cc <= 'z') 6501 cc -= 0x20; 6502 int val = 0; 6503 if(cc >= '0' && cc <= '9') 6504 val = cc - '0'; 6505 else 6506 val = cc - 'A' + 10; 6507 6508 size += power * val; 6509 power *= 16; 6510 } 6511 6512 chunkSize = size; 6513 assert(size >= 0); 6514 6515 if(loc + 2 > a.length) { 6516 ir.popFront(0, a.length + loc + 2); 6517 a = ir.front(); 6518 } 6519 6520 a = ir.consume(loc + 2); 6521 6522 if(chunkSize == 0) { // we're done with the response 6523 // if we got here, will change must be true.... 6524 more_footers: 6525 loc = locationOf(a, "\r\n"); 6526 if(loc == -1) { 6527 ir.popFront(); 6528 a = ir.front; 6529 goto more_footers; 6530 } else { 6531 assert(loc == 0); 6532 ir.consume(loc + 2); 6533 goto finish; 6534 } 6535 } else { 6536 // if we got here, will change must be true.... 6537 if(a.length < chunkSize + 2) { 6538 ir.popFront(0, chunkSize + 2); 6539 a = ir.front(); 6540 } 6541 6542 ret ~= (a[0..chunkSize]); 6543 6544 if(!(a.length > chunkSize + 2)) { 6545 ir.popFront(0, chunkSize + 2); 6546 a = ir.front(); 6547 } 6548 assert(a[chunkSize] == 13); 6549 assert(a[chunkSize+1] == 10); 6550 a = ir.consume(chunkSize + 2); 6551 chunkSize = 0; 6552 goto another_chunk; 6553 } 6554 6555 finish: 6556 return ret; 6557 } 6558 6559 // I want to be able to get data from multiple sources the same way... 6560 interface ByChunkRange { 6561 bool empty(); 6562 void popFront(); 6563 const(ubyte)[] front(); 6564 } 6565 6566 ByChunkRange byChunk(const(ubyte)[] data) { 6567 return new class ByChunkRange { 6568 override bool empty() { 6569 return !data.length; 6570 } 6571 6572 override void popFront() { 6573 if(data.length > 4096) 6574 data = data[4096 .. $]; 6575 else 6576 data = null; 6577 } 6578 6579 override const(ubyte)[] front() { 6580 return data[0 .. $ > 4096 ? 4096 : $]; 6581 } 6582 }; 6583 } 6584 6585 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6586 const(ubyte)[] f; 6587 6588 f = ir.front; 6589 if(f.length > atMost) 6590 f = f[0 .. atMost]; 6591 6592 return new class ByChunkRange { 6593 override bool empty() { 6594 return atMost == 0; 6595 } 6596 6597 override const(ubyte)[] front() { 6598 return f; 6599 } 6600 6601 override void popFront() { 6602 ir.consume(f.length); 6603 atMost -= f.length; 6604 auto a = ir.front(); 6605 6606 if(a.length <= atMost) { 6607 f = a; 6608 atMost -= a.length; 6609 a = ir.consume(a.length); 6610 if(atMost != 0) 6611 ir.popFront(); 6612 if(f.length == 0) { 6613 f = ir.front(); 6614 } 6615 } else { 6616 // we actually have *more* here than we need.... 6617 f = a[0..atMost]; 6618 atMost = 0; 6619 ir.consume(atMost); 6620 } 6621 } 6622 }; 6623 } 6624 6625 version(cgi_with_websocket) { 6626 // http://tools.ietf.org/html/rfc6455 6627 6628 /++ 6629 WEBSOCKET SUPPORT: 6630 6631 Full example: 6632 --- 6633 import arsd.cgi; 6634 6635 void websocketEcho(Cgi cgi) { 6636 if(cgi.websocketRequested()) { 6637 if(cgi.origin != "http://arsdnet.net") 6638 throw new Exception("bad origin"); 6639 auto websocket = cgi.acceptWebsocket(); 6640 6641 websocket.send("hello"); 6642 websocket.send(" world!"); 6643 6644 auto msg = websocket.recv(); 6645 while(msg.opcode != WebSocketOpcode.close) { 6646 if(msg.opcode == WebSocketOpcode.text) { 6647 websocket.send(msg.textData); 6648 } else if(msg.opcode == WebSocketOpcode.binary) { 6649 websocket.send(msg.data); 6650 } 6651 6652 msg = websocket.recv(); 6653 } 6654 6655 websocket.close(); 6656 } else { 6657 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6658 } 6659 } 6660 6661 mixin GenericMain!websocketEcho; 6662 --- 6663 +/ 6664 6665 class WebSocket { 6666 Cgi cgi; 6667 6668 private this(Cgi cgi) { 6669 this.cgi = cgi; 6670 6671 Socket socket = cgi.idlol.source; 6672 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6673 } 6674 6675 // returns true if data available, false if it timed out 6676 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6677 if(!waitForNextMessageWouldBlock()) 6678 return true; 6679 if(isDataPending(timeout)) 6680 return true; // this is kinda a lie. 6681 6682 return false; 6683 } 6684 6685 public bool lowLevelReceive() { 6686 auto bfr = cgi.idlol; 6687 top: 6688 auto got = bfr.front; 6689 if(got.length) { 6690 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6691 receiveBuffer.length += receiveBufferUsedLength + got.length; 6692 6693 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6694 receiveBufferUsedLength += got.length; 6695 bfr.consume(got.length); 6696 6697 return true; 6698 } 6699 6700 if(bfr.sourceClosed) 6701 return false; 6702 6703 bfr.popFront(0); 6704 if(bfr.sourceClosed) 6705 return false; 6706 goto top; 6707 } 6708 6709 6710 bool isDataPending(Duration timeout = 0.seconds) { 6711 Socket socket = cgi.idlol.source; 6712 6713 auto check = new SocketSet(); 6714 check.add(socket); 6715 6716 auto got = Socket.select(check, null, null, timeout); 6717 if(got > 0) 6718 return true; 6719 return false; 6720 } 6721 6722 // note: this blocks 6723 WebSocketFrame recv() { 6724 return waitForNextMessage(); 6725 } 6726 6727 6728 6729 6730 private void llclose() { 6731 cgi.close(); 6732 } 6733 6734 private void llsend(ubyte[] data) { 6735 cgi.write(data); 6736 cgi.flush(); 6737 } 6738 6739 void unregisterActiveSocket(WebSocket) {} 6740 6741 /* copy/paste section { */ 6742 6743 private int readyState_; 6744 private ubyte[] receiveBuffer; 6745 private size_t receiveBufferUsedLength; 6746 6747 private Config config; 6748 6749 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6750 enum OPEN = 1; /// The connection is open and ready to communicate. 6751 enum CLOSING = 2; /// The connection is in the process of closing. 6752 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6753 6754 /++ 6755 6756 +/ 6757 /// Group: foundational 6758 static struct Config { 6759 /++ 6760 These control the size of the receive buffer. 6761 6762 It starts at the initial size, will temporarily 6763 balloon up to the maximum size, and will reuse 6764 a buffer up to the likely size. 6765 6766 Anything larger than the maximum size will cause 6767 the connection to be aborted and an exception thrown. 6768 This is to protect you against a peer trying to 6769 exhaust your memory, while keeping the user-level 6770 processing simple. 6771 +/ 6772 size_t initialReceiveBufferSize = 4096; 6773 size_t likelyReceiveBufferSize = 4096; /// ditto 6774 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6775 6776 /++ 6777 Maximum combined size of a message. 6778 +/ 6779 size_t maximumMessageSize = 10 * 1024 * 1024; 6780 6781 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6782 string origin; /// Origin URL to send with the handshake, if desired. 6783 string protocol; /// the protocol header, if desired. 6784 6785 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6786 } 6787 6788 /++ 6789 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6790 +/ 6791 int readyState() { 6792 return readyState_; 6793 } 6794 6795 /++ 6796 Closes the connection, sending a graceful teardown message to the other side. 6797 +/ 6798 /// Group: foundational 6799 void close(int code = 0, string reason = null) 6800 //in (reason.length < 123) 6801 in { assert(reason.length < 123); } do 6802 { 6803 if(readyState_ != OPEN) 6804 return; // it cool, we done 6805 WebSocketFrame wss; 6806 wss.fin = true; 6807 wss.opcode = WebSocketOpcode.close; 6808 wss.data = cast(ubyte[]) reason.dup; 6809 wss.send(&llsend); 6810 6811 readyState_ = CLOSING; 6812 6813 llclose(); 6814 } 6815 6816 /++ 6817 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6818 +/ 6819 /// Group: foundational 6820 void ping() { 6821 WebSocketFrame wss; 6822 wss.fin = true; 6823 wss.opcode = WebSocketOpcode.ping; 6824 wss.send(&llsend); 6825 } 6826 6827 // automatically handled.... 6828 void pong() { 6829 WebSocketFrame wss; 6830 wss.fin = true; 6831 wss.opcode = WebSocketOpcode.pong; 6832 wss.send(&llsend); 6833 } 6834 6835 /++ 6836 Sends a text message through the websocket. 6837 +/ 6838 /// Group: foundational 6839 void send(in char[] textData) { 6840 WebSocketFrame wss; 6841 wss.fin = true; 6842 wss.opcode = WebSocketOpcode.text; 6843 wss.data = cast(ubyte[]) textData.dup; 6844 wss.send(&llsend); 6845 } 6846 6847 /++ 6848 Sends a binary message through the websocket. 6849 +/ 6850 /// Group: foundational 6851 void send(in ubyte[] binaryData) { 6852 WebSocketFrame wss; 6853 wss.fin = true; 6854 wss.opcode = WebSocketOpcode.binary; 6855 wss.data = cast(ubyte[]) binaryData.dup; 6856 wss.send(&llsend); 6857 } 6858 6859 /++ 6860 Waits for and returns the next complete message on the socket. 6861 6862 Note that the onmessage function is still called, right before 6863 this returns. 6864 +/ 6865 /// Group: blocking_api 6866 public WebSocketFrame waitForNextMessage() { 6867 do { 6868 auto m = processOnce(); 6869 if(m.populated) 6870 return m; 6871 } while(lowLevelReceive()); 6872 6873 throw new ConnectionClosedException("Websocket receive timed out"); 6874 //return WebSocketFrame.init; // FIXME? maybe. 6875 } 6876 6877 /++ 6878 Tells if [waitForNextMessage] would block. 6879 +/ 6880 /// Group: blocking_api 6881 public bool waitForNextMessageWouldBlock() { 6882 checkAgain: 6883 if(isMessageBuffered()) 6884 return false; 6885 if(!isDataPending()) 6886 return true; 6887 while(isDataPending()) 6888 lowLevelReceive(); 6889 goto checkAgain; 6890 } 6891 6892 /++ 6893 Is there a message in the buffer already? 6894 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6895 If `false`, check [isDataPending] as the next step. 6896 +/ 6897 /// Group: blocking_api 6898 public bool isMessageBuffered() { 6899 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6900 auto s = d; 6901 if(d.length) { 6902 auto orig = d; 6903 auto m = WebSocketFrame.read(d); 6904 // that's how it indicates that it needs more data 6905 if(d !is orig) 6906 return true; 6907 } 6908 6909 return false; 6910 } 6911 6912 private ubyte continuingType; 6913 private ubyte[] continuingData; 6914 //private size_t continuingDataLength; 6915 6916 private WebSocketFrame processOnce() { 6917 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6918 auto s = d; 6919 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6920 WebSocketFrame m; 6921 if(d.length) { 6922 auto orig = d; 6923 m = WebSocketFrame.read(d); 6924 // that's how it indicates that it needs more data 6925 if(d is orig) 6926 return WebSocketFrame.init; 6927 m.unmaskInPlace(); 6928 switch(m.opcode) { 6929 case WebSocketOpcode.continuation: 6930 if(continuingData.length + m.data.length > config.maximumMessageSize) 6931 throw new Exception("message size exceeded"); 6932 6933 continuingData ~= m.data; 6934 if(m.fin) { 6935 if(ontextmessage) 6936 ontextmessage(cast(char[]) continuingData); 6937 if(onbinarymessage) 6938 onbinarymessage(continuingData); 6939 6940 continuingData = null; 6941 } 6942 break; 6943 case WebSocketOpcode.text: 6944 if(m.fin) { 6945 if(ontextmessage) 6946 ontextmessage(m.textData); 6947 } else { 6948 continuingType = m.opcode; 6949 //continuingDataLength = 0; 6950 continuingData = null; 6951 continuingData ~= m.data; 6952 } 6953 break; 6954 case WebSocketOpcode.binary: 6955 if(m.fin) { 6956 if(onbinarymessage) 6957 onbinarymessage(m.data); 6958 } else { 6959 continuingType = m.opcode; 6960 //continuingDataLength = 0; 6961 continuingData = null; 6962 continuingData ~= m.data; 6963 } 6964 break; 6965 case WebSocketOpcode.close: 6966 readyState_ = CLOSED; 6967 if(onclose) 6968 onclose(); 6969 6970 unregisterActiveSocket(this); 6971 break; 6972 case WebSocketOpcode.ping: 6973 pong(); 6974 break; 6975 case WebSocketOpcode.pong: 6976 // just really references it is still alive, nbd. 6977 break; 6978 default: // ignore though i could and perhaps should throw too 6979 } 6980 } 6981 6982 // the recv thing can be invalidated so gotta copy it over ugh 6983 if(d.length) { 6984 m.data = m.data.dup(); 6985 } 6986 6987 import core.stdc.string; 6988 memmove(receiveBuffer.ptr, d.ptr, d.length); 6989 receiveBufferUsedLength = d.length; 6990 6991 return m; 6992 } 6993 6994 private void autoprocess() { 6995 // FIXME 6996 do { 6997 processOnce(); 6998 } while(lowLevelReceive()); 6999 } 7000 7001 7002 void delegate() onclose; /// 7003 void delegate() onerror; /// 7004 void delegate(in char[]) ontextmessage; /// 7005 void delegate(in ubyte[]) onbinarymessage; /// 7006 void delegate() onopen; /// 7007 7008 /++ 7009 7010 +/ 7011 /// Group: browser_api 7012 void onmessage(void delegate(in char[]) dg) { 7013 ontextmessage = dg; 7014 } 7015 7016 /// ditto 7017 void onmessage(void delegate(in ubyte[]) dg) { 7018 onbinarymessage = dg; 7019 } 7020 7021 /* } end copy/paste */ 7022 7023 7024 } 7025 7026 /++ 7027 Returns true if the request headers are asking for a websocket upgrade. 7028 7029 If this returns true, and you want to accept it, call [acceptWebsocket]. 7030 +/ 7031 bool websocketRequested(Cgi cgi) { 7032 return 7033 "sec-websocket-key" in cgi.requestHeaders 7034 && 7035 "connection" in cgi.requestHeaders && 7036 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 7037 && 7038 "upgrade" in cgi.requestHeaders && 7039 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 7040 ; 7041 } 7042 7043 /++ 7044 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 7045 +/ 7046 WebSocket acceptWebsocket(Cgi cgi) { 7047 assert(!cgi.closed); 7048 assert(!cgi.outputtedResponseData); 7049 cgi.setResponseStatus("101 Switching Protocols"); 7050 cgi.header("Upgrade: WebSocket"); 7051 cgi.header("Connection: upgrade"); 7052 7053 string key = cgi.requestHeaders["sec-websocket-key"]; 7054 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 7055 7056 import std.digest.sha; 7057 auto hash = sha1Of(key); 7058 auto accept = Base64.encode(hash); 7059 7060 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 7061 7062 cgi.websocketMode = true; 7063 cgi.write(""); 7064 7065 cgi.flush(); 7066 7067 return new WebSocket(cgi); 7068 } 7069 7070 // FIXME get websocket to work on other modes, not just embedded_httpd 7071 7072 /* copy/paste in http2.d { */ 7073 enum WebSocketOpcode : ubyte { 7074 continuation = 0, 7075 text = 1, 7076 binary = 2, 7077 // 3, 4, 5, 6, 7 RESERVED 7078 close = 8, 7079 ping = 9, 7080 pong = 10, 7081 // 11,12,13,14,15 RESERVED 7082 } 7083 7084 public struct WebSocketFrame { 7085 private bool populated; 7086 bool fin; 7087 bool rsv1; 7088 bool rsv2; 7089 bool rsv3; 7090 WebSocketOpcode opcode; // 4 bits 7091 bool masked; 7092 ubyte lengthIndicator; // don't set this when building one to send 7093 ulong realLength; // don't use when sending 7094 ubyte[4] maskingKey; // don't set this when sending 7095 ubyte[] data; 7096 7097 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7098 WebSocketFrame msg; 7099 msg.fin = true; 7100 msg.opcode = opcode; 7101 msg.data = cast(ubyte[]) data.dup; 7102 7103 return msg; 7104 } 7105 7106 private void send(scope void delegate(ubyte[]) llsend) { 7107 ubyte[64] headerScratch; 7108 int headerScratchPos = 0; 7109 7110 realLength = data.length; 7111 7112 { 7113 ubyte b1; 7114 b1 |= cast(ubyte) opcode; 7115 b1 |= rsv3 ? (1 << 4) : 0; 7116 b1 |= rsv2 ? (1 << 5) : 0; 7117 b1 |= rsv1 ? (1 << 6) : 0; 7118 b1 |= fin ? (1 << 7) : 0; 7119 7120 headerScratch[0] = b1; 7121 headerScratchPos++; 7122 } 7123 7124 { 7125 headerScratchPos++; // we'll set header[1] at the end of this 7126 auto rlc = realLength; 7127 ubyte b2; 7128 b2 |= masked ? (1 << 7) : 0; 7129 7130 assert(headerScratchPos == 2); 7131 7132 if(realLength > 65535) { 7133 // use 64 bit length 7134 b2 |= 0x7f; 7135 7136 // FIXME: double check endinaness 7137 foreach(i; 0 .. 8) { 7138 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7139 rlc >>>= 8; 7140 } 7141 7142 headerScratchPos += 8; 7143 } else if(realLength > 125) { 7144 // use 16 bit length 7145 b2 |= 0x7e; 7146 7147 // FIXME: double check endinaness 7148 foreach(i; 0 .. 2) { 7149 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7150 rlc >>>= 8; 7151 } 7152 7153 headerScratchPos += 2; 7154 } else { 7155 // use 7 bit length 7156 b2 |= realLength & 0b_0111_1111; 7157 } 7158 7159 headerScratch[1] = b2; 7160 } 7161 7162 //assert(!masked, "masking key not properly implemented"); 7163 if(masked) { 7164 // FIXME: randomize this 7165 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7166 headerScratchPos += 4; 7167 7168 // we'll just mask it in place... 7169 int keyIdx = 0; 7170 foreach(i; 0 .. data.length) { 7171 data[i] = data[i] ^ maskingKey[keyIdx]; 7172 if(keyIdx == 3) 7173 keyIdx = 0; 7174 else 7175 keyIdx++; 7176 } 7177 } 7178 7179 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7180 llsend(headerScratch[0 .. headerScratchPos]); 7181 llsend(data); 7182 } 7183 7184 static WebSocketFrame read(ref ubyte[] d) { 7185 WebSocketFrame msg; 7186 7187 auto orig = d; 7188 7189 WebSocketFrame needsMoreData() { 7190 d = orig; 7191 return WebSocketFrame.init; 7192 } 7193 7194 if(d.length < 2) 7195 return needsMoreData(); 7196 7197 ubyte b = d[0]; 7198 7199 msg.populated = true; 7200 7201 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7202 b >>= 4; 7203 msg.rsv3 = b & 0x01; 7204 b >>= 1; 7205 msg.rsv2 = b & 0x01; 7206 b >>= 1; 7207 msg.rsv1 = b & 0x01; 7208 b >>= 1; 7209 msg.fin = b & 0x01; 7210 7211 b = d[1]; 7212 msg.masked = (b & 0b1000_0000) ? true : false; 7213 msg.lengthIndicator = b & 0b0111_1111; 7214 7215 d = d[2 .. $]; 7216 7217 if(msg.lengthIndicator == 0x7e) { 7218 // 16 bit length 7219 msg.realLength = 0; 7220 7221 if(d.length < 2) return needsMoreData(); 7222 7223 foreach(i; 0 .. 2) { 7224 msg.realLength |= d[0] << ((1-i) * 8); 7225 d = d[1 .. $]; 7226 } 7227 } else if(msg.lengthIndicator == 0x7f) { 7228 // 64 bit length 7229 msg.realLength = 0; 7230 7231 if(d.length < 8) return needsMoreData(); 7232 7233 foreach(i; 0 .. 8) { 7234 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7235 d = d[1 .. $]; 7236 } 7237 } else { 7238 // 7 bit length 7239 msg.realLength = msg.lengthIndicator; 7240 } 7241 7242 if(msg.masked) { 7243 7244 if(d.length < 4) return needsMoreData(); 7245 7246 msg.maskingKey = d[0 .. 4]; 7247 d = d[4 .. $]; 7248 } 7249 7250 if(msg.realLength > d.length) { 7251 return needsMoreData(); 7252 } 7253 7254 msg.data = d[0 .. cast(size_t) msg.realLength]; 7255 d = d[cast(size_t) msg.realLength .. $]; 7256 7257 return msg; 7258 } 7259 7260 void unmaskInPlace() { 7261 if(this.masked) { 7262 int keyIdx = 0; 7263 foreach(i; 0 .. this.data.length) { 7264 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7265 if(keyIdx == 3) 7266 keyIdx = 0; 7267 else 7268 keyIdx++; 7269 } 7270 } 7271 } 7272 7273 char[] textData() { 7274 return cast(char[]) data; 7275 } 7276 } 7277 /* } */ 7278 } 7279 7280 7281 version(Windows) 7282 { 7283 version(CRuntime_DigitalMars) 7284 { 7285 extern(C) int setmode(int, int) nothrow @nogc; 7286 } 7287 else version(CRuntime_Microsoft) 7288 { 7289 extern(C) int _setmode(int, int) nothrow @nogc; 7290 alias setmode = _setmode; 7291 } 7292 else static assert(0); 7293 } 7294 7295 version(Posix) { 7296 import core.sys.posix.unistd; 7297 version(CRuntime_Musl) {} else { 7298 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7299 } 7300 } 7301 7302 7303 // FIXME: these aren't quite public yet. 7304 //private: 7305 7306 // template for laziness 7307 void startAddonServer()(string arg) { 7308 version(OSX) { 7309 assert(0, "Not implemented"); 7310 } else version(linux) { 7311 import core.sys.posix.unistd; 7312 pid_t pid; 7313 const(char)*[16] args; 7314 args[0] = "ARSD_CGI_ADDON_SERVER"; 7315 args[1] = arg.ptr; 7316 posix_spawn(&pid, "/proc/self/exe", 7317 null, 7318 null, 7319 args.ptr, 7320 null // env 7321 ); 7322 } else version(Windows) { 7323 wchar[2048] filename; 7324 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7325 if(len == 0 || len == filename.length) 7326 throw new Exception("could not get process name to start helper server"); 7327 7328 STARTUPINFOW startupInfo; 7329 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7330 PROCESS_INFORMATION processInfo; 7331 7332 import std.utf; 7333 7334 // I *MIGHT* need to run it as a new job or a service... 7335 auto ret = CreateProcessW( 7336 filename.ptr, 7337 toUTF16z(arg), 7338 null, // process attributes 7339 null, // thread attributes 7340 false, // inherit handles 7341 0, // creation flags 7342 null, // environment 7343 null, // working directory 7344 &startupInfo, 7345 &processInfo 7346 ); 7347 7348 if(!ret) 7349 throw new Exception("create process failed"); 7350 7351 // when done with those, if we set them 7352 /* 7353 CloseHandle(hStdInput); 7354 CloseHandle(hStdOutput); 7355 CloseHandle(hStdError); 7356 */ 7357 7358 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7359 } 7360 7361 // template for laziness 7362 /* 7363 The websocket server is a single-process, single-thread, event 7364 I/O thing. It is passed websockets from other CGI processes 7365 and is then responsible for handling their messages and responses. 7366 Note that the CGI process is responsible for websocket setup, 7367 including authentication, etc. 7368 7369 It also gets data sent to it by other processes and is responsible 7370 for distributing that, as necessary. 7371 */ 7372 void runWebsocketServer()() { 7373 assert(0, "not implemented"); 7374 } 7375 7376 void sendToWebsocketServer(WebSocket ws, string group) { 7377 assert(0, "not implemented"); 7378 } 7379 7380 void sendToWebsocketServer(string content, string group) { 7381 assert(0, "not implemented"); 7382 } 7383 7384 7385 void runEventServer()() { 7386 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7387 } 7388 7389 void runTimerServer()() { 7390 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7391 } 7392 7393 version(Posix) { 7394 alias LocalServerConnectionHandle = int; 7395 alias CgiConnectionHandle = int; 7396 alias SocketConnectionHandle = int; 7397 7398 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7399 } else version(Windows) { 7400 alias LocalServerConnectionHandle = HANDLE; 7401 version(embedded_httpd_threads) { 7402 alias CgiConnectionHandle = SOCKET; 7403 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7404 } else version(fastcgi) { 7405 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7406 enum INVALID_CGI_CONNECTION_HANDLE = null; 7407 } else version(scgi) { 7408 alias CgiConnectionHandle = SOCKET; 7409 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7410 } else { /* version(plain_cgi) */ 7411 alias CgiConnectionHandle = HANDLE; 7412 enum INVALID_CGI_CONNECTION_HANDLE = null; 7413 } 7414 alias SocketConnectionHandle = SOCKET; 7415 } 7416 7417 version(with_addon_servers_connections) 7418 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7419 version(Posix) { 7420 import core.sys.posix.unistd; 7421 import core.sys.posix.sys.un; 7422 7423 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7424 if(sock == -1) 7425 throw new Exception("socket " ~ to!string(errno)); 7426 7427 scope(failure) 7428 close(sock); 7429 7430 cloexec(sock); 7431 7432 // add-on server processes are assumed to be local, and thus will 7433 // use unix domain sockets. Besides, I want to pass sockets to them, 7434 // so it basically must be local (except for the session server, but meh). 7435 sockaddr_un addr; 7436 addr.sun_family = AF_UNIX; 7437 version(linux) { 7438 // on linux, we will use the abstract namespace 7439 addr.sun_path[0] = 0; 7440 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7441 } else { 7442 // but otherwise, just use a file cuz we must. 7443 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7444 } 7445 7446 bool alreadyTried; 7447 7448 try_again: 7449 7450 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7451 if(!alreadyTried && errno == ECONNREFUSED) { 7452 // try auto-spawning the server, then attempt connection again 7453 startAddonServer(arg); 7454 import core.thread; 7455 Thread.sleep(50.msecs); 7456 alreadyTried = true; 7457 goto try_again; 7458 } else 7459 throw new Exception("connect " ~ to!string(errno)); 7460 } 7461 7462 return sock; 7463 } else version(Windows) { 7464 return null; // FIXME 7465 } 7466 } 7467 7468 version(with_addon_servers_connections) 7469 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7470 version(Posix) { 7471 import core.sys.posix.unistd; 7472 close(handle); 7473 } else version(Windows) 7474 CloseHandle(handle); 7475 } 7476 7477 void runSessionServer()() { 7478 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7479 } 7480 7481 import core.stdc.errno; 7482 7483 struct IoOp { 7484 @disable this(); 7485 @disable this(this); 7486 7487 /* 7488 So we want to be able to eventually handle generic sockets too. 7489 */ 7490 7491 enum Read = 1; 7492 enum Write = 2; 7493 enum Accept = 3; 7494 enum ReadSocketHandle = 4; 7495 7496 // Your handler may be called in a different thread than the one that initiated the IO request! 7497 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7498 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7499 private void delegate(IoOp*) closeHandler; 7500 private void delegate(IoOp*) completeHandler; 7501 private int internalFd; 7502 private int operation; 7503 private int bufferLengthAllocated; 7504 private int bufferLengthUsed; 7505 private ubyte[1] internalBuffer; // it can be overallocated! 7506 7507 ubyte[] allocatedBuffer() return { 7508 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7509 } 7510 7511 ubyte[] usedBuffer() return { 7512 return allocatedBuffer[0 .. bufferLengthUsed]; 7513 } 7514 7515 void reset() { 7516 bufferLengthUsed = 0; 7517 } 7518 7519 int fd() { 7520 return internalFd; 7521 } 7522 } 7523 7524 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7525 import core.stdc.stdlib; 7526 7527 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7528 if(ptr is null) 7529 assert(0); // out of memory! 7530 7531 auto op = cast(IoOp*) ptr; 7532 7533 op.handler = handler; 7534 op.internalFd = fd; 7535 op.operation = operation; 7536 op.bufferLengthAllocated = bufferSize; 7537 op.bufferLengthUsed = 0; 7538 7539 import core.memory; 7540 7541 GC.addRoot(ptr); 7542 7543 return op; 7544 } 7545 7546 void freeIoOp(ref IoOp* ptr) { 7547 7548 import core.memory; 7549 GC.removeRoot(ptr); 7550 7551 import core.stdc.stdlib; 7552 free(ptr); 7553 ptr = null; 7554 } 7555 7556 version(Posix) 7557 version(with_addon_servers_connections) 7558 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7559 7560 //import std.stdio : writeln; writeln(cast(string) data); 7561 7562 import core.sys.posix.unistd; 7563 7564 auto ret = write(connection, data.ptr, data.length); 7565 if(ret != data.length) { 7566 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7567 // the file is closed, remove it 7568 eis.fileClosed(connection); 7569 } else 7570 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7571 } 7572 } 7573 version(Windows) 7574 version(with_addon_servers_connections) 7575 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7576 // FIXME 7577 } 7578 7579 bool isInvalidHandle(CgiConnectionHandle h) { 7580 return h == INVALID_CGI_CONNECTION_HANDLE; 7581 } 7582 7583 /+ 7584 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7585 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7586 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7587 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7588 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7589 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7590 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7591 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7592 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7593 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7594 7595 +/ 7596 7597 /++ 7598 You can customize your server by subclassing the appropriate server. Then, register your 7599 subclass at compile time with the [registerEventIoServer] template, or implement your own 7600 main function and call it yourself. 7601 7602 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7603 +/ 7604 version(with_addon_servers_connections) 7605 interface EventIoServer { 7606 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7607 void handleLocalConnectionClose(IoOp* op); 7608 void handleLocalConnectionComplete(IoOp* op); 7609 void wait_timeout(); 7610 void fileClosed(int fd); 7611 7612 void epoll_fd(int fd); 7613 } 7614 7615 // the sink should buffer it 7616 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7617 static if(is(T == struct)) { 7618 foreach(member; __traits(allMembers, T)) 7619 serialize(sink, __traits(getMember, t, member)); 7620 } else static if(is(T : int)) { 7621 // no need to think of endianness just because this is only used 7622 // for local, same-machine stuff anyway. thanks private lol 7623 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7624 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7625 // these are common enough to optimize 7626 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7627 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7628 sink(cast(ubyte[]) t[]); 7629 } else static if(is(T : A[], A)) { 7630 // generic array is less optimal but still prolly ok 7631 int len = cast(int) t.length; 7632 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7633 foreach(item; t) 7634 serialize(sink, item); 7635 } else static assert(0, T.stringof); 7636 } 7637 7638 // all may be stack buffers, so use cautio 7639 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7640 static if(is(T == struct)) { 7641 T t; 7642 foreach(member; __traits(allMembers, T)) 7643 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7644 dg(t); 7645 } else static if(is(T : int)) { 7646 // no need to think of endianness just because this is only used 7647 // for local, same-machine stuff anyway. thanks private lol 7648 T t; 7649 auto data = get(t.sizeof); 7650 t = (cast(T[]) data)[0]; 7651 dg(t); 7652 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7653 // these are common enough to optimize 7654 int len; 7655 auto data = get(len.sizeof); 7656 len = (cast(int[]) data)[0]; 7657 7658 /* 7659 typeof(T[0])[2000] stackBuffer; 7660 T buffer; 7661 7662 if(len < stackBuffer.length) 7663 buffer = stackBuffer[0 .. len]; 7664 else 7665 buffer = new T(len); 7666 7667 data = get(len * typeof(T[0]).sizeof); 7668 */ 7669 7670 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7671 7672 dg(t); 7673 } else static if(is(T == E[], E)) { 7674 T t; 7675 int len; 7676 auto data = get(len.sizeof); 7677 len = (cast(int[]) data)[0]; 7678 t.length = len; 7679 foreach(ref e; t) { 7680 deserialize!E(get, (ele) { e = ele; }); 7681 } 7682 dg(t); 7683 } else static assert(0, T.stringof); 7684 } 7685 7686 unittest { 7687 serialize((ubyte[] b) { 7688 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7689 }, 1); 7690 serialize((ubyte[] b) { 7691 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7692 }, 56674); 7693 ubyte[1000] buffer; 7694 int bufferPoint; 7695 void add(scope ubyte[] b) { 7696 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7697 bufferPoint += b.length; 7698 } 7699 ubyte[] get(int sz) { 7700 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7701 bufferPoint += sz; 7702 return b; 7703 } 7704 serialize(&add, "test here"); 7705 bufferPoint = 0; 7706 deserialize!string(&get, (t) { assert(t == "test here"); }); 7707 bufferPoint = 0; 7708 7709 struct Foo { 7710 int a; 7711 ubyte c; 7712 string d; 7713 } 7714 serialize(&add, Foo(403, 37, "amazing")); 7715 bufferPoint = 0; 7716 deserialize!Foo(&get, (t) { 7717 assert(t.a == 403); 7718 assert(t.c == 37); 7719 assert(t.d == "amazing"); 7720 }); 7721 bufferPoint = 0; 7722 } 7723 7724 /* 7725 Here's the way the RPC interface works: 7726 7727 You define the interface that lists the functions you can call on the remote process. 7728 The interface may also have static methods for convenience. These forward to a singleton 7729 instance of an auto-generated class, which actually sends the args over the pipe. 7730 7731 An impl class actually implements it. A receiving server deserializes down the pipe and 7732 calls methods on the class. 7733 7734 I went with the interface to get some nice compiler checking and documentation stuff. 7735 7736 I could have skipped the interface and just implemented it all from the server class definition 7737 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7738 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7739 7740 7741 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7742 7743 Realistically though the bodies would just be 7744 connection.call(this.mangleof, args...) sooooo. 7745 7746 FIXME: overloads aren't supported 7747 */ 7748 7749 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7750 interface SessionObject {} 7751 7752 private immutable void delegate(string[])[string] scheduledJobHandlers; 7753 private immutable void delegate(string[])[string] websocketServers; 7754 7755 version(with_breaking_cgi_features) 7756 mixin(q{ 7757 7758 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7759 static import std.traits; 7760 7761 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7762 static foreach(idx, member; __traits(derivedMembers, T)) { 7763 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7764 mixin( q{ 7765 std.traits.ReturnType!(__traits(getMember, T, member)) 7766 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7767 { 7768 SerializationBuffer buffer; 7769 auto i = cast(ushort) idx; 7770 serialize(&buffer.sink, i); 7771 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7772 foreach(param; params) 7773 serialize(&buffer.sink, param); 7774 7775 auto sendable = buffer.sendable; 7776 7777 version(Posix) {{ 7778 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7779 7780 if(ret == -1) { 7781 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7782 } else if(ret == 0) { 7783 throw new Exception("Connection to addon server lost"); 7784 } if(ret < sendable.length) 7785 throw new Exception("Send failed to send all"); 7786 assert(ret == sendable.length); 7787 }} // FIXME Windows impl 7788 7789 static if(!is(typeof(return) == void)) { 7790 // there is a return value; we need to wait for it too 7791 version(Posix) { 7792 ubyte[3000] revBuffer; 7793 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7794 auto got = revBuffer[0 .. ret]; 7795 7796 int dataLocation; 7797 ubyte[] grab(int sz) { 7798 auto dataLocation1 = dataLocation; 7799 dataLocation += sz; 7800 return got[dataLocation1 .. dataLocation]; 7801 } 7802 7803 typeof(return) retu; 7804 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7805 return retu; 7806 } else { 7807 // FIXME Windows impl 7808 return typeof(return).init; 7809 } 7810 7811 } 7812 }}); 7813 } 7814 7815 private static typeof(this) singletonInstance; 7816 private LocalServerConnectionHandle connectionHandle; 7817 7818 static typeof(this) connection() { 7819 if(singletonInstance is null) { 7820 singletonInstance = new typeof(this)(); 7821 singletonInstance.connect(); 7822 } 7823 return singletonInstance; 7824 } 7825 7826 void connect() { 7827 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7828 } 7829 7830 void disconnect() { 7831 closeLocalServerConnection(connectionHandle); 7832 } 7833 } 7834 7835 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7836 ushort calledIdx; 7837 string calledFunction; 7838 7839 int dataLocation; 7840 ubyte[] grab(int sz) { 7841 if(sz == 0) assert(0); 7842 auto d = data[dataLocation .. dataLocation + sz]; 7843 dataLocation += sz; 7844 return d; 7845 } 7846 7847 again: 7848 7849 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7850 deserialize!string(&grab, (a) { calledFunction = a; }); 7851 7852 import std.traits; 7853 7854 sw: switch(calledIdx) { 7855 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7856 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 7857 case idx: 7858 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7859 7860 Parameters!(__traits(getMember, Interface, memberName)) params; 7861 foreach(ref param; params) 7862 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7863 7864 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7865 __traits(getMember, this_, memberName)(params); 7866 } else { 7867 auto ret = __traits(getMember, this_, memberName)(params); 7868 SerializationBuffer buffer; 7869 serialize(&buffer.sink, ret); 7870 7871 auto sendable = buffer.sendable; 7872 7873 version(Posix) { 7874 auto r = send(fd, sendable.ptr, sendable.length, 0); 7875 if(r == -1) { 7876 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7877 } else if(r == 0) { 7878 throw new Exception("Connection to addon client lost"); 7879 } if(r < sendable.length) 7880 throw new Exception("Send failed to send all"); 7881 7882 } // FIXME Windows impl 7883 } 7884 break sw; 7885 } 7886 default: assert(0); 7887 } 7888 7889 if(dataLocation != data.length) 7890 goto again; 7891 } 7892 7893 7894 private struct SerializationBuffer { 7895 ubyte[2048] bufferBacking; 7896 int bufferLocation; 7897 void sink(scope ubyte[] data) { 7898 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7899 bufferLocation += data.length; 7900 } 7901 7902 ubyte[] sendable() return { 7903 return bufferBacking[0 .. bufferLocation]; 7904 } 7905 } 7906 7907 /* 7908 FIXME: 7909 add a version command line arg 7910 version data in the library 7911 management gui as external program 7912 7913 at server with event_fd for each run 7914 use .mangleof in the at function name 7915 7916 i think the at server will have to: 7917 pipe args to the child 7918 collect child output for logging 7919 get child return value for logging 7920 7921 on windows timers work differently. idk how to best combine with the io stuff. 7922 7923 will have to have dump and restore too, so i can restart without losing stuff. 7924 */ 7925 7926 /++ 7927 A convenience object for talking to the [BasicDataServer] from a higher level. 7928 See: [Cgi.getSessionObject]. 7929 7930 You pass it a `Data` struct describing the data you want saved in the session. 7931 Then, this class will generate getter and setter properties that allow access 7932 to that data. 7933 7934 Note that each load and store will be done as-accessed; it doesn't front-load 7935 mutable data nor does it batch updates out of fear of read-modify-write race 7936 conditions. (In fact, right now it does this for everything, but in the future, 7937 I might batch load `immutable` members of the Data struct.) 7938 7939 At some point in the future, I might also let it do different backends, like 7940 a client-side cookie store too, but idk. 7941 7942 Note that the plain-old-data members of your `Data` struct are wrapped by this 7943 interface via a static foreach to make property functions. 7944 7945 See_Also: [MockSession] 7946 +/ 7947 interface Session(Data) : SessionObject { 7948 @property string sessionId() const; 7949 7950 /++ 7951 Starts a new session. Note that a session is also 7952 implicitly started as soon as you write data to it, 7953 so if you need to alter these parameters from their 7954 defaults, be sure to explicitly call this BEFORE doing 7955 any writes to session data. 7956 7957 Params: 7958 idleLifetime = How long, in seconds, the session 7959 should remain in memory when not being read from 7960 or written to. The default is one day. 7961 7962 NOT IMPLEMENTED 7963 7964 useExtendedLifetimeCookie = The session ID is always 7965 stored in a HTTP cookie, and by default, that cookie 7966 is discarded when the user closes their browser. 7967 7968 But if you set this to true, it will use a non-perishable 7969 cookie for the given idleLifetime. 7970 7971 NOT IMPLEMENTED 7972 +/ 7973 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7974 7975 /++ 7976 Regenerates the session ID and updates the associated 7977 cookie. 7978 7979 This is also your chance to change immutable data 7980 (not yet implemented). 7981 +/ 7982 void regenerateId(); 7983 7984 /++ 7985 Terminates this session, deleting all saved data. 7986 +/ 7987 void terminate(); 7988 7989 /++ 7990 Plain-old-data members of your `Data` struct are wrapped here via 7991 the property getters and setters. 7992 7993 If the member is a non-string array, it returns a magical array proxy 7994 object which allows for atomic appends and replaces via overloaded operators. 7995 You can slice this to get a range representing a $(B const) view of the array. 7996 This is to protect you against read-modify-write race conditions. 7997 +/ 7998 static foreach(memberName; __traits(allMembers, Data)) 7999 static if(is(typeof(__traits(getMember, Data, memberName)))) 8000 mixin(q{ 8001 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 8002 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 8003 }); 8004 8005 } 8006 8007 /++ 8008 An implementation of [Session] that works on real cgi connections utilizing the 8009 [BasicDataServer]. 8010 8011 As opposed to a [MockSession] which is made for testing purposes. 8012 8013 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 8014 +/ 8015 class BasicDataServerSession(Data) : Session!Data { 8016 private Cgi cgi; 8017 private string sessionId_; 8018 8019 public @property string sessionId() const { 8020 return sessionId_; 8021 } 8022 8023 protected @property string sessionId(string s) { 8024 return this.sessionId_ = s; 8025 } 8026 8027 private this(Cgi cgi) { 8028 this.cgi = cgi; 8029 if(auto ptr = "sessionId" in cgi.cookies) 8030 sessionId = (*ptr).length ? *ptr : null; 8031 } 8032 8033 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 8034 assert(sessionId is null); 8035 8036 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 8037 8038 import std.random, std.conv; 8039 sessionId = to!string(uniform(1, long.max)); 8040 8041 BasicDataServer.connection.createSession(sessionId, idleLifetime); 8042 setCookie(); 8043 } 8044 8045 protected void setCookie() { 8046 cgi.setCookie( 8047 "sessionId", sessionId, 8048 0 /* expiration */, 8049 "/" /* path */, 8050 null /* domain */, 8051 true /* http only */, 8052 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 8053 } 8054 8055 void regenerateId() { 8056 if(sessionId is null) { 8057 start(); 8058 return; 8059 } 8060 import std.random, std.conv; 8061 auto oldSessionId = sessionId; 8062 sessionId = to!string(uniform(1, long.max)); 8063 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 8064 setCookie(); 8065 } 8066 8067 void terminate() { 8068 BasicDataServer.connection.destroySession(sessionId); 8069 sessionId = null; 8070 setCookie(); 8071 } 8072 8073 static foreach(memberName; __traits(allMembers, Data)) 8074 static if(is(typeof(__traits(getMember, Data, memberName)))) 8075 mixin(q{ 8076 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8077 if(sessionId is null) 8078 return typeof(return).init; 8079 8080 import std.traits; 8081 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8082 if(v.length == 0) 8083 return typeof(return).init; 8084 import std.conv; 8085 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8086 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8087 // basically. Assuming the session is POD this should be fine. 8088 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8089 } 8090 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8091 if(sessionId is null) 8092 start(); 8093 import std.conv; 8094 import std.traits; 8095 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8096 return value; 8097 } 8098 }); 8099 } 8100 8101 /++ 8102 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8103 Simply stores the data in its instance members. 8104 +/ 8105 class MockSession(Data) : Session!Data { 8106 pure { 8107 @property string sessionId() const { return "mock"; } 8108 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8109 void regenerateId() {} 8110 void terminate() {} 8111 8112 private Data store_; 8113 8114 static foreach(memberName; __traits(allMembers, Data)) 8115 static if(is(typeof(__traits(getMember, Data, memberName)))) 8116 mixin(q{ 8117 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8118 return __traits(getMember, store_, memberName); 8119 } 8120 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8121 return __traits(getMember, store_, memberName) = value; 8122 } 8123 }); 8124 } 8125 } 8126 8127 /++ 8128 Direct interface to the basic data add-on server. You can 8129 typically use [Cgi.getSessionObject] as a more convenient interface. 8130 +/ 8131 version(with_addon_servers_connections) 8132 interface BasicDataServer { 8133 /// 8134 void createSession(string sessionId, int lifetime); 8135 /// 8136 void renewSession(string sessionId, int lifetime); 8137 /// 8138 void destroySession(string sessionId); 8139 /// 8140 void renameSession(string oldSessionId, string newSessionId); 8141 8142 /// 8143 void setSessionData(string sessionId, string dataKey, string dataValue); 8144 /// 8145 string getSessionData(string sessionId, string dataKey); 8146 8147 /// 8148 static BasicDataServerConnection connection() { 8149 return BasicDataServerConnection.connection(); 8150 } 8151 } 8152 8153 version(with_addon_servers_connections) 8154 class BasicDataServerConnection : BasicDataServer { 8155 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8156 } 8157 8158 version(with_addon_servers) 8159 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8160 8161 void createSession(string sessionId, int lifetime) { 8162 sessions[sessionId.idup] = Session(lifetime); 8163 } 8164 void destroySession(string sessionId) { 8165 sessions.remove(sessionId); 8166 } 8167 void renewSession(string sessionId, int lifetime) { 8168 sessions[sessionId].lifetime = lifetime; 8169 } 8170 void renameSession(string oldSessionId, string newSessionId) { 8171 sessions[newSessionId.idup] = sessions[oldSessionId]; 8172 sessions.remove(oldSessionId); 8173 } 8174 void setSessionData(string sessionId, string dataKey, string dataValue) { 8175 if(sessionId !in sessions) 8176 createSession(sessionId, 3600); // FIXME? 8177 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8178 } 8179 string getSessionData(string sessionId, string dataKey) { 8180 if(auto session = sessionId in sessions) { 8181 if(auto data = dataKey in (*session).values) 8182 return *data; 8183 else 8184 return null; // no such data 8185 8186 } else { 8187 return null; // no session 8188 } 8189 } 8190 8191 8192 protected: 8193 8194 struct Session { 8195 int lifetime; 8196 8197 string[string] values; 8198 } 8199 8200 Session[string] sessions; 8201 8202 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8203 auto data = op.usedBuffer; 8204 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8205 return false; 8206 } 8207 8208 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8209 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8210 void wait_timeout() {} 8211 void fileClosed(int fd) {} // stateless so irrelevant 8212 void epoll_fd(int fd) {} 8213 } 8214 8215 /++ 8216 See [schedule] to make one of these. You then call one of the methods here to set it up: 8217 8218 --- 8219 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8220 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8221 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8222 --- 8223 +/ 8224 version(with_addon_servers_connections) 8225 struct ScheduledJobHelper { 8226 private string func; 8227 private string[] args; 8228 private bool consumed; 8229 8230 private this(string func, string[] args) { 8231 this.func = func; 8232 this.args = args; 8233 } 8234 8235 ~this() { 8236 assert(consumed); 8237 } 8238 8239 /++ 8240 Schedules the job to be run at the given time. 8241 +/ 8242 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8243 consumed = true; 8244 8245 auto conn = ScheduledJobServerConnection.connection; 8246 import std.file; 8247 auto st = SysTime(when, timezone); 8248 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8249 } 8250 8251 /++ 8252 Schedules the job to run at least after the specified delay. 8253 +/ 8254 void delay(Duration delay) { 8255 consumed = true; 8256 8257 auto conn = ScheduledJobServerConnection.connection; 8258 import std.file; 8259 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8260 } 8261 8262 /++ 8263 Runs the job in the background ASAP. 8264 8265 $(NOTE It may run in a background thread. Don't segfault!) 8266 +/ 8267 void asap() { 8268 consumed = true; 8269 8270 auto conn = ScheduledJobServerConnection.connection; 8271 import std.file; 8272 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8273 } 8274 8275 /+ 8276 /++ 8277 Schedules the job to recur on the given pattern. 8278 +/ 8279 void recur(string spec) { 8280 8281 } 8282 +/ 8283 } 8284 8285 /++ 8286 First step to schedule a job on the scheduled job server. 8287 8288 The scheduled job needs to be a top-level function that doesn't read any 8289 variables from outside its arguments because it may be run in a new process, 8290 without any context existing later. 8291 8292 You MUST set details on the returned object to actually do anything! 8293 +/ 8294 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8295 /// 8296 ScheduledJobHelper schedule(T args) { 8297 // this isn't meant to ever be called, but instead just to 8298 // get the compiler to type check the arguments passed for us 8299 auto sample = delegate() { 8300 fn(args); 8301 }; 8302 string[] sargs; 8303 foreach(arg; args) 8304 sargs ~= to!string(arg); 8305 return ScheduledJobHelper(fn.mangleof, sargs); 8306 } 8307 8308 shared static this() { 8309 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8310 import std.traits; 8311 Parameters!fn args; 8312 foreach(idx, ref arg; args) 8313 arg = to!(typeof(arg))(sargs[idx]); 8314 fn(args); 8315 }; 8316 } 8317 } 8318 8319 /// 8320 interface ScheduledJobServer { 8321 /// Use the [schedule] function for a higher-level interface. 8322 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8323 /// 8324 void cancelJob(int jobId); 8325 } 8326 8327 version(with_addon_servers_connections) 8328 class ScheduledJobServerConnection : ScheduledJobServer { 8329 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8330 } 8331 8332 version(with_addon_servers) 8333 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8334 // FIXME: we need to handle SIGCHLD in this somehow 8335 // whenIs is 0 for relative, 1 for absolute 8336 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8337 auto nj = nextJobId; 8338 nextJobId++; 8339 8340 version(linux) { 8341 import core.sys.linux.timerfd; 8342 import core.sys.linux.epoll; 8343 import core.sys.posix.unistd; 8344 8345 8346 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8347 if(fd == -1) 8348 throw new Exception("fd timer create failed"); 8349 8350 foreach(ref arg; args) 8351 arg = arg.idup; 8352 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8353 8354 itimerspec value; 8355 value.it_value.tv_sec = when; 8356 value.it_value.tv_nsec = 0; 8357 8358 value.it_interval.tv_sec = 0; 8359 value.it_interval.tv_nsec = 0; 8360 8361 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8362 throw new Exception("couldn't set fd timer"); 8363 8364 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8365 jobs.remove(nj); 8366 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8367 close(fd); 8368 8369 8370 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8371 8372 return true; 8373 }); 8374 scope(failure) 8375 freeIoOp(op); 8376 8377 epoll_event ev; 8378 ev.events = EPOLLIN | EPOLLET; 8379 ev.data.ptr = op; 8380 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8381 throw new Exception("epoll_ctl " ~ to!string(errno)); 8382 8383 jobs[nj] = job; 8384 return nj; 8385 } else assert(0); 8386 } 8387 8388 protected void cancelJob(int jobId) { 8389 version(linux) { 8390 auto job = jobId in jobs; 8391 if(job is null) 8392 return; 8393 8394 jobs.remove(jobId); 8395 8396 version(linux) { 8397 import core.sys.linux.timerfd; 8398 import core.sys.linux.epoll; 8399 import core.sys.posix.unistd; 8400 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8401 close(job.timerfd); 8402 } 8403 } 8404 jobs.remove(jobId); 8405 } 8406 8407 int nextJobId = 1; 8408 static struct Job { 8409 string executable; 8410 string func; 8411 string[] args; 8412 int timerfd; 8413 int id; 8414 } 8415 Job[int] jobs; 8416 8417 8418 // event io server methods below 8419 8420 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8421 auto data = op.usedBuffer; 8422 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8423 return false; 8424 } 8425 8426 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8427 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8428 void wait_timeout() {} 8429 void fileClosed(int fd) {} // stateless so irrelevant 8430 8431 int epoll_fd_; 8432 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8433 int epoll_fd() { return epoll_fd_; } 8434 } 8435 8436 /// 8437 version(with_addon_servers_connections) 8438 interface EventSourceServer { 8439 /++ 8440 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8441 8442 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8443 8444 See_Also: 8445 [sendEvent] 8446 +/ 8447 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8448 /* 8449 If lastEventId is missing or empty, you just get new events as they come. 8450 8451 If it is set from something else, it sends all since then (that are still alive) 8452 down the pipe immediately. 8453 8454 The reason it can come from the header is that's what the standard defines for 8455 browser reconnects. The reason it can come from a query string is just convenience 8456 in catching up in a user-defined manner. 8457 8458 The reason the header overrides the query string is if the browser tries to reconnect, 8459 it will send the header AND the query (it reconnects to the same url), so we just 8460 want to do the restart thing. 8461 8462 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8463 */ 8464 string lastEventId = cgi.lastEventId; 8465 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8466 lastEventId = cgi.get["lastEventId"]; 8467 8468 cgi.setResponseContentType("text/event-stream"); 8469 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8470 cgi.flush(); 8471 8472 cgi.closed = true; 8473 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8474 scope(exit) 8475 closeLocalServerConnection(s); 8476 8477 version(fastcgi) 8478 throw new Exception("sending fcgi connections not supported"); 8479 else { 8480 auto fd = cgi.getOutputFileHandle(); 8481 if(isInvalidHandle(fd)) 8482 throw new Exception("bad fd from cgi!"); 8483 8484 EventSourceServerImplementation.SendableEventConnection sec; 8485 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8486 8487 version(Posix) { 8488 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8489 assert(res == sec.sizeof); 8490 } else version(Windows) { 8491 // FIXME 8492 } 8493 } 8494 } 8495 8496 /++ 8497 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8498 8499 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8500 8501 Params: 8502 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8503 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8504 data = the event data. Available in JS as `event.data`. 8505 lifetime = the amount of time to keep this event for replaying on the event server. 8506 8507 See_Also: 8508 [sendEventToEventServer] 8509 +/ 8510 public static void sendEvent(string url, string event, string data, int lifetime) { 8511 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8512 scope(exit) 8513 closeLocalServerConnection(s); 8514 8515 EventSourceServerImplementation.SendableEvent sev; 8516 sev.populate(url, event, data, lifetime); 8517 8518 version(Posix) { 8519 auto ret = send(s, &sev, sev.sizeof, 0); 8520 assert(ret == sev.sizeof); 8521 } else version(Windows) { 8522 // FIXME 8523 } 8524 } 8525 8526 /++ 8527 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8528 8529 See_Also: [disconnect] 8530 +/ 8531 void connect(string url, string forwardUrl); 8532 8533 /++ 8534 Disconnects `forwardUrl` from `url` 8535 8536 See_Also: [connect] 8537 +/ 8538 void disconnect(string url, string forwardUrl); 8539 } 8540 8541 /// 8542 version(with_addon_servers) 8543 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8544 8545 protected: 8546 8547 void connect(string url, string forwardUrl) { 8548 pipes[url] ~= forwardUrl; 8549 } 8550 void disconnect(string url, string forwardUrl) { 8551 auto t = url in pipes; 8552 if(t is null) 8553 return; 8554 foreach(idx, n; (*t)) 8555 if(n == forwardUrl) { 8556 (*t)[idx] = (*t)[$-1]; 8557 (*t) = (*t)[0 .. $-1]; 8558 break; 8559 } 8560 } 8561 8562 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8563 if(receivedFd != -1) { 8564 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8565 8566 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8567 8568 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8569 8570 auto url = got.url.idup; 8571 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8572 8573 // FIXME: catch up on past messages here 8574 } else { 8575 auto data = op.usedBuffer; 8576 auto event = cast(SendableEvent*) data.ptr; 8577 8578 if(event.magic == 0xdeadbeef) { 8579 handleInputEvent(event); 8580 8581 if(event.url in pipes) 8582 foreach(pipe; pipes[event.url]) { 8583 event.url = pipe; 8584 handleInputEvent(event); 8585 } 8586 } else { 8587 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8588 } 8589 } 8590 return false; 8591 } 8592 void handleLocalConnectionClose(IoOp* op) { 8593 fileClosed(op.fd); 8594 } 8595 void handleLocalConnectionComplete(IoOp* op) {} 8596 8597 void wait_timeout() { 8598 // just keeping alive 8599 foreach(url, connections; eventConnectionsByUrl) 8600 foreach(connection; connections) 8601 if(connection.needsChunking) 8602 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8603 else 8604 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8605 } 8606 8607 void fileClosed(int fd) { 8608 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8609 foreach(idx, conn; connections) { 8610 if(fd == conn.fd) { 8611 connections[idx] = connections[$-1]; 8612 connections = connections[0 .. $ - 1]; 8613 continue outer; 8614 } 8615 } 8616 } 8617 } 8618 8619 void epoll_fd(int fd) {} 8620 8621 8622 private: 8623 8624 8625 struct SendableEventConnection { 8626 ubyte responseChunked; 8627 8628 int urlLength; 8629 char[256] urlBuffer = 0; 8630 8631 int lastEventIdLength; 8632 char[32] lastEventIdBuffer = 0; 8633 8634 char[] url() return { 8635 return urlBuffer[0 .. urlLength]; 8636 } 8637 void url(in char[] u) { 8638 urlBuffer[0 .. u.length] = u[]; 8639 urlLength = cast(int) u.length; 8640 } 8641 char[] lastEventId() return { 8642 return lastEventIdBuffer[0 .. lastEventIdLength]; 8643 } 8644 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8645 in { 8646 assert(url.length < this.urlBuffer.length); 8647 assert(lastEventId.length < this.lastEventIdBuffer.length); 8648 } 8649 do { 8650 this.responseChunked = responseChunked ? 1 : 0; 8651 this.urlLength = cast(int) url.length; 8652 this.lastEventIdLength = cast(int) lastEventId.length; 8653 8654 this.urlBuffer[0 .. url.length] = url[]; 8655 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8656 } 8657 } 8658 8659 struct SendableEvent { 8660 int magic = 0xdeadbeef; 8661 int urlLength; 8662 char[256] urlBuffer = 0; 8663 int typeLength; 8664 char[32] typeBuffer = 0; 8665 int messageLength; 8666 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8667 int _lifetime; 8668 8669 char[] message() return { 8670 return messageBuffer[0 .. messageLength]; 8671 } 8672 char[] type() return { 8673 return typeBuffer[0 .. typeLength]; 8674 } 8675 char[] url() return { 8676 return urlBuffer[0 .. urlLength]; 8677 } 8678 void url(in char[] u) { 8679 urlBuffer[0 .. u.length] = u[]; 8680 urlLength = cast(int) u.length; 8681 } 8682 int lifetime() { 8683 return _lifetime; 8684 } 8685 8686 /// 8687 void populate(string url, string type, string message, int lifetime) 8688 in { 8689 assert(url.length < this.urlBuffer.length); 8690 assert(type.length < this.typeBuffer.length); 8691 assert(message.length < this.messageBuffer.length); 8692 } 8693 do { 8694 this.urlLength = cast(int) url.length; 8695 this.typeLength = cast(int) type.length; 8696 this.messageLength = cast(int) message.length; 8697 this._lifetime = lifetime; 8698 8699 this.urlBuffer[0 .. url.length] = url[]; 8700 this.typeBuffer[0 .. type.length] = type[]; 8701 this.messageBuffer[0 .. message.length] = message[]; 8702 } 8703 } 8704 8705 struct EventConnection { 8706 int fd; 8707 bool needsChunking; 8708 } 8709 8710 private EventConnection[][string] eventConnectionsByUrl; 8711 private string[][string] pipes; 8712 8713 private void handleInputEvent(scope SendableEvent* event) { 8714 static int eventId; 8715 8716 static struct StoredEvent { 8717 int id; 8718 string type; 8719 string message; 8720 int lifetimeRemaining; 8721 } 8722 8723 StoredEvent[][string] byUrl; 8724 8725 int thisId = ++eventId; 8726 8727 if(event.lifetime) 8728 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8729 8730 auto connectionsPtr = event.url in eventConnectionsByUrl; 8731 EventConnection[] connections; 8732 if(connectionsPtr is null) 8733 return; 8734 else 8735 connections = *connectionsPtr; 8736 8737 char[4096] buffer; 8738 char[] formattedMessage; 8739 8740 void append(const char[] a) { 8741 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8742 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8743 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8744 } 8745 8746 import std.algorithm.iteration; 8747 8748 if(connections.length) { 8749 append("id: "); 8750 append(to!string(thisId)); 8751 append("\n"); 8752 8753 append("event: "); 8754 append(event.type); 8755 append("\n"); 8756 8757 foreach(line; event.message.splitter("\n")) { 8758 append("data: "); 8759 append(line); 8760 append("\n"); 8761 } 8762 8763 append("\n"); 8764 } 8765 8766 // chunk it for HTTP! 8767 auto len = toHex(formattedMessage.length); 8768 buffer[4 .. 6] = "\r\n"[]; 8769 buffer[4 - len.length .. 4] = len[]; 8770 buffer[6 + formattedMessage.length] = '\r'; 8771 buffer[6 + formattedMessage.length + 1] = '\n'; 8772 8773 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8774 // done 8775 8776 // FIXME: send back requests when needed 8777 // FIXME: send a single ":\n" every 15 seconds to keep alive 8778 8779 foreach(connection; connections) { 8780 if(connection.needsChunking) { 8781 nonBlockingWrite(this, connection.fd, chunkedMessage); 8782 } else { 8783 nonBlockingWrite(this, connection.fd, formattedMessage); 8784 } 8785 } 8786 } 8787 } 8788 8789 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8790 version(Posix) { 8791 8792 import core.sys.posix.unistd; 8793 import core.sys.posix.fcntl; 8794 import core.sys.posix.sys.un; 8795 8796 import core.sys.posix.signal; 8797 signal(SIGPIPE, SIG_IGN); 8798 8799 static extern(C) void sigchldhandler(int) { 8800 int status; 8801 import w = core.sys.posix.sys.wait; 8802 w.wait(&status); 8803 } 8804 signal(SIGCHLD, &sigchldhandler); 8805 8806 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8807 if(sock == -1) 8808 throw new Exception("socket " ~ to!string(errno)); 8809 8810 scope(failure) 8811 close(sock); 8812 8813 cloexec(sock); 8814 8815 // add-on server processes are assumed to be local, and thus will 8816 // use unix domain sockets. Besides, I want to pass sockets to them, 8817 // so it basically must be local (except for the session server, but meh). 8818 sockaddr_un addr; 8819 addr.sun_family = AF_UNIX; 8820 version(linux) { 8821 // on linux, we will use the abstract namespace 8822 addr.sun_path[0] = 0; 8823 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8824 } else { 8825 // but otherwise, just use a file cuz we must. 8826 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8827 } 8828 8829 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8830 throw new Exception("bind " ~ to!string(errno)); 8831 8832 if(listen(sock, 128) == -1) 8833 throw new Exception("listen " ~ to!string(errno)); 8834 8835 makeNonBlocking(sock); 8836 8837 version(linux) { 8838 import core.sys.linux.epoll; 8839 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8840 if(epoll_fd == -1) 8841 throw new Exception("epoll_create1 " ~ to!string(errno)); 8842 scope(failure) 8843 close(epoll_fd); 8844 } else { 8845 import core.sys.posix.poll; 8846 } 8847 8848 version(linux) 8849 eis.epoll_fd = epoll_fd; 8850 8851 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8852 scope(exit) 8853 freeIoOp(acceptOp); 8854 8855 version(linux) { 8856 epoll_event ev; 8857 ev.events = EPOLLIN | EPOLLET; 8858 ev.data.ptr = acceptOp; 8859 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8860 throw new Exception("epoll_ctl " ~ to!string(errno)); 8861 8862 epoll_event[64] events; 8863 } else { 8864 pollfd[] pollfds; 8865 IoOp*[int] ioops; 8866 pollfds ~= pollfd(sock, POLLIN); 8867 ioops[sock] = acceptOp; 8868 } 8869 8870 import core.time : MonoTime, seconds; 8871 8872 MonoTime timeout = MonoTime.currTime + 15.seconds; 8873 8874 while(true) { 8875 8876 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8877 8878 int timeout_milliseconds = 0; // -1; // infinite 8879 8880 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8881 if(timeout_milliseconds < 0) 8882 timeout_milliseconds = 0; 8883 8884 //writeln("waiting for ", name); 8885 8886 version(linux) { 8887 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8888 if(nfds == -1) { 8889 if(errno == EINTR) 8890 continue; 8891 throw new Exception("epoll_wait " ~ to!string(errno)); 8892 } 8893 } else { 8894 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8895 size_t lastIdx = 0; 8896 } 8897 8898 if(nfds == 0) { 8899 eis.wait_timeout(); 8900 timeout += 15.seconds; 8901 } 8902 8903 foreach(idx; 0 .. nfds) { 8904 version(linux) { 8905 auto flags = events[idx].events; 8906 auto ioop = cast(IoOp*) events[idx].data.ptr; 8907 } else { 8908 IoOp* ioop; 8909 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8910 if(thing.revents) { 8911 ioop = ioops[thing.fd]; 8912 lastIdx += tidx + 1; 8913 break; 8914 } 8915 } 8916 } 8917 8918 //writeln(flags, " ", ioop.fd); 8919 8920 void newConnection() { 8921 // on edge triggering, it is important that we get it all 8922 while(true) { 8923 auto size = cast(socklen_t) addr.sizeof; 8924 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8925 if(ns == -1) { 8926 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8927 // all done, got it all 8928 break; 8929 } 8930 throw new Exception("accept " ~ to!string(errno)); 8931 } 8932 cloexec(ns); 8933 8934 makeNonBlocking(ns); 8935 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8936 niop.closeHandler = &eis.handleLocalConnectionClose; 8937 niop.completeHandler = &eis.handleLocalConnectionComplete; 8938 scope(failure) freeIoOp(niop); 8939 8940 version(linux) { 8941 epoll_event nev; 8942 nev.events = EPOLLIN | EPOLLET; 8943 nev.data.ptr = niop; 8944 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8945 throw new Exception("epoll_ctl " ~ to!string(errno)); 8946 } else { 8947 bool found = false; 8948 foreach(ref pfd; pollfds) { 8949 if(pfd.fd < 0) { 8950 pfd.fd = ns; 8951 found = true; 8952 } 8953 } 8954 if(!found) 8955 pollfds ~= pollfd(ns, POLLIN); 8956 ioops[ns] = niop; 8957 } 8958 } 8959 } 8960 8961 bool newConnectionCondition() { 8962 version(linux) 8963 return ioop.fd == sock && (flags & EPOLLIN); 8964 else 8965 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8966 } 8967 8968 if(newConnectionCondition()) { 8969 newConnection(); 8970 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8971 while(true) { 8972 int in_fd; 8973 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8974 if(got == -1) { 8975 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8976 // all done, got it all 8977 if(ioop.completeHandler) 8978 ioop.completeHandler(ioop); 8979 break; 8980 } 8981 throw new Exception("recv " ~ to!string(errno)); 8982 } 8983 8984 if(got == 0) { 8985 if(ioop.closeHandler) { 8986 ioop.closeHandler(ioop); 8987 version(linux) {} // nothing needed 8988 else { 8989 foreach(ref pfd; pollfds) { 8990 if(pfd.fd == ioop.fd) 8991 pfd.fd = -1; 8992 } 8993 } 8994 } 8995 close(ioop.fd); 8996 freeIoOp(ioop); 8997 break; 8998 } 8999 9000 ioop.bufferLengthUsed = cast(int) got; 9001 ioop.handler(ioop, in_fd); 9002 } 9003 } else if(ioop.operation == IoOp.Read) { 9004 while(true) { 9005 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 9006 if(got == -1) { 9007 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9008 // all done, got it all 9009 if(ioop.completeHandler) 9010 ioop.completeHandler(ioop); 9011 break; 9012 } 9013 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 9014 } 9015 9016 if(got == 0) { 9017 if(ioop.closeHandler) 9018 ioop.closeHandler(ioop); 9019 close(ioop.fd); 9020 freeIoOp(ioop); 9021 break; 9022 } 9023 9024 ioop.bufferLengthUsed = cast(int) got; 9025 if(ioop.handler(ioop, ioop.fd)) { 9026 close(ioop.fd); 9027 freeIoOp(ioop); 9028 break; 9029 } 9030 } 9031 } 9032 9033 // EPOLLHUP? 9034 } 9035 } 9036 } else version(Windows) { 9037 9038 // set up a named pipe 9039 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 9040 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 9041 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 9042 9043 } else static assert(0); 9044 } 9045 9046 9047 version(with_sendfd) 9048 // copied from the web and ported from C 9049 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 9050 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 9051 msghdr msg; 9052 iovec[1] iov; 9053 9054 version(OSX) { 9055 //msg.msg_accrights = cast(cattr_t) &sendfd; 9056 //msg.msg_accrightslen = int.sizeof; 9057 } else version(Android) { 9058 } else { 9059 union ControlUnion { 9060 cmsghdr cm; 9061 char[CMSG_SPACE(int.sizeof)] control; 9062 } 9063 9064 ControlUnion control_un; 9065 cmsghdr* cmptr; 9066 9067 msg.msg_control = control_un.control.ptr; 9068 msg.msg_controllen = control_un.control.length; 9069 9070 cmptr = CMSG_FIRSTHDR(&msg); 9071 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 9072 cmptr.cmsg_level = SOL_SOCKET; 9073 cmptr.cmsg_type = SCM_RIGHTS; 9074 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 9075 } 9076 9077 msg.msg_name = null; 9078 msg.msg_namelen = 0; 9079 9080 iov[0].iov_base = ptr; 9081 iov[0].iov_len = nbytes; 9082 msg.msg_iov = iov.ptr; 9083 msg.msg_iovlen = 1; 9084 9085 return sendmsg(fd, &msg, 0); 9086 } 9087 9088 version(with_sendfd) 9089 // copied from the web and ported from C 9090 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9091 msghdr msg; 9092 iovec[1] iov; 9093 ssize_t n; 9094 int newfd; 9095 9096 version(OSX) { 9097 //msg.msg_accrights = cast(cattr_t) recvfd; 9098 //msg.msg_accrightslen = int.sizeof; 9099 } else version(Android) { 9100 } else { 9101 union ControlUnion { 9102 cmsghdr cm; 9103 char[CMSG_SPACE(int.sizeof)] control; 9104 } 9105 ControlUnion control_un; 9106 cmsghdr* cmptr; 9107 9108 msg.msg_control = control_un.control.ptr; 9109 msg.msg_controllen = control_un.control.length; 9110 } 9111 9112 msg.msg_name = null; 9113 msg.msg_namelen = 0; 9114 9115 iov[0].iov_base = ptr; 9116 iov[0].iov_len = nbytes; 9117 msg.msg_iov = iov.ptr; 9118 msg.msg_iovlen = 1; 9119 9120 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9121 return n; 9122 9123 version(OSX) { 9124 //if(msg.msg_accrightslen != int.sizeof) 9125 //*recvfd = -1; 9126 } else version(Android) { 9127 } else { 9128 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9129 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9130 if (cmptr.cmsg_level != SOL_SOCKET) 9131 throw new Exception("control level != SOL_SOCKET"); 9132 if (cmptr.cmsg_type != SCM_RIGHTS) 9133 throw new Exception("control type != SCM_RIGHTS"); 9134 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9135 } else 9136 *recvfd = -1; /* descriptor was not passed */ 9137 } 9138 9139 return n; 9140 } 9141 /* end read_fd */ 9142 9143 9144 /* 9145 Event source stuff 9146 9147 The api is: 9148 9149 sendEvent(string url, string type, string data, int timeout = 60*10); 9150 9151 attachEventListener(string url, int fd, lastId) 9152 9153 9154 It just sends to all attached listeners, and stores it until the timeout 9155 for replaying via lastEventId. 9156 */ 9157 9158 /* 9159 Session process stuff 9160 9161 it stores it all. the cgi object has a session object that can grab it 9162 9163 session may be done in the same process if possible, there is a version 9164 switch to choose if you want to override. 9165 */ 9166 9167 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9168 alias handler = dispatchHandler; 9169 string urlPrefix; 9170 bool rejectFurther; 9171 immutable(DispatcherDetails) details; 9172 } 9173 9174 private string urlify(string name) pure { 9175 return beautify(name, '-', true); 9176 } 9177 9178 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9179 if(name == "id") 9180 return allLowerCase ? name : "ID"; 9181 9182 char[160] buffer; 9183 int bufferIndex = 0; 9184 bool shouldCap = true; 9185 bool shouldSpace; 9186 bool lastWasCap; 9187 foreach(idx, char ch; name) { 9188 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9189 9190 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9191 if(lastWasCap) { 9192 // two caps in a row, don't change. Prolly acronym. 9193 } else { 9194 if(idx) 9195 shouldSpace = true; // new word, add space 9196 } 9197 9198 lastWasCap = true; 9199 } else { 9200 lastWasCap = false; 9201 } 9202 9203 if(shouldSpace) { 9204 buffer[bufferIndex++] = space; 9205 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9206 shouldSpace = false; 9207 } 9208 if(shouldCap) { 9209 if(ch >= 'a' && ch <= 'z') 9210 ch -= 32; 9211 shouldCap = false; 9212 } 9213 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9214 ch += 32; 9215 buffer[bufferIndex++] = ch; 9216 } 9217 return buffer[0 .. bufferIndex].idup; 9218 } 9219 9220 /* 9221 string urlFor(alias func)() { 9222 return __traits(identifier, func); 9223 } 9224 */ 9225 9226 /++ 9227 UDA: The name displayed to the user in auto-generated HTML. 9228 9229 Default is `beautify(identifier)`. 9230 +/ 9231 struct DisplayName { 9232 string name; 9233 } 9234 9235 /++ 9236 UDA: The name used in the URL or web parameter. 9237 9238 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9239 +/ 9240 struct UrlName { 9241 string name; 9242 } 9243 9244 /++ 9245 UDA: default format to respond for this method 9246 +/ 9247 struct DefaultFormat { string value; } 9248 9249 class MissingArgumentException : Exception { 9250 string functionName; 9251 string argumentName; 9252 string argumentType; 9253 9254 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9255 this.functionName = functionName; 9256 this.argumentName = argumentName; 9257 this.argumentType = argumentType; 9258 9259 super("Missing Argument: " ~ this.argumentName, file, line, next); 9260 } 9261 } 9262 9263 /++ 9264 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9265 9266 History: 9267 Added December 15, 2021 (dub v10.5) 9268 +/ 9269 class ResourceNotFoundException : Exception { 9270 string resourceType; 9271 string resourceId; 9272 9273 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9274 this.resourceType = resourceType; 9275 this.resourceId = resourceId; 9276 9277 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9278 } 9279 9280 } 9281 9282 /++ 9283 This can be attached to any constructor or function called from the cgi system. 9284 9285 If it is present, the function argument can NOT be set from web params, but instead 9286 is set to the return value of the given `func`. 9287 9288 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9289 the current request. Otherwise, it must take zero arguments. 9290 9291 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9292 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9293 the cgi object. 9294 9295 const arguments are also supported. 9296 +/ 9297 struct ifCalledFromWeb(alias func) {} 9298 9299 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9300 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9301 9302 // FIXME: any array of structs should also be settable or gettable from csv as well. 9303 9304 // FIXME: think more about checkboxes and bools. 9305 9306 import std.traits; 9307 9308 Parameters!method params; 9309 alias idents = ParameterIdentifierTuple!method; 9310 alias defaults = ParameterDefaults!method; 9311 9312 const(string)[] names; 9313 const(string)[] values; 9314 9315 // first, check for missing arguments and initialize to defaults if necessary 9316 9317 static if(is(typeof(method) P == __parameters)) 9318 foreach(idx, param; P) {{ 9319 // see: mustNotBeSetFromWebParams 9320 static if(is(param : Cgi)) { 9321 static assert(!is(param == immutable)); 9322 cast() params[idx] = cgi; 9323 } else static if(is(param == Session!D, D)) { 9324 static assert(!is(param == immutable)); 9325 cast() params[idx] = cgi.getSessionObject!D(); 9326 } else { 9327 bool populated; 9328 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9329 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9330 static if(is(typeof(func(cgi)))) 9331 params[idx] = func(cgi); 9332 else 9333 params[idx] = func(); 9334 9335 populated = true; 9336 } 9337 } 9338 9339 if(!populated) { 9340 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9341 params[idx] = param.getAutomaticallyForCgi(cgi); 9342 populated = true; 9343 } 9344 } 9345 9346 if(!populated) { 9347 auto ident = idents[idx]; 9348 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9349 if(ident !in cgi.get) { 9350 static if(is(defaults[idx] == void)) { 9351 static if(is(param == bool)) 9352 params[idx] = false; 9353 else 9354 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9355 } else 9356 params[idx] = defaults[idx]; 9357 } 9358 } else { 9359 if(ident !in cgi.post) { 9360 static if(is(defaults[idx] == void)) { 9361 static if(is(param == bool)) 9362 params[idx] = false; 9363 else 9364 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9365 } else 9366 params[idx] = defaults[idx]; 9367 } 9368 } 9369 } 9370 } 9371 }} 9372 9373 // second, parse the arguments in order to build up arrays, etc. 9374 9375 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9376 static if(is(T == struct)) { 9377 if(name == paramName) { 9378 *what = T.init; 9379 return true; 9380 } else { 9381 // could be a child. gonna allow either obj.field OR obj[field] 9382 9383 string afterName; 9384 9385 if(name[paramName.length] == '[') { 9386 int count = 1; 9387 auto idx = paramName.length + 1; 9388 while(idx < name.length && count > 0) { 9389 if(name[idx] == '[') 9390 count++; 9391 else if(name[idx] == ']') { 9392 count--; 9393 if(count == 0) break; 9394 } 9395 idx++; 9396 } 9397 9398 if(idx == name.length) 9399 return false; // malformed 9400 9401 auto insideBrackets = name[paramName.length + 1 .. idx]; 9402 afterName = name[idx + 1 .. $]; 9403 9404 name = name[0 .. paramName.length]; 9405 9406 paramName = insideBrackets; 9407 9408 } else if(name[paramName.length] == '.') { 9409 paramName = name[paramName.length + 1 .. $]; 9410 name = paramName; 9411 int p = 0; 9412 foreach(ch; paramName) { 9413 if(ch == '.' || ch == '[') 9414 break; 9415 p++; 9416 } 9417 9418 afterName = paramName[p .. $]; 9419 paramName = paramName[0 .. p]; 9420 } else { 9421 return false; 9422 } 9423 9424 if(paramName.length) 9425 // set the child member 9426 switch(paramName) { 9427 foreach(idx, memberName; __traits(allMembers, T)) 9428 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9429 // data member! 9430 case memberName: 9431 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9432 } 9433 default: 9434 // ok, not a member 9435 } 9436 } 9437 9438 return false; 9439 } else static if(is(T == enum)) { 9440 *what = to!T(value); 9441 return true; 9442 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9443 *what = to!T(value); 9444 return true; 9445 } else static if(is(T == bool)) { 9446 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9447 return true; 9448 } else static if(is(T == K[], K)) { 9449 K tmp; 9450 if(name == paramName) { 9451 // direct - set and append 9452 if(setVariable(name, paramName, &tmp, value)) { 9453 (*what) ~= tmp; 9454 return true; 9455 } else { 9456 return false; 9457 } 9458 } else { 9459 // child, append to last element 9460 // FIXME: what about range violations??? 9461 auto ptr = &(*what)[(*what).length - 1]; 9462 return setVariable(name, paramName, ptr, value); 9463 9464 } 9465 } else static if(is(T == V[K], K, V)) { 9466 // assoc array, name[key] is valid 9467 if(name == paramName) { 9468 // no action necessary 9469 return true; 9470 } else if(name[paramName.length] == '[') { 9471 int count = 1; 9472 auto idx = paramName.length + 1; 9473 while(idx < name.length && count > 0) { 9474 if(name[idx] == '[') 9475 count++; 9476 else if(name[idx] == ']') { 9477 count--; 9478 if(count == 0) break; 9479 } 9480 idx++; 9481 } 9482 if(idx == name.length) 9483 return false; // malformed 9484 9485 auto insideBrackets = name[paramName.length + 1 .. idx]; 9486 auto afterName = name[idx + 1 .. $]; 9487 9488 auto k = to!K(insideBrackets); 9489 V v; 9490 if(auto ptr = k in *what) 9491 v = *ptr; 9492 9493 name = name[0 .. paramName.length]; 9494 //writeln(name, afterName, " ", paramName); 9495 9496 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9497 if(ret) { 9498 (*what)[k] = v; 9499 return true; 9500 } 9501 } 9502 9503 return false; 9504 } else { 9505 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9506 } 9507 9508 //return false; 9509 } 9510 9511 void setArgument(string name, string value) { 9512 int p; 9513 foreach(ch; name) { 9514 if(ch == '.' || ch == '[') 9515 break; 9516 p++; 9517 } 9518 9519 auto paramName = name[0 .. p]; 9520 9521 sw: switch(paramName) { 9522 static if(is(typeof(method) P == __parameters)) 9523 foreach(idx, param; P) { 9524 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9525 // cannot be set from the outside 9526 } else { 9527 case idents[idx]: 9528 static if(is(param == Cgi.UploadedFile)) { 9529 params[idx] = cgi.files[name]; 9530 } else static if(is(param : const Cgi.UploadedFile[])) { 9531 (cast() params[idx]) = cgi.filesArray[name]; 9532 } else { 9533 setVariable(name, paramName, ¶ms[idx], value); 9534 } 9535 break sw; 9536 } 9537 } 9538 default: 9539 // ignore; not relevant argument 9540 } 9541 } 9542 9543 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9544 names = cgi.allGetNamesInOrder; 9545 values = cgi.allGetValuesInOrder; 9546 } else { 9547 names = cgi.allPostNamesInOrder; 9548 values = cgi.allPostValuesInOrder; 9549 } 9550 9551 foreach(idx, name; names) { 9552 setArgument(name, values[idx]); 9553 } 9554 9555 static if(is(ReturnType!method == void)) { 9556 typeof(null) ret; 9557 dg(params); 9558 } else { 9559 auto ret = dg(params); 9560 } 9561 9562 // FIXME: format return values 9563 // options are: json, html, csv. 9564 // also may need to wrap in envelope format: none, html, or json. 9565 return ret; 9566 } 9567 9568 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9569 static if(is(T : const(Cgi))) { 9570 return true; 9571 } else static if(is(T : const(Session!D), D)) { 9572 return true; 9573 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9574 return true; 9575 } else { 9576 foreach(uda; attrs) 9577 static if(is(uda == ifCalledFromWeb!func, alias func)) 9578 return true; 9579 return false; 9580 } 9581 } 9582 9583 private bool hasIfCalledFromWeb(attrs...)() { 9584 foreach(uda; attrs) 9585 static if(is(uda == ifCalledFromWeb!func, alias func)) 9586 return true; 9587 return false; 9588 } 9589 9590 /++ 9591 Implies POST path for the thing itself, then GET will get the automatic form. 9592 9593 The given customizer, if present, will be called as a filter on the Form object. 9594 9595 History: 9596 Added December 27, 2020 9597 +/ 9598 template AutomaticForm(alias customizer) { } 9599 9600 /++ 9601 This is meant to be returned by a function that takes a form POST submission. You 9602 want to set the url of the new resource it created, which is set as the http 9603 Location header for a "201 Created" result, and you can also set a separate 9604 destination for browser users, which it sets via a "Refresh" header. 9605 9606 The `resourceRepresentation` should generally be the thing you just created, and 9607 it will be the body of the http response when formatted through the presenter. 9608 The exact thing is up to you - it could just return an id, or the whole object, or 9609 perhaps a partial object. 9610 9611 Examples: 9612 --- 9613 class Test : WebObject { 9614 @(Cgi.RequestMethod.POST) 9615 CreatedResource!int makeThing(string value) { 9616 return CreatedResource!int(value.to!int, "/resources/id"); 9617 } 9618 } 9619 --- 9620 9621 History: 9622 Added December 18, 2021 9623 +/ 9624 struct CreatedResource(T) { 9625 static if(!is(T == void)) 9626 T resourceRepresentation; 9627 string resourceUrl; 9628 string refreshUrl; 9629 } 9630 9631 /+ 9632 /++ 9633 This can be attached as a UDA to a handler to add a http Refresh header on a 9634 successful run. (It will not be attached if the function throws an exception.) 9635 This will refresh the browser the given number of seconds after the page loads, 9636 to the url returned by `urlFunc`, which can be either a static function or a 9637 member method of the current handler object. 9638 9639 You might use this for a POST handler that is normally used from ajax, but you 9640 want it to degrade gracefully to a temporarily flashed message before reloading 9641 the main page. 9642 9643 History: 9644 Added December 18, 2021 9645 +/ 9646 struct Refresh(alias urlFunc) { 9647 int waitInSeconds; 9648 9649 string url() { 9650 static if(__traits(isStaticFunction, urlFunc)) 9651 return urlFunc(); 9652 else static if(is(urlFunc : string)) 9653 return urlFunc; 9654 } 9655 } 9656 +/ 9657 9658 /+ 9659 /++ 9660 Sets a filter to be run before 9661 9662 A before function can do validations of params and log and stop the function from running. 9663 +/ 9664 template Before(alias b) {} 9665 template After(alias b) {} 9666 +/ 9667 9668 /+ 9669 Argument conversions: for the most part, it is to!Thing(string). 9670 9671 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9672 they are passed 9673 9674 arr=foo&arr=bar <-- notice the same name. 9675 9676 Structs are first declared with an empty thing, then have their members set individually, 9677 with dot notation. The members are not required, just the initial declaration. 9678 9679 struct Foo { 9680 int a; 9681 string b; 9682 } 9683 void test(Foo foo){} 9684 9685 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9686 9687 Arrays of structs use this declaration. 9688 9689 void test(Foo[] foo) {} 9690 9691 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9692 9693 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9694 declaration is ignored. 9695 9696 Mind that order matters! The declaration MUST come first in the string. 9697 9698 Arrays of struct members follow this rule recursively. 9699 9700 struct Foo { 9701 int[] a; 9702 } 9703 9704 foo&foo.a=1&foo.a=2&foo&foo.a=1 9705 9706 9707 Associative arrays are formatted with brackets, after a declaration, like structs: 9708 9709 foo&foo[key]=value&foo[other_key]=value 9710 9711 9712 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9713 do not support the strict ordering requirements to work with these struct protocols. 9714 9715 FIXME: also perhaps accept application/json to better work with outside trash. 9716 9717 9718 Return values are also auto-formatted according to user-requested type: 9719 for json, it loops over and converts. 9720 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9721 +/ 9722 9723 /++ 9724 A web presenter is responsible for rendering things to HTML to be usable 9725 in a web browser. 9726 9727 They are passed as template arguments to the base classes of [WebObject] 9728 9729 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9730 and override it. Use forwarding and specialization to customize it. 9731 9732 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9733 class templates and your overridden templates work with each other. 9734 9735 --- 9736 class MyPresenter : WebPresenter!(MyPresenter) { 9737 @Override 9738 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9739 // present the CustomType 9740 } 9741 @Override 9742 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9743 // handle everything else via the super class, which will call 9744 // back to your class when appropriate 9745 super.presentSuccessfulReturnAsHtml(cgi, ret); 9746 } 9747 } 9748 --- 9749 9750 The meta argument in there can be overridden by your own facility. 9751 9752 +/ 9753 class WebPresenter(CRTP) { 9754 9755 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9756 /// If you override a plain method, use `override`. If a template, use `@Override`. 9757 enum Override; 9758 9759 string script() { 9760 return ` 9761 `; 9762 } 9763 9764 string style() { 9765 return ` 9766 :root { 9767 --mild-border: #ccc; 9768 --middle-border: #999; 9769 --accent-color: #f2f2f2; 9770 --sidebar-color: #fefefe; 9771 } 9772 ` ~ genericFormStyling() ~ genericSiteStyling(); 9773 } 9774 9775 string genericFormStyling() { 9776 return 9777 q"css 9778 table.automatic-data-display { 9779 border-collapse: collapse; 9780 border: solid 1px var(--mild-border); 9781 } 9782 9783 table.automatic-data-display td { 9784 vertical-align: top; 9785 border: solid 1px var(--mild-border); 9786 padding: 2px 4px; 9787 } 9788 9789 table.automatic-data-display th { 9790 border: solid 1px var(--mild-border); 9791 border-bottom: solid 1px var(--middle-border); 9792 padding: 2px 4px; 9793 } 9794 9795 ol.automatic-data-display { 9796 margin: 0px; 9797 list-style-position: inside; 9798 padding: 0px; 9799 } 9800 9801 dl.automatic-data-display { 9802 9803 } 9804 9805 .automatic-form { 9806 max-width: 600px; 9807 } 9808 9809 .form-field { 9810 margin: 0.5em; 9811 padding-left: 0.5em; 9812 } 9813 9814 .label-text { 9815 display: block; 9816 font-weight: bold; 9817 margin-left: -0.5em; 9818 } 9819 9820 .submit-button-holder { 9821 padding-left: 2em; 9822 } 9823 9824 .add-array-button { 9825 9826 } 9827 css"; 9828 } 9829 9830 string genericSiteStyling() { 9831 return 9832 q"css 9833 * { box-sizing: border-box; } 9834 html, body { margin: 0px; } 9835 body { 9836 font-family: sans-serif; 9837 } 9838 header { 9839 background: var(--accent-color); 9840 height: 64px; 9841 } 9842 footer { 9843 background: var(--accent-color); 9844 height: 64px; 9845 } 9846 #site-container { 9847 display: flex; 9848 } 9849 main { 9850 flex: 1 1 auto; 9851 order: 2; 9852 min-height: calc(100vh - 64px - 64px); 9853 padding: 4px; 9854 padding-left: 1em; 9855 } 9856 #sidebar { 9857 flex: 0 0 16em; 9858 order: 1; 9859 background: var(--sidebar-color); 9860 } 9861 css"; 9862 } 9863 9864 import arsd.dom; 9865 Element htmlContainer() { 9866 auto document = new Document(q"html 9867 <!DOCTYPE html> 9868 <html class="no-script"> 9869 <head> 9870 <script>document.documentElement.classList.remove("no-script");</script> 9871 <style>.no-script requires-script { display: none; }</style> 9872 <title>D Application</title> 9873 <meta name="viewport" content="initial-scale=1, width=device-width" /> 9874 <link rel="stylesheet" href="style.css" /> 9875 </head> 9876 <body> 9877 <header></header> 9878 <div id="site-container"> 9879 <main></main> 9880 <div id="sidebar"></div> 9881 </div> 9882 <footer></footer> 9883 <script src="script.js"></script> 9884 </body> 9885 </html> 9886 html", true, true); 9887 9888 return document.requireSelector("main"); 9889 } 9890 9891 /// Renders a response as an HTTP error with associated html body 9892 void renderBasicError(Cgi cgi, int httpErrorCode) { 9893 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9894 auto c = htmlContainer(); 9895 c.innerText = getHttpCodeText(httpErrorCode); 9896 cgi.setResponseContentType("text/html; charset=utf-8"); 9897 cgi.write(c.parentDocument.toString(), true); 9898 } 9899 9900 template methodMeta(alias method) { 9901 enum methodMeta = null; 9902 } 9903 9904 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9905 switch(format) { 9906 case "html": 9907 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9908 break; 9909 case "json": 9910 import arsd.jsvar; 9911 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9912 var json; 9913 foreach(index, type; Types) { 9914 if(ret.contains == index) 9915 json = ret.payload[index]; 9916 } 9917 } else { 9918 var json = ret; 9919 } 9920 var envelope = json; // var.emptyObject; 9921 /* 9922 envelope.success = true; 9923 envelope.result = json; 9924 envelope.error = null; 9925 */ 9926 cgi.setResponseContentType("application/json"); 9927 cgi.write(envelope.toJson(), true); 9928 break; 9929 default: 9930 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9931 } 9932 } 9933 9934 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9935 /// in the default presenter - allowing the function to have full low-level control over the 9936 /// response. 9937 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9938 // nothing intentionally! 9939 } 9940 9941 /// Redirections are forwarded to [Cgi.setResponseLocation] 9942 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9943 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9944 } 9945 9946 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9947 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9948 cgi.setResponseStatus(getHttpCodeText(201)); 9949 if(ret.resourceUrl.length) 9950 cgi.header("Location: " ~ ret.resourceUrl); 9951 if(ret.refreshUrl.length) 9952 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9953 static if(!is(R == void)) 9954 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9955 } 9956 9957 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9958 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9959 bool outputted = false; 9960 foreach(index, type; Types) { 9961 if(ret.contains == index) { 9962 assert(!outputted); 9963 outputted = true; 9964 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9965 } 9966 } 9967 if(!outputted) 9968 assert(0); 9969 } 9970 9971 /++ 9972 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9973 +/ 9974 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9975 cgi.setCache(true); // not necessarily true but meh 9976 if(auto fn = ret.filename()) { 9977 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9978 } 9979 cgi.setResponseContentType(ret.contentType); 9980 cgi.write(ret.getData(), true); 9981 } 9982 9983 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9984 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9985 auto container = this.htmlContainer(); 9986 container.appendChild(formatReturnValueAsHtml(ret)); 9987 cgi.write(container.parentDocument.toString(), true); 9988 } 9989 9990 /++ 9991 9992 History: 9993 Added January 23, 2023 (dub v11.0) 9994 +/ 9995 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9996 switch(format) { 9997 case "html": 9998 presentExceptionAsHtml(cgi, t, meta); 9999 break; 10000 case "json": 10001 presentExceptionAsJsonImpl(cgi, t); 10002 break; 10003 default: 10004 } 10005 } 10006 10007 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 10008 cgi.setResponseStatus("500 Internal Server Error"); 10009 cgi.setResponseContentType("application/json"); 10010 import arsd.jsvar; 10011 var v = var.emptyObject; 10012 v.type = typeid(t).toString; 10013 v.msg = t.msg; 10014 v.fullString = t.toString(); 10015 cgi.write(v.toJson(), true); 10016 } 10017 10018 10019 /++ 10020 If you override this, you will need to cast the exception type `t` dynamically, 10021 but can then use the template arguments here to refer back to the function. 10022 10023 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 10024 method on the live object. You could, in theory, change arguments and retry, but I 10025 provide that information mostly with the expectation that you will use them to make 10026 useful forms or richer error messages for the user. 10027 10028 History: 10029 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 10030 I removed this in favor of a `Meta` param. 10031 10032 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 10033 10034 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 10035 10036 If you used the func for something, move that something into your `methodMeta` template. 10037 10038 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 10039 enabling an easier implementation of [presentExceptionalReturn]. 10040 +/ 10041 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 10042 Form af; 10043 /+ 10044 foreach(attr; __traits(getAttributes, func)) { 10045 static if(__traits(isSame, attr, AutomaticForm)) { 10046 af = createAutomaticFormForFunction!(func)(dg); 10047 } 10048 } 10049 +/ 10050 presentExceptionAsHtmlImpl(cgi, t, af); 10051 } 10052 10053 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 10054 if(auto e = cast(ResourceNotFoundException) t) { 10055 auto container = this.htmlContainer(); 10056 10057 container.addChild("p", e.msg); 10058 10059 if(!cgi.outputtedResponseData) 10060 cgi.setResponseStatus("404 Not Found"); 10061 cgi.write(container.parentDocument.toString(), true); 10062 } else if(auto mae = cast(MissingArgumentException) t) { 10063 if(automaticForm is null) 10064 goto generic; 10065 auto container = this.htmlContainer(); 10066 if(cgi.requestMethod == Cgi.RequestMethod.POST) 10067 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 10068 container.appendChild(automaticForm); 10069 10070 cgi.write(container.parentDocument.toString(), true); 10071 } else { 10072 generic: 10073 auto container = this.htmlContainer(); 10074 10075 // import std.stdio; writeln(t.toString()); 10076 10077 container.appendChild(exceptionToElement(t)); 10078 10079 container.addChild("h4", "GET"); 10080 foreach(k, v; cgi.get) { 10081 auto deets = container.addChild("details"); 10082 deets.addChild("summary", k); 10083 deets.addChild("div", v); 10084 } 10085 10086 container.addChild("h4", "POST"); 10087 foreach(k, v; cgi.post) { 10088 auto deets = container.addChild("details"); 10089 deets.addChild("summary", k); 10090 deets.addChild("div", v); 10091 } 10092 10093 10094 if(!cgi.outputtedResponseData) 10095 cgi.setResponseStatus("500 Internal Server Error"); 10096 cgi.write(container.parentDocument.toString(), true); 10097 } 10098 } 10099 10100 Element exceptionToElement(Throwable t) { 10101 auto div = Element.make("div"); 10102 div.addClass("exception-display"); 10103 10104 div.addChild("p", t.msg); 10105 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10106 10107 auto pre = div.addChild("pre"); 10108 string s; 10109 s = t.toString(); 10110 Element currentBox; 10111 bool on = false; 10112 foreach(line; s.splitLines) { 10113 if(!on && line.startsWith("-----")) 10114 on = true; 10115 if(!on) continue; 10116 if(line.indexOf("arsd/") != -1) { 10117 if(currentBox is null) { 10118 currentBox = pre.addChild("details"); 10119 currentBox.addChild("summary", "Framework code"); 10120 } 10121 currentBox.addChild("span", line ~ "\n"); 10122 } else { 10123 pre.addChild("span", line ~ "\n"); 10124 currentBox = null; 10125 } 10126 } 10127 10128 return div; 10129 } 10130 10131 /++ 10132 Returns an element for a particular type 10133 +/ 10134 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10135 import std.traits; 10136 10137 auto div = Element.make("div"); 10138 div.addClass("form-field"); 10139 10140 static if(is(T : const Cgi.UploadedFile)) { 10141 Element lbl; 10142 if(displayName !is null) { 10143 lbl = div.addChild("label"); 10144 lbl.addChild("span", displayName, "label-text"); 10145 lbl.appendText(" "); 10146 } else { 10147 lbl = div; 10148 } 10149 auto i = lbl.addChild("input", name); 10150 i.attrs.name = name; 10151 i.attrs.type = "file"; 10152 i.attrs.multiple = "multiple"; 10153 } else static if(is(T == Cgi.UploadedFile)) { 10154 Element lbl; 10155 if(displayName !is null) { 10156 lbl = div.addChild("label"); 10157 lbl.addChild("span", displayName, "label-text"); 10158 lbl.appendText(" "); 10159 } else { 10160 lbl = div; 10161 } 10162 auto i = lbl.addChild("input", name); 10163 i.attrs.name = name; 10164 i.attrs.type = "file"; 10165 } else static if(is(T == enum)) { 10166 Element lbl; 10167 if(displayName !is null) { 10168 lbl = div.addChild("label"); 10169 lbl.addChild("span", displayName, "label-text"); 10170 lbl.appendText(" "); 10171 } else { 10172 lbl = div; 10173 } 10174 auto i = lbl.addChild("select", name); 10175 i.attrs.name = name; 10176 10177 foreach(memberName; __traits(allMembers, T)) 10178 i.addChild("option", memberName); 10179 10180 } else static if(is(T == struct)) { 10181 if(displayName !is null) 10182 div.addChild("span", displayName, "label-text"); 10183 auto fieldset = div.addChild("fieldset"); 10184 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10185 fieldset.addChild("input", name); 10186 foreach(idx, memberName; __traits(allMembers, T)) 10187 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10188 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10189 } 10190 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10191 Element lbl; 10192 if(displayName !is null) { 10193 lbl = div.addChild("label"); 10194 lbl.addChild("span", displayName, "label-text"); 10195 lbl.appendText(" "); 10196 } else { 10197 lbl = div; 10198 } 10199 Element i; 10200 if(udaSuggestion) { 10201 i = udaSuggestion(); 10202 lbl.appendChild(i); 10203 } else { 10204 i = lbl.addChild("input", name); 10205 } 10206 i.attrs.name = name; 10207 static if(isSomeString!T) 10208 i.attrs.type = "text"; 10209 else 10210 i.attrs.type = "number"; 10211 if(i.tagName == "textarea") 10212 i.textContent = to!string(T.init); 10213 else 10214 i.attrs.value = to!string(T.init); 10215 } else static if(is(T == bool)) { 10216 Element lbl; 10217 if(displayName !is null) { 10218 lbl = div.addChild("label"); 10219 lbl.addChild("span", displayName, "label-text"); 10220 lbl.appendText(" "); 10221 } else { 10222 lbl = div; 10223 } 10224 auto i = lbl.addChild("input", name); 10225 i.attrs.type = "checkbox"; 10226 i.attrs.value = "true"; 10227 i.attrs.name = name; 10228 } else static if(is(T == K[], K)) { 10229 auto templ = div.addChild("template"); 10230 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10231 if(displayName !is null) 10232 div.addChild("span", displayName, "label-text"); 10233 auto btn = div.addChild("button"); 10234 btn.addClass("add-array-button"); 10235 btn.attrs.type = "button"; 10236 btn.innerText = "Add"; 10237 btn.attrs.onclick = q{ 10238 var a = document.importNode(this.parentNode.firstChild.content, true); 10239 this.parentNode.insertBefore(a, this); 10240 }; 10241 } else static if(is(T == V[K], K, V)) { 10242 div.innerText = "assoc array not implemented for automatic form at this time"; 10243 } else { 10244 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10245 } 10246 10247 10248 return div; 10249 } 10250 10251 /// creates a form for gathering the function's arguments 10252 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10253 10254 auto form = cast(Form) Element.make("form"); 10255 10256 form.method = "POST"; // FIXME 10257 10258 form.addClass("automatic-form"); 10259 10260 string formDisplayName = beautify(__traits(identifier, method)); 10261 foreach(attr; __traits(getAttributes, method)) 10262 static if(is(typeof(attr) == DisplayName)) 10263 formDisplayName = attr.name; 10264 form.addChild("h3", formDisplayName); 10265 10266 import std.traits; 10267 10268 //Parameters!method params; 10269 //alias idents = ParameterIdentifierTuple!method; 10270 //alias defaults = ParameterDefaults!method; 10271 10272 static if(is(typeof(method) P == __parameters)) 10273 foreach(idx, _; P) {{ 10274 10275 alias param = P[idx .. idx + 1]; 10276 10277 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10278 string displayName = beautify(__traits(identifier, param)); 10279 Element function() element; 10280 foreach(attr; __traits(getAttributes, param)) { 10281 static if(is(typeof(attr) == DisplayName)) 10282 displayName = attr.name; 10283 else static if(is(typeof(attr) : typeof(element))) { 10284 element = attr; 10285 } 10286 } 10287 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10288 if(i.querySelector("input[type=file]") !is null) 10289 form.setAttribute("enctype", "multipart/form-data"); 10290 } 10291 }} 10292 10293 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10294 10295 return form; 10296 } 10297 10298 /// creates a form for gathering object members (for the REST object thing right now) 10299 Form createAutomaticFormForObject(T)(T obj) { 10300 auto form = cast(Form) Element.make("form"); 10301 10302 form.addClass("automatic-form"); 10303 10304 form.addChild("h3", beautify(__traits(identifier, T))); 10305 10306 import std.traits; 10307 10308 //Parameters!method params; 10309 //alias idents = ParameterIdentifierTuple!method; 10310 //alias defaults = ParameterDefaults!method; 10311 10312 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10313 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10314 string displayName = beautify(memberName); 10315 Element function() element; 10316 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10317 static if(is(typeof(attr) == DisplayName)) 10318 displayName = attr.name; 10319 else static if(is(typeof(attr) : typeof(element))) 10320 element = attr; 10321 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10322 10323 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10324 }}} 10325 10326 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10327 10328 return form; 10329 } 10330 10331 /// 10332 Element formatReturnValueAsHtml(T)(T t) { 10333 import std.traits; 10334 10335 static if(is(T == typeof(null))) { 10336 return Element.make("span"); 10337 } else static if(is(T : Element)) { 10338 return t; 10339 } else static if(is(T == MultipleResponses!Types, Types...)) { 10340 foreach(index, type; Types) { 10341 if(t.contains == index) 10342 return formatReturnValueAsHtml(t.payload[index]); 10343 } 10344 assert(0); 10345 } else static if(is(T == Paginated!E, E)) { 10346 auto e = Element.make("div").addClass("paginated-result"); 10347 e.appendChild(formatReturnValueAsHtml(t.items)); 10348 if(t.nextPageUrl.length) 10349 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10350 return e; 10351 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10352 return Element.make("span", to!string(t), "automatic-data-display"); 10353 } else static if(is(T == V[K], K, V)) { 10354 auto dl = Element.make("dl"); 10355 dl.addClass("automatic-data-display associative-array"); 10356 foreach(k, v; t) { 10357 dl.addChild("dt", to!string(k)); 10358 dl.addChild("dd", formatReturnValueAsHtml(v)); 10359 } 10360 return dl; 10361 } else static if(is(T == struct)) { 10362 auto dl = Element.make("dl"); 10363 dl.addClass("automatic-data-display struct"); 10364 10365 foreach(idx, memberName; __traits(allMembers, T)) 10366 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10367 dl.addChild("dt", beautify(memberName)); 10368 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10369 } 10370 10371 return dl; 10372 } else static if(is(T == bool)) { 10373 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10374 } else static if(is(T == E[], E)) { 10375 static if(is(E : RestObject!Proxy, Proxy)) { 10376 // treat RestObject similar to struct 10377 auto table = cast(Table) Element.make("table"); 10378 table.addClass("automatic-data-display"); 10379 string[] names; 10380 foreach(idx, memberName; __traits(derivedMembers, E)) 10381 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10382 names ~= beautify(memberName); 10383 } 10384 table.appendHeaderRow(names); 10385 10386 foreach(l; t) { 10387 auto tr = table.appendRow(); 10388 foreach(idx, memberName; __traits(derivedMembers, E)) 10389 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10390 static if(memberName == "id") { 10391 string val = to!string(__traits(getMember, l, memberName)); 10392 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10393 } else { 10394 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10395 } 10396 } 10397 } 10398 10399 return table; 10400 } else static if(is(E == struct)) { 10401 // an array of structs is kinda special in that I like 10402 // having those formatted as tables. 10403 auto table = cast(Table) Element.make("table"); 10404 table.addClass("automatic-data-display"); 10405 string[] names; 10406 foreach(idx, memberName; __traits(allMembers, E)) 10407 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10408 names ~= beautify(memberName); 10409 } 10410 table.appendHeaderRow(names); 10411 10412 foreach(l; t) { 10413 auto tr = table.appendRow(); 10414 foreach(idx, memberName; __traits(allMembers, E)) 10415 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10416 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10417 } 10418 } 10419 10420 return table; 10421 } else { 10422 // otherwise, I will just make a list. 10423 auto ol = Element.make("ol"); 10424 ol.addClass("automatic-data-display"); 10425 foreach(e; t) 10426 ol.addChild("li", formatReturnValueAsHtml(e)); 10427 return ol; 10428 } 10429 } else static if(is(T : Object)) { 10430 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10431 return Element.make("div", t.toHtml()); 10432 else 10433 return Element.make("div", t.toString()); 10434 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10435 10436 assert(0); 10437 } 10438 10439 } 10440 10441 /++ 10442 The base class for the [dispatcher] function and object support. 10443 +/ 10444 class WebObject { 10445 //protected Cgi cgi; 10446 10447 protected void initialize(Cgi cgi) { 10448 //this.cgi = cgi; 10449 } 10450 } 10451 10452 /++ 10453 Can return one of the given types, decided at runtime. The syntax 10454 is to declare all the possible types in the return value, then you 10455 can `return typeof(return)(...value...)` to construct it. 10456 10457 It has an auto-generated constructor for each value it can hold. 10458 10459 --- 10460 MultipleResponses!(Redirection, string) getData(int how) { 10461 if(how & 1) 10462 return typeof(return)(Redirection("http://dpldocs.info/")); 10463 else 10464 return typeof(return)("hi there!"); 10465 } 10466 --- 10467 10468 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10469 +/ 10470 struct MultipleResponses(T...) { 10471 private size_t contains; 10472 private union { 10473 private T payload; 10474 } 10475 10476 static foreach(index, type; T) 10477 public this(type t) { 10478 contains = index; 10479 payload[index] = t; 10480 } 10481 10482 /++ 10483 This is primarily for testing. It is your way of getting to the response. 10484 10485 Let's say you wanted to test that one holding a Redirection and a string actually 10486 holds a string, by name of "test": 10487 10488 --- 10489 auto valueToTest = your_test_function(); 10490 10491 valueToTest.visit( 10492 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10493 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10494 ); 10495 --- 10496 10497 History: 10498 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10499 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10500 +/ 10501 void visit(Handlers...)(Handlers handlers) { 10502 template findHandler(type, int count, HandlersToCheck...) { 10503 static if(HandlersToCheck.length == 0) 10504 enum findHandler = -1; 10505 else { 10506 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10507 enum findHandler = count; 10508 else 10509 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10510 } 10511 } 10512 foreach(index, type; T) { 10513 enum handlerIndex = findHandler!(type, 0, Handlers); 10514 static if(handlerIndex == -1) 10515 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10516 else { 10517 if(index == this.contains) 10518 handlers[handlerIndex](this.payload[index]); 10519 } 10520 } 10521 } 10522 10523 /+ 10524 auto toArsdJsvar()() { 10525 import arsd.jsvar; 10526 return var(null); 10527 } 10528 +/ 10529 } 10530 10531 // FIXME: implement this somewhere maybe 10532 struct RawResponse { 10533 int code; 10534 string[] headers; 10535 const(ubyte)[] responseBody; 10536 } 10537 10538 /++ 10539 You can return this from [WebObject] subclasses for redirections. 10540 10541 (though note the static types means that class must ALWAYS redirect if 10542 you return this directly. You might want to return [MultipleResponses] if it 10543 can be conditional) 10544 +/ 10545 struct Redirection { 10546 string to; /// The URL to redirect to. 10547 int code = 303; /// The HTTP code to return. 10548 } 10549 10550 /++ 10551 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10552 10553 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10554 the presenter in the dispatcher. 10555 10556 FIXME: explain this better 10557 10558 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10559 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10560 the runtime result of that is undefined. 10561 10562 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10563 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10564 10565 $(WARNING 10566 --- 10567 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10568 // even if you put different URL udas on it, the current code ignores them. 10569 void foo(int a) {} 10570 void foo(string a) {} 10571 --- 10572 ) 10573 10574 See_Also: [serveRestObject], [serveStaticFile] 10575 +/ 10576 auto serveApi(T)(string urlPrefix) { 10577 assert(urlPrefix[$ - 1] == '/'); 10578 return serveApiInternal!T(urlPrefix); 10579 } 10580 10581 private string nextPieceFromSlash(ref string remainingUrl) { 10582 if(remainingUrl.length == 0) 10583 return remainingUrl; 10584 int slash = 0; 10585 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10586 slash++; 10587 10588 // I am specifically passing `null` to differentiate it vs empty string 10589 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10590 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10591 // so if it is the last item, the dot can be used to load an alternative view 10592 // otherwise tho the dot is considered part of the identifier 10593 // FIXME 10594 10595 // again notice "" vs null here! 10596 if(slash == remainingUrl.length) 10597 remainingUrl = null; 10598 else 10599 remainingUrl = remainingUrl[slash + 1 .. $]; 10600 10601 return ident; 10602 } 10603 10604 /++ 10605 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10606 +/ 10607 enum AddTrailingSlash; 10608 /// ditto 10609 enum RemoveTrailingSlash; 10610 10611 private auto serveApiInternal(T)(string urlPrefix) { 10612 10613 import arsd.dom; 10614 import arsd.jsvar; 10615 10616 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10617 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10618 10619 try { 10620 // see duplicated code below by searching subresource_ctor 10621 // also see mustNotBeSetFromWebParams 10622 10623 static if(is(typeof(T.__ctor) P == __parameters)) { 10624 P params; 10625 10626 foreach(pidx, param; P) { 10627 static if(is(param : Cgi)) { 10628 static assert(!is(param == immutable)); 10629 cast() params[pidx] = cgi; 10630 } else static if(is(param == Session!D, D)) { 10631 static assert(!is(param == immutable)); 10632 cast() params[pidx] = cgi.getSessionObject!D(); 10633 10634 } else { 10635 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10636 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10637 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10638 static if(is(typeof(func(cgi)))) 10639 params[pidx] = func(cgi); 10640 else 10641 params[pidx] = func(); 10642 } 10643 } 10644 } else { 10645 10646 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10647 params[pidx] = param.getAutomaticallyForCgi(cgi); 10648 } else static if(is(param == string)) { 10649 auto ident = nextPieceFromSlash(remainingUrl); 10650 params[pidx] = ident; 10651 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10652 } 10653 } 10654 } 10655 10656 auto obj = new T(params); 10657 } else { 10658 auto obj = new T(); 10659 } 10660 10661 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10662 } catch(Throwable t) { 10663 switch(cgi.request("format", "html")) { 10664 case "html": 10665 static void dummy() {} 10666 presenter.presentExceptionAsHtml(cgi, t, null); 10667 return true; 10668 case "json": 10669 var envelope = var.emptyObject; 10670 envelope.success = false; 10671 envelope.result = null; 10672 envelope.error = t.toString(); 10673 cgi.setResponseContentType("application/json"); 10674 cgi.write(envelope.toJson(), true); 10675 return true; 10676 default: 10677 throw t; 10678 // return true; 10679 } 10680 // return true; 10681 } 10682 10683 assert(0); 10684 } 10685 10686 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10687 10688 obj.initialize(cgi); 10689 10690 /+ 10691 Overload rules: 10692 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10693 statically. 10694 10695 Moreover, some args vs no args can be overloaded dynamically. 10696 +/ 10697 10698 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10699 /+ 10700 auto orig = remainingUrl; 10701 assert(0, 10702 (orig is null ? "__null" : orig) 10703 ~ " .. " ~ 10704 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10705 +/ 10706 10707 if(methodNameFromUrl is null) 10708 methodNameFromUrl = "__null"; 10709 10710 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10711 10712 if(remainingUrl.length) 10713 hack ~= "/"; 10714 10715 switch(hack) { 10716 foreach(methodName; __traits(derivedMembers, T)) 10717 static if(methodName != "__ctor") 10718 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10719 static if(is(typeof(overload) P == __parameters)) 10720 static if(is(typeof(overload) R == return)) 10721 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10722 { 10723 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10724 case urlNameForMethod: 10725 10726 static if(is(R : WebObject)) { 10727 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10728 10729 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10730 10731 // subresource_ctor 10732 // also see mustNotBeSetFromWebParams 10733 10734 P params; 10735 10736 string ident; 10737 10738 foreach(pidx, param; P) { 10739 static if(is(param : Cgi)) { 10740 static assert(!is(param == immutable)); 10741 cast() params[pidx] = cgi; 10742 } else static if(is(param == typeof(presenter))) { 10743 cast() param[pidx] = presenter; 10744 } else static if(is(param == Session!D, D)) { 10745 static assert(!is(param == immutable)); 10746 cast() params[pidx] = cgi.getSessionObject!D(); 10747 } else { 10748 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10749 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10750 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10751 static if(is(typeof(func(cgi)))) 10752 params[pidx] = func(cgi); 10753 else 10754 params[pidx] = func(); 10755 } 10756 } 10757 } else { 10758 10759 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10760 params[pidx] = param.getAutomaticallyForCgi(cgi); 10761 } else static if(is(param == string)) { 10762 ident = nextPieceFromSlash(remainingUrl); 10763 if(ident is null) { 10764 // trailing slash mandated on subresources 10765 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10766 return true; 10767 } else { 10768 params[pidx] = ident; 10769 } 10770 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10771 } 10772 } 10773 } 10774 10775 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10776 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10777 } else { 10778 // 404 it if any url left - not a subresource means we don't get to play with that! 10779 if(remainingUrl.length) 10780 return false; 10781 10782 bool automaticForm; 10783 10784 foreach(attr; __traits(getAttributes, overload)) 10785 static if(is(attr == AddTrailingSlash)) { 10786 if(remainingUrl is null) { 10787 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10788 return true; 10789 } 10790 } else static if(is(attr == RemoveTrailingSlash)) { 10791 if(remainingUrl !is null) { 10792 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10793 return true; 10794 } 10795 10796 } else static if(__traits(isSame, AutomaticForm, attr)) { 10797 automaticForm = true; 10798 } 10799 10800 /+ 10801 int zeroArgOverload = -1; 10802 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10803 bool calledWithZeroArgs = true; 10804 foreach(k, v; cgi.get) 10805 if(k != "format") { 10806 calledWithZeroArgs = false; 10807 break; 10808 } 10809 foreach(k, v; cgi.post) 10810 if(k != "format") { 10811 calledWithZeroArgs = false; 10812 break; 10813 } 10814 10815 // first, we need to go through and see if there is an empty one, since that 10816 // changes inside. But otherwise, all the stuff I care about can be done via 10817 // simple looping (other improper overloads might be flagged for runtime semantic check) 10818 // 10819 // an argument of type Cgi is ignored for these purposes 10820 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10821 static if(is(typeof(overload) P == __parameters)) 10822 static if(P.length == 0) 10823 zeroArgOverload = cast(int) idx; 10824 else static if(P.length == 1 && is(P[0] : Cgi)) 10825 zeroArgOverload = cast(int) idx; 10826 }} 10827 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10828 bool overloadHasBeenCalled = false; 10829 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10830 bool callFunction = true; 10831 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10832 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10833 callFunction = false; 10834 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10835 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10836 callFunction = false; 10837 10838 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10839 10840 bool hadAnyMethodRestrictions = false; 10841 bool foundAcceptableMethod = false; 10842 foreach(attr; __traits(getAttributes, overload)) { 10843 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10844 hadAnyMethodRestrictions = true; 10845 if(attr == cgi.requestMethod) 10846 foundAcceptableMethod = true; 10847 } 10848 } 10849 10850 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10851 callFunction = false; 10852 10853 /+ 10854 The overloads we really want to allow are the sane ones 10855 from the web perspective. Which is likely on HTTP verbs, 10856 for the most part, but might also be potentially based on 10857 some args vs zero args, or on argument names. Can't really 10858 do argument types very reliable through the web though; those 10859 should probably be different URLs. 10860 10861 Even names I feel is better done inside the function, so I'm not 10862 going to support that here. But the HTTP verbs and zero vs some 10863 args makes sense - it lets you define custom forms pretty easily. 10864 10865 Moreover, I'm of the opinion that empty overload really only makes 10866 sense on GET for this case. On a POST, it is just a missing argument 10867 exception and that should be handled by the presenter. But meh, I'll 10868 let the user define that, D only allows one empty arg thing anyway 10869 so the method UDAs are irrelevant. 10870 +/ 10871 if(callFunction) 10872 +/ 10873 10874 auto format = cgi.request("format", defaultFormat!overload()); 10875 auto wantsFormFormat = format.startsWith("form-"); 10876 10877 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10878 // Should I still show the form on a json thing? idk... 10879 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10880 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10881 return true; 10882 } 10883 10884 try { 10885 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10886 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10887 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10888 } catch(Throwable t) { 10889 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10890 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10891 } 10892 return true; 10893 //}} 10894 10895 //cgi.header("Accept: POST"); // FIXME list the real thing 10896 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10897 //return true; 10898 } 10899 } 10900 } 10901 case "GET script.js": 10902 cgi.setResponseContentType("text/javascript"); 10903 cgi.gzipResponse = true; 10904 cgi.write(presenter.script(), true); 10905 return true; 10906 case "GET style.css": 10907 cgi.setResponseContentType("text/css"); 10908 cgi.gzipResponse = true; 10909 cgi.write(presenter.style(), true); 10910 return true; 10911 default: 10912 return false; 10913 } 10914 10915 assert(0); 10916 } 10917 return DispatcherDefinition!internalHandler(urlPrefix, false); 10918 } 10919 10920 string defaultFormat(alias method)() { 10921 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10922 foreach(attr; __traits(getAttributes, method)) { 10923 static if(is(typeof(attr) == DefaultFormat)) { 10924 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10925 return attr.value; 10926 } 10927 } 10928 return "html"; 10929 } 10930 10931 struct Paginated(T) { 10932 T[] items; 10933 string nextPageUrl; 10934 } 10935 10936 template urlNamesForMethod(alias method, string default_) { 10937 string[] helper() { 10938 auto verb = Cgi.RequestMethod.GET; 10939 bool foundVerb = false; 10940 bool foundNoun = false; 10941 10942 string def = default_; 10943 10944 bool hasAutomaticForm = false; 10945 10946 foreach(attr; __traits(getAttributes, method)) { 10947 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10948 verb = attr; 10949 if(foundVerb) 10950 assert(0, "Multiple http verbs on one function is not currently supported"); 10951 foundVerb = true; 10952 } 10953 static if(is(typeof(attr) == UrlName)) { 10954 if(foundNoun) 10955 assert(0, "Multiple url names on one function is not currently supported"); 10956 foundNoun = true; 10957 def = attr.name; 10958 } 10959 static if(__traits(isSame, attr, AutomaticForm)) { 10960 hasAutomaticForm = true; 10961 } 10962 } 10963 10964 if(def is null) 10965 def = "__null"; 10966 10967 string[] ret; 10968 10969 static if(is(typeof(method) R == return)) { 10970 static if(is(R : WebObject)) { 10971 def ~= "/"; 10972 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10973 ret ~= v ~ " " ~ def; 10974 } else { 10975 if(hasAutomaticForm) { 10976 ret ~= "GET " ~ def; 10977 ret ~= "POST " ~ def; 10978 } else { 10979 ret ~= to!string(verb) ~ " " ~ def; 10980 } 10981 } 10982 } else static assert(0); 10983 10984 return ret; 10985 } 10986 enum urlNamesForMethod = helper(); 10987 } 10988 10989 10990 enum AccessCheck { 10991 allowed, 10992 denied, 10993 nonExistant, 10994 } 10995 10996 enum Operation { 10997 show, 10998 create, 10999 replace, 11000 remove, 11001 update 11002 } 11003 11004 enum UpdateResult { 11005 accessDenied, 11006 noSuchResource, 11007 success, 11008 failure, 11009 unnecessary 11010 } 11011 11012 enum ValidationResult { 11013 valid, 11014 invalid 11015 } 11016 11017 11018 /++ 11019 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 11020 11021 WARNING: this is not stable. 11022 +/ 11023 class RestObject(CRTP) : WebObject { 11024 11025 import arsd.dom; 11026 import arsd.jsvar; 11027 11028 /// Prepare the object to be shown. 11029 void show() {} 11030 /// ditto 11031 void show(string urlId) { 11032 load(urlId); 11033 show(); 11034 } 11035 11036 /// Override this to provide access control to this object. 11037 AccessCheck accessCheck(string urlId, Operation operation) { 11038 return AccessCheck.allowed; 11039 } 11040 11041 ValidationResult validate() { 11042 // FIXME 11043 return ValidationResult.valid; 11044 } 11045 11046 string getUrlSlug() { 11047 import std.conv; 11048 static if(is(typeof(CRTP.id))) 11049 return to!string((cast(CRTP) this).id); 11050 else 11051 return null; 11052 } 11053 11054 // The functions with more arguments are the low-level ones, 11055 // they forward to the ones with fewer arguments by default. 11056 11057 // POST on a parent collection - this is called from a collection class after the members are updated 11058 /++ 11059 Given a populated object, this creates a new entry. Returns the url identifier 11060 of the new object. 11061 +/ 11062 string create(scope void delegate() applyChanges) { 11063 applyChanges(); 11064 save(); 11065 return getUrlSlug(); 11066 } 11067 11068 void replace() { 11069 save(); 11070 } 11071 void replace(string urlId, scope void delegate() applyChanges) { 11072 load(urlId); 11073 applyChanges(); 11074 replace(); 11075 } 11076 11077 void update(string[] fieldList) { 11078 save(); 11079 } 11080 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11081 load(urlId); 11082 applyChanges(); 11083 update(fieldList); 11084 } 11085 11086 void remove() {} 11087 11088 void remove(string urlId) { 11089 load(urlId); 11090 remove(); 11091 } 11092 11093 abstract void load(string urlId); 11094 abstract void save(); 11095 11096 Element toHtml(Presenter)(Presenter presenter) { 11097 import arsd.dom; 11098 import std.conv; 11099 auto obj = cast(CRTP) this; 11100 auto div = Element.make("div"); 11101 div.addClass("Dclass_" ~ CRTP.stringof); 11102 div.dataset.url = getUrlSlug(); 11103 bool first = true; 11104 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11105 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11106 if(!first) div.addChild("br"); else first = false; 11107 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11108 } 11109 return div; 11110 } 11111 11112 var toJson() { 11113 import arsd.jsvar; 11114 var v = var.emptyObject(); 11115 auto obj = cast(CRTP) this; 11116 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11117 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11118 v[memberName] = __traits(getMember, obj, memberName); 11119 } 11120 return v; 11121 } 11122 11123 /+ 11124 auto structOf(this This) { 11125 11126 } 11127 +/ 11128 } 11129 11130 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11131 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11132 11133 /++ 11134 Base class for REST collections. 11135 +/ 11136 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11137 /// You might subclass this and use the cgi object's query params 11138 /// to implement a search filter, for example. 11139 /// 11140 /// FIXME: design a way to auto-generate that form 11141 /// (other than using the WebObject thing above lol 11142 // it'll prolly just be some searchParams UDA or maybe an enum. 11143 // 11144 // pagination too perhaps. 11145 // 11146 // and sorting too 11147 IndexResult index() { return IndexResult.init; } 11148 11149 string[] sortableFields() { return null; } 11150 string[] searchableFields() { return null; } 11151 11152 struct IndexResult { 11153 Obj[] results; 11154 11155 string[] sortableFields; 11156 11157 string previousPageIdentifier; 11158 string nextPageIdentifier; 11159 string firstPageIdentifier; 11160 string lastPageIdentifier; 11161 11162 int numberOfPages; 11163 } 11164 11165 override string create(scope void delegate() applyChanges) { assert(0); } 11166 override void load(string urlId) { assert(0); } 11167 override void save() { assert(0); } 11168 override void show() { 11169 index(); 11170 } 11171 override void show(string urlId) { 11172 show(); 11173 } 11174 11175 /// Proxy POST requests (create calls) to the child collection 11176 alias PostProxy = Obj; 11177 } 11178 11179 /++ 11180 Serves a REST object, similar to a Ruby on Rails resource. 11181 11182 You put data members in your class. cgi.d will automatically make something out of those. 11183 11184 It will call your constructor with the ID from the URL. This may be null. 11185 It will then populate the data members from the request. 11186 It will then call a method, if present, telling what happened. You don't need to write these! 11187 It finally returns a reply. 11188 11189 Your methods are passed a list of fields it actually set. 11190 11191 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11192 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11193 with relative linking. But meh.) 11194 11195 GET /items -> index. all values not set. 11196 GET /items/id -> get. only ID will be set, other params ignored. 11197 POST /items -> create. values set as given 11198 PUT /items/id -> replace. values set as given 11199 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11200 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11201 PATCH /items/id -> update. values set as given, list of changed fields passed 11202 or POST /items/id with cgi.post["_method"] == "PATCH" 11203 DELETE /items/id -> destroy. only ID guaranteed to be set 11204 or POST /items/id with cgi.post["_method"] == "DELETE" 11205 11206 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11207 redirect you away from it. 11208 11209 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11210 11211 I will also let you change the default, if you must. 11212 11213 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11214 11215 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11216 that follow the same thing. They may be individual resources or collections themselves. 11217 11218 Your class is expected to have at least the following methods: 11219 11220 FIXME: i kinda wanna add a routes object to the initialize call 11221 11222 create 11223 Create returns the new address on success, some code on failure. 11224 show 11225 index 11226 update 11227 remove 11228 11229 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11230 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11231 11232 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11233 11234 NOT IMPLEMENTED 11235 11236 11237 Really, a collection is a resource with a bunch of subresources. 11238 11239 GET /items 11240 index because it is GET on the top resource 11241 11242 GET /items/foo 11243 item but different than items? 11244 11245 class Items { 11246 11247 } 11248 11249 ... but meh, a collection can be automated. not worth making it 11250 a separate thing, let's look at a real example. Users has many 11251 items and a virtual one, /users/current. 11252 11253 the individual users have properties and two sub-resources: 11254 session, which is just one, and comments, a collection. 11255 11256 class User : RestObject!() { // no parent 11257 int id; 11258 string name; 11259 11260 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11261 // but you can override them to do it differently. 11262 11263 // any member which is of type RestObject can be linked automatically via href btw. 11264 11265 void show() {} 11266 void show(string urlId) {} // automated! GET of this specific thing 11267 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11268 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11269 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11270 void remove(string urlId) {} // DELETE 11271 11272 void load(string urlId) {} // the default implementation of show() populates the id, then 11273 11274 this() {} 11275 11276 mixin Subresource!Session; 11277 mixin Subresource!Comment; 11278 } 11279 11280 class Session : RestObject!() { 11281 // the parent object may not be fully constructed/loaded 11282 this(User parent) {} 11283 11284 } 11285 11286 class Comment : CollectionOf!Comment { 11287 this(User parent) {} 11288 } 11289 11290 class Users : CollectionOf!User { 11291 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11292 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11293 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11294 } 11295 11296 +/ 11297 auto serveRestObject(T)(string urlPrefix) { 11298 assert(urlPrefix[0] == '/'); 11299 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11300 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11301 string url = cgi.pathInfo[urlPrefix.length .. $]; 11302 11303 if(url.length && url[$ - 1] == '/') { 11304 // remove the final slash... 11305 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11306 return true; 11307 } 11308 11309 return restObjectServeHandler!T(cgi, presenter, url); 11310 } 11311 return DispatcherDefinition!internalHandler(urlPrefix, false); 11312 } 11313 11314 /+ 11315 /// Convenience method for serving a collection. It will be named the same 11316 /// as type T, just with an s at the end. If you need any further, just 11317 /// write the class yourself. 11318 auto serveRestCollectionOf(T)(string urlPrefix) { 11319 assert(urlPrefix[0] == '/'); 11320 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11321 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11322 } 11323 +/ 11324 11325 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11326 string urlId = null; 11327 if(url.length && url[0] == '/') { 11328 // asking for a subobject 11329 urlId = url[1 .. $]; 11330 foreach(idx, ch; urlId) { 11331 if(ch == '/') { 11332 urlId = urlId[0 .. idx]; 11333 break; 11334 } 11335 } 11336 } 11337 11338 // FIXME handle other subresources 11339 11340 static if(is(T : CollectionOf!(C), C)) { 11341 if(urlId !is null) { 11342 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11343 } 11344 } 11345 11346 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11347 11348 auto obj = new T(); 11349 obj.initialize(cgi); 11350 // FIXME: populate reflection info delegates 11351 11352 11353 // FIXME: I am not happy with this. 11354 switch(urlId) { 11355 case "script.js": 11356 cgi.setResponseContentType("text/javascript"); 11357 cgi.gzipResponse = true; 11358 cgi.write(presenter.script(), true); 11359 return true; 11360 case "style.css": 11361 cgi.setResponseContentType("text/css"); 11362 cgi.gzipResponse = true; 11363 cgi.write(presenter.style(), true); 11364 return true; 11365 default: 11366 // intentionally blank 11367 } 11368 11369 11370 11371 11372 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11373 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11374 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11375 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11376 } 11377 } 11378 void applyChanges() { 11379 applyChangesTemplate(cgi, obj); 11380 } 11381 11382 string[] modifiedList; 11383 11384 void writeObject(bool addFormLinks) { 11385 if(cgi.request("format") == "json") { 11386 cgi.setResponseContentType("application/json"); 11387 cgi.write(obj.toJson().toString, true); 11388 } else { 11389 auto container = presenter.htmlContainer(); 11390 if(addFormLinks) { 11391 static if(is(T : CollectionOf!(C), C)) 11392 container.appendHtml(` 11393 <form> 11394 <button type="submit" name="_method" value="POST">Create New</button> 11395 </form> 11396 `); 11397 else 11398 container.appendHtml(` 11399 <a href="..">Back</a> 11400 <form> 11401 <button type="submit" name="_method" value="PATCH">Edit</button> 11402 <button type="submit" name="_method" value="DELETE">Delete</button> 11403 </form> 11404 `); 11405 } 11406 container.appendChild(obj.toHtml(presenter)); 11407 cgi.write(container.parentDocument.toString, true); 11408 } 11409 } 11410 11411 // FIXME: I think I need a set type in here.... 11412 // it will be nice to pass sets of members. 11413 11414 try 11415 switch(cgi.requestMethod) { 11416 case Cgi.RequestMethod.GET: 11417 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11418 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11419 11420 // automatic forms here for usable basic auto site from browser. 11421 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11422 switch(cgi.request("_method", "GET")) { 11423 case "GET": 11424 static if(is(T : CollectionOf!(C), C)) { 11425 auto results = obj.index(); 11426 if(cgi.request("format", "html") == "html") { 11427 auto container = presenter.htmlContainer(); 11428 auto html = presenter.formatReturnValueAsHtml(results.results); 11429 container.appendHtml(` 11430 <form> 11431 <button type="submit" name="_method" value="POST">Create New</button> 11432 </form> 11433 `); 11434 11435 container.appendChild(html); 11436 cgi.write(container.parentDocument.toString, true); 11437 } else { 11438 cgi.setResponseContentType("application/json"); 11439 import arsd.jsvar; 11440 var json = var.emptyArray; 11441 foreach(r; results.results) { 11442 var o = var.emptyObject; 11443 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11444 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11445 o[memberName] = __traits(getMember, r, memberName); 11446 } 11447 11448 json ~= o; 11449 } 11450 cgi.write(json.toJson(), true); 11451 } 11452 } else { 11453 obj.show(urlId); 11454 writeObject(true); 11455 } 11456 break; 11457 case "PATCH": 11458 obj.load(urlId); 11459 goto case; 11460 case "PUT": 11461 case "POST": 11462 // an editing form for the object 11463 auto container = presenter.htmlContainer(); 11464 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11465 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11466 } else { 11467 auto form = presenter.createAutomaticFormForObject(obj); 11468 } 11469 form.attrs.method = "POST"; 11470 form.setValue("_method", cgi.request("_method", "GET")); 11471 container.appendChild(form); 11472 cgi.write(container.parentDocument.toString(), true); 11473 break; 11474 case "DELETE": 11475 // FIXME: a delete form for the object (can be phrased "are you sure?") 11476 auto container = presenter.htmlContainer(); 11477 container.appendHtml(` 11478 <form method="POST"> 11479 Are you sure you want to delete this item? 11480 <input type="hidden" name="_method" value="DELETE" /> 11481 <input type="submit" value="Yes, Delete It" /> 11482 </form> 11483 11484 `); 11485 cgi.write(container.parentDocument.toString(), true); 11486 break; 11487 default: 11488 cgi.write("bad method\n", true); 11489 } 11490 break; 11491 case Cgi.RequestMethod.POST: 11492 // this is to allow compatibility with HTML forms 11493 switch(cgi.request("_method", "POST")) { 11494 case "PUT": 11495 goto PUT; 11496 case "PATCH": 11497 goto PATCH; 11498 case "DELETE": 11499 goto DELETE; 11500 case "POST": 11501 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11502 auto p = new obj.PostProxy(); 11503 void specialApplyChanges() { 11504 applyChangesTemplate(cgi, p); 11505 } 11506 string n = p.create(&specialApplyChanges); 11507 } else { 11508 string n = obj.create(&applyChanges); 11509 } 11510 11511 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11512 cgi.setResponseLocation(newUrl); 11513 cgi.setResponseStatus("201 Created"); 11514 cgi.write(`The object has been created.`); 11515 break; 11516 default: 11517 cgi.write("bad method\n", true); 11518 } 11519 // FIXME this should be valid on the collection, but not the child.... 11520 // 303 See Other 11521 break; 11522 case Cgi.RequestMethod.PUT: 11523 PUT: 11524 obj.replace(urlId, &applyChanges); 11525 writeObject(false); 11526 break; 11527 case Cgi.RequestMethod.PATCH: 11528 PATCH: 11529 obj.update(urlId, &applyChanges, modifiedList); 11530 writeObject(false); 11531 break; 11532 case Cgi.RequestMethod.DELETE: 11533 DELETE: 11534 obj.remove(urlId); 11535 cgi.setResponseStatus("204 No Content"); 11536 break; 11537 default: 11538 // FIXME: OPTIONS, HEAD 11539 } 11540 catch(Throwable t) { 11541 presenter.presentExceptionAsHtml(cgi, t); 11542 } 11543 11544 return true; 11545 } 11546 11547 /+ 11548 struct SetOfFields(T) { 11549 private void[0][string] storage; 11550 void set(string what) { 11551 //storage[what] = 11552 } 11553 void unset(string what) {} 11554 void setAll() {} 11555 void unsetAll() {} 11556 bool isPresent(string what) { return false; } 11557 } 11558 +/ 11559 11560 /+ 11561 enum readonly; 11562 enum hideonindex; 11563 +/ 11564 11565 /++ 11566 Returns true if I recommend gzipping content of this type. You might 11567 want to call it from your Presenter classes before calling cgi.write. 11568 11569 --- 11570 cgi.setResponseContentType(yourContentType); 11571 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11572 cgi.write(yourData, true); 11573 --- 11574 11575 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11576 11577 11578 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11579 11580 History: 11581 Added January 28, 2023 (dub v11.0) 11582 +/ 11583 bool gzipRecommendedForContentType(string contentType) { 11584 if(contentType.startsWith("text/")) 11585 return true; 11586 if(contentType.startsWith("application/javascript")) 11587 return true; 11588 11589 return false; 11590 } 11591 11592 /++ 11593 Serves a static file. To be used with [dispatcher]. 11594 11595 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11596 +/ 11597 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11598 // https://baus.net/on-tcp_cork/ 11599 // man 2 sendfile 11600 assert(urlPrefix[0] == '/'); 11601 if(filename is null) 11602 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11603 if(contentType is null) { 11604 contentType = contentTypeFromFileExtension(filename); 11605 } 11606 11607 static struct DispatcherDetails { 11608 string filename; 11609 string contentType; 11610 } 11611 11612 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11613 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11614 cgi.setCache(true); 11615 cgi.setResponseContentType(details.contentType); 11616 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11617 cgi.write(std.file.read(details.filename), true); 11618 return true; 11619 } 11620 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11621 } 11622 11623 /++ 11624 Serves static data. To be used with [dispatcher]. 11625 11626 History: 11627 Added October 31, 2021 11628 +/ 11629 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11630 assert(urlPrefix[0] == '/'); 11631 if(contentType is null) { 11632 contentType = contentTypeFromFileExtension(urlPrefix); 11633 } 11634 11635 static struct DispatcherDetails { 11636 immutable(void)[] data; 11637 string contentType; 11638 } 11639 11640 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11641 cgi.setCache(true); 11642 cgi.setResponseContentType(details.contentType); 11643 cgi.write(details.data, true); 11644 return true; 11645 } 11646 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11647 } 11648 11649 string contentTypeFromFileExtension(string filename) { 11650 if(filename.endsWith(".png")) 11651 return "image/png"; 11652 if(filename.endsWith(".apng")) 11653 return "image/apng"; 11654 if(filename.endsWith(".svg")) 11655 return "image/svg+xml"; 11656 if(filename.endsWith(".jpg")) 11657 return "image/jpeg"; 11658 if(filename.endsWith(".html")) 11659 return "text/html"; 11660 if(filename.endsWith(".css")) 11661 return "text/css"; 11662 if(filename.endsWith(".js")) 11663 return "application/javascript"; 11664 if(filename.endsWith(".wasm")) 11665 return "application/wasm"; 11666 if(filename.endsWith(".mp3")) 11667 return "audio/mpeg"; 11668 if(filename.endsWith(".pdf")) 11669 return "application/pdf"; 11670 return null; 11671 } 11672 11673 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11674 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11675 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11676 assert(urlPrefix[0] == '/'); 11677 assert(urlPrefix[$-1] == '/'); 11678 11679 static struct DispatcherDetails { 11680 string directory; 11681 bool recursive; 11682 } 11683 11684 if(directory is null) 11685 directory = urlPrefix[1 .. $]; 11686 11687 if(directory.length == 0) 11688 directory = "./"; 11689 11690 assert(directory[$-1] == '/'); 11691 11692 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11693 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11694 11695 if(details.recursive) { 11696 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11697 if(file.indexOf("\\") != -1) 11698 return false; 11699 11700 import std.path; 11701 11702 file = std.path.buildNormalizedPath(file); 11703 enum upOneDir = ".." ~ std.path.dirSeparator; 11704 11705 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11706 if(file == ".." || file.startsWith(upOneDir)) 11707 return false; 11708 if(std.path.isAbsolute(file)) 11709 return false; 11710 11711 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11712 11713 // once it passes these filters it is probably ok. 11714 } else { 11715 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11716 return false; 11717 } 11718 11719 auto contentType = contentTypeFromFileExtension(file); 11720 11721 auto fn = details.directory ~ file; 11722 if(std.file.exists(fn)) { 11723 //if(contentType.indexOf("image/") == 0) 11724 //cgi.setCache(true); 11725 //else if(contentType.indexOf("audio/") == 0) 11726 cgi.setCache(true); 11727 cgi.setResponseContentType(contentType); 11728 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11729 cgi.write(std.file.read(fn), true); 11730 return true; 11731 } else { 11732 return false; 11733 } 11734 } 11735 11736 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11737 } 11738 11739 /++ 11740 Redirects one url to another 11741 11742 See_Also: [dispatcher], [serveStaticFile] 11743 +/ 11744 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11745 assert(urlPrefix[0] == '/'); 11746 static struct DispatcherDetails { 11747 string redirectTo; 11748 string code; 11749 } 11750 11751 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11752 cgi.setResponseLocation(details.redirectTo, true, details.code); 11753 return true; 11754 } 11755 11756 11757 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11758 } 11759 11760 /// Used exclusively with `dispatchTo` 11761 struct DispatcherData(Presenter) { 11762 Cgi cgi; /// You can use this cgi object. 11763 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11764 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11765 } 11766 11767 /++ 11768 Dispatches the URL to a specific function. 11769 +/ 11770 auto handleWith(alias handler)(string urlPrefix) { 11771 // cuz I'm too lazy to do it better right now 11772 static class Hack : WebObject { 11773 static import std.traits; 11774 @UrlName("") 11775 auto handle(std.traits.Parameters!handler args) { 11776 return handler(args); 11777 } 11778 } 11779 11780 return urlPrefix.serveApiInternal!Hack; 11781 } 11782 11783 /++ 11784 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11785 11786 --- 11787 bool other(DD)(DD dd) { 11788 return dd.dispatcher!( 11789 "/whatever".serveRedirect("/success"), 11790 "/api/".serveApi!MyClass 11791 ); 11792 } 11793 --- 11794 11795 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11796 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11797 Or, of course, you could just use the exact type in your own code. 11798 11799 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11800 good job. 11801 11802 11803 +/ 11804 auto dispatchTo(alias handler)(string urlPrefix) { 11805 assert(urlPrefix[0] == '/'); 11806 assert(urlPrefix[$-1] != '/'); 11807 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11808 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11809 } 11810 11811 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11812 } 11813 11814 /++ 11815 See [serveStaticFile] if you want to serve a file off disk. 11816 11817 History: 11818 Added January 28, 2023 (dub v11.0) 11819 +/ 11820 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11821 assert(urlPrefix[0] == '/'); 11822 11823 static struct DispatcherDetails { 11824 immutable(ubyte)[] data; 11825 string contentType; 11826 string filenameToSuggestAsDownload; 11827 } 11828 11829 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11830 cgi.setCache(true); 11831 cgi.setResponseContentType(details.contentType); 11832 if(details.filenameToSuggestAsDownload.length) 11833 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11834 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11835 cgi.write(details.data, true); 11836 return true; 11837 } 11838 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11839 } 11840 11841 /++ 11842 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11843 11844 History: 11845 Added January 28, 2023 (dub v11.0) 11846 +/ 11847 alias KeepExistingPresenter = typeof(null); 11848 11849 /++ 11850 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11851 this issues the given errorCode and stops processing. 11852 11853 --- 11854 bool hasAdminPermissions(Cgi cgi) { 11855 return true; 11856 } 11857 11858 mixin DispatcherMain!( 11859 "/admin".dispatchSubsection!( 11860 passFilterOrIssueError!(hasAdminPermissions, 403), 11861 KeepExistingPresenter, 11862 "/".serveApi!AdminFunctions 11863 ) 11864 ); 11865 --- 11866 11867 History: 11868 Added January 28, 2023 (dub v11.0) 11869 +/ 11870 template passFilterOrIssueError(alias filter, int errorCode) { 11871 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11872 if(filter(dd.cgi)) 11873 return true; 11874 dd.presenter.renderBasicError(dd.cgi, errorCode); 11875 return false; 11876 } 11877 } 11878 11879 /++ 11880 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11881 and then be dispatched to their own handlers. 11882 11883 --- 11884 /+ 11885 // a long-form filter function 11886 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11887 // you are permitted to call mutable methods on the Cgi object 11888 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11889 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11890 11891 if(checkPermissionOnRequest(dd.cgi)) { 11892 return true; // OK, allow processing to continue 11893 } else { 11894 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11895 return false; // and stop further processing into this subsection 11896 } 11897 } 11898 +/ 11899 11900 // but you can also do short-form filters: 11901 11902 bool permissionCheck(Cgi cgi) { 11903 return ("ok" in cgi.get) !is null; 11904 } 11905 11906 // handler for the subsection 11907 class AdminClass : WebObject { 11908 int foo() { return 5; } 11909 } 11910 11911 // handler for the main site 11912 class TheMainSite : WebObject {} 11913 11914 mixin DispatcherMain!( 11915 "/admin".dispatchSubsection!( 11916 // converts our short-form filter into a long-form filter 11917 passFilterOrIssueError!(permissionCheck, 403), 11918 // can use a new presenter if wanted for the subsection 11919 KeepExistingPresenter, 11920 // and then provide child route dispatchers 11921 "/".serveApi!AdminClass 11922 ), 11923 // and back to the top level 11924 "/".serveApi!TheMainSite 11925 ); 11926 --- 11927 11928 Note you can encapsulate sections in files like this: 11929 11930 --- 11931 auto adminDispatcher(string urlPrefix) { 11932 return urlPrefix.dispatchSubsection!( 11933 .... 11934 ); 11935 } 11936 11937 mixin DispatcherMain!( 11938 "/admin".adminDispatcher, 11939 // and so on 11940 ) 11941 --- 11942 11943 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11944 11945 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11946 11947 11948 History: 11949 Added January 28, 2023 (dub v11.0) 11950 +/ 11951 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11952 assert(urlPrefix[0] == '/'); 11953 assert(urlPrefix[$-1] != '/'); 11954 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11955 static if(!is(PreRequestFilter == typeof(null))) { 11956 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11957 return true; // we handled it by rejecting it 11958 } 11959 11960 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11961 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11962 } else { 11963 auto newPresenter = new NewPresenter(); 11964 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11965 } 11966 } 11967 11968 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11969 } 11970 11971 /++ 11972 A URL dispatcher. 11973 11974 --- 11975 if(cgi.dispatcher!( 11976 "/api/".serveApi!MyApiClass, 11977 "/objects/lol".serveRestObject!MyRestObject, 11978 "/file.js".serveStaticFile, 11979 "/admin/".dispatchTo!adminHandler 11980 )) return; 11981 --- 11982 11983 11984 You define a series of url prefixes followed by handlers. 11985 11986 You may want to do different pre- and post- processing there, for example, 11987 an authorization check and different page layout. You can use different 11988 presenters and different function chains. See [dispatchSubsection] for details. 11989 11990 [dispatchTo] will send the request to another function for handling. 11991 +/ 11992 template dispatcher(definitions...) { 11993 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11994 static if(is(Presenter == typeof(null))) { 11995 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11996 auto presenter = new GenericWebPresenter(); 11997 } else 11998 alias presenter = presenterArg; 11999 12000 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 12001 } 12002 12003 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 12004 // I can prolly make this more efficient later but meh. 12005 foreach(definition; definitions) { 12006 if(definition.rejectFurther) { 12007 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 12008 auto ret = definition.handler( 12009 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12010 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12011 if(ret) 12012 return true; 12013 } 12014 } else if( 12015 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 12016 // cgi.d dispatcher urls must be complete or have a /; 12017 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 12018 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 12019 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 12020 ) { 12021 auto ret = definition.handler( 12022 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12023 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12024 if(ret) 12025 return true; 12026 } 12027 } 12028 return false; 12029 } 12030 } 12031 12032 }); 12033 12034 private struct StackBuffer { 12035 char[1024] initial = void; 12036 char[] buffer; 12037 size_t position; 12038 12039 this(int a) { 12040 buffer = initial[]; 12041 position = 0; 12042 } 12043 12044 void add(in char[] what) { 12045 if(position + what.length > buffer.length) 12046 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 12047 buffer[position .. position + what.length] = what[]; 12048 position += what.length; 12049 } 12050 12051 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 12052 add(w1); 12053 add(w2); 12054 add(w3); 12055 } 12056 12057 void add(long v) { 12058 char[16] buffer = void; 12059 auto pos = buffer.length; 12060 bool negative; 12061 if(v < 0) { 12062 negative = true; 12063 v = -v; 12064 } 12065 do { 12066 buffer[--pos] = cast(char) (v % 10 + '0'); 12067 v /= 10; 12068 } while(v); 12069 12070 if(negative) 12071 buffer[--pos] = '-'; 12072 12073 auto res = buffer[pos .. $]; 12074 12075 add(res[]); 12076 } 12077 12078 char[] get() @nogc { 12079 return buffer[0 .. position]; 12080 } 12081 } 12082 12083 // duplicated in http2.d 12084 private static string getHttpCodeText(int code) pure nothrow @nogc { 12085 switch(code) { 12086 case 200: return "200 OK"; 12087 case 201: return "201 Created"; 12088 case 202: return "202 Accepted"; 12089 case 203: return "203 Non-Authoritative Information"; 12090 case 204: return "204 No Content"; 12091 case 205: return "205 Reset Content"; 12092 case 206: return "206 Partial Content"; 12093 // 12094 case 300: return "300 Multiple Choices"; 12095 case 301: return "301 Moved Permanently"; 12096 case 302: return "302 Found"; 12097 case 303: return "303 See Other"; 12098 case 304: return "304 Not Modified"; 12099 case 305: return "305 Use Proxy"; 12100 case 307: return "307 Temporary Redirect"; 12101 case 308: return "308 Permanent Redirect"; 12102 12103 // 12104 case 400: return "400 Bad Request"; 12105 case 401: return "401 Unauthorized"; 12106 case 402: return "402 Payment Required"; 12107 case 403: return "403 Forbidden"; 12108 case 404: return "404 Not Found"; 12109 case 405: return "405 Method Not Allowed"; 12110 case 406: return "406 Not Acceptable"; 12111 case 407: return "407 Proxy Authentication Required"; 12112 case 408: return "408 Request Timeout"; 12113 case 409: return "409 Conflict"; 12114 case 410: return "410 Gone"; 12115 case 411: return "411 Length Required"; 12116 case 412: return "412 Precondition Failed"; 12117 case 413: return "413 Payload Too Large"; 12118 case 414: return "414 URI Too Long"; 12119 case 415: return "415 Unsupported Media Type"; 12120 case 416: return "416 Range Not Satisfiable"; 12121 case 417: return "417 Expectation Failed"; 12122 case 418: return "418 I'm a teapot"; 12123 case 421: return "421 Misdirected Request"; 12124 case 422: return "422 Unprocessable Entity (WebDAV)"; 12125 case 423: return "423 Locked (WebDAV)"; 12126 case 424: return "424 Failed Dependency (WebDAV)"; 12127 case 425: return "425 Too Early"; 12128 case 426: return "426 Upgrade Required"; 12129 case 428: return "428 Precondition Required"; 12130 case 431: return "431 Request Header Fields Too Large"; 12131 case 451: return "451 Unavailable For Legal Reasons"; 12132 12133 case 500: return "500 Internal Server Error"; 12134 case 501: return "501 Not Implemented"; 12135 case 502: return "502 Bad Gateway"; 12136 case 503: return "503 Service Unavailable"; 12137 case 504: return "504 Gateway Timeout"; 12138 case 505: return "505 HTTP Version Not Supported"; 12139 case 506: return "506 Variant Also Negotiates"; 12140 case 507: return "507 Insufficient Storage (WebDAV)"; 12141 case 508: return "508 Loop Detected (WebDAV)"; 12142 case 510: return "510 Not Extended"; 12143 case 511: return "511 Network Authentication Required"; 12144 // 12145 default: assert(0, "Unsupported http code"); 12146 } 12147 } 12148 12149 12150 /+ 12151 /++ 12152 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12153 12154 It relies on jsvar.d and dom.d. 12155 12156 12157 You can get javascript out of it to call. The generated functions need to look 12158 like 12159 12160 function name(a,b,c,d,e) { 12161 return _call("name", {"realName":a,"sds":b}); 12162 } 12163 12164 And _call returns an object you can call or set up or whatever. 12165 +/ 12166 bool apiDispatcher()(Cgi cgi) { 12167 import arsd.jsvar; 12168 import arsd.dom; 12169 } 12170 +/ 12171 version(linux) 12172 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12173 /* 12174 Copyright: Adam D. Ruppe, 2008 - 2023 12175 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12176 Authors: Adam D. Ruppe 12177 12178 Copyright Adam D. Ruppe 2008 - 2023. 12179 Distributed under the Boost Software License, Version 1.0. 12180 (See accompanying file LICENSE_1_0.txt or copy at 12181 http://www.boost.org/LICENSE_1_0.txt) 12182 */